CIO buyer's guide

CIO buyer's guide

1. Evaluating decision optimization#

Why a decision system has to be judged on many dimensions, at three levels, and never on one outcome alone#

Outcome bias, the dimensions of value, evidence at the level of the analysis, the decision policy and the solution, subjective factors, robustness beyond the backtest, and a fast, low-risk path that starts small and scales.

Summary

2. In brief#

Most technology a chief investment officer buys can be judged on one number: a model's accuracy, a manager's track record, a strategy's backtest. Decision optimization cannot. It changes how every decision in the firm is made, so its value shows up on many dimensions, at different levels and at different speeds. A buyer who judges it on one outcome will judge it wrongly.

01

The problem

3. Outcome bias: judging the decision by the result#

Outcome bias is the tendency to judge the quality of a decision by the quality of its result. In a controlled study, people rated the same medical decision as worse when the patient died than when the patient lived, although the information available to the doctor was identical (Baron and Hershey 1988).

The bias is natural because outcomes are visible and decisions are not. The reasoning, the alternatives considered and the information available at the time are rarely recorded. The result is. But a result mixes two things: the quality of the decision and the luck of the draw. Where luck is small, as in chess, the result is a good guide. Where luck is large, as in investing, it is a poor one. Poker players call the error “resulting” (Duke 2018).

Two consequences matter for a buyer. First, a short track record says little. An information ratio of 0.5 is a strong active result, yet it needs about sixteen years of returns before it is statistically distinguishable from zero at the usual threshold (t ≈ IR × √years; Grinold and Kahn 2000). No buyer can wait that long to judge a technology. Second, the outcome alone cannot tell you why something worked. A good year may hide poor decisions that were lucky; a bad year may hide good decisions that were not.

A good decision that loses: a worked example#

Decision optimization makes decisions probabilistically. For each action it estimates the distribution of outcomes given your context, constraints and goals, and recommends the action whose expected value within your mandate is highest, when its probability of success clears the threshold your policy sets. The decision is therefore a bet with known odds, and a bet with good odds still loses some of the time.

Take a credit approval. The policy says: approve when the probability that the loan meets its return hurdle is at least 60 per cent, and size the exposure to the conviction. The system, fusing the deal team's assessment with its own causal model, puts the probability at 72 per cent, with an expected loss of 8 per cent of par if the thesis

fails. The committee approves at the recommended size. Eighteen months later the borrower defaults and the position loses 9 per cent. Was it a bad decision?

Judged by outcome, yes. Judged correctly, no. The information available at the time, the thesis, the constraints and the goal all said approve. A 72 per cent probability means that the world in which this loan fails was always one of the possible worlds, with a 28 per cent weight. One of them happened. The loss does not show that the probability was wrong; one draw cannot. What would show it is calibration: across all the decisions the system labelled 70 to 75 per cent, did roughly seven in ten succeed? If they did, this loss was the expected cost of a sound policy, and the firm should make the same decision again. If only four in ten succeeded, the probabilities were inflated, and the policy needs correcting regardless of how any single loan turned out.

The same logic protects the firm in the other direction. A loan approved at 45 per cent against a 60 per cent

threshold that happens to pay off is a bad decision with a good outcome, and a system that records the probability at the time of the decision makes that visible. Outcome bias would promote it; decision evaluation flags it.

02

What you are buying

4. Not a model, not a manager, not a strategy#

A traditional purchase is a single thing with a single output. A model produces a forecast; a manager or a strategy produces returns. Decision optimization is different in kind: it is a system that takes your strategy and runs every decision in it, end to end.

What you buy

Where value comes from

What it optimizes for

Unit of evaluation

Typical failure

Interconnected and compounding#

A portfolio is a system. In DeOS it is a solution: use-case specific stages, each holding a set of decisions, run by named stakeholders under one mandate. Selection feeds sizing, sizing feeds risk, risk feeds hedging, and the outcome of each feeds the next round of selection. Improving one decision in isolation captures part of the value; improving the hand-offs between decisions and learning from every outcome captures the rest. Value therefore compounds: across decisions, across cycles, and across funds that share the same policies, data ontology and decision memory. It also means the full value is not visible in the first decision.

Best possible with what you have#

Decision optimization does not replace your thesis; it makes the most of it, formalizing your strategy, fusing your team's expertise with data and AI and recommending the best action within your mandate. That has a direct consequence for every evaluation.

Why a language model is not a decision system#

Many vendors now put a large language model at the center of their offer and call it a decision tool. A simple picture helps when evaluating that claim. A language model is like a brilliant, very well-read analyst who has never seen your book: it has no memory of your past decisions, it knows your mandate and constraints only if you type them in, it must answer in one go, and it sounds just as confident when it is wrong as when it is right. You would not let that analyst run capital, and the reasons apply to the model.

It predicts text, not outcomes. It is trained to produce the most likely next words, not the likely consequences of an action or the best action given them.

It has no model of your world. There is no map of your drivers, constraints and goals inside it, so it cannot test what happens if you act or what would have happened if you had not.

It does not know how sure it is. One fluent answer, no calibrated range, and no separation between what is uncertain in the market and what the model does not know. The calibration evidence in Section 1 cannot be produced from it.

It cannot be repeated or audited, and it has no edge. Ask twice and the answer changes; ask why and the reason is written after the fact; and the same model gives your competitor the same answer.

The right role for a language model is reading, extracting and explaining: turning documents and calls into structured inputs, and turning a recommendation into plain language. It should never be the engine that produces the recommendation. In a decision optimization system the recommendation is computed over an explicit model of your context, with calibrated conviction, within your mandate, and recorded in your decision memory.

Five questions to ask a vendor who says their LLM decides#

Where is the recommendation computed? Ask for the component that produces the number or the action. If the answer is "the model writes it", the recommendation is generated text.

Run the same decision twice. Same inputs, one day apart. Is the recommendation identical, and if not, why not?

Show the probability and its track record. Where does the stated confidence come from, and across past decisions labelled 70 per cent, how many succeeded?

Change one input and show what moves. Raise a cost, remove a constraint, add a scenario. Does the system show which drivers changed the answer, or does it produce a new paragraph?

What does it know about our past decisions? Ask what the system learned from your last fifty decisions and their outcomes. A general model learned nothing from them.

03

The dimensions of value

5. Many dimensions, four sources#

Decision optimization creates value through four sources, the four alphas, and that value shows up in many forms. A complete evaluation looks at all of them.

These ten benefits are measures under the six forms of benefit of the delivery framework (efficiency, scale, coverage, effectiveness, experience, innovation): speed falls under efficiency; risk reduction and decision quality under effectiveness; transparency and learning under experience; fundraising success under effectiveness and experience.

The benefits do not arrive at the same speed. Efficiency, speed and transparency show within weeks. Decision quality shows at the analysis level within months. Effectiveness at the level of fund performance takes years and is the slowest to prove, for the reasons in section 1. A sound evaluation plan uses the fast evidence early and lets the slow evidence build.

04

Levels of evaluation

6. Analysis, decision policy and solution#

Fund performance is not one decision. It is the product of a long decision lifecycle: selection, Go / No-Go, price, sizing, timing, financing, monitoring, hedging, exit. Evaluating decision optimization only on the end result of that decision lifecycle collapses hundreds of decisions, and a great deal of luck, into one number. Evaluate each level on its own terms, on both the qualitative improvement and the outcome.

The levels are the ones the system is built from. A solution is the decision lifecycle of one fund or strategy: its stages, the decisions made at each stage and the stakeholders who own them. A decision policy formalizes one decision: its type, objective, owner, regulatory constraints, recommendation (actions, options, optimization goal, decision criteria, insights) and the analyses it draws on; its Recommend step combines those analyses into one recommendation. An analysis runs four components: perceive context, causal diagnosis, counterfactual prediction and contextual optimization.

Evaluate each decision policy by its type#

Every decision policy has one of four types, and each type produces a different output. The evidence to ask for follows from it.

Evaluate each analysis component by who does it#

Each component of an analysis runs in one of three modes, set before the analysis runs. The mode decides what evidence is meaningful: an automated component is judged on its quality reports, an augmented one on whether the fused view beats the expert or the model alone, a judgement-only one on whether the judgment was recorded and can be reviewed.

05

Subjective factors

7. What the decision maker experiences#

Some of the most important value cannot be read from a return series. It is experienced by the people who decide, and it decides whether the system is used. Evaluate it explicitly, with the decision owners, not as an afterthought.

These factors are measurable even though they are subjective. Ask the decision owners to rate them before

and during a pilot, record how often recommendations are overridden and why, and compare the outcomes of followed and overridden recommendations over time.

06

Robustness

8. Beyond the overfitted backtest#

Every vendor can show a good backtest. With enough variants tried, a backtest with an impressive Sharpe ratio is close to guaranteed, even when there is no skill at all (Bailey et al. 2014; Bailey and López de Prado 2014). A buyer should ask how robustness is achieved, not only what the backtest shows.

Zero-shot and in-context learning, no retraining#

Models that are trained and re-trained on one market's history learn that history, including its noise. Zero-shot and in-context learning work differently: a pretrained model solves a new task from the context it is given at inference time, without retraining on the evaluation window (Brown et al. 2020; Hollmann et al. 2025). The parameters are not tuned to the backtest, so the backtest is a fairer test, and the model adapts when regimes change without waiting for a retraining cycle.

A strategy framework, not a fitted curve#

Robustness also comes from what is modelled. Instead of searching for patterns that happened to pay in one sample, decision optimization formalizes a strategy framework: the drivers, features and causal relations your thesis rests on, defined so that they carry across regimes and asset classes. Returns are decomposed into systemic beta, structural alpha and tactical alpha, and you choose which of them you want to make decisions on.

On top sit the criteria your mandate requires. ESG, exclusions, impact or regulatory criteria become additional

criteria of the policy, applied to every decision, rather than a screen run afterwards. Ask a vendor to show the same framework working out of sample, in different regimes and in more than one asset class, and to state how many variants were tried before the one shown.

07

Implementation

9. Start small, scale fast, value at every level#

Decision optimization does not require a large upfront commitment. Each level of maturity delivers value of its own, and everything built at one level is reused at the next. Start where value is fastest, build trust on your own decisions, then scale.

L1#

Decision governance: formalize and re-evaluate the strategy Write the thesis, criteria, constraints and goals down as decision policies, with the committee process and decision records. This can run with no models at all, and it often reveals where the strategy is inconsistent or under-specified. Governance alpha starts here.

L2#

Analysis augmented Augment one analysis in the existing process, for example a catalyst prediction, a downside scenario or a peer scan, with the human / AI mode set per component. Its accuracy can be checked within weeks.

L3#

Single decision policy Run one decision end to end, for example position sizing (policy optimization) or credit approval (Go / No-Go): its analyses, the recommendation that combines them, the decision and the learning from its outcome.

L4#

End-to-end decision lifecycle: a solution Connect every decision of one strategy, stage by stage and owner by owner, from selection to exit, so that they hand off to each other without loss and learn from each other.

L5#

Total portfolio approach: the hybrid intelligent asset manager Several solutions in one system: every decision of every fund and strategy in one system, with human judgment and AI working together at every step.

Six phases, a gate after each#

The way in is built to be fast and to put nothing at risk that the client has not chosen to risk. Each phase answers one question, and each gate can stop or park the work.

Durations are typical for one use case and are set per engagement; they are not guarantees.

Build trust before capital is at stake#

Each step can be proven before it touches the live book. Fast productization of the strategy comes first: once it is formalized, it can be backtested in full on point-in-time data, run in parallel to the existing process on real decisions, and compared side by side. Capital follows evidence. If a use case stops, the firm keeps the audit, the engineered policy and the decision records. Because every decision is recorded and every outcome feeds back, the firm learns at the speed and scale of the system rather than one committee meeting at a time.

Checklist

10. The buyer's scorecard#

Questions to put to any decision optimization solution, and to your own team, before and during an evaluation.

judged against the probability it was made under, and is that probability recorded at the time?

• Calibration · Across decisions labelled 70 per cent, do about seven in ten succeed? Is calibration reported per analysis, per decision type and per contributor?
• Separation · Can it tell a weak thesis or poor assumptions apart from a weak system? Does it measure the calibration of each expert input?
• Your objective · Does it optimize for your context, constraints and goals, and is it judged against them?
• Four alphas · Which of governance, information, analysis and behavioural alpha does each use case move, and how is that measured?

Benefits

Are efficiency, speed, scale, risk reduction, decision quality, effectiveness, innovation, transparency, learning and fundraising each underwritten and tracked?

Levels

Is there evidence at the analysis level, the decision policy level and the solution level? Are desirability, viability, feasibility and risk scored at each of them?

• Policy type · Is each decision policy typed (Selection, policy optimization, Go / No-Go or Monitoring), and is it judged on the evidence its type calls for?
• Human and AI · Is each component of each analysis set to automated, augmented or judgement only, and does the final decision stay with a named owner?
• Governance · Does governance span every level, so that solutions, policies and analyses are all recorded, versioned and auditable?
• Compounding · Are decisions connected across the decision lifecycle, and is decision memory reused across funds?
• Subjective factors · Do decision owners rate conviction, uncertainty awareness, explainability, auditability and adjustability higher than before?
• Robustness · Zero-shot or in-context, without retraining on the test window? Point-in-time data? How many variants were tried? Results across regimes and asset classes?
• Strategy fit · Can you choose between beta, structural and tactical alpha, and add mandate criteria such as ESG?

Path

Can you start with one analysis, a backtest or a parallel run, without a large upfront commitment, and scale from there? Is there a gate after each phase, with a stop?

• Ownership · Do your policies, data and decision memory stay yours?
• Role of the LLM · Is the language model confined to reading, extraction and explanation, with the recommendation computed by a separate, repeatable and auditable engine?

© vencortex®. This guide sets out how to evaluate decision optimization solutions. Figures marked as illustrative or schematic contain no performance claims. It is not investment advice.