Measure AI Value in Terms Leadership Trusts

Which decision the value evidence has to support

Value evidence is only worth collecting when someone is going to act on it. The decision usually takes one of three forms: keep the service running as it is, change how it works, or stop it and move the budget elsewhere. Each of those decisions needs different evidence, and a measure that cannot distinguish between them is decoration. A dashboard of usage may be reassuring, but it will not tell a finance lead whether a service should be expanded, and it will not tell a service owner which part of the workflow to fix first.

This page is for the person who must answer a question such as "is this worth what it costs?" on a schedule they did not choose. That is typically a finance partner, a service owner who reports to a steering group, or a leader who has to justify a second round of spending after a first pilot. The approach described here is about building a measure that survives that scrutiny, not about selecting a particular metric from a list.

What each family of measures can and cannot show

Most value discussions mix up four kinds of evidence. Each is useful for something, and each becomes misleading when it is asked to answer a question it was never built for.

The gap between activity and outcome is where most weak claims begin. Usage can rise because a tool is useful, or because it is mandatory, or because people are retrying a failed request. A measure that treats all three as success will eventually be challenged by someone who knows the workflow, and the challenge will land on the whole program rather than on the single figure.

Designing a baseline before the service changes anything

The most common reason value cannot be shown is that nobody measured the process before the tool arrived. Once the workflow has changed, the baseline is gone, and any comparison has to rely on memory or on a different period with different demand. Designing the baseline is therefore part of the service, not an afterthought.

A usable baseline records the volume of the work, the time from request to completion, the rate of rework or escalation, and the cost of the people involved, across a period long enough to include normal peaks. It should be captured from the systems of record where possible, and the method of capture should be written down so that the comparison can be repeated. Where a comparison group is available, such as a team or region that does not use the tool yet, it strengthens the case considerably, because it separates the effect of the tool from the effect of a busy quarter.

When a baseline was not captured, say so plainly. Some useful evidence can still be gathered from logs, case histories and sampled records, but the claim should be framed as an estimate with its limits written next to it.

Reading quality and operating signals together

Quality signals are usually the most informative and the least recorded. Track how often a reviewer accepted an output without change, how often they edited it, and how often they set it aside. Track the reasons, not only the counts. A rise in corrections concentrated in one case type is more useful than an overall rate, because it tells the owner where the workflow needs attention.

• Corrections and rejections, categorized by the reason the reviewer gave.
• Support questions about the service, grouped by whether they concern the tool or the process around it.
• Workarounds that users have built, such as copying output into another tool before using it.
• Access events and denied requests, which show whether the service is being used within its approved scope.
• Unresolved cases handed back to manual work, and the reason each one was handed back.

Those signals rarely look like value on their own. They become value evidence when they are linked to a decision: a workaround that appears in one team is a reason to change the workflow, and a category of rejections that never falls is a reason to stop investing in that case type.

Connecting each measure to an owner and an action

A measure without an owner decays. Before a measure goes into a report, name the person who will explain it, the period it covers, the source it is drawn from, and the action that will follow if it moves the wrong way. The action matters most. If a measure moves and nothing happens, the measure has not been connected to a decision and should be retired or redesigned.

The governance of those measures is closely tied to the way the service is run. The operating model design page covers who owns the business outcome, data, engineering, security, human review, support, change and evidence when an AI capability becomes part of a service, and it is worth agreeing who decides on the basis of the evidence before the first report is due. Where the evidence depends on human review, the observability and oversight page covers making an AI workflow understandable after use, which is what lets a review be recorded in a form that can be counted.

Withdrawing a value claim when the evidence changes

Value claims have a shelf life. A figure that was accurate when the service launched can become misleading once the data source changes, the user group widens, the workflow is redesigned or the underlying model is replaced. A defensible program therefore records the conditions under which each claim was made and reviews the claim when those conditions change. When a claim no longer holds, withdrawing it promptly is more credible than defending it for another cycle. Stakeholders tend to trust a program that revises its own numbers far more than one that never changes them.

Questions finance and service leaders ask

What if no baseline was collected?

Build the best estimate the existing records allow, label it as an estimate, and begin a comparison from now. A future decision can rest on a clean comparison even when the past cannot.

Can time saved be counted as value?

Only if the saved time is released to work that matters, or the organization has decided to reduce cost. Time that is simply absorbed by other tasks does not appear in any budget, and a claim built on it will not survive a finance review.

How often should the measures be reviewed?

Early in a service, often each month, because the workflow is still changing. Once it stabilizes, a quarterly review is usually enough, with an immediate review whenever the model, data, user group or consequence changes.

Who should challenge the numbers?

Someone who knows the workflow but did not build the measure. A reviewer who never sees the report will not notice a flaw that the team has learned to ignore.

Next step for a value review

Bring the service, the question leadership is asking and any figures already in circulation. Those three items are usually enough to show which claims can be supported, which need a baseline that does not yet exist, and which should be withdrawn. Where the first step is still choosing which service deserves measurement, the use-case discovery page covers mapping decisions, work effort, data and failure consequences to find AI use cases worth pursuing. Discuss your AI value measurement question with us when a report is due and the current evidence does not yet answer it.