Put AI Outputs Through Real Quality Checks

What does a release decision for an AI workflow actually require?

A release decision for an AI workflow is a statement that the workflow may now affect real work under stated conditions. It is not a statement that the model is good. The person signing it, usually a product owner or a service owner with a quality lead beside them, needs to know three things: what the workflow is allowed to do, how it has behaved on cases that resemble the real ones, and what will happen when it behaves badly after release. Testing exists to produce that knowledge. A decision made on a demonstration, however convincing, has not been tested in the sense that matters. Retrieval-based assistants face this problem in a sharper form; knowledge and retrieval readiness deals with the source, access and uncertainty questions that sit behind it. The wider AI and machine-learning services route lists the related AI work.

The decision also needs a defined owner for each failure type. A wrong answer that is caught by the user is a different kind of event from a wrong action that has already been written to a record. The release conditions should say which events stop the workflow automatically, which go to a named person for triage and which are logged for the next review. Those routes are part of the quality design, not an afterthought to it.

How to build a set of evaluation cases that tells you something

An evaluation set is only as useful as the cases in it. Teams often gather examples from whoever is available, which produces a set dominated by easy and typical requests. The result looks good and says little about the cases where the workflow is most likely to fail. A better set is built deliberately from several sources: real requests from the past, deliberately awkward inputs written by people who know the process, and cases that the business has flagged as expensive when they go wrong.

For each case the expected behavior should be written before the workflow is run against it. That discipline prevents the common habit of accepting whatever output appears and then judging it reasonable. It also makes disagreements between reviewers visible, which is useful information in itself.

Why a single accuracy figure hides the failures that matter

An overall score averages cases that carry very different consequences. A workflow that is right on nearly every routine request and wrong on the one case that triggers a costly refund has a high score and an unacceptable release profile. Scores are useful for tracking change between versions. They are poor evidence for a release decision on their own. A better summary breaks results down by case type and by consequence, and lists every high-consequence failure individually, with the reviewer's reasoning.

Reviewer agreement deserves the same attention. If two competent reviewers disagree about whether an output is acceptable, the acceptance criterion is not yet specific enough. Tightening the criterion, and rerunning the disputed cases, is a better use of time than debating the score.

Testing the failure paths, not just the happy path

Much evaluation effort goes into what the workflow should do. Less goes into what it does when things break. The failure tests worth including are the ones that resemble production conditions: a tool that times out, a source that returns an error instead of data, a prompt that was changed for a minor reason and never retested, a user who repeats the same request until a different answer appears, and input that is deliberately off-topic. For each one the expected result should be a defined response, such as a clarifying question, a refusal, an escalation or a stop. An unplanned response is a finding in its own right.

Repeated attempts deserve particular care. A workflow that gives a refusal on the first try and a compliant answer on the third has a consistency problem that a single-run test will never reveal. Running the same case several times and recording the spread is a cheap way to find it.

What to put in a go, no-go and pilot decision

A release decision should be a short written record that a second person can read. It should state the scope of the release, the evaluation cases used and their results by type, the failures that were accepted with the reason for accepting them, the monitoring signals that will be watched after release, and the conditions under which the workflow will be disabled. A pilot decision differs in one respect: it names the users, the case volume and the review effort that the pilot will consume, and it states what result would justify widening or stopping it.

• The scope of the release, written as the decisions the workflow may support.
• The evaluation results by case type, with every high-consequence failure listed.
• The accepted residual failures, each with an owner and a reason.
• The support path for users who receive a poor output.
• The re-evaluation trigger, such as a change of model, prompt, source or user group.

How does evaluation continue after the release?

Evaluation does not end at release. The workflow's inputs change, its sources are updated, and users find ways of asking questions that the original set did not anticipate. A living evaluation set takes in new cases from production, especially the corrected ones, and reruns the full set whenever something material changes. Monitoring that tells you when that is needed belongs with observability and human oversight, and the two should share the same case records so that a production problem becomes a regression test.

Where does quality assurance for AI sit within wider testing?

AI quality assurance borrows from conventional test practice: defined entry and exit criteria, traceable cases, defect triage and regression control. It differs in that the output is not fully deterministic and that the correct answer is sometimes a judgment. Teams that already run a disciplined quality assurance and testing function can extend it to cover AI behavior rather than building a parallel process. On ServiceNow estates, automated testing and release assurance is about which workflows are protected by repeatable tests and how results affect a release decision, which makes it a natural place to hold AI-specific regression checks.

Which questions should a quality lead answer before signing a release?

Are the evaluation cases written by people who did not build the workflow?

They should be, at least in part. Cases written by the builder tend to reflect the builder's assumptions about the input, and they miss the awkward inputs that real users produce.

What if the reviewers cannot agree on an acceptable output?

Treat the disagreement as a specification gap. Write a sharper acceptance condition, rerun the disputed cases against it and record the change.

How many times should a case be run?

Enough to see whether the output is stable. For cases where the answer matters, several runs are worth the cost. A case that passes once and fails the next time is not a passing case.

Is a pilot a substitute for evaluation?

No. A pilot is a controlled way of gathering real cases under supervision. It produces evidence only when the evaluation criteria were set before the pilot began and the pilot records outcomes against them.

If an AI workflow is close to release and the release decision is not yet supported by a written evaluation, bring the workflow, the cases you already have and the failure path that worries you most, so the acceptance conditions and what the pilot must prove can be settled before the workflow is widened. Discuss AI release quality.