Govern AI Risk With Controls That Scale

What has to be approved before an AI tool touches live work?

An approval record should be specific enough that someone who was not in the room can tell what was allowed. It names the use case, the users, the decision or task the output supports, the actions the tool must never take, the accountable owner, the person with approval authority and the conditions that send the use case back for review. Those conditions are where many records are weakest. A useful list includes a material change to the model or prompt, a new data source, a new user group, a new integration, a rise in challenged outputs and a change in the consequence of a wrong answer. Without them, approval is a single event that nobody revisits, even after the work has grown well beyond what was approved.

Approval should also separate suggesting from acting. A tool that drafts a message for a person to send carries a different risk from one that sends the message itself. The approval record should say which of the two the tool does, because the controls that follow depend on it.

Which control covers which risk?

Controls are easier to design when each one answers a specific question. The table below sets out the main control areas, what each one answers, the evidence that should exist and the gap that usually appears when the control is weak.

How should data and access be limited and recorded?

Data controls start with the question of what the tool actually receives, not what the design document says it receives. Retrieval features often pull more text than the user expects, including fields that were never meant to leave their original record. The practical method is to trace a sample of real requests end to end and record every field that reaches the model, the retrieval index or a log file. Sensitive fields can then be masked, excluded or restricted to roles that already hold that access in the source system. Retention needs an owner as well. Prompts, outputs and logs often persist longer than the business process that created them, and a retention rule written for the source system may not cover the copies the AI tool keeps.

Access follows the same discipline. A reviewer may need to read outputs but should not be able to change the underlying record. A tool that reads from a knowledge base should be limited to the sources it was approved to use, not every document the service account can see. Tool permissions deserve particular attention when the AI can call other systems. Each permitted action should be listed, and any action that changes a record, sends a message outside the organization or moves money should require a separate approval and a logged confirmation step.

What does a sound evaluation look like before release?

An evaluation starts with a question the business can answer, such as whether a draft reply is accurate enough to send after light editing, or whether a classification is reliable enough to route work without review. The test set should be drawn from real cases, including the awkward ones: incomplete input, contradictory fields, unusual wording, languages the team did not plan for and requests that should be refused. Acceptance conditions must be written before the test runs. A team that sets the threshold after seeing the results will tend to accept whatever it produced.

Reviewers should be calibrated before their judgements count as evidence. Two reviewers who score the same outputs differently will produce a result that depends on who happened to score it. A short calibration round, where reviewers discuss disagreements and agree a common reading of the acceptance conditions, is inexpensive and makes the results far more useful. After release, a sample of live outputs should be re-scored on a regular schedule, and a drop in quality should trigger a defined response rather than a general concern.

How should human oversight work day to day?

Human review fails in predictable ways. The reviewer may be given too many items to check properly, may lack the context to judge the output, may assume the tool is usually right and approve without reading, or may have no practical way to override it. None of these is solved by placing a person in the loop and calling the process governed. The reviewer needs authority to reject or edit, enough time to do the work, the source material the tool used and a clear action for outputs that are wrong or uncertain. Overrides should be logged with a short reason, because the pattern of overrides is often the earliest sign that the tool has drifted from what the process needs.

Oversight should also say what the reviewer may not do. If a reviewer can approve an output that affects a person's access, the approval should be recorded against the reviewer's name and the decision rule that applied, so that a later audit can see who decided and on what basis.

How should an AI incident be handled?

Incidents in AI work fall into a few recognizable groups. An output reaches a person and gives them wrong or harmful information. Sensitive data appears in an output, a log or a retrieved passage where it should not. An automated action changes a record or sends a message incorrectly. A user finds a way to make the tool act outside its approved scope. Each group needs a severity scale, an owner and a first response. For most of them the first response is containment: switch the affected feature off, route work back to the manual process and preserve the logs before anything is changed.

After containment, the team should establish what happened from the retained records, decide whether affected people or records need correction, and fix the cause before the feature returns. The fix should be tested against the case that failed and against the surrounding cases that could fail for the same reason. The post-incident record should feed back into the approval conditions. If an incident shows that a data boundary was looser than the approval assumed, the approval needs revisiting, not only the prompt. A disable switch that the owner can use without a technical release is worth building before go-live, because the decision to stop is hardest to take under pressure.

What are the limits of these controls?

Controls reduce risk, but they do not remove it. An evaluation covers the cases that were tested, and live work will produce cases nobody anticipated. A model or hosted service can change behavior after an update, which is why change control has to cover vendor changes as well as internal ones. Controls also take time. A review step that nobody can staff will be bypassed, and a control set designed for the highest-risk use case will slow down low-risk work if it is applied to everything. The right level is set by the consequence of a wrong answer, and it should be revisited as that consequence changes. The controls described here are a design pattern for organizations to adapt. They are not a record of results achieved for any particular client.

Frequently asked questions about AI governance controls

Is human review enough on its own?

No. A reviewer needs authority, context, time, the evidence behind the output and a defined action when the output is wrong or uncertain. Without those, review becomes a formality that adds delay without adding protection.

Do low-risk tools need the full control set?

They need proportionate controls. A tool that suggests internal wording for a person to edit needs a clear approval record, a data boundary and light evaluation. A tool that changes records or affects access needs the full set, including change control and a tested disable route.

Who should own AI incidents?

The business owner of the use case should own the decision to stop or restore the feature, while the platform or security function owns the investigation and technical fix. Naming both before go-live avoids an argument about responsibility during the incident itself.

How often should controls be reviewed?

Review them when the use case changes, and at a fixed interval even when it does not. The change triggers catch new data, users and integrations, while the scheduled review catches drift that no single change explains.

Next step: design the control model before the rollout

If a use case is already in front of users, or is close to it, the most useful next step is to write the approval record, map the data and access boundary and agree the disable route before expanding the rollout. Get in touch about your AI control model once the use case, the data it touches and the review it needs can be described. The AI and ML services page covers AI consulting and implementation, from use-case selection and data preparation through to evaluation. For the decisions that come before the controls, the enterprise AI strategy page covers use-case selection, data boundaries and human oversight, and the ServiceNow support and maintenance page covers keeping the platform running.