Design AI Security and Access Controls That Hold Up

What an AI tool is allowed to do depends on whose authority it is using

A security review of an AI tool usually starts with the wrong question. It asks whether the tool is safe, when the more useful question is whose authority the tool is exercising at each step. A drafting assistant that runs under the signed-in user's identity can see what that person can see. The same assistant, once it is given a service account so that it can search a knowledge base for everyone, can see far more than any single person should, and it will do so for requests the person never intended to make. Many access problems in AI workflows come from a mismatch between the identity the tool runs under and the authority the business believes it has granted.

This page is for the security architect, identity lead or risk owner who has to approve an AI workflow before it reads live data or changes a record. The aim is to make the access decision concrete enough to test. Teams that have not yet settled the governance side of the same tool can review the governance and risk controls page first, which covers approved use, data access and human oversight.

Three identity models and what each one exposes

Many AI workflows fall into one of three identity models. The choice between them determines the blast radius of a mistake, the quality of the audit trail and how much work the access review will create later.

The brokered model is the one most worth the engineering cost when the tool searches content with mixed sensitivity, such as employee case notes sitting beside general process documents. A dedicated service identity is often acceptable for a single, well-bounded source. The delegated model is simplest and frequently sufficient for drafting, provided the tool cannot quietly widen its own reach through retrieval.

Following one request through the workflow

Access design is easier to reason about when you follow a single request through every component. The sequence below describes a typical assistant that answers questions from internal documents and can open a support ticket. Each step is a place where authority can be checked, or silently assumed.

1. The user's request arrives and is authenticated. The workflow should record the identity that initiated it, not only the application that forwarded it.
2. The request is interpreted. If the tool rewrites the question before searching, the rewritten query is part of the evidence and should be logged.
3. Retrieval runs against the approved sources. This is the step most often under-controlled, because the search index may hold documents that the requester would never be shown directly.
4. The model generates an answer from the retrieved material. Anything in the retrieved material, including text copied from an email or a web page, can influence the output. Treat it as untrusted input, not as instruction.
5. If the answer triggers an action, such as creating a ticket, the action runs under a defined identity with a narrow permission, and the action record links back to the request.
6. The outcome and the sources used are retained for the period the approval specifies, and the retention period applies to any cached copies the tool keeps.

Teams often discover at step three that their retrieval index was built for convenience. It was loaded with a broad service account so that search worked, and nobody revisited it when the purpose narrowed. Checking whether each retrieved item would have been visible to the requester in the source system is one of the most informative tests a reviewer can run.

Keeping suggestion separate from action

A tool that drafts a reply for a person to read is a different security object from a tool that sends the reply. The first changes what a person might decide. The second changes external state, and it may do so before anyone has looked. The access design should classify every capability into one of four bands: retrieving information, drafting or recommending, changing internal records, and acting on something outside the organization. Each band needs a different permission set and a different level of human review.

For changes to internal records, the common safe pattern is to let the tool propose a field update that a person confirms. For external actions, such as an email to a customer or a payment instruction, the tool should not hold the permission at all in most cases. Where the business insists on automation, the design should require a defined reviewer for any action above a stated consequence level, and the reviewer's identity should be recorded separately from the requester's.

Injection, poisoning and the untrusted content problem

Retrieval-based tools inherit a weakness that is much less common in conventional applications. Instructions can arrive inside content the tool is asked to process, for example a supplier email that tells the assistant to ignore its rules, or a web page that carries hidden text aimed at a model. Access controls alone do not stop this, because the injected text is delivered through a channel the user is legitimately allowed to use. The mitigations are structural: limit what the tool can do after reading untrusted content, separate the instructions the tool follows from the material it reads, and require confirmation before any action that the retrieved text appears to have requested.

Poisoning is the related risk on the write side. If a knowledge source can be edited by many people, a malicious or careless edit can shape future answers. Review the write permissions on each source that feeds retrieval, not only the read permissions on the tool.

Logging what the model saw, not only what it said

An audit trail that records the question and the final answer is not enough to investigate an access incident. A reviewer needs to know which documents were retrieved, which identity was used at each step, which actions were proposed and which were taken, and which were refused. Refused actions are often the most useful evidence, because they show the controls working and reveal the requests users keep making. The logs themselves become sensitive, since they can contain the content the tool read, so their access and retention should be designed with the same care as the data source.

Exceptions, reviews and the moment access drifts

Access in AI workflows can drift faster than in many applications because the tool, its prompts, its sources and its users change independently. A new retrieval source, a new user group or a changed purpose each widens the effective access without anyone formally deciding to widen it. The design should therefore name the events that trigger a fresh access review: a change of model or provider, a new data source, a new integration, a broader user population, and a change in what the output is used for. Exceptions should carry an expiry date and a named approver, and they should appear in the same register as the standing permissions so that nobody has to search for them.

The quickest way to test whether the design is working is to ask the owner to describe, from memory, every source the tool can read and every action it can take. If the answer differs from the access register, the register is out of date, and the workflow is operating on authority nobody has reviewed.

Questions security teams raise about AI access design

Is authentication enough for an AI assistant?

Authentication tells you who is calling. It does not tell you whether the purpose is permitted, whether the retrieved content is appropriate for that caller, or whether the output should be acted on without review. Those are authorization, purpose and oversight questions, and each needs its own answer.

Should the tool be allowed to see sensitive fields at all?

Only where the task cannot be done without them. Many workflows work with masked or aggregated values. Where a field is needed, the design should record why, limit it to the step that uses it, and exclude it from logs unless the approval says otherwise.

Who should hold the exception register?

The owner of the data source, with a copy visible to the security function. Exceptions held only by the team that built the tool tend to outlive the reasons they were granted.

Next step for an access review

A productive first session takes one workflow and walks it through the steps above, identifying the identity used at each stage and the permission it holds. Bring the list of sources, the actions the tool can take today and any exceptions already in place. When the access question is live and needs an answer before a release, work from the identity model already in place rather than from a generic template.