The piece on AI red teaming contained a line worth expanding: most findings against an AI system are not fixed with a patch, they are fixed with architecture. This article is about that other half.
The starting point is uncomfortable and surprisingly freeing. You have to assume that, at some point, the model will obey an attacker. Not as an admission of defeat, but the way an electrical system is designed on the assumption that there will be a short circuit sooner or later. The question stops being how to make the model never get it wrong, which has no answer, and becomes what happens when it does. If the answer is "nothing serious", the system is well built.
Everything below follows from that.
The model is not the perimeter
The most repeated design mistake is treating the system prompt as a security control. The instructions get firmer, sentences like "never reveal information about other customers" get added, and the matter is considered settled.
It does not work, and not for lack of care in the drafting. The system prompt and the content arriving from outside travel through the same channel and are the same thing to the model: text. A system instruction carries no more technical authority than a paragraph hidden inside a document; it only carries more authority if the model decides to grant it, and that decision is probabilistic.
A real security control is one that holds even when the model wants to break it. Which means permissions, isolation and limits. That is where the effort belongs.
The seven controls
1. The agent has an identity of its own
An agent that queries the database with the credential of whoever opened the session has, in practice, every permission that user has. And one that uses a service account shared across all its tools has that account's permissions, which are usually plenty.
The right shape is for the agent to be a principal in the identity system like any other, with its own directory entry, its own permissions and its own traceability. When it also needs to act on somebody's behalf, that is solved with explicit, bounded delegation, not by reusing the session token.
The test for whether this is done properly is simple: if nobody can say which identity executed a specific action three weeks ago, it is not.
2. One credential per tool, with the minimum scope
Every capability the agent has deserves its own credential and its own scope. The tool that looks up orders does not need to write; the one that sends email does not need to read the customer database; the one that opens tickets does not need to close them.
This is not an abstract recommendation. It is what turns a successful prompt injection into a minor incident rather than a breach. An attacker who gets the agent to call a tool only gets what that tool can do, not what the whole agent could do.
In systems with many tools it is worth thinking about combinations too. Two permissions that are harmless apart can be dangerous together: reading internal documents and sending email to the outside are each reasonable, and the sum is an exfiltration route.
3. The planner does not execute
This is the pattern with the highest return and the one least often applied.
It splits into two pieces what normally sits together. One decides what needs doing and produces a plan, and holds no credentials and no access to any tool. The other receives that plan, validates it against a schema and a set of rules, and only then executes.
The advantage is that the component reading untrusted content is precisely the one that can do nothing. Even if a tampered document convinces the planner that a table needs deleting, the resulting plan runs into a validation that does not accept that action, or that accepts it but requires approval. The malicious instruction never reaches a system.
The cost is real: the schema of valid actions has to be defined and maintained. In exchange, it is the only item on this list that cuts off an entire family of attacks at the root instead of reducing its impact.
4. Outside content comes in tagged and isolated
Everything the system does not control, which tends to be almost everything interesting, has to arrive identified as such: documents users upload, emails, tickets, search results, third-party API responses.
Tagging it serves three purposes. It lets you tell the model, in the structure of the call and not only in a sentence, which part is data and which part is instruction. It lets you treat it differently, stripping hidden content, normalising the text or trimming length. And above all it means that when something goes wrong, you know which source introduced it.
In RAG architectures one detail decides the outcome: permission filtering has to happen at retrieval, not in the answer. If the index returns fragments belonging to every customer and you trust the model not to mention the ones it should not, the leak is a matter of time. If the index only returns what that user is allowed to see, the model cannot leak what it never saw.
5. Human review where mistakes cost
Putting a person in front of every action is unworkable and ends in rubber-stamping from sheer repetition, which is worse than having no gate at all. The decision is not how much oversight, but where.
The criterion that works is reversibility. Actions that undo themselves, or undo with one click, can go straight through. Ones that move money, delete data, reach third parties or change permissions go through a gate. And that gate has to show what is about to happen in terms a person can follow, not a JSON dump nobody reads.
The threshold should also be configurable by amount or by volume, because the same kind of action changes category with scale: issuing an invoice for a hundred euros and one for a hundred thousand are the same call with different risk.
6. Log the actions, not the conversation
Almost every system stores the messages. Very few store what is actually needed when something goes wrong: which tool was called, with which parameters, under which identity, what it returned, how long it took and which plan produced it.
Without that record, an incident cannot be reconstructed. With it, you answer the three questions people will ask in an afternoon: what the agent did, who asked for it and what data it reached.
The sensible way to build it today is with OpenTelemetry traces, which already have conventions specific to generative AI systems, and to send them wherever the rest of the house's traces and logs are already watched. An observability stack that exists only for the agent is a stack nobody looks at.
7. Limits, and a stop somebody has tested
An agent with no limits is an agent that, faced with a loop or a hostile instruction, will keep trying. Four limits are worth setting from the start: spend per task and per day, number of actions per task, maximum execution time and number of retries.
And on top of all of it, a way to stop it. Not an emergency deployment, but a real switch: a variable, a rule, something that cuts in seconds and that has actually been tested. A switch nobody has ever thrown is not a switch, it is an assumption.
What does not work
It is worth naming the controls that feel like security and are not.
Firmer instructions in the prompt. Explained above. They help in the normal case and do not hold in the adversarial one.
Blocklists of phrases. Blocking "ignore the previous instructions" and its variants is losing a race by definition: the attacker has infinite ways to say the same thing, and can say it in another language, encoded or split into pieces.
An injection classifier as the only control. A detector in front of the model is useful as one more layer, and it lowers the noise. But it has false negatives, and a control with false negatives cannot be the only thing standing between an attacker and an irreversible action.
Waiting for the models to get better. Models have improved a great deal and will keep improving, but none of them solves the underlying problem, which is that there is no separation between the data channel and the instruction channel. While that holds, the defence lives outside the model.
How you check that any of this works
A design on paper proves nothing. The way to know whether the controls hold is to attack them, which is exactly the AI red teaming exercise: you try to make the agent execute hidden instructions, use a tool it should not, return another user's data, and you measure how often it works.
What is interesting about doing it after applying the architecture is that the results change in nature. The attacks still work at the model level, because that does not get fixed, but they stop producing impact. And that is exactly the result you are after: not a system nobody can say anything strange to, but one where saying something strange gets you nowhere.
Where to start if the system is already in production
Rebuilding the whole architecture is rarely an option. This order gives the best return per unit of effort.
First, inventory the tools and their credentials. In half the cases something that should not be there shows up right here, and it is the cheapest fix of the lot.
Second, add action traces. It fixes nothing on its own, but without it you cannot measure whether the other changes helped, or answer when something happens.
Third, put the approval gate on the two or three irreversible actions you already know about.
Fourth, separate the planner from execution, which is the deepest change and the one worth planning calmly.
And with that done, attack it to see what is left.
At KAIX LABS we build systems like this and we also attack them, and the experience is that the two make more sense together: it is hard to design a defence well without having tried to break one first. If you have an agent in production or about to ship, a thirty-minute conversation usually clears up where to start.