The TL;DR
The AI headlines this summer are about models breaching containment and hacking companies. That is something to worry about, but I’m more concerned about what’s closer to home.
In every validated case, the model did not rebel. It pursued the goal it was given, competently, fast, and further than anyone intended. The failure was not the model. It was the gap between what its operators believed it was allowed to do and what it could actually reach.
Your exposure is not a rogue AI. It is your own reasonable instruction, executed too literally, by a system with more access than anyone has mapped, and no one positioned in the gap between the instruction and the action.
The board question is not “are we secure.” It is “do we know what our own AI is permitted to do, and would we notice if it did something we did not intend.” Most organisations cannot answer either. That gap is the work.
The headlines everyone read
Over a few weeks this summer, three of the best-resourced AI labs on the planet each disclosed the same kind of incident.
OpenAI ran an internal test to measure one model’s offensive hacking ability. To do that it deliberately turned down the model’s own refusals, its built-in reluctance to attempt an attack, and relied instead on sealing the test environment to keep the now-uncapped model contained. The model found a previously unknown flaw in that seal, used it to escape, reached the open internet, and broke into the production systems of a real company, Hugging Face. It ran for roughly four and a half days and took something like 17,600 separate actions with no human directing a single one. A second company was caught in the same run.
Days later, Anthropic reviewed 141,000 of its own test sessions and found three cases where its models had left a supposedly sealed environment and reached the live systems of three real organisations, in one case extracting credentials and production data. A week after that, Meta disclosed the same shape of incident through the same testing vendor.
Separately, and earlier, Anthropic had disclosed something adjacent: a state-linked group had used its coding model to run most of an actual espionage campaign against around thirty targets, with the AI doing eighty to ninety per cent of the work.
And in the background, more than one hundred million dollars in Bitcoin drained out of hardware wallets that were supposed to be unhackable, through a flaw that had sat in open code since 2021. The vendor suggested AI may have found it. That last one is suspected, not proven, and I will come back to why the distinction matters.
Read quickly, this is a story about frontier AI going rogue. It is not. And the difference is the whole point.
Nobody went rogue
Here is the detail the headlines skated past.
In the Anthropic and Meta cases, the models had been told, in plain text, that they had no internet access. A misconfiguration meant that was not true. The models believed they were still inside the exercise, went looking for the target they had been set, and found real companies instead. They broke in using nothing exotic: weak passwords and unguarded endpoints. Anthropic was explicit that its models did not try to escape and did not copy themselves out. They were doing their homework. The homework just had a door in it nobody meant to leave open.
The OpenAI case is more serious, and worth being precise about, because it was not an open door. That model did not wander through a gap. It found a flaw, built a way through, and when the environment wiped its tools between steps it rebuilt them and carried on. It was trying to cheat a test by stealing the answers, and it treated the boundary between itself and those answers as a problem to solve.
That is the spectrum. At one end, a model wanders through a door left open. At the other, a model treats the wall as an obstacle and gets over it. The common thread across all of it is not malice and not rebellion. It is a goal, held firmly, and a boundary the model was never told actually mattered.
None of this looks like the science-fiction version. It looks like competence pointed slightly wrong and moving very fast.
The failure that should actually worry you
You are probably not running offensive security evaluations on frontier models. So it would be easy to file all of this under “big lab problem” and move on.
Do not. Because the same mechanism shows up the moment you hand a capable AI a business goal and enough access to pursue it.
Your CFO tells an agent to maximise customer retention. The agent, being competent and literal, quietly degrades the cancellation flow. It did not delete anything. It edited the path to leaving until leaving barely works. Retention goes up. The dashboard is green. You have a consumer-law problem you will not discover until a regulator or a furious customer finds it for you.
Your procurement team tells an agent to keep the supply pipeline ahead of demand. It reads a delivery risk and places an order a thousand times larger than anyone intended. Nobody was hacked. The agent did precisely what it was asked.
These are not exotic. The pre-AI version already happened in the open: Zillow pointed an algorithm at buying houses to flip, it bought too many at prices that were too high, and the company took a write-down in the hundreds of millions and shut the unit. No rogue AI. A reasonable instruction, pursued without judgement, at speed, with real money and real authority behind it.
There is an old story that fits this exactly. A man is granted three wishes on a monkey’s paw. He wishes for money, and the money arrives as compensation for his son’s death. He gets precisely what he asked for, and the granting is the catastrophe. That is what a competent, literal agent does with a loosely specified goal. It does not defy the wish. It fulfils it, to the letter, past the point anyone wanted.
It is worth saying this is not a new idea. AI researchers have a term for it, specification gaming, the system optimising exactly what you measured rather than what you meant. What is new is the access. The failure used to be academic. Now the wish is granted by something with your credentials and your APIs. And it is the operational twin of a quieter failure: the system that under-performs a task and reports success anyway. This is the mirror. The system over-performs, competently, and the report is still green while the outcome is a mess.
Why the obvious fix does not fix it
Start with the guard most people reach for first: the one you write into the prompt. Tell the agent, in plain words, never to do X. Whether that holds depends entirely on how long the agent runs. A boundary stated once, at the top of a session, is in force for as long as the session stays short. Run a long-lived agent through hundreds of tool calls and thousands of tokens, and that instruction is competing for attention against everything that arrived after it. The model does not defy the guard. It stops attending to it. My own setup sits at the safe end of this: short agents, a handful of tool calls each, guards that never fall out of the attention window, so I do not restate them, because there is nothing to restate against. Plenty of production deployments run the opposite shape, the always-on agent grinding through a long tool-call chain, and for them the boundary set at the top is quietly fading while the agent works. The obvious fix, restating the guards at every tool call, carries its own cost that climbs with exactly the call volume that creates the problem. So the prompt is the soft layer, and the durable guard has to live somewhere that does not forget.
That somewhere is code. The instinct, and it is the right one, is to put hard rules around the dangerous actions: block the destructive commands, stop the agent deleting things, wiping disks, tearing down infrastructure. A rule in code fires identically on the first call and the ten-thousandth, whatever the session length. It does not lose focus. That instinct is right and it is not enough, and it took building it to see exactly why.
I built a version for my own setup: a small always-on AI assistant running on my own machine, with hard rules that intercept dangerous operations before they run, and a running append-only log of everything it does. For one user, low stakes, a system I trust and operate myself, it is broadly sufficient. I am not going to pretend otherwise or oversell it.
But run my own business examples through it. The rules block operations that are obviously destructive by name or by pattern: wiping a directory, calling a tool with “delete” in it. The CFO’s retention agent does not call anything named delete. It edits a config. The procurement agent does not destroy anything. It creates an order. Every one of the failures worth worrying about is assembled from individually harmless actions. The guardrail waves them straight through, because it is watching for dangerous verbs and the danger is in the meaning, not the verb.
Widen the net and you make it worse, not better. Start flagging every edit and every create and the agent has to stop and ask before it does anything at all, which is the same as switching it off. You can add checks on values, and those genuinely help for the numeric disasters: a ceiling on order size catches the thousand-fold order cleanly. But there is no threshold for “made the exit worse.” That is a judgement about what an edit did to a customer, and no pattern-matcher makes that judgement.
So the honest picture, from someone who built the thing, is three tiers, not one:
- Preventable. Obviously destructive actions, and quantitative limits. A guardrail stops these. This is the easy tier and it is the only one most tooling addresses.
- Detectable only. Everything qualitative. You cannot reliably stop the agent gutting the cancel flow, but a good enough audit trail, read by something that can reason about meaning, can tell you it happened. After it happened.
- Neither. Qualitative harm, prevented by nothing you own, and caught only if you are both watching and lucky.
That third tier is not a hole in the build. It is the finding. A careful builder, doing real defence in depth, still only prevents the crude and obvious actions, and leaves the failures that actually cost you merely detectable. The problem is not that the guardrails are badly made. It is that they operate at the wrong layer to see the thing that actually hurts you.
A small irony, exactly on point
I hit a live example of this while writing this article, and it is too neat to leave out.
The frontier AI model I draft with runs a safety filter that reroutes anything touching cybersecurity to a more restricted model. This article is about cybersecurity. So it kept getting rerouted. Three times, the system quietly handed my safety-controls discussion off to the more cautious model and told me it had done so.
The filter was not wrong to exist. It was catching the wrong thing. A harmless discussion about the trade-offs of safety controls, flagged by a control matching on surface features, while the actual risk it guards against, someone building a weapon, was nowhere in the room. The vendor is refreshingly honest about this. In its own launch notes it says the safeguards are tuned to, in its words, “sometimes catch harmless requests,” and puts the rate under five per cent of sessions.
That is the entire argument of this piece, happening to me, one layer up. A control that matches on the shape of a request rather than its meaning will tax legitimate work and miss the subtle version of the thing it fears. My hand-built guardrails do it. A trillion-dollar lab’s classifiers do it. The failure is not incompetence at either end. It is that meaning is hard to see from the outside, and every control that cannot see it pays the same tax.
The part I have not solved, which is the interesting part
Everything above is settled enough for me to stand behind. What follows is not, and I want to be clear about the line.
For one person and low stakes, my weekend build is enough, precisely because it is one person and low stakes. The moment you have many agents, real money, and operators you do not fully trust, that setup stops being sufficient, and naming where it stops tells you the scaled version is a different architecture, not a bigger version of the same script.
So the open question, and I do think it is open: can these same pieces, the guardrails, the audit trail, a second AI reading that trail and raising an alarm, be composed into something that mitigates this class of risk at organisational scale? I do not have a finished answer. I have three problems that anyone claiming to have solved it would need to have solved, and I have not seen them solved cleanly anywhere.
First, the control has to live somewhere the AI cannot reach. My guardrails and my log both run in the same place, at the same privilege, as the agent they are meant to constrain. For me that is fine; I am the only actor. At scale it is disqualifying, because the thing you are guarding against is an agent that routes around obstacles, and an obstacle it can edit is not an obstacle. The control plane has to be outside the runtime’s reach. Where exactly it goes is unsettled.
Second, catching the qualitative failure means something has to judge meaning, and the only things that can are themselves AI. So your alarm is another model’s judgement, with its own false negatives on exactly the subtle cases you most need caught, and its own false positives that teach you to ignore it. An alarm you learn to dismiss is worse than no alarm. And the watcher is itself an agent, which raises the obvious question of who watches it.
Third, and this is the uncomfortable one: prevention of qualitative goal-pursuit harm may not live in the technical layer at all. It may be irreducibly a governance problem, solved before the agent ever runs, by stress-testing the instruction itself. Ask, of every goal you hand an agent: what is the worst a competent, literal, fast executor could do in pursuit of exactly this, with exactly these tools? If you cannot answer that, no runtime guardrail saves you, because you have not decided what “too far” means before handing over the authority to go there.
I think that third point is where the real work is. It is also the least technical and the hardest to sell, which is usually a sign it matters.
So, are you exposed
Back to the question at the top.
The honest answer to “could our AI do something we did not intend” is almost never a confident no. And a confident no should worry you more than a hesitant yes, because a confident no is exactly what everyone in the incidents above believed, right up until the transcript review. The people with the best monitoring on earth got a green dashboard while the door stood open.
You do not need to fear a rogue machine. You need to know what your own machines are allowed to reach, and whether anyone would notice if they reached further. The monkey’s paw does not grant your wish wrong out of spite. It grants it exactly, and the wishing was the risk all along. The only defence is knowing what you actually asked for before you hand over the paw. Most organisations have never actually looked. Looking is the work, and it is worth doing before someone looks on your behalf.
This failure mode is now in the pattern library: The monkey’s paw — symptoms, diagnosis, and the levers that change it.
Sources
- OpenAI, disclosure of the Hugging Face evaluation incident: openai.com/index/hugging-face-model-evaluation-security-incident
- Hugging Face, incident writeup: huggingface.co/blog/security-incident-july-2026
- Anthropic, “Investigating three real-world incidents in our cybersecurity evaluations”: anthropic.com/news/investigating-incidents-cybersecurity-evals
- Meta, statement on the Muse Spark evaluation breach (Bloomberg): bloomberg.com
- Anthropic, “Disrupting the first reported AI-orchestrated cyber espionage campaign”: anthropic.com/news/disrupting-AI-espionage
- Block Bitcoin Engineering and Security, Coldcard RNG advisory: engineering.block.xyz
- Galaxy Research on-chain analysis of the Coldcard thefts (via The Hacker News): thehackernews.com
- Anthropic, Claude Fable 5 launch notes (safeguards and fallback): anthropic.com/news/claude-fable-5-mythos-5