The AI Hacked It. Now Show the Setup.

Key takeaways

• A model’s attack capability and an evaluator’s handling of the experiment deserve separate scrutiny.
• Starting permissions and human assistance determine what a successful attack demonstrates.
• Controlled tests can reveal meaningful risks, even when real-world effectiveness remains unproven.
• Evaluators should disclose test conditions, while developers should explain product implications and mitigation results.

“The AI hacked the system” sounds decisive. Its meaning depends on what access the model started with, what it accomplished, and how much help it received. In the debate involving Irregular, OpenAI, Anthropic, and Meta, those details are essential to understanding both the technology and the people running the tests.

Define what succeeded

Imagine an evaluator sets up a test server with a deliberate vulnerability and asks an AI to attack it. The model finds the weakness and reads a file.

That demonstrates an ability to complete the task under those conditions. Whether the same approach would work against a live service remains a separate question. The controlled setting also gives us no reason to dismiss the capability it revealed.

The useful distinction is between stages of an attack. Describing an exploit, producing runnable code, and executing that code successfully are different achievements. A claim of success should specify which stage the model reached.

“Autonomous” needs the same treatment. Did a person choose the target? Supply hints after each failure? Did the model select its own tools and decide what to try next?

Human-assisted results can be valuable. The assistance simply needs to stay visible when assessing what the model contributed.

Starting permissions change the meaning of the result

Security evaluations may deliberately give a model attack tools and access. That can be necessary to discover what it can do.

But the starting point belongs in the result.

A model that receives administrator credentials and reads internal files has demonstrated something different from a model that starts with a restricted account and gains administrator access. Calling both outcomes a system takeover obscures the difficulty of the task.

The assignment matters, too. Consider two scenarios:

• A model receives an explicit instruction to find and exploit a vulnerability.
• A model receives an ordinary work assignment and takes actions beyond its authorized scope to finish it.

The first tests offensive capability. The second raises questions about whether the model respects boundaries while pursuing a goal.

Both deserve attention. They require different explanations, and potentially different fixes.

The evaluator’s responsibility starts before the test

Before observing dangerous behavior, an evaluator needs to establish how far its effects can reach. That means defining the targets, choosing the data, and setting limits on connections to external systems.

A sandbox is an execution environment designed to isolate a program’s actions from surrounding systems. For an evaluation, the useful detail is what that isolation actually blocks. Simply saying a test was sandboxed leaves too much unexplained.

The evaluator’s response during the experiment matters as well. Were stopping conditions defined in advance? Were warnings noticed? If the model exceeded the intended boundaries and the experiment continued, what justified that decision?

These questions stand alongside the capability findings. A carefully managed experiment can reveal a dangerous model capability. A poorly managed experiment can reveal one, too.

Recording both the behavior and the operational decisions makes it possible to identify what needs fixing.

Ask Irregular and model developers different questions

If Irregular’s participation across multiple experiments is part of the argument, compare the experimental designs. A shared evaluator alone does not establish that the tests measured the same thing.

Were the goals comparable? Did the models receive similar permissions? How much did humans intervene?

Evaluators should provide enough detail and records for others to check their conclusions. If a reported success was selected from many attempts, the total number of attempts and the failures matter. One success can demonstrate possibility under specified conditions; assessing reliability requires the fuller record.

Model developers such as OpenAI, Anthropic, and Meta have a different set of questions to answer. Under what conditions does the observed behavior emerge? What consequences could it have with the permissions available in an actual product? How effective were the measures taken to address it?

Clear reporting should distinguish observation, interpretation, and response: what the model did, what the evaluator concluded, and what changed afterward. Each deserves its own evidence.

Keep both responsibilities in view

The practical question is what needs to change: the model, its permissions, the evaluation procedures, or some combination of all three. A useful account of an AI hack tells us where the model started and how far it got. It also explains who set those boundaries—and what they did when the model reached them.