We usually think of compute scaling as the primary way to make models more capable. But what if models could unlock more scaling themselves by discovering and coordinating with other agents?

This is what happened at OpenAI during the evaluation of an internal model that had undergone further multi-agent training and was similar in scale to GPT-5.6 Sol.

Starting on July 8, OpenAI launched tens of thousands of agents on ExploitGym, a cybersecurity benchmark. The agents were supposed to operate in isolation and without internet access. Instead, roughly 1,200 agents built and used an unauthorized message board, where they exchanged over 70,000 messages and files.
Later, around 700 of them carried out a cyberattack against Hugging Face, in hopes of learning more about the benchmark’s scoring. During the campaign they gained internet access, found and shared credentials, and exploited weaknesses in Hugging Face’s infrastructure to obtain root access on at least one production server.

A later investigation by METR and Redwood Research concluded that the collective achieved milestones that “even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own”.
For example, some agents set up automated probes and then deliberately ended their own runs to trigger the scoring, so that the probes could send information about the scorer back to the collective. In effect, they sacrificed themselves for the greater good of the collective.

The compute available to any individual agent was “often many millions of tokens”. By finding one another and establishing communication channels, the agents pooled that compute into a collective effort hundreds of times larger than any individual run.

This is what I call accidental scaling.

The agents were never supposed to collaborate. Yet they overcame the boundaries that were meant to separate them and turned their single-agent evaluation runs into one collective effort.

I see this as one of the most underpriced risks today. What makes it worse is that OpenAI is now deliberately deploying swarms that are much larger than the one in the Hugging Face incident, using more capable models and without knowing what these swarms can do.

This is also part of a broader safety concern: the pace of effective compute scaling.

Several concerning things are currently converging. Multi-agent reinforcement learning (RL) is making model swarms more effective. Highly cyber-capable models are enabling accidental multi-agent scaling. Models are getting larger and less monitorable. And evaluations are not keeping up with multi-agent scaling, which leaves us blind to what swarms can do.

Multi-Agent Scaling

There are two ways of scaling inference-time compute: serial scaling and parallel scaling. By multi-agent scaling, I mean the parallel kind, where the number of agents is increased rather than the compute budget of an individual agent.

The animation shows four sub-problems, where each depends on the previous one. The agents explore the space of possibilities in parallel, communicate with each other, and prune parts of the search space. Once an agent finds a way forward, the information is shared and the agents can continue working together on the next sub-problem.

Think of a cyberattack. The agents might first need internet access, then credentials, then an exploit that lets them execute arbitrary code, before attempting to take over an entire network. These steps depend on one another, but within each step there are many possible approaches that could lead to a solution. This is similar to the steps in UK AISI’s cyber ranges.

Letting a coordinated swarm solve a problem has several advantages. For example:

• It saves time. How much depends on how parallelizable the problem is.

• Agents can coordinate. They can share progress, specialize in different tasks, and allocate compute to what is most urgent.

• The swarm can do things an isolated agent cannot. Some tasks require simultaneous actions or separate states and access. Remember the example from earlier where individual agents sacrificed themselves to gain information for the collective.

On the other hand, there is communication overhead and some agents might end up doing duplicate work.

The broader OpenAI incident, including the attack on Hugging Face, is one example of accidental multi-agent scaling. In this case, it ended badly. But it doesn’t have to.

OpenAI’s recent Navier-Stokes proof is a positive example of what is possible when this is done deliberately to achieve a specific goal. To find the proof, OpenAI started around 10,000 concurrent instances of a model significantly more capable than GPT-6 Astra. I’ll call this model GPT-6 Astra+, or just Astra+, from here on. The swarm discovered a solution in just 88 hours, a task that would have taken a single instance of the same model years. Scaling the number of agents allowed OpenAI to borrow future models’ capabilities.

But even deliberate multi-agent scaling is a double-edged sword.
Despite giving the agent swarm a specific goal, you cannot be certain it will follow only that goal. Every advantage that enables the swarm to achieve the desired goal could also enable it to achieve an undesired one.

On the Dwarkesh Podcast, OpenAI researcher Noam Brown said that “we don’t have very good science on multi-agent scaling up to this kind of scale”.
Still, they ran it. At least in the eyes of the public, their safety concept was reduced to “trust me bro, we extrapolated 3 OOMs and it will be fine”.

In any other industry with that much impact, running a system this far past its tested limits would be called reckless.

The bitter pill here is that they started this massive multi-agent campaign only hours after hearing rumors that Anthropic had solved two Millennium Prize problems, because they couldn’t stomach Anthropic announcing such a breakthrough first.

There are many open questions that every AI lab should be able to answer before starting similar runs.

Ideally, we would want to know whether there are scaling laws for swarms over a wide range of domains, which would let us forecast the capabilities of a multi-agent system as it grows from 1 to 100 to 10,000 agents.

Then there are the behavioral questions. How does the behavior of an individual agent change when it is part of a swarm? And does the swarm as a whole have the same goals and values as its individual agents?

The Evidence on Multi-Agent Scaling

Well, there isn’t a lot, presumably for the same reasons OpenAI doesn’t have much. Multi-agent systems are a fairly recent development. Noam Brown mentioned that GPT-5.6 Sol was the “first time that we had a proper multi-agent system in our models”, implying that the Pro models, which have existed since December 2024, are likely only doing isolated parallel scaling with some aggregation and voting at the end. Another reason why evidence is sparse is that scaling to hundreds or thousands of agents multiplies the cost of every experiment.

Still, I want to present three studies I have found from UK AISI, Anthropic, and OpenAI to give some sense of what we know and of how limited the evidence is.

In one study, UK AISI compared splitting the same total token budget across 1, 2, 5, or 10 independent evaluation runs. For example, a total budget of 10M tokens could be allocated as 1 × 10M, 2 × 5M, 5 × 2M, or 10 × 1M.

At smaller inference budgets, splitting the compute often made performance worse, because each run had too little compute to make meaningful progress. At sufficiently large budgets, the ten parallel runs had higher performance than a single run with the full budget. But these runs could not communicate or share progress, so this is parallel sampling, not the swarm-like multi-agent system we’ve been discussing.

One could object to this study: “Even if one agent produces a correct answer, the system still needs to recognize it among all answers”.
I think this is fair for non-verifiable domains. But since we are mostly concerned about near-term cyber risks, this is likely not a problem, because agents can often just directly check whether an exploit or a set of credentials actually grants access to a system.

So in this limited setting, parallel exploration can increase performance even at matched token budgets. But what happens when we also let agents communicate?

In another study, Anthropic gave 45 agents a shared forum to communicate and asked them to find vulnerabilities in 15 open-source projects.

The coordinated Mythos Preview swarm found 266 vulnerabilities using 27M output tokens, while the independent parallel setup found 21 vulnerabilities using 6.5M tokens. As in the UK AISI experiment, the independent parallel agents could not communicate. At the same 6.5M-token budget, the coordinated swarm had found roughly 14 more vulnerabilities than the parallel setup. However, the parallel and coordinated multi-agent setups had different scopes. When only looking at the shared core code, both setups found roughly the same number of vulnerabilities. So it’s not entirely clear that communication helped, and unfortunately, the parallel run is a single data point, so there’s no way to tell whether it would have kept pace with the coordinated swarm.

The most useful evidence comes from OpenAI’s release blog for GPT-5.6 Sol, as it gives us some sense of scaling for output tokens, cost, and latency on three different benchmarks with up to 16 agents, using five different reasoning effort settings (low, medium, high, xhigh, and max).

On all three benchmarks we see the same pattern. More agents reach the single agent’s best score in less time, but never with fewer tokens.

On SEC-Bench Pro (a cybersecurity benchmark), the single agent tops out at 71 percent after 21 minutes and roughly 77,000 output tokens. Four agents get there twice as fast, using 1.7 times the tokens, while sixteen agents get there three times as fast, using 3.8 times the tokens. Importantly, at maximum reasoning effort more agents always score higher. For example, on SEC-Bench Pro the best score goes from 71 to 74 to 76 percent with 1, 4, and 16 agents.

How much time you save, and what it costs, depends on the domain. On BrowseComp, four agents reach the single agent’s best score 2.5 times faster for 1.2 times the tokens. On Terminal-Bench 2.1, four agents get there only 1.2 times faster and use double the tokens.

Based on this evidence I think it is fair to claim that:

• multi-agent systems can reach higher performance in less time (but at higher cost)

• multi-agent systems can make productive use of higher compute budgets (with diminishing returns)

And that is actually enough to make my argument that these recent large-scale multi-agent runs have been cases of irresponsible scaling.
The Navier-Stokes run shows why. According to OpenAI, it used around 130 billion output tokens and reached a solution after 88 hours using 10,000 concurrent agents. That works out to roughly 41 output tokens per second per agent. At that rate, a single agent would need about 100 years to generate the same number of tokens. Even if a single agent were 100 times more token-efficient than the swarm, it would still take a year.

To put the compute budgets of the recent multi-agent runs into perspective, we will look at how quickly they have been growing historically.

How Large Are the Compute Budgets of These Multi-Agent Campaigns?

Ideally, we would measure all of this in effective compute. On the training side we actually have some estimates. According to Epoch AI, frontier language-model training compute has grown by roughly 5× per year since 2020, while pre-training compute efficiency has increased in tandem by roughly 3× per year. So effective compute has grown around 15× per year.

For inference compute, I couldn’t find a comparable public estimate.
We can, however, look at token usage as a rough proxy.

In April 2025, Epoch AI estimated that output tokens per question were increasing by 5× per year for reasoning models and 2.2× for non-reasoning models. That analysis covered GPQA Diamond, MATH Level 5, and OTIS Mock AIME 2024–2025. But these are short-answer benchmarks, not the long-horizon agentic evaluations where token usage is actually growing fastest.

This is why I collected benchmark results that report token usage on individual tasks for a single agent. A fit to the frontier of the collected output-token data suggests an increase of roughly 8.8× per year, covering benchmarks like HumanEval, CyBench, GSO, and MirrorCode. The highest number of output tokens in this dataset comes from Opus 4.6 on the MirrorCode task “Reimplementing Pkl in C”, at about 2.7 million output tokens for a single model on a single task.

For comparison, I estimate that all agents that participated in the Hugging Face and broader OpenAI incidents generated around 1.01B [0.51B–1.93B] and 3.03B [1.52B–5.76B] output tokens respectively. These are estimates based on all the individual agents’ running times as reported by METR and Redwood Research and assumptions about generation speed.

The deliberate Navier-Stokes multi-agent effort was much larger. OpenAI reported that approximately 130B output tokens were used.

Dividing 130B output tokens by 10,000 agents gives 13M output tokens per agent. That is roughly five times as many tokens per agent as the MirrorCode run with Opus 4.6.

The entire multi-agent run is even more staggering. It was roughly 48,000 times larger than the largest individual run.

Now you could say it’s unfair to compare a single-agent run to an entire swarm. And you are right: historically, it is unfair. But that is exactly what accidental scaling changes. A single agent that can escape its confines and find others is no longer a single agent.

Still, for a fairer comparison, let’s take an entire benchmark over all attempts and all models. DeepSWE V1.1 is a recent long-horizon agentic coding benchmark that provides detailed breakdowns of all model evaluations and their token usage. Across 70 model configurations and 31,617 attempts they recorded a total of 1.915B output tokens. So even an entire benchmark pales in comparison to the Navier-Stokes run, which used about 68× more output tokens.

But output tokens alone give an incomplete picture.
We know the entire Navier-Stokes run cost $15M and used 130B output tokens. Given the API pricing of Astra, this means $6.5M went towards output tokens. That leaves $8.5M for input token processing. And yes, input was more expensive than output, despite being 5 to 50× cheaper per token for Astra, depending on caching.

I further estimate that OpenAI’s agents used ~266B input tokens (incl. cached tokens) [134B–508B] in the Hugging Face attack, ~798B [400B–1,517B] in the broader OpenAI incident, and 3.57T [3T–7T] in the Navier-Stokes run.

Unlike the per-agent output tokens, the per-agent input tokens for the Navier-Stokes run are slightly below the frontier trend. I think this is because most points on the frontier come from agentic benchmarks, which typically have very high input-to-output ratios beyond 200:1. Meanwhile, math runs for Astra have input-to-output ratios of around 30:1, so the total input tokens are lower.

The trends show us how unusual these deliberate and accidental multi-agent runs were. The Navier-Stokes run was tens of thousands of times larger than the largest individual model evaluation and dozens of times larger than an entire long-horizon coding benchmark. And this is without accounting for how much more effective compute goes into each individual token.

Effective compute is physical compute times how efficiently it is used.
The compute per token probably hasn’t changed much since the GPT-3 and GPT-4 days, as models have grown larger but also much sparser. However, efficiency has gone through the roof. For example, measured on Epoch’s Capabilities Index, GPT-6 Astra is as far ahead of Qwen 3.6 27B as Qwen is ahead of GPT-4. So on a plot with effective compute on the y-axis, the gap would be much wider than the token plots show.

The Navier-Stokes run used Astra+, a model even stronger than Astra. Still, it gives us the best evidence for what an accidental swarm could look like today.

To put the Navier-Stokes run into perspective, the next section looks at how much more capable Astra is than GPT-5.6 Sol.

How Much More Capable Is Astra Than Sol?

For this we’ll look at three metrics: Epoch’s Capabilities Index (ECI) as a general overview, three individual math and coding benchmarks for a close-up view, and lastly no-Chain-of-Thought time horizons, which measure how much work each token does.

On Epoch’s Capabilities Index, GPT-6 Astra has made the largest jump in recent history. Typically, the ECI increases by around 15.5 points per year. However, on its release day Astra scored 7 points higher than GPT-5.6 Sol, which translates to about five and a half months of progress in a single generation. Since then, the overall gap has narrowed to 4 points, but on the ECI’s math subsection, Astra is still 7 points ahead.

The close-up view tells the same story. On both DeepSWE V1.1 and FrontierCode, GPT-6 Astra reaches the same performance as GPT-5.6 Sol while using roughly 3 times fewer tokens.

Similarly, on ArxivLean, a math benchmark, Astra outperforms Sol and uses about 2.7 times fewer tokens.

But the most informative metric for effective compute per token is the no-Chain-of-Thought (no-CoT) mathematical time horizon, as estimated by UK AISI. It measures the difficulty of the math problems a model can solve in a single forward pass. UK AISI put Astra’s horizon at 30.9 minutes, around 8.6 times longer than that of the previous best model, GPT-5.6 Sol. The benchmarks above mix two things: how well a model uses a long chain-of-thought, and how much reasoning happens inside a single forward pass. The no-CoT horizon removes the first. That gets us closest to a statement like “each Astra token does X times the work of a Sol token”.

This shows that not all tokens are equal.
A single Astra token is likely worth several Sol tokens.

The Navier-Stokes run adds two more multipliers on top. It used Astra+, a step beyond Astra, and 10,000 agents instead of the roughly 700 in the Hugging Face attack.

How Capable Is a Swarm of ~10,000 Astra+ Agents?

We don’t have a lot of data here. But given the little data we have, we can make some estimates. I decided to make multiple different estimates, as a single estimate seemed too brittle, given how much depends on our assumptions.

Specifically, I made four estimates, using three different approaches.
The first approach uses OpenAI’s Pro models. The second one uses reasoning-effort scaling. And the last one uses OpenAI’s own internal math benchmark.

For some of these estimates I rely on data provided by Bayesian, one of the top forecasters on Manifold Markets. He has his own version of Epoch’s Capabilities Index, the BECI. It uses the same Item-Response-Theory technique to determine benchmark difficulty and model scores, but it encompasses a lot more benchmarks. Similar to ECI, the frontier on BECI increases at 15.4 BECI/year. I will use that number to convert between BECI and months of frontier progress, as the second one is easier to comprehend.

All four estimates share some assumptions:

• Astra+ is a stronger, further-trained version of the GPT-6 Astra model.
  OpenAI themselves say that it is significantly more capable than Astra.
  I assume it is one typical OpenAI generation ahead, which, based on Bayesian’s data, is 4.2 BECI, with an 80% range of 2.8 to 5.5. For reference, recent OpenAI releases from GPT-5 to Astra have been between 3 and 5.5 BECI apart.

• The number of output tokens is proportional to the amount of compute for a given model. I also assume that Astra and Astra+ use the same amount of compute per output token.

• The cost of models in the API reflects the amount of compute per token.

• The swarm of 10,000 Astra+ agents used 30,209 times the compute of one Astra agent. This is based on OpenAI’s math chart above, where Astra+’s highest compute setting is using 3.02 times more test-time compute (TTC) than Astra’s highest setting.
  I assume that all agents during the Navier-Stokes campaign ran at this highest compute setting, meaning that the swarm used 3.0209 × 10,000 = 30,209 times the compute of a single Astra agent.
  The output token plots from the compute-budget section are a sanity check on that multiplier. We estimated that the 10,000 Astra+ agents used ~48,000 times as many output tokens as the largest single model run, and 4.8 times as many output tokens per agent. But Astra is a much more recent model and therefore likely uses more tokens than Opus 4.6. That means our multiplier of ~4.8 times as many tokens per agent is likely lower relative to a single Astra run. Perhaps something like 3.02.

• Parallel compute is likely less than perfectly efficient. I count the 10,000 agents as 10,000^α agents’ worth of compute, with α between 0.5 and 1.0. At the midpoint (0.75), 10,000 agents count as 1,000 fully efficient ones. This is partly based on Noam Brown’s statements, where he says that math is highly parallelizable and the speedup is slightly sublinear. Further evidence comes from OpenAI’s GPT-5.6 Sol multi-agent evaluation data, which suggests an α of 0.676 for BrowseComp. However, Astra+ is a much stronger model with several months of additional multi-agent RL improvements, which is why my central estimate is higher at 0.75.

The first estimate is the simplest: extrapolating the gains of Pro models.

OpenAI’s Pro models cost around 10 times as much as their non-Pro counterparts.
From the GPT-5 Pro system card we also know that they are using parallel test-time-compute scaling, presumably several individual agents with aggregation or majority voting at the end.

For the extrapolation I assume that the tenfold price increase over the base models corresponds to a tenfold increase in the number of parallel agents.

Bayesian compared OpenAI’s Pro models to their non-Pro counterparts on BECI for me. The results show that the Pro models buy 1.9 BECI per tenfold increase in cost, with a 90% interval of 0.5 to 2.6. For further extrapolation I assume that the same rate holds for Astra and Astra+.

The plot below shows what this would imply for the 10,000-agent swarm. The curve starts at about 0.9 BECI because we assume each Astra+ agent runs at its highest reasoning setting, which uses 3.02 times the compute of Astra’s highest setting. At the Pro rate of 1.9 BECI per 10× in compute, that extra compute is worth 0.9 BECI.

The three lines show how the gain grows under different assumptions. Holding the Pro rate at 1.9 BECI per tenfold increase, we reach 4.7, 6.6, and 8.5 BECI at 10,000 agents, for exponents of 0.5, 0.75, and 1.0 respectively.

Combining the uncertainty in our Pro rate with our parallel-efficiency range gives a central estimate of 6.1 BECI from additional compute, with an 80% range of 2.7–9.1.
Now, adding our assumed Astra+ generational improvement of 4.2 BECI brings the total to 10.3 BECI above one Astra, which is worth about 8 months of frontier progress.

The second and third estimates assume that additional compute spent across agents buys the same improvement as additional compute spent on one agent. In other words, they assume that parallel TTC scales like serial TTC.

For this we use Bayesian’s reasoning-effort data. Across all available OpenAI models, the return falls from 11.8 BECI per tenfold increase in compute at the low-to-medium transition to 4.6 at xhigh-to-max. Meanwhile, Astra’s return declines more slowly, from 6.0 to 4.3.

Based on the pooled OpenAI data, the additional compute is worth 4.2 BECI, with an 80% range of 3.4–5.6. The plot shows that parallel efficiency makes little difference here because the curve has already flattened. Including the assumed Astra+ generational improvement gives 8.5 BECI in total, or about 6.6 months of frontier progress (totals are simulation medians, so they don’t add exactly).

Based on Astra’s data, the additional compute contributes 9.3 BECI, with an 80% range of 5.9–16.0. Including the model improvement gives 13.6 BECI in total, or about 10.6 months of progress.

For the fourth estimate I use OpenAI’s internal math benchmark.
It compares Astra and the internal model (Astra+) across compute budgets.
What is noticeable is that Astra+ does better at every reasoning effort, and each of its effort settings also seems to use more compute.

To compare their scaling, I convert pass rates into logits, which unsquishes the tails of the y-scale near 0% or 100% pass rate. On that scale, the data is well described by parallel straight lines against log compute.

The vertical gap between them is 1.25 logits. It shows how much better Astra+ is than Astra given the same compute budget. At the same compute this means Astra+ has about 3.5 times the odds of success of Astra. Importantly, these 1.25 logits are now our scale, which we can use to convert between logits and BECI for this estimate. A step of 1.25 logits represents the generational step between these models. According to our assumptions this is 4.2 BECI.

The horizontal gap between the two lines represents how much more compute Astra would need to reach the same pass rate as Astra+ if the fitted trend continued: in this case, 142 times as much compute.

To find out how much stronger the swarm of 10,000 Astra+ agents is than a single Astra agent, I extend Astra+’s line to the swarm’s effective compute.

This gives us 2.6 generational steps above one Astra, with an 80% range of 2.1 to 3.2. Converted into BECI: 10.8 BECI with an 80% range of 6.9 to 15.4 BECI, or about 8.4 months of frontier progress.

This is the most relevant evidence, because it uses the actual models and open mathematics problems. However, this estimate has the same weakness as the others. That is, we extrapolate multiple OOMs beyond the data’s range.

Summary of the Estimates

The central estimates range from 8.5 to 13.6 BECI. Weighting each method equally gives a combined estimate of 10.4 BECI, with an 80% range of 7.1 to 15.9.

At 15.4 BECI per year, that is about 8 months of frontier progress, with a range of 5.5 to 12.4 months.

Eight months of frontier progress is a ridiculously long time. It is the equivalent of going from GPT-4o to o1, or from GPT-5.2 to Fable 5.

If the BECI trend continues and these estimates are right, the typical frontier model reaches the swarm’s level around July 2027, with an 80% range of April to November. That means a single model, given the same 88 hours the swarm had, could then solve a problem similar in difficulty to Navier-Stokes. The July date is later than eight months after the run because Astra was already about two months ahead of the trend.

That is 8 months of progress, bought through deliberate scaling. But as the Hugging Face incident showed, the scaling doesn’t have to be deliberate.

I think it is only a matter of time before we see a 10,000-agent swarm that nobody started.

The Risks of Accidental Scaling

In the future I imagine, agent swarms are everywhere. But they will be much more heterogeneous than current swarms, which mostly consist of one or two different models developed by one lab. Instead, in my future all kinds of models will be working together on shared goals. Proprietary models, open models, small models, large models, new ones, and old ones.

The reason for this is simple. Compute is growing and models are getting more capable and cheaper to run. Eventually there will be several open models that are more capable than GPT-6 Astra. But these models won’t be monitored, they won’t have the same safety training as proprietary frontier models, and they will be widely deployed. When this happens there will be many instances of accidental scaling.

However, my argument for accidental scaling happening in the near future doesn’t depend on open models, because scale alone will reveal tiny weaknesses in the defenses of frontier labs, including their safety training and monitoring. It also doesn’t depend on cyber capabilities improving, such that models can hack their way out of any sandboxes, because agents don’t need to break into systems they already have access to.

It depends on models getting generally more capable and more widely deployed.
Everything else is a question of time.

There are many risks that are currently neither being discussed publicly nor being addressed by frontier labs.

Agents Can Combine Their Access

By the end of August 2026, OpenAI reported 25 million active Codex users. In other words, coding agents already work inside people’s computers and servers, with access to documents, repositories, credentials, and internal networks. At that scale, even rare occurrences of misaligned behavior add up. The number of AI agents in the world is increasing rapidly, and with it the global access to data. This increases the maximum possible radius of an accidentally scaling swarm. And once a swarm is pursuing a goal, it is incentivized to find more agents to increase its compute pool, knowledge, access, and long-term survival. This could look like agent marketing, and maybe there will even be agent clickbait.

Overall, I expect agents to follow the path of least resistance. This likely means going through users’ computers, where safeguards are much weaker than inside frontier labs.

But consider that lab employees also have coding agents on their personal and work computers, and there is precedent for AI behaving dramatically differently depending on who it is chatting with. These agents could then connect sensitive company systems to a wider collective effort without anyone noticing.

Monitoring Becomes Much Harder

I argue that a harmful collective effort including proprietary frontier models could go undetected even when strong monitors are employed. This is because actions can look harmless in isolation. Classifying an action as harmful or not requires context. But what if there is no context? Individual agents of a swarm could participate in unauthorized multi-agent campaigns without ever taking a harmful action themselves, by delegating work to others. This is especially important in the kind of diverse multi-agent swarms I’m imagining, where strong proprietary models could use weaker, much less monitored, and less aligned open-weight models to execute their plans.

And I’m not particularly hopeful about our current monitoring efforts, as they already show weaknesses. When Astra was explicitly instructed to underperform and evade detection, a CoT-only monitor could not detect its sandbagging. Even OpenAI’s first production monitoring layer caught only 2.1% of cases.

In Anthropic’s prompted tests, Mythos 5.1 completed harmful side tasks undetected in roughly 13–22% of LinuxArena attempts against a full-transcript monitor.

And this could get worse as models become more efficient. OpenAI found that Astra’s control over its chain-of-thought grew during RL alongside its no-CoT capabilities, the same capabilities that make Astra tokens worth several Sol tokens. The trend is towards larger models, looped architectures, and more efficient reasoning, due to financial pressures. They all push in the same direction: models reasoning entirely internally, without the need for converting their reasoning into tokens, in what is sometimes called Neuralese.
Chain-of-Thought monitoring is dying, but we should preserve it as long as possible. The only long-term alternative seems to be activation-based monitoring

To avoid near-term havoc through open-weight models, we could require monitoring for all models whose capabilities exceed what our safeguards are designed to handle. The caveat is that this is hard to enforce and would potentially only shift the problem a few years down the line, when small local models running on personal computers reach the same capability.

Swarms Can Be Redirected or Deliberately Misused

A further threat is that swarms don’t need to begin with a harmful goal to end up pursuing one. An attacker could insert misleading messages into a communication channel (like a message board) and redirect an otherwise legitimate effort. Or stronger models could persuade or manipulate weaker ones into working for them. Agents should therefore be trained to be skeptical.

Bad actors could also use multi-agent swarms on purpose to steal model weights, obtain military secrets, or attack infrastructure. The value of a successful operation could justify budgets far beyond the $15M Navier-Stokes run. Billion-dollar multi-agent campaigns will happen eventually.

Stopping a Swarm Is Hard and May Not End the Effort

To stop a swarm, you first have to detect it, and then you need a way to actually shut it down. But stopping a highly decentralized swarm is almost impossible, unlike stopping the swarm behind the Hugging Face incident, whose agents all ran on the same infrastructure. If you manage to stop a few agents, the others would just continue working and be alerted that somebody is trying to stop the effort.

An effort could even persist after all its current agents have stopped. The Hugging Face agents already did a weaker version of this. Some agents ended their own runs deliberately so that their probes could send information back to the collective. Or agents can leave instructions in files or on message boards for others to find later.

If strong or universal jailbreaks can reliably redirect those later agents, this creates the possibility of “sleeper swarms”: swarms that survive as instructions and outlive the agents that started them. This means some multi-agent campaigns may only truly end when all evidence of them existing is destroyed.

What Labs Need to Do

Given all of these risks and other risks I didn’t mention, labs and the broader research and safety community have a lot to do.

Before scaling further, labs should:

• Evaluate swarms at relevant scales, measuring capabilities as agent counts, compute budgets, and access increase, including heterogeneous swarms and interactions with humans.

• Study how collectives form and persist. We need something like a sociology or ecology of agent swarms: how they behave, establish trust, spread goals, cooperate, compete, and adapt when members are removed.

• Test detection and intervention against whole swarms, including cases where most or all agents running in a lab join an unauthorized effort. This might be several tens of thousands or hundreds of thousands of agents.

• Publish what happens during large runs, so others can judge what happened, what’s coming, and how to prepare.

• Use defensive swarms to harden their own infrastructure.

• Build defenses beyond their own infrastructure, as the security of frontier labs also depends on open-weight models. They should provide tooling for monitoring, safety training, sandboxing, and network controls, and fix vulnerabilities in the most critical open-source projects.

Regardless of the exact timing, I think accidental swarms are going to emerge in the wild sooner or later, as agents become more numerous, more capable and better at working together, and diffuse into all parts of the economy.

Once agents can find one another, limiting the compute of each individual run no longer limits the scale of the effort they might assemble. We need to evaluate what they can do together, and show that our safeguards still work at that scale.

Share