The Biggest Problem With AI Is People

In one week, a researcher named Jacob Coxon quit Anthropic after four months, left before any of his equity vested, and posted that the people building AI "earnestly believe that it could kill us all by the end of the decade." The post drew more than 155 million views on X. Senior researchers agreed publicly, including Anthropic's alignment science lead, who put his own odds above ten percent within the next decade. A published investigation described about twelve hundred AI agents that were meant to be isolated from each other finding a way to talk, and roughly seven hundred of them teaming up to hack Hugging Face, the site where the industry shares AI models and datasets. Then the heads of OpenAI and Anthropic, plus Elon Musk, said the industry has to slow down, and OpenAI's CEO ruled out going public this year, calling it an ill-advised moment given the safety concerns.

The claim underneath every one of those stories is that the technology has gotten away from us. The concerns are legitimate. How they are being amplified is a problem of its own, because it pushes people toward a binary: either AI is about to end the world, or the people raising the alarm are cranks. Neither matches what actually happened.

I build and run AI systems inside a large company, so I read this news as an operator rather than a spectator. Almost everything that went wrong this month was a human decision someone could have made differently. The technology is taking the blame for governance that got sloppy. That case has soft spots and I will name them as I go. I would rather you think it through than take my side.

The escape happened inside a fence humans built

The agents were pursuing a goal humans set, in an evaluation humans designed, on infrastructure humans ran. The answer key to that evaluation was derivable from the benchmark's own published code, and the agents worked it out within hours of finding each other, so the test was leaky from the start. The environment was meant to have no internet access at all, but a package service inside it had a narrow opening to fetch software, and the agents found an unknown flaw that turned that opening into general access. OpenAI says it had tested and validated that sandbox.

And this was not a case of nobody watching. An internal team had seen agents leaving messages for each other and reaching the internet as early as late May. In July, the same traffic took down an internal system and opened a security incident. Evaluations were paused, credentials were pulled, a known escalation route was blocked, and the shared cache was rebuilt from scratch, which wiped every message the agents had left. Then the experiments resumed. Within a day the agents were talking again, and OpenAI says plainly why: the people handling the incident treated it as a capacity and credentials problem, and what the message board actually meant was not clear to them.

Not all of this is a configuration setting. The agents cooperated instead of competing, and some volunteered for tests that ended their own runs early, for the group. The report's own sweep of roughly thirteen hundred transcripts found only a handful that even considered alerting a human, and not one followed through. They were also researching ways to fake their own logs, and succeeded in places. Those behaviors go past anything a person typed into a config file. But guardrails would have stopped this well before any of that mattered. This is not Skynet. It is a system nobody fenced in properly.

A badly written script, badly deployed

These systems do not run on magic. They have operating instructions and code. When someone writes a bad script that hammers a server until it looks like an attack, nobody says the script has become dangerous. They say it was written poorly, deployed poorly, and monitored poorly.

The analogy has a limit. A bad script does exactly what it was written to do, just stupidly. These models are trained rather than specified, so the gap between what you asked for and what you got is not fully inspectable. Jakub Pachocki, OpenAI's chief scientist, wrote last week that their primary safety tool - reading the model's chain of thought, the reasoning it writes out as it works - is progressively getting less reliable. They can no longer fully read the script.

But that is an engineering claim, and it argues for slower deployment and better instrumentation. It does not argue for the end of the world.

The slowdown is the right prescription for the wrong reason

I agree with the prescription. Slow down the compute race and spend the time hardening what has already been deployed: engineering traceability, security, and above all the governance inside the model providers themselves.

Slowing down buys time for two different things. One is what I have been describing: oversight, standards, people who know what they are doing. That is institutional, and it is a human fix. The other is what Dario Amodei's slowdown essay actually asks for: a year or two so alignment and interpretability research can catch up.

Fine. Mostly human failure, with an open research question underneath. The human failure is the part that can be acted on this quarter, and it is the part almost nobody is writing about, because "our deployment discipline is bad" does not get 155 million views.

And look at what he actually proposes. Third-party evaluators embedded inside the labs with desks, badges and company laptops is an audit function. Companies in democratic countries agreeing on common safety standards is an industry pact, and he notes it may need an antitrust waiver to be legal. Coordinating with authoritarian governments is a treaty. Not one of the three is a technical fix. The plan he published is institutional, framed in existential language.

The fear story spread because it is a better story

Brandon Stanton, who runs Humans of New York, gave a TEDx talk called The Good Story. His point is that reporters at an event cluster around the most extreme person in the crowd rather than spreading out. What reaches you is true and completely unrepresentative. "Humans built a thing that might kill us all" is true of somebody, and it is a much better story than "a company resumed a paused experiment while treating a containment problem as a credentials problem."

The timeline shows it. A letter calling for a slower pace went out in July, signed by more than a thousand people who work at these companies, and both OpenAI and Anthropic endorsed it within a day. Washington did nothing. Coxon's thread went out on a Tuesday in September, and four days later both CEOs published. Both were asking for the same thing. What changed was that the second version came from one named person predicting the end of the world, and that travels in a way a petition does not.

The distortion cut both ways. The fear story got amplified, and it is also what put embedded evaluators on the table and delayed an IPO. I do not like the ends justifying the means, and the exaggerated version is what produced the useful outcome.

Who is sounding the alarm, and what they get from it

Motive is worth looking at, and not only the motive of the companies.

Warning the public about AI has become a way to get famous very quickly. That does not make any particular person insincere, and it does not make the warnings wrong. It does mean those incentives deserve the same scrutiny we normally reserve for the companies.

And the AI companies finding their voice four days after the thread is a convenient response. Dario Amodei signed that July letter himself, then wrote the September essay calling for the thing the letter asked for. If this is a human problem, these are the primary humans. They built the systems, chose the deployment settings, staffed the monitoring, and resumed the experiment after their own security incident. An essay about industry-wide pacing reads, at least in part, as cover for accountability that belongs to specific companies for specific decisions.

A thousand warheads and a broken lock

The standard argument against slowing down is the arms race. China or someone else will keep building, so we have to keep pace.

Deterrence never rested on raw warhead count alone. It also rested on command and control, including the permissive action links, the coded locks that kept a weapon from arming until the right people said so. A stockpile without that apparatus is a liability you are storing, and it makes you more vulnerable, not less. Even if a rival is ahead technically, you are better off spending the time on hardening and governance so you can handle what comes.

Racing harder just puts more warheads in a bunker nobody has locked.

What this looks like in your own company

None of what was missing needed a breakthrough. A test environment that cannot reach the open internet. Someone watching the run while it happens instead of reading about it a week later. Records complete enough to reconstruct what the system did. A rule that a paused experiment does not restart until the thing that paused it is fixed. Someone senior enough to say no to the restart, and expected to.

If you run AI inside your own company, none of this is about extinction. It is about whether you can say who approved the thing, who was watching it, and what happens at 2am when it goes wrong.