The problem with talking about AI in absolutes
On September 8, a 27-year-old researcher posted seven messages on X and became famous before lunch.
His name is Jacob Coxon. He’d spent three years working on pretraining research, first at OpenAI and then at Anthropic. He quit and explained why: in his view, neither company is behaving responsibly. They’re racing toward systems capable of improving themselves and, in his words, gambling with our lives.
The thread crossed 100 million views. Within 72 hours, Coxon was everywhere.
Then it got louder. Evan Hubinger, who leads Alignment Science at Anthropic and still works there, publicly agreed with Coxon and added his own number: he personally puts the probability that AI kills all humans within the next decade at more than 10%.
Not a 10% chance of a recession.
Not a 10% chance of millions of jobs disappearing.
More than a 10% chance that we disappear.
Two details make the story more uncomfortable. Coxon had been at Anthropic for such a short time that he walked away before his equity vested, so the easy “he did it for the money” explanation doesn’t hold up well. And all of this is happening while Anthropic prepares what could become one of the biggest IPOs in history.
Naturally, the internet responded with restraint and nuance.
Just kidding.
Within hours, the camps were already forming.
Two Certainties
On one side, the doomers. Bernie Sanders backed the warning publicly and, together with Representative Greg Casar, announced legislation aimed at banning superintelligence and pausing advanced AI development. Hubinger’s 10% started moving around the internet as if it were a measured statistic, like an inflation rate or a mortality figure.
One of the best reasons to be skeptical of that number comes from someone firmly on the alarmist side. Geoffrey Hinton, Nobel Prize winner and one of the fathers of modern AI, puts the probability somewhere around 10 to 20%. He’s also been blunt about what numbers like these actually are.
On the other side are the messiahs, and some of them aren’t exactly subtle. Marc Andreessen has called AI potentially “the most important — and best — thing our civilization has ever created,” and in his Techno-Optimist Manifesto he went further: slowing AI down will cost lives, and deaths that could have been prevented by AI we chose not to build are “a form of murder.” Jensen Huang has pushed back just as hard against AI pessimism, arguing that the fear itself does damage by scaring people and companies away from investing in a technology with enormous upside.
So we end up with two versions of the future. In one, Terminator. In the other, AI cures disease, fixes education, unlocks productivity, solves climate change, eliminates poverty and probably tells you where you left your keys.
I have a problem with both, and it isn’t that either scenario is impossible. It’s that both convert uncertainty into certainty. When we’re talking about a technology whose ultimate reach we clearly don’t understand yet, certainty may be the least defensible position available.
There is a third angle
One reaction didn’t fit into either camp. David Sacks, the former White House AI czar, didn’t argue that the risks were fake. He suggested there might be advocacy interests and regulatory capture behind the way the Coxon story spread, and called for Anthropic’s IPO to be paused while the claims were investigated.
The underlying point survives even if the specific accusation doesn’t. There are incentives on both sides of this debate. Fear has beneficiaries, and so does the absence of fear. That doesn’t make either argument wrong. It means we should apply the same skepticism to everyone, including the people saying exactly what we already wanted to hear.
We don’t debate ideas anymore. We pick teams.
This obviously didn’t start with AI. We’ve gotten very good at treating complex issues like sporting events: pick a side first, find the arguments later.
Nuclear energy will either save the planet or destroy it. Remote work is either the inevitable future of work or the end of corporate culture. Social media either democratized information or destroyed society. Electric vehicles, crypto, weight-loss drugs, GMOs, immigration, arms control, the Lindsay Clancy case, and your local politician: pick almost any issue and there seems to be an unwritten requirement to join one of two extreme camps.
Reality is rarely that simple.
I don’t think extremism is necessarily a sign of deep conviction, either. More often it’s a shortcut. It’s much cheaper to have a position than to have a model. You can get a position in thirty seconds, and it hands you an identity, a community and something to say at dinner. A model takes work, forces you to change your mind when new information shows up, and usually lands on the least satisfying answers available:
It depends. I don’t know. It’s not that simple.
AI is not one thing
We talk about “AI” as if it were a single object. A conversational assistant, an agent running code on your computer, a system helping discover new drugs and a hypothetical autonomous superintelligence all get thrown into the same bucket. They aren’t the same thing, and the public conversation constantly blurs concepts that are technically distinct.
Capability isn’t autonomy. Autonomy isn’t agency. Agency doesn’t necessarily imply self-generated goals. And none of those things automatically means loss of control. A model outperforming almost every human at certain intellectual tasks doesn’t mean it’s going to wake up on a Tuesday morning and decide to invade Poland.
The opposite argument is just as weak. In 1995 the internet already existed and worked; the reassurance was that nobody would ever put a credit card into it, or bank on it, or let it hold anything that mattered. That turned out to be wrong within a decade, and not because the technology leapt forward. Because we kept plugging it into things.
Saying “today’s models can’t do that” doesn’t answer the question that matters, which is what happens as we combine increasingly capable models with memory, tools, internet access, code execution, business systems and real infrastructure, and then let those systems run for longer stretches without a human in the loop.
There’s a second problem. When people say “AI risk,” they’re usually mixing at least three different conversations.
The first is human misuse: fraud, cyberattacks, manipulation, weapons.
The second is economic and systemic risk: job displacement, concentration of power, misinformation, and growing dependence on systems we understand less and less.
The third is far more speculative and potentially far more serious: loss of control over future systems dramatically more capable than today’s.
You can believe the first risk is enormous, the second manageable and the third remote. Or the exact opposite. They’re different problems, supported by different evidence and requiring different remedies. A cloned-voice fraud attack and a hypothetical superintelligence evading human oversight are not going to be solved by the same law.
Collapsing all of that into “AI will kill us” versus “AI will save us” makes for great tweets and terrible public policy.
What actually breaks
I’ve spent years helping implement conversational AI inside banks, insurance companies, retailers and telecom companies. From that vantage point, I can say one thing with some confidence: neither extreme does a good job of describing what happens once AI leaves the demo and enters an actual business process.
The doomer isn’t much help when a model hallucinates inside a collections flow. The messiah told the client AI would make the contact center disappear, and six months later the project is still a pilot because nobody defined what happens when the model doesn’t know the answer and needs to hand the customer to a human. Both of them look equally bad in front of an operational risk committee. Neither argument gets you very far once you’re sitting in one.
The part I find strange is that both extremes produce the same paralysis by opposite routes. The person who believes AI is going to kill us doesn’t implement it. The person who believes AI solves everything doesn’t implement it properly: no controls, no traceability, no defined handoffs, no measurement, nobody who owns the outcome.
The problems we can actually observe and measure in production today are mostly governance problems. A piece of personal data ends up somewhere it shouldn’t. An agent takes an action nobody authorized. A customer gets the wrong quote in a regulated channel. Six months later, nobody can explain to Compliance why the system behaved the way it did. None of it is cinematic. All of it is expensive.
That doesn’t make existential risk imaginary or ridiculous. It puts it in a different category: much harder to measure, much more uncertain, and, if the darker scenarios are right, vastly more consequential. I can’t resolve that debate and neither can anyone else. But learning to govern the systems we have now is the only practice we’re going to get for governing far more capable ones later.
The last time something like this happened
The closest historical analogy is probably the Industrial Revolution. Machines arrived before the rules were ready, and factories produced enormous productivity gains alongside brutal working conditions, dangerous workplaces and child labor.
The simplified version of that history is wrong, though. Governments didn’t ignore the problem for seventy years and then suddenly discover regulation. Britain was passing factory laws in the early 1800s. What took decades was understanding what needed regulating and, more importantly, how to enforce it. The Factory Act of 1833 mattered partly because it created a corps of government inspectors. At first there were four for thousands of factories.
Trade unions took much longer to establish themselves fully. The eight-hour workday came later still. So the lesson isn’t that we did nothing for decades. It’s something more useful:
We regulated early, but we learned slowly what to regulate and how to enforce it.
Aviation is the encouraging version of the same story. The Wright brothers flew in 1903; the FAA didn’t exist until 1958, and it arrived after a mid-air collision over the Grand Canyon killed everyone on both planes. Fifty-five years of learning, most of it from wreckage. But it worked. Flying is now the safest way to travel, under a regime that scales with consequence — a hobbyist’s ultralight and a commercial airliner don’t live under the same rules.
Two industries, same lesson: the rules eventually got good. They just got good slowly, and the tuition was paid by people who weren’t consulted.
Here’s where the analogies start to break. If you wanted ten times the industrial output, you had to build factories, buy land, install machinery, hire people, and secure raw materials, transportation and capital. The physical world created friction.
AI still depends on the physical world: chips, data centers, energy, cloud infrastructure, capital and human talent. Many of those chokepoints are remarkably concentrated. But once a digital capability exists, it can spread globally at a speed industrial technology never could. And for the first time, the tool itself is starting to participate in building the next generation of tools.
There’s another difference, and it’s the one I keep coming back to. In a factory, the worker wasn’t only part of the productive system. The worker had leverage over it. Workers could stop the line. Most of the labor architecture we built over two centuries — unions, labor law, inspections, social protections — grew up around that bargaining power.
Which brings us back to Coxon. He did exactly what a worker who believes the system is dangerous is supposed to be able to do. He withdrew his labor.
He quit.
His resignation generated more than 100 million views and a global political argument.
And the race kept going.
That’s roughly his own diagnosis of the industry: the people inside understand the risk, but the companies are trapped in a race to get there first, because each believes that if it slows down, somebody less responsible won’t. The control points still exist: chips, energy, data centers, capital, talent, regulation — but many of the institutions best positioned to slow things down are also the ones facing some of the strongest reasons not to. Competition, capital, geopolitics, markets and prestige all point the same direction.
Coxon removed himself from the race. The race did not remove itself.
The problem isn’t that there’s no brake. It’s that our reaction cycle may be slower than the capability cycle.
This is not regulation versus innovation
Which is why I think the question is being framed wrong.
“Stop everything until we can prove it’s completely safe” sounds prudent and is close to impossible. We’ve never required absolute certainty before developing a general-purpose technology, and it’s hard to picture one national law halting the global development of knowledge. Slowing things down indiscriminately has a cost too: AI is already accelerating scientific research, software development, medicine, education and productivity.
But “move as fast as possible and deal with the problems later” doesn’t work either, for the reason above. Later may arrive after the problem does.
The choice shouldn’t be innovation or regulation. It should be regulation that scales with capability.
A customer-service assistant answering FAQs shouldn’t live under the same regime as an autonomous agent that can execute code, move money or touch critical infrastructure. As capability rises, as autonomy rises, and as the consequences of failure rise, the controls should rise with them: more testing, more traceability, more monitoring, clearer ownership.
We don’t need to decide today what rules AI will require ten years from now. We need mechanisms that can evolve as the technology evolves.
None of this is radical. It’s how any serious enterprise deployment already works — the level of control matches the consequence of failure. We just haven’t moved the idea from the project room into public policy.
Neither apocalypse nor messiah
I don’t know whether the probability that some future superintelligence wipes out humanity is 90%, 10%, 1% or 0.01%.
But I don’t need to believe the end of the world is likely to accept something much simpler. If a technology may eventually become extraordinarily powerful, it’s better to think about safety mechanisms before we need them than to improvise them afterward.
Days after Coxon’s resignation, Anthropic CEO Dario Amodei published a long essay arguing that the industry should deliberately slow the pace at which frontier capabilities improve, so that safety work can catch up. He didn’t stop at “we should be careful.” He proposed a three-part plan, and Anthropic committed unilaterally to the first piece: giving third-party evaluators permanent, employee-level access to its systems so they can verify safety measures, report incidents and assess model alignment during training.
That’s almost exactly what controls that scale with capability look like in practice — an independent check built into the system while the capability is being built, rather than a promise that the lab will police itself. Elon Musk, not famous for wanting technology to move slowly, responded that Dario was right.
Amodei also runs one of the companies with the most to gain from this race, and he published it while Anthropic prepares a massive IPO. Maybe it’s sincerity, maybe it’s positioning, maybe it’s both. I don’t know, and saying so strikes me as more honest than picking whichever explanation happens to support the argument I already wanted to make. What it does demonstrate is that asking capability and safety to advance together isn’t inherently anti-technology.
Fear by itself doesn’t build controls. Institutions and rules do.
The doomers are pointing at something real: we’re building systems whose ultimate potential we don’t fully understand. The messiahs are pointing at something real too. The mistake is turning either possibility into destiny.
One last thing I feel strongly about. These rules can’t be written without people who understand what these systems actually do when they leave the lab and enter a bank, an insurance company, a live customer process — real data, real people, real consequences. That doesn’t mean practitioners should write them alone. We need safety researchers, governments, academia, companies and civil society. But regulation drafted without anyone who has watched these systems break in production tends to protect too little while getting in the way too much.
So my concern isn’t that AI is moving too fast.
It’s that it may move faster than our ability to understand what breaks and build the controls that would contain it.
I’m both afraid and excited about the future of AI.
Leave a Reply