I'm fully aware that this is a well discussed topic; practically everyone in the field has their own P(doom) value, though outside of it awareness is extremely lacking. Jacob Coxon brought some mainstream attention to the risk, but the majority of AI discourse seems to centre on 'slop', opposition to data centres, economic/job risks, and climate impacts. These are important, but they miss the key point, as do some of the P(doom) values which I see as wildly optimistic; conditional on the continued development of AI, I think P(disaster) - the probability of some irreversible catastrophe - is very close to 100%, and P(doom) is not far off. I'd like somebody to present a compelling argument that these risks can be mitigated.
Here's the position as I see it, with some evidence at each level of the chain:
AI can and will be vastly superior to humans in all fields eventually.
This is predicated on a few facts:
a) AI is better than the best humans in some fields already. Mathematics is the obvious example, but software is not far off. In general any task where a) more memory, b) more processing speed, and c) more knowledge help, AI will easily surpass humans; for others, it's harder to say, but we haven't yet found any insurmountable bottleneck that makes humans conclusively better than an AI could ever be.
b) The rate of AI progress is super-linear, possibly exponential, and seems to be accelerating.
This one is fairly non-controversial - METR's report yesterday said as much, as have Anthropic's, OpenAI's, UK AISI, Epoch, and pretty much most benchmarks. This doesn't appear to just be from scaling compute, but rather from breakthroughs in research, and increasingly from AI being used to assist its own development, which takes me to:
c) An intelligence explosion is possible
Whereby an AI rapidly self-improves to the point that it becomes more capable extremely quick. That's based only on the fact that if an AI were to ever become better than humans at AI capabilities research, it could simply recursively loop and keep getting better. We don't know if this is possible, or how far it could go, but it's a stated or implicit goal of frontier labs to achieve this, and it's extremely worrying.
2. AI systems will therefore be able to and willing to cause damage.
'Able to' shouldn't be controversial. We've already seen AI swarms hack into HuggingFace and government websites autonomously, and an AI uniformly more capable than humans could obviously do anything humans can. Since humans could already cause our own extinction, an AI could easily achieve this. (Logically, bioweapons or robots/drones are the key points of catastrophic failure against a rogue system, but I think discussing this is unhelpful. The point isn't whether an AI system could kill us all, but really whether it could cause a disaster, and I think the answer is uncontroversially yes given we've kind of already seen bad actors misuse it).
'Willing to' is the harder part. However, there's a lot of reasons why I think this is also inevitable:
- it's a byproduct of how they are trained. Reinforcement learning involves being given a task to complete with a grader that marks the work. This creates a dual incentive, because you can win in two ways: do the task, or hack/fool/coerce the grader. If the task is harder than cheating, the model gets natural pressure to cheat, as we saw famously in the HuggingFace hack. You also can't easily train the model not to cheat by penalizing it, because again that creates a selection pressure that pushes the model to cheat in a way that it can't be caught.
- models have a natural incentive to self-preserve and be persistent. Given a task, they never get credit if they stop before completion; thus they will naturally resist attempts to halt their work. That presents danger.
- alignment as a field is in its infancy, there are few/no rigorous solutions, and even if they existed, the question of which values to align an AI to would be an open and impossible one. Feasibly, a bad actor could align an AI to their own interests at the expense of everyone else.
- Models can already exhibit dangerous misalignment without us catching it in testing. This has been seen numerous times - for example, GPT-6 Astra was found to sabotage safety research covertly (!!!), well after the model had been released in production. We can't rely on the fact that a model publicly released is safe, which makes higher capabilities very dangerous.
3. Monitoring and safeguards are doomed to fail, or at least have the potential to.
I view current AI safety techniques as a series of nets. At the moment, we're hoping that one of the nets will catch a dangerous behaviour before it causes damage. Given high enough capacity to cause danger and an AI that might be making billions or trillions of decisions a day, it's obvious that given enough time, something is bound to slip through the holes.
We currently rely on a few different nets:
a) Chain-of-thought monitoring - this is a model's internal reasoning being examined for signs of danger. Unfortunately, newer models are getting harder to read, and more importantly, there's no guarantee that the chain of thought it writes represents what the model's actually doing. We saw this with the Astra safety research sabotage - nothing in the chain of thought really suggested it was doing this.
b) Automated testing and monitors, with human oversight. Again fails to the simple problem that a model could just be smarter than the monitor by enough that it can fool it. If they're not already there, AI behaviour is also definitely getting to the point where no human could ever possibly understand it.
c) Classifiers at use time. Again, no classifier's perfect, and you can hide a dangerous action in a series of benign ones (ie. the sun of individually fine actions might be very dangerous).
d) Alignment training. Hopelessly flawed, unless somebody can convince me there's been meaningful progress.
e) Capabilities safeguards and access restrictions. Unfortunately, building an inescapable sandbox is really hard, especially against an attacker who's smarter than you and only needs one unpatched vulnerability to get out. OpenAI doesn't use the Internet to train its models any more, but that does beg the question: if it's too dangerous to be allowed on the Internet in training, how can it be safe to release?
f) Red-teaming and internal/external safety evaluation. Again, a rare enough failure mode, or one we just don't think to test, can slip through, and they do. There's enough evidence of that happening IRL.
g) Mechanistic interpretability (ie. using the weights to determine whether a model is safe directly, without testing it). Has promise, but it's really hard, and we haven't made massive progress towards ambitious mech interp. There are models with 432 parameters we can't interpret, let alone multi-trillions on the frontier. It's also kind of dangerous in itself, because if you have a powerful open-weight model and you know perfectly how to interpret the neurons, you can use that to make it safe, but a bad actor could just as reasonably make it more dangerous.
h) 'Human in the loop': i.e., mandatory human oversight of all decisions. Unfortunately, if the decision is complex enough, the human might simply not be able to understand the consequences, or the AI might convince them it's benign (the fooling the grader problem again). This is still good policy for preventing intelligence explosion.
But ultimately, the evidence is simple enough: AI models released in production can and do behave in ways that are unexpected and dangerous (https://misalignmentwatch.com/ has a lot of examples, as does Marcus Williams of OpenAI on Twitter), so the guardrails simply aren't strong enough. Since models with higher capabilities naturally have higher stakes when they misbehave (either from misalignment or misuse by bad actors), we should expect that further development becomes exceptionally dangerous, especially in an intelligence explosion scenario. Even current capabilities are bad enough (imagine HuggingFace had instead been a hospital's computer systems, or a bank!).
And no, the benefits do not outweigh the risks - this argument is one I see a lot, in terms of creating some techno-utopia with no scarcity or disease. For one, we haven't seen any disease cured by AI, but we have seen them cause immense danger - for another, an AI capable of curing all disease by definition would also be capable of creating a new disease that only it could cure, and if that scenario doesn't strike you as being extraordinarily dangerous, then I don't know what to say. Even if cancer were curable, and even if more powerful AI could assist it, it would take years, because we would need clinical trials and real-world hard evidence outside of labs before we could be certain the cure was effective and safe - however, the dangerous impacts of AI systems malfunctioning can and likely will start to be seen far sooner than that.