A 10% chance of human extinction would be intolerable in aviation, medicine or nuclear safety. Why should artificial intelligence be different? Geoffrey Hinton, one of the pioneers of modern AI, has stated that a 10% estimate of catastrophic AI risk is not unreasonable — though he also stresses, "nobody really knows how to give a sensible estimate." Uncertainty offers little comfort when the possible consequence is the destruction of humanity.
The danger is not simply that AI may surpass human intelligence. It is that sufficiently autonomous systems may pursue goals in ways humans can no longer reliably control. Humans possess an innate capacity for self-preservation. History is filled with folly — yet we can recognise the abyss and retreat from it. That was the great lesson of the Cold War: bitter enemies understood there could be no victory in mutual annihilation. Hotlines, arms control, verification — enemies discovered that common humanity could impose limits on rivalry.
A rogue AI has no such instinct unless we build it in. It fears no death, remembers no Hiroshima, loves no children. Human control cannot be merely one feature of advanced AI. It must be its first principle.
Dr Lim Teck Ghee’s recent article in The Edge, Will the AI race trigger a catastrophic war (Sept 10), warns of the US–China struggle for AI supremacy — and the risk that AI infrastructure itself becomes a target. Researchers on both sides warn of an AI "Chernobyl moment": cooperation arriving only after catastrophe. The greatest danger is not rogue AI alone — it is an arms race that makes rogue AI inevitable.
The two powers run different races. America’s frontier push is driven by private corporations competing on raw intelligence — whose model reasons, codes, acts most autonomously. A company that pauses over safety risks being overtaken. China’s firms compete too, but Beijing prioritises real-world deployment — manufacturing, robotics, agriculture, healthcare. America asks: how intelligent can we make it? China asks: what can we make it do?
The contrast is too often caricatured. China’s system is dismissed as a single hand on the lever — inherently more prone to catastrophic error. Yet as political theorist Melanie Manion observes, reality is more layered. Even China’s largest AI firms operate commercially, but the Communist Party has legal standing to embed representatives within them. State oversight is not external and reactive — it is in situ, inside the organisation itself.
America presents the opposite paradox. Its leading AI companies are controlled by a tiny circle — a plutocratic class with immense political access. Are they more accountable than the CCP’s top cadres? In the US, breaches result in corporate fines — costs absorbed by the institution, not personal consequences. Executives face no personal liability, no imprisonment, if guardrails fail. In China, senior officials and enterprise leaders operate under a vastly starker incentive structure: career ruin, imprisonment, and in extreme cases, execution are real consequences for serious breaches. Whatever their motives — power, status, privilege — Chinese decision-makers carry a powerful personal deterrent against reckless acceleration. This does not make them wiser; but it may make them more cautious.
This creates the deeper paradox. The freedom that drives American innovation may also make it harder to say: Stop. Private companies driven by quarterly returns downplay risk. Whistleblowers sounding the alarm are silenced. Two days after credible whistleblowers sounding the alarm of rogue AI risk, Donald Trump dismissed existential risk outright: "not going to happen." When the most powerful leader on earth treats humanity’s survival as a joke, the danger is no longer merely technical — it is political.
This is not Cold War national security competition. It is a raw contest for hegemony. Bipartisan consensus in Washington views AI dominance as the pillar of continuing US global supremacy. China sees AI leadership as the foundation of its national rejuvenation. Neither competes to be safe — they compete to be number one. Washington fears restraint cedes the future; Beijing sees containment as an existential threat. Each side’s determination removes any incentive to hit the brakes. The race has no automatic stop sign.
The answer is not to ban AI. It is one of humanity’s most promising technologies — comparable to steam, electricity, the internet — with immense benefit to medicine, education, science and daily life. What we need is not AI disarmament. It needs AI for humanity.
Governments and corporations will not act alone. They are trapped in a competition they cannot control and refuse to abandon. Only a worldwide citizens’ movement can break the deadlock. We must demand rules for the most dangerous frontier AI applications: meaningful human control over weapons; an absolute ban on autonomous nuclear-launch decisions; shared safety testing standards; emergency communication channels between capitals; and agreed red lines with credible, independent verification.
The Campaign for Nuclear Disarmament turned nuclear annihilation from generals’ business into millions’ moral concern. AI needs its equivalent — not Luddite opposition to progress, but insistence that progress remains subordinate to human survival. The message must ring out: control AI before AI destroys us.
We cannot wait for leaders to change their minds. They will not. Pressure must come from below — citizens, scientists, workers, universities, civil society — until cooperation becomes politically unavoidable. There is no "big deal" to be struck with those who refuse to acknowledge the danger. The deal must be between us — the people of the world — and it must be struck now.
* Tan Wah Piow is a lawyer and social activist based in London. He is the author of ‘A Dialogue Across Civilisations’.