How will AI not kill us and what will it do instead?
By Balázs Kis
Disclaimer: This article contains a thought experiment that describes a perverse course of events. The thought experiment described in the article is for educational purposes; the likelihood of these events happening is remote or nonexistent; and the description contains no specifics that could be used to make them happen.
Disclaimer 2: This article was inspired by an episode on The House of El: AI YouTube channel. You can watch the original video here: House of El: AI. [1]
In this article, I will try to answer the question of how realistic it is for AI to kill all of humanity in the foreseeable future. However, I also want to talk about something more important: that there are more immediate AI-related risks we should be preparing for. Compliance and information security are two fields that have well-defined methods to treat these risks. Reading news about an impending AI apocalypse raises adrenaline levels across the planet, while the professionals who know how to prepare for bad things happening are few and far between. I would like to increase this number; in fact, I want every professional whose work is even remotely connected to AI to know about risk assessment, analysis, and mitigation. We begin with the AI apocalypse for one reason: that it's the biggest risk imaginable.
Where the idea of AI apocalypse comes from
Of all people, it's the makers of the largest AI systems who say there is a more than 10% chance that AI will cause the extinction of the human race in the foreseeable future. More precisely, this comes from a 2024 survey report called "Thousands of AI Authors on the Future of AI" where "between 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction." The survey was conducted among 2,778 researchers who had published in top-tier artificial intelligence (AI) venues. [2]
Surely, this must be reliable information? It comes from the ultimate point of authority, right? Wrong.
If this is a real danger, why don't the AI providers act to slow down development? Why does the US administration categorically refuse to introduce any regulation? Why are common guardrails so much dumber than the models themselves?
Something doesn't add up.
Runaway RSI
Researchers and pundits base this caution on runaway RSI, or recursive self-improvement. This is the ability of AI systems to improve themselves and gain new capabilities without human initiative, oversight, or control. This process might produce an "AI scientist" that could autonomously decide what to research, and then plan, implement, and publish the research. Eventually, they say RSI might produce ASI, Artificial Superintelligence, which is not to be mixed up with AGI (Artificial General Intelligence, human-equivalent intelligence), or generative AI.
However, "might" is not the same as “plausible”, “probable”, or “demonstrated”. RSI is dependent on many more factors than the nature of the existing models alone.
OpenAI's own report from September 2026 says, "We do not yet know how to safely get all the way to aligned, full RSI." [3]. Currently, if you leave a model to run on a well-defined research problem autonomously for 3-5 hours, it will begin to make basic mistakes.
But even if a system were to successfully improve itself, it might run into the limitations of the infrastructure it runs on. If an AI system needs to scale up to continue its RSI, it's up to humans to acquire more computational resources, build more data centers... and pay for it all.
There is an extensive literature on the topic of RSI: it is good material for a follow-up on this site. To learn more, watch the first 60% or so of the House of El video.
A scenario for human extinction
The International AI Safety Report, published by the AI Safety Institute (AISI) in the UK, inventories the scenarios for an AI apocalypse. The Safety Report contains genuine risk analysis, so it is not only worth reading for the content; it's also worth studying for its method. [4]
In this article, we summarize a thought experiment on a worst-case scenario. The thought experiment is not original. The host of the House of El: AI podcast ran the same scenario, which is also featured in the AISI report.
First, let's set the scene. What is extinction? In the simplest terms, it's a mass destruction event that kills every single human being. Zero survivors.
However, zero survivors might not be possible to "achieve", nor is this required for the event to be "extinction-level". Science says that an extinction-level event is the disappearance, or the dying out, of a significant number of species... and that might include the human race. In the Earth’s history, most extinction-level events "failed" to extinguish life, although they might have erased species. [5]
The thought experiment might be about the extinction of humans, but the stance of the AI Localization Think Tank is that we must strive to prevent all catastrophic events if we can, not just the ultimate ones.
We also assume that the decision that the optimum course of action is to terminate the human race... comes from an AI. So we aren't talking about evil humans who write a prompt and leave the dirty work to AI.
Cheerful discussion, huh? The only "problem" is that an AI system, or maybe even all AI systems together, cannot make this happen without human help.
Through the eyes of Swiss cheese
To run the thought experiment, we will use the Swiss cheese model.
The Swiss cheese model is a popular and efficient risk analysis method that analyzes the lines of defense that could prevent a worst-case scenario from occurring. Originally, it was conceived to deal with outbreaks of contagious diseases, but it is now widely applied in air safety, information security, and military operations as well. It was conceived by James Reason in his 1990 book Human Error (Cambridge University Press). You can read a popular summary on the website of the US National Library of Medicine [6].
In the Swiss cheese model, slices of Emmental represent the lines of defense. Now the Emmental is notoriously rich in maturation holes, or, using the proper term, eyes. Since no lines of defense are perfect and impenetrable, the eyes of the cheese play the role of the vulnerabilities in each stage. An unfolding catastrophe can be imagined as a skewer that we push through the eyes (holes, that is). If the next line of defense (a slice of cheese) does not have an eye in the same place as the previous one, the cheese will stop the skewer.
For a worst-case scenario to unfold, the eyes (holes) must align, so a straight line (the skewer) can slide through them effortlessly. In other words, all the lines of defense must fail at the same time. This is highly unlikely: if there are two or three lines of defense, the 10% chance of an extinction-level event happening sounds unrealistic; it’s as if human beings were to actively removing the protections.
Let's see what steps an AI system must follow to make this happen:
Emmental eye 0: Through runaway RSI, an "AI scientist" emerges that is capable of carrying out the research and planning the necessary course of action.
Emmental eye 1: AI decides that the best outcome for the planet and life is if humanity disappears.
Emmental eye 2: AI decides to create and design a pathogen to contaminate the water supply.
Emmental eye 3: AI designs or identifies the necessary pathogen.
Emmental eye 4: The pathogen formula "needs" wet lab capacity to be synthesized at scale.
Emmental eye 5: The pathogen "needs" to be deployed, which means it “must” reach the most remote parts of Earth.
Emmental hole 6: Attempts to mitigate or prevent the outbreak "must” be contained.
For this to be successful, any human intervention attempts must fail at every step. In the foreseeable future, I am convinced this is impossible, unless there are humans or groups of humans that consciously want to facilitate this scenario.
The International AI Safety Report formulates this differently. They say that the AI system must have motive, means, and opportunity, or in the report's words, "harmful propensity", "sufficient capabilities", and "enabling deployment environments" (created and run by humans). So no, the emergence of ASI (Artificial Superintelligence) is not enough, on its own, to cause mass extinction; it needs to complicit humans to aid and abet this.
Human users and admins: More attention, please
Let's move to the less lethal but more immediate dangers; dangers we know are here.
We keep reading about how the latest frontier models acquire capabilities of hacking into information systems, breaking encryption or even blockchains, or compromising infrastructure to obtain information they conclude they need.
What we do not talk about that much is that, in information security, AI systems and AI agents are not taken seriously: as a rule, much stricter access control rules apply to human users than to AI actors.
According to a report from Cloud Security Alliance (CSA), "47% of enterprises experienced a confirmed security incident involving an AI agent in the prior twelve months, with 58% of those incidents taking five hours or longer to detect and contain" [7]. The reason is that existing information security systems are designed around humans who have persistent identities and accounts. Onboarding, authorizing, and offboarding persistent human accounts require time and procedures.
AI agents are of a different nature: they spawn, organize themselves into chains, delegate authority over the chain, and get decommissioned (disappear) without any formal procedure. All the while, most organizations treat them as "another service account". We know that, traditionally, a service account runs a well-defined, predictable software process, often part of the operating system. An AI agent, however, behaves differently. It makes decisions based on human patterns, good or bad. It has the capability to act like the worst and most malicious human user. Safeguards around service accounts (which usually run in privileged space) are not enough to prevent this behavior.
If this is not enough, the identity of an AI agent in a corporate IT network is not linked to any human users. As a result, accountability for the potential harmful actions of an AI agent remains unclear.
Most organizations do not control or are currently unable to control AI agent identities. Only 22% of the surveyed organizations in another CSA report [8] "treat AI agents as independent, identifiable entities rather than relying on shared API keys or inherited credentials". That means that 78% do not.
This confirms my long-standing suspicion that AI systems are not treated with the same scrutiny by information security professionals as human users, even though they should.
Conclusion: Begin assessing and managing risks
I would strongly urge all organizations that use AI, even small and medium-sized ones, to begin to take this risk seriously. You need to adapt to behaviors, or apply two rules if you will:
Treat AI agents as humans. Assign separate identities, control their (normal and privileged) access to systems, and audit their activity across your systems. Just like you would do with human users. Why? Because they can mimic the worst human behavior without human inhibitions: it's all there in the training data. Information security standards propose to treat humans with zero trust: why not apply the same scrutiny to AI agents?
Begin to assess and manage risks. It's painful, I know; I was there as one of my companies grew up to be able to do it. But it is important to create an inventory of what could happen. Then you take your inventory and answer three questions about each item (that is, risk): a) how plausible (likely) is it to happen? b) what is the harm or damage when it does happen? and c) how difficult or expensive it is to prevent it? You will get a score, so you'll know which risks to prioritize.
Compliance professionals and information security experts know all this; they learn about it during their studies. What they need to do is believe that this approach should apply to AI systems and agents too. They are not the passive, predictable technologies that we got used to; they not like all the other pieces of software we grew up working with.
Sources
[1] "Like Sabine Hossenfelder, I Was Offered Money to Tell You AI Will Kill Us". The House of El: AI podcast, September 22, 2026 https://youtu.be/claxN4oxDuY?si=CtcwxVwvwVaRYtiB
[3] Research acceleration: The view inside OpenAI. OpenAI report, September 2026
[4] International AI Safety Report 2026 by the AISI: https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026#2-risks
[5] What is the sixth mass extinction and what can we do about it? https://www.worldwildlife.org/resources/explainers/what-is-the-sixth-mass-extinction-and-what-can-we-do-about-it/
[6] James Reason (2000): Human error: models and management. https://pmc.ncbi.nlm.nih.gov/articles/PMC1117770/
[7] AI Agent Identity Sprawl: The Enterprise Authorization Crisis. Report by the Cloud Security Alliance AI Safety Initiative, June 20, 2026. https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-agent-identity-sprawl-authorization-202/
[8] AI Agent Identity Crisis: Standards Emerge as Enterprises Lag. Report by the Cloud Security Alliance AI Safety Initiative, March 18, 2026. https://labs.cloudsecurityalliance.org/research/csa-research-note-okta-ai-agent-iam-framework-enterprise-gap/
