The AI doom scenario you've never heard of may be the hardest to stop

It seems that almost daily, a big tech company releases a new AI-driven digital assistant. Muse by Meta, Gemini by Google, dots by OpenAI. By the time this article is published, a new one from [insert tech company here] will probably have a press release touting its ability to manage your work and life — to take things off your plate, automate your processes, speed up your tedious tasks. It sounds helpful and, to lower the bar, harmless. 

But some researchers say this is the beginning of a treacherous path for humankind, one where bots take more and more off our plate until all that's left is an empty vessel, and we human users become nothing more than thoughtless "approvers." 

It's not just a threat to jobs. They warn that  allowing our skills and knowledge to deteriorate, and surrendering our critical thinking to AI, poses an existential danger to humanity.

One term for this path is "gradual disempowerment" — a scenario where AI displaces human labor in the economy, replaces human decision-making in all key societal concerns, erodes human creativity, and degrades human cognition along the way. 

And though the word "gradual" is in the name, that does not mean the problem is something to worry about far in the future, according to experts. 

"I put gradual disempowerment on the very top of my list," said Jay Chooi, the founder and CEO of Robocurve, a company that tracks and measures AI-driven robotics capabilities. 

Chooi believes gradual disempowerment is just as worrisome as the threat of engineered pandemics and more destructive cyberattacks, which could be accelerated by capable AI.

"It's a civilization-scale problem," Chooi said of gradual disempowerment. "A lot of people feel very powerless, because it requires a lot of effort and a whole-of-society approach to solve it."

What is "gradual disempowerment" and how could AI cause it?

The term "gradual disempowerment" stems from a paper published in early 2025.

At the heart of the gradual disempowerment scenario is AI technology continuing to advance in capabilities, as it has rapidly since the launch of ChatGPT in 2022. 

AI models may soon be better at nearly all cognitive tasks than humans — solving frontier math problems, doing hard science, coding, writing and creating art and music, to name a few of the areas where AI increasingly excels. At the same time, advances in robotics could lead to AI-powered robots performing better than humans at physical labor like packing boxes, constructing homes, building cars, painting murals, mining for rare minerals, and even building IKEA furniture. 

We like to think that in an ideal world, the best applicant generally gets the job, and they are paid based on the level of demand for their skills. But imagine if a human is never the best person for the job. That's a world that some AI prognosticators and whistleblowers expect.

It's a world where an AI agent or AI-enabled robot is better at every task and sub-task that it takes to do my job and your job. They move faster than us, think deeper, have more patience and stamina. They more cordially answer phones, do more thorough research, and more dexterously wield a scalpel or a nail file. Plus, they never need to sleep or take sick days, they never show up hungover or moody, and they never threaten to leave for a better job. 

To picture gradual disempowerment more clearly, here's a scenario: Let's say you run Hammer Co., a company that manufactures hammers. You have line workers who shape the steel heads and wooden handles and then fit them together, and managers who keep the factory running smoothly. You have people who maintain your supply chain and teams to handle marketing, sales, accounting, and all the other functions that help your business thrive.

But then a tech company comes out with AI-powered Hammer Bots. The robot workers can make hammers far faster than your human workers, while AI agents can run the business side with little oversight. They'll work all night, never make mistakes, and cost far less than your happy, well-paid staff. You don't want to fire your human workers. You see them around town and have known their families for years, after all. 

But your competitor gets the Hammer Bots. Most customers just want the cheapest hammers — they don't care who or what makes them. As CEO of Hammer Co. you either compete or die. So you fire all your humans and get Hammer Bots to replace them. 

Now, try to envision this is the case for every company, and every job. With normal free-market incentives, the company that stays loyal to its human employees will be outcompeted by the company that gets AI to do the work instead. 

Here's where gradual disempowerment comes into play.

The fewer humans that participate in labor, and the more work that's outsourced to AI agents, the more humanity becomes dependent on AI for decision-making and the more human skills deteriorate. How long until  the finer points of hammer-making will be forgotten?

With AI making all our complex decisions for us, humans become less capable of understanding and overseeing those decisions or of pushing back against them. 

"If people aren't actually practicing the skills that the agents are doing for them, they lose the ability to have that skill, and so now you become reliant," said Margaret Mitchell, an AI research scientist at Hugging Face and co-author of another recent paper on the importance of keeping humans in the loop in an AI world. 

"What happens when something goes wrong? There's no human you can point to who can fix it. So now you've become fully dependent on these systems to fix themselves, and what if that's not possible?"

A risk to democratic governance

It's not just economics and labor at risk. Take the military, for example. All or most decisions in war involve humans today, even if AI contributes to the process, but the balance is likely to shift as the technology becomes more capable. 

If the United States uses AI to make decisions in battle, its enemies across the field will be pressured to do the same. The side that leans more heavily on humans will be slower and less efficient. Human soldiers need food and rest. Robot soldiers will not. Human decision-makers are limited in the scenarios they can conceive and plan for; AIs can take into account all of human knowledge. 

"The states need citizens. They need us because we pay taxes and we fight for the states in the military," said Jan Kulveit, a co-author of the gradual disempowerment paper and co-founder of ACS Research Group. 

States also use people to enforce laws. But what if law enforcement is all automated in the future? 

"If, or when, existing states don't need their citizens, they're less likely to be democratic," said Kulveit. 

Kulveit compares theoretical future countries that are mostly run by AI to oil states today. He said the oil states tend to be authoritarian because they don't rely on most of their citizens to pay taxes and don't need citizens' votes to rule. States that rely on AI to run the economy and military could have similar outcomes, said Kulveit. 

If that is the case, and citizens no longer are the backbone of labor or warfare in a world of gradual disempowerment, then humans lose perhaps their greatest leverage against those in power. 

"A lot of people are talking about rogue AI hacking into data centers and taking over the world, but it is not in the mainstream consciousness that like AI could literally take over the world just by continuing to do what it's supposed to do, which is automating everything," said Jay Chooi of Robocurve.

"Organizations are all going to have a very strong economic pressure, and political pressure, to phase out humans and move work more and more into robots or AI."

How does humanity avoid gradual disempowerment?

Humans ceding control to AI, say the gradual disempowerment authors, is almost a natural transition once very advanced AI is here. 

Gradual disempowerment "does not need to emerge from a deliberate scheme or powergrab by AI systems," wrote Kulveit and his colleagues. "In the short-term, it is being incentivized by the perceived value that AI systems can bring to economic, cultural and state functions."

"We can't be in the situation where we are competing with the machines," said Kulveit. "Because if we are just competing with the machines, we will probably lose."

The gradual disempowerment authors believe stopping our disempowerment may be a heavy lift for humanity, since it involves complex systems that feed into each other — government, culture, and the economy. 

"This may be a bigger challenge than merely preventing AI systems from pursuing overtly harmful goals," they write. 

One first step in slowing disempowerment is keeping indicators of how much work is being ceded to AI, and of what kind of specific jobs AI systems are capable of handling. If thousands of workers are being displaced by AI in, say, call center jobs, it helps to know that in granular detail. 

But measurement isn't enough without action, Mitchell said. She argues that removing humans from the loop in AI processes both worsens oversight and degrades the skills of the human overseer. Therefore, companies and leadership in all fields must take proactive steps to keep humans engaged. 

"This general practice within tech is we want our systems to be frictionless," said Mitchell, referring to the idea that everything should move fast and without obstacles. "There's this sense that the less friction the better, but there's been a push within the more ethical AI community that strategic friction can be good because it can draw people's attention to critical issues that otherwise they might overlook."

Mitchell argues that strategic friction — keeping humans in the work loop at predetermined steps — can be done both by developers of AI, companies like OpenAI and Anthropic and Google, and by companies and organizations that deploy AI. Thoughtful and purposeful design of AI can benefit society as a whole, and the individuals who use it. 

Kulveit agrees that AI should be thoughtfully designed to add value to humans, not necessarily replace our every task. 

"You want to carve out places where we agree that we will not allow competitive pressures to remove humans from the loop," he said. "It would be nice if the companies train the AIs to be more complementary to humans. Currently, the AIs are implicitly trained to be a replacement for humans. But you can imagine training the AIs so they are better at explaining to us what's happening." 

Kulveit is describing AI that doesn't just spew out the answers to whatever question you ask it, at the very least in sectors where it is critical to include human expertise and decision-making, like war. Quick and easy answers can be helpful, but they can also make you dependent and credulous, believing anything the chatbot tells you, whether right or wrong. 

But AI can be more than that. It can help you not just by doing the work for you, but by educating you along the way. That way the human worker continues to learn while using AI as a partner, not a substitute. 

"There has been such a singular focus on capability or functionality of these systems to the detriment of people, to the detriment of users," said Mitchell. "If your only goal is to get a high score on a benchmark, then you are not going to be taking into account cognitive support design. But if your goal is to make a system that works really well for people — like if the race is for a utility as opposed to benchmark score — then you end up designing fundamentally different systems."

If gradual disempowerment really is a large-scale risk, compared to more talked-about doom scenarios — like AI helping to design bioweapons or chemical weapons — it may suffer from a relative lack of vividness. 

"We know what it means for a robot to take over the world. We know what it means for countries to nuke each other," said Chooi. "AI taking over the world is very legible in the human consciousness. But gradual disempowerment? No one has made a movie about gradual disempowerment."

But to Chooi, gradual disempowerment is very easy to picture. 

"All the organizations have the incentives to adopt it and phase out humans," he said. "This is a very logical conclusion. It's not very hard to see it."