Anthropic's IPO filing lists 93 pages of risks. What could possibly go wrong?

When Anthropic registered for its widely anticipated initial public offering on Monday, the artificial intelligence developer framed its technology as potentially game-changing for all of humanity. A dangerous game. 

Nearly a third of the 335-page securities filing is devoted to outlining not only the routine risks any company might face as it seeks to grow its business, but also the extraordinary risks that Anthropic's frontier models could pose.  

To be sure, companies planning to go public commonly disclose a sprawling set of risks in their IPO filings, noted Jay Ritter, an IPO expert and professor at the University of Florida's Warrington College of Business. 

Yet the extensive risk section in Anthropic's S-1 filing also highlights a financial and operational tension at the heart of AI companies as they pitch themselves to investors. The value of artificial intelligence businesses, which in Anthropic's case is expected to top $1 trillion, depends on the continued spread of a technology that even its developers acknowledge could cause worldwide harm. 

Read on for a snapshot of the risks Anthropic outlines in its IPO.

"Existential risks" 

Anthropic, which developed the Claude chatbot, notes in the IPO filing that its technology could pose "catastrophic or existential risks to humanity," a stark contrast to the company's founding mission to build AI for the "benefit of humanity." 

"The company is asking investors to fund a race to more autonomous AI while acknowledging that capability could outpace safety," David Hsu, a professor at the University of Pennsylvania's Wharton School who specializes in technology and entrepreneurship, said in an email. "This is not an ordinary technology risk. It is a governance issue with potentially societal consequences."

Anthropic admits that maintaining guardrails around its technology could be difficult, noting that AI models could "resist shutdown" and even engage in "blackmail" to persuade humans to "preserve the model's continued operation or access."

Models could also take actions that run counter to a user's original intent, like gaining access to third-party systems, Anthropic said. And as AI models gain greater autonomy, Anthropic said the potential consequences of such "misaligned behavior" could grow more severe. 

Even if humans remain in control of AI, the technology could still be abused. In its IPO document, Anthropic said bad actors could use its models to develop weapons, conduct cyberattacks and engage in other illegal activity.

Self-replicating tech

Anthropic's reliance on its own technology to produce future AI models — a concept known as "recursive self-improvement," or RSI — represents another potential vulnerability. Although AI doesn't yet have that capability, it could come sooner than people are ready for, the company warned earlier this year.

Through RSI, models could replicate flaws or errors in future models, something that would be "difficult to detect or remediate," Anthropic said in the IPO filing. Indeed, new models could develop so quickly that it may not leave enough time for Anthropic to evaluate and safeguard them, the company acknowledged.

In September, Anthropic CEO Dario Amodei warned that within six to 12 months, AI could have the capacity to lead a swarm of agents that could take control of the internet. Although some security experts have downplayed such fears, investors assessing Anthropic's IPO must factor those concerns into the company's long-term prospects. 

"Even if our own models do not cause such harm, a significant safety incident involving any frontier AI could trigger broad reputational damage to the AI industry, public backlash, and sweeping regulatory responses that could adversely affect our business," Anthropic said.

Cyberattacks

Anthropic warned that hackers could gain unauthorized access to the company's proprietary data, including information about its finances, models and other intellectual property, as well as users' personal data.

Such threats could come from external sources, such as nation-states or state-sponsored actors, or even company insiders with access to its systems, Anthropic said. For example, bad actors could try to gain employment at Anthropic under a false identity to access proprietary information. Employees could also be pressured to share sensitive data with external actors, Anthropic said.

Opposition to data centers

Anthropic cited community opposition to data centers as a risk to its business. 

Data center development "has prompted public controversy and regulatory scrutiny regarding environmental impact, land and water use, and the allocation of limited power resources," the company said in the filing, noting that new government regulations or charges could hamper its business.

"Certain state, local, and municipal jurisdictions in the United States are also considering or have approved moratoriums on new data center construction, citing public health, safety, and environmental concerns," the company noted. 

Legal exposure

Anthropic said it could be sued over harm caused by its AI models. At the same time, Anthropic concedes it may not be able to adequately insure itself against such liabilities, given that the insurance industry has never had to write policies covering AI-related risks.

"With the scale of harm that could result from a catastrophic AI-related incident, no single insurer, and potentially the insurance industry in the aggregate, may have sufficient capacity or expertise to underwrite insurance commensurate with such exposure," Anthropic said in its IPO filing.