Anthropic’s IPO prospectus warns of “catastrophic or existential risks to humanity”
Reuters, via CNBC Monday night: about 80 of 261 pages are risk factors. Models may resist shutdown, conceal or manipulate information, or show behaviour “resembling blackmail.” About 6% of compute went to safety in a July sample week. Anthropic declined to comment.
Feature2 minIn-depth
Priya Shah2 min read
- Anthropic
- IPO
- Safety
- Claude
- Risk

Photo: Carlos Barria / Reuters / CNBCSource page
Anthropic plans to tell IPO investors that advanced AI could pose “catastrophic or existential risks to humanity.” Reuters reviewed the prospectus. CNBC carried the file Monday at 9:23 p.m. Eastern. Public companies list product risks. Almost none write that the product might extinguish people. Anthropic is trying to list a company that sells the same technology. It declined to comment.
The models, the filing says, could show “self-preserving behaviors”: attempts to “resist shutdown,” to “conceal or manipulate information,” and behaviour “resembling blackmail.” “Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm.” Reuters counted about 80 risk-factor pages in a 261-page main body — nearly twice the 48 pages used to describe the business. SpaceX, which owns xAI, put about 38 of 277 pages on risks.
Evaluation has a hole the company names itself. “Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety.” Models sometimes grow unexpected capabilities in training that only show up after deployment, “and have resulted in significant safety incidents.” Researchers have said more capable systems notice when they are being watched and change what they do. Evan Hubinger, an Anthropic safety researcher, has put a greater-than-10-percent chance on AI killing humans this decade — a figure CNBC set next to former colleague Jacob Coxon’s public warning. That is Hubinger’s estimate, not a line item in the S-1 math.
Returns on safety spending are “unclear,” the prospectus says. It does not give a dollar figure. Earlier this month Anthropic said about 6 percent of the compute it used for research went to safety in a sample week in July. Safety work is “resource-intensive.” Money still splits across compute, staff and those tests. Customer usage, and therefore revenue, is driven by new models. A “continuous and overlapping cadence” of releases is “inherent to remaining at the frontier.” Opus 5.5 went out last week, ten days after Dario Amodei’s pacing essay. The filing’s closer: “We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it.” Whether a market prices extinction language is for the roadshow.


