The safety week has a name now: recursive self-improvement
A Friday CNBC report puts RSI at the center of the existential argument. Hubinger’s 10 percent, two labs saying it is happening faster than they thought, and still no scientific plan.
Deep dive2 minIn-depth
Maya Chen2 min read
- OpenAI
- Anthropic
- Safety
- RSI

Photo: CNBCSource page
By Friday evening the week had a narrower noun than “safety.” CNBC’s morning report, landing in Europe as the afternoon briefing, put recursive self-improvement at the center of the existential argument. RSI is not a metaphor. It is the claim that models are already helping to build the next models, and that the loop can close until the people who started it are no longer the ones steering.
Evan Hubinger, an alignment lead at Anthropic, wrote that he thinks there is more than a 10 percent chance AI could kill all humans within a decade — then, in a follow-up, said the thing he is actually worried about is superintelligence arriving through RSI, “as we have said is happening faster than we thought.” That second sentence is the news. The first is the temperature.
Faster than they thought is now a corporate sentence
Both labs have said, in recent months, that autonomous model improvement is ahead of their own forecasts. Anthropic posted in June that internal data showed Claude accelerating AI development — a possible path to a system that builds a more capable successor. An August note said engineers now ship, on average, eight times as much code per quarter as they did from 2021 to 2025. That is not RSI as a finished product. It is RSI as a slope.
There is not yet a viable scientific plan to solve risks from recursively self-improving AI. Please look up!
— Anna Wang, AGI safety and alignment, Anthropic, cited by CNBC
Jasmine Wang, on OpenAI’s alignment staff, wrote that it is hard to overstate how dangerous it is to speed toward RSI. Jakub Pachocki’s Saturday note — capability jumps as large as the last ones, systems that increasingly drive their own development — sits behind the Friday recap rather than in front of it. The morning Signal on Altman being “open to slowing” is the corporate weather. This is the mechanism the weather is about.
Three futures, one they will not price
Anthropic’s RSI post sketched three paths. Frontier progress stalls and capability spreads; the lab does not think that is likely. Humans stay in control while the world still changes; that one, it called likely. Or systems reach full recursive self-improvement and people play a substantially diminished role. How alignment gets solved in that third future, the company said, is what it is least certain about. Least certain is not a product timeline. It is a gap you can drive a listing through.
Vincent Conitzer at Carnegie Mellon told CNBC that models can already introduce some new ideas, which makes the acceleration point hard to forecast. Readers should treat Friday’s piece as a map of what staff are willing to say in public, not as proof that the loop has closed. Watch the training calendar and the code-per-engineer charts. The adjectives will take care of themselves.


