RSI: The New AGI — Equally Elusive

The concept of “recursion” has recently become a prominent subject in AI circles. Two separate startups have embraced this terminology, while many others are starting to include Recursive Self-Improvement (RSI) in their strategic agendas. Much like AGI in past debates, RSI has become a shorthand for a potential AI breakthrough—although interpretations of its exact definition differ.

In straightforward terms, RSI refers to an AI system that can continually enhance itself. When AI systems can manage their upgrade cycles more efficiently than humans, this could lead to a self-sustaining loop, limited only by their available computational resources, making human intervention redundant or even counterproductive.

This vision is either concerning or exciting, depending on one’s perspective, but it’s one that many AI laboratories are keen to explore.

Earlier this month, the prominent AI researcher Richard Socher unveiled Recursive Superintelligence, specifically citing RSI as a key goal. “Our primary objective is to develop truly recursive, self-improving superintelligence on a large scale,” Socher disclosed to TechCrunch during the launch, “which implies the entire ideation, implementation, and validation process of research ideas would be automated.”

Several eminent researchers are also pursuing this vision, aiming for breakthroughs that would facilitate recursive self-improvement.

Alex Karpathy, a well-recognized figure from Tesla and OpenAI, is particularly active in this domain. He is working on agent swarms designed to train language models on fundamental tasks for a project he refers to as Auto-Research. Karpathy has been notably open about the project, frequently sharing updates on Twitter and providing essential components via a public GitHub repository. So far, the focus has mainly been on making minor enhancements to a GPT-2 scale model—though Karpathy admitted in March, “It’s not groundbreaking research (yet)”—it has generated sufficient interest to draw in numerous fellow researchers advocating for the RSI vision. With Karpathy now engaged in pre-training at Anthropic, he will have plenty of opportunities to develop these concepts on a larger scale.

Adaption, co-founded by Sara Hooker, a former team member at Cohere and Google, recently introduced a similar tool named AutoScientist, aimed at automating frontier training. Similar to Karpathy’s auto-research project, this system trains agents to make incremental improvements—but Adaption’s goal is to streamline the training of a comprehensive frontier model. Should researchers advance the frontier, the system could rapidly evolve into something akin to true RSI.

Doris Xin, founder of Disarray, sparked significant interest in RSI when her self-trained machine learning agent secured 28 medals in a recent Kaggle competition, surpassing many human-trained agents. She identifies reliability as the primary obstacle.

“I would argue that with infinite computation and an unlimited time horizon, we are already there,” Xin stated. “It’s important to clarify that this isn’t inherently a creative endeavor. It’s primarily about engineering.”

Not there yet

Nonetheless, substantial evidence indicates that the AI sector remains far from achieving genuinely recursive systems and continues to struggle with effectively communicating its progress to a cautious public. This sentiment was echoed by Google CEO Sundar Pichai during a recent podcast discussion.

“It’s a continuum, and we are all undoubtedly progressing,” Pichai noted. “However, the way people describe R.S.I. suggests a new level of acceleration, which would have profound implications, and we aren’t quite there yet.”

However, this continuum encompasses a significant number of self-improving AI systems. In January, one of Anthropic’s leading developers for Claude Code reported that “close to 100%” of his team’s code was generated by the tool—a candid acknowledgment that Claude Code is essentially writing itself.

Utilizing an AI tool doesn’t automatically imply replacing engineers—but Anthropic seems to be approaching that threshold. In a recent survey regarding the Mythos preview, five out of 18 Anthropic engineers believed that with further enhancements, this version of Mythos could soon function as a replacement for an L4 engineer—a mid-level programmer capable of handling complex projects independently.

Still, some predictable limitations persist.

“Some of Claude’s key weaknesses compared to an L4 are: managing prolonged ambiguous tasks, understanding organizational priorities, taste, verification, instruction-following, and epistemics,” the report states.

In other words, its deficiencies focus on self-direction, which is crucial for RSI. Nonetheless, apart from these areas, Claude is well-equipped to step in.

Similar to the AGI terminology that came before it, the AI industry cannot definitively ascertain how far it is from showcasing a meaningful recursive system. When the Georgetown Center for Security and Emerging Technologies convened a group of experts to discuss RSI last year, they found a significant divide in viewpoints—some anticipated an imminent “superintelligence” surge, while others expected a gradual approach leading to a plateau. Nevertheless, all concurred that recursion complicates future forecasts.

Helen Toner, director of CSET and a former board member at OpenAI, explained to TechCrunch that merely employing AI tools for AI research does not qualify as RSI. “They’re just maximizing the use of AI to its fullest potential,” Toner stated. “This differs from the traditional definition of RSI, which implies that human intervention becomes unnecessary.”

Toner referenced a recent article by METR’s Ayeja Cotra, which outlines various milestones along the path toward AI research autonomy. One step, termed “adequacy,” would occur when a system can conduct research independently of human intervention—even if the resulting work isn’t as valuable or efficient. “Parity” is achieved when an AI-only system performs as well as a human-only system, while “supremacy,” the final stage, is reached when an AI-only system exceeds the effectiveness of a collaborative human-AI research effort.

Ultimately, Cotra concludes that AI is approaching the adequacy threshold of being capable of producing some work autonomously—similar to the incremental changes initiated by Karpathy’s Auto-Research system. “I wouldn’t be shocked if this milestone has already been achieved, and I expect it to happen within the next few years,” Cotra claims.

While she is less certain about the timing for achieving parity, she believes that once it happens, it will “significantly speed up the pace of AI progress, culminating in AI research supremacy within the following year.”

Bumps in the road

Given that much of AI development relies on scaling laws, there’s a tendency among many to assume that RSI will follow a similar path. Toner suggests that many involved in RSI view it as a smooth ascent, where continuous scaling is possible.

However, even if AI researchers achieve incremental progress, they will face greater challenges in relinquishing full control over the research process. Toner ties this to the history of computing, where human involvement has gradually diminished while still guiding the overall process.

“We moved from machine languages to assembly languages and then to compiled languages; over time, we’ve distanced ourselves further from the core mechanisms of the computer,” Toner elaborates. “Yet, humans still intuitively oversee the operation.”

Advancing beyond this paradigm will pose substantial engineering and alignment challenges. Despite significant investments, no unlimited computational resources exist, and the fundamental trade-off between human labor and machine intelligence will be difficult to navigate.

As for a fully recursive AI system in line with apocalyptic visions? The prevailing consensus among researchers seems to be that, much like AGI, it is still not a reality.