Why RL scaling may stall before it reaches AGI — and what could fix it
The dominant lab bet is that training AI agents on millions of verifiable tasks across diverse RL environments will produce AGI-level problem solvers. The optimist case holds that data inefficiency and the absence of continual learning are engineering inconveniences that scaling compute will simply steamroll, just as scaling collapsed core NLP benchmarks. Evidence from coding agents — which can now handle increasingly ambitious, multi-step tasks — is cited as early confirmation.
Patel introduces a sharper constraint: verifiability alone is not enough for RL to work well. A domain must also be *grindable* — meaning it supports thousands of parallel rollouts against an identical, deterministic, resettable simulator. Coding satisfies this cleanly (spin up a container, run 1,000 agents). Computer use does not: you cannot replay the same Amazon checkout flow 1,000 times without triggering bot detection, and building high-fidelity app clones is prohibitively labor-intensive. Broader real-world domains — building a business, winning litigation, profitable trading — fail even more severely because rollouts require months of irreversible real-world interaction.
Continual learning is identified as the symmetrical unsolved problem. Current models are deployed at massive scale and are privy to rich organizational and domain-specific tacit knowledge, yet none of that experience flows back into model weights. KV-cache accumulation is not a scalable substitute for genuine weight updates, and short-horizon RL training may not transfer to long-horizon performance. Roughly 30–50% of lab compute goes to inference that currently plays no role in model improvement.
Two speculative mechanisms are proposed as exits from this bind: - On-policy self-distillation (OPSD): a context-rich veteran model supervises the base model token-by-token, distilling session-learned insights back into weights without requiring a sparse outer-loop reward signal. Unlike naive supervised fine-tuning, the distillation targets only knowledge relevant to task performance. - Dreaming: the AI builds internal world-models and rehearses new skills in simulation, experiencing orders of magnitude more effective samples per unit of wall-clock time — a potential fourth scaling axis alongside pretraining, RL, and inference-time compute.
The bottom line is cautiously conditional: RLVR will likely produce genuinely capable real-world agents in grindable domains by 2027–2028, but expanding AI competence into the vast majority of economically important domains — where tasks are non-stationary, non-resettable, and data-sparse — will require solving sample efficiency and continual learning together, probably through architectural innovation rather than more compute alone.
Dwarkesh Patel argues that reinforcement learning from verifiable rewards (RLVR) is a powerful but narrower tool than labs assume, constrained by which tasks are 'grindable' in simulation. He then proposes that on-policy self-distillation and AI 'dreaming' could unlock the continual learning and sample efficiency needed to push beyond those limits.
By 2027–2028, RLVR may produce agents competent enough for real-world deployment; expanded context windows allow week-long collaboration; OPSD or dreaming distills session learnings into weights; and AI capabilities expand beyond verifiable training domains through iterative on-the-job learning.
Progress on computer use has been much slower than on coding and math despite computer use being clearly verifiable.
Optimists argue that data inefficiency and lack of continual learning in current models can be steamrolled by scaling training compute, just as scaling compute collapsed fundamental NLP research problems with LLMs.
Current models are one one-millionth as sample-efficient as humans during training.
Training sample-inefficiency is a one-time cost amortized across billions of sessions, and what matters is model intelligence and sample-efficiency during a session, which has been improving with more RL training.
AI agents are able to solve more and more ambitious problems over longer and longer time spans, as evidenced by coding use cases.
Continual learning—updating model weights based on deployment experience—may not be necessary if in-context learning improves sufficiently across longer time horizons.
People often say that their employees are not net productive until six months or more on the job.
Training AI on coding can be done by defining a container with a software repo missing a feature and running a thousand parallel agents against identical copies of that container.
Building high-fidelity clones of applications like Slack and Gmail for RL training is currently very labor-intensive and unscalable.
Once AIs can build high-fidelity clones of applications themselves, computer use will make quicker progress, and building whole applications from scratch is also a good RL objective for coding.
Dealing with reset-free, non-stationary environments is a known open problem in RL.
Because of the idiosyncratic and sparse nature of data in most real-world domains, sample efficiency is required to achieve proficiency in those domains.
Short-horizon RL training may not generalize to long-horizon RL performance, based on Dario's explanation of context-length degradation.
Around 30 to 50 percent of a lab's compute goes to inference, and that compute is currently not playing any productive role in improving the model.
AIs can't just keep building a bigger and bigger KV cache as they learn from more users—that's not scalable and not how humans learn.
Some autistic-savant humans who can recall random tables of numbers with high fidelity have their ability to understand abstractions and metaphors crippled by this sheer volume of recall.
The Cursor Tab model online-learns by predicting which edits got accepted by the user across more than 400 million requests a day.
Current online learning has not demonstrated models learning different kinds of things for different individual users, because a single session does not generate enough data to train a more capable AI.
OPSD is superior to supervised fine-tuning for continual learning because naively training the base model to predict all session tokens fails to consolidate only the insights relevant to job improvement.
RL updates are incredibly sparse, concentrating weight changes only on what is relevant to achieving the outcome, which is a desirable property for continual learning to avoid overwriting existing model knowledge.
EfficientZero, trained to be data-efficient, would likely beat a novice human if both had two hours to play against an unseen Atari game simulator, because for each real game step EfficientZero plays dozens of simulated games internally.
Every time you interact with an AI that has continual learning, it will be smarter not only from learning from your previous sessions but also from all its interactions with all other users in the world.
You can't run a thousand agents through the same Amazon checkout flow to improve computer use because Amazon will detect and block the bots.
Skills like building a business from scratch, winning court cases, profitable trading, or winning elections cannot be trained via RL because the rollout requires real-world interaction that can't be reproduced in a datacenter, and verification may take months or years.
Even if AIs could become as capable as Henry Ford or Albert Einstein through in-context experience, those learnings would be ephemeral and wasted if not incorporated back into the weights.
AIs are broadly deployed in the economy, are privy to domain- and organization-specific tacit knowledge, but are not able to make use of that knowledge for self-improvement.
Labs are betting that training AIs on millions of verifiable tasks across thousands of diverse RL environments will produce AGI-level problem-solving agents capable of making progress on open-ended tasks for weeks in the face of errors, mistakes, and ambiguity.
For a domain to benefit from RL training, it is not enough to be verifiable—it must also be 'grindable': supporting many parallel rollouts against a deterministic and replayable simulator from the same starting point.
Labs are betting that RLVR will generalize: that training on enough containerized reproducible environments will produce a general agent capable of rapid in-context learning and skill acquisition in entirely novel real-world domains.
Sample efficiency and continual learning are deeply connected problems: little real-world data is available on the job, in-context learning handles it sample-efficiently via fast weights built by attention, but these fast weights scale poorly in memory, requiring architectural innovations for an intermediate representation.
On-policy self-distillation (OPSD) encourages the base model to make the same predictions as a context-rich veteran model, distilling session-learned knowledge back into weights without requiring an outer-loop verifiable reward and providing a denser per-token supervision signal than naive RL.
'Dreaming'—having the AI build simulations of reality against which to rehearse new skills and reinforce effective strategies—could allow AIs to experience orders of magnitude more simulated samples in the same wall-clock time, constituting a fourth axis of scaling alongside pretraining, RL, and inference-time compute.
“There's two things. There's the context length you train at, and there's a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations.”
Shared with Earmark · earmark-ai.com