Recursive Language Models: detailed notes in simple words
What this paper is about
Large language models are good at thinking and writing, but they hit a hard wall when the text you give them gets very long. The usual fix is to make the model’s context window bigger. Here, they do not try to stuff more text into the neural network. They treat a huge prompt as something sitting outside the model, in a programming environment the model can explore with code, and they let the model call copies of itself on pieces of that prompt. They call this setup a Recursive Language Model, or RLM. From the outside it still looks like a normal language model: you give it a string and it returns a string. On the inside it is a loop of code writing, code running, peeking at data, and recursive sub-model calls. The paper shows that this can handle inputs more than ten times longer than a model’s normal context window, and that on several hard long-context tasks it beats both plain frontier models and popular agent-style scaffolds, often by large margins, without exploding cost.
The problem they are trying to solve
Frontier models have limited context windows. Even inside those windows, quality often falls as the prompt gets longer. The paper refers to this degradation as context rot. Figure 1 in the paper makes this vivid. On a simple needle-in-a-haystack task, GPT-5 can stay strong as input length grows. On OOLONG, where nearly every line matters, performance drops sooner. On OOLONG-Pairs, which needs pairwise reasoning over the data, the drop is even harsher.
Figure 1 shows how GPT-5 performance degrades as inputs get longer and tasks get harder, while RLM maintains strong performance even beyond the 272K context window limit. So the effective usable context is not one fixed number. It depends on how hard the task is and how densely the answer is spread through the text. The authors also point out a practical pressure: people are starting to use language models for long-horizon work that may need tens or hundreds of millions of tokens. Training and architecture will slowly raise context limits, but the paper asks whether inference-time scaling can raise effective context by orders of magnitude right now.
Why common long-context fixes are not enough
The most popular general trick for long inputs is compaction or condensation. When history or documents get too long, the system summarizes and throws away bulk text to free room. That works when early details can safely be forgotten. It fails when the answer needs dense access to many details across the whole prompt. Retrieval agents and coding agents can pull snippets from a filesystem or document store, but they still eventually fill the model’s window and fall back to compaction. Self-delegation methods let a model spawn sub-agents, but many of them force the model to verbalize each sub-call in natural language inside its own context. That means the number of subtasks is limited by how much the root model can write and keep in memory. The authors argue that prior coding, retrieval, and sub-agent designs miss a combination of three things: a symbolic handle to the full prompt outside the neural context, the ability to produce very long outputs by assembling them in an external environment rather than generating every token through the root model, and symbolic recursion, meaning code can launch model calls inside loops over programmatically chosen slices of the input.
The key idea of Recursive Language Models
An RLM starts from a base neural language model M with a maximum context size K. The user prompt P can be arbitrarily long. Instead of pasting P into M, the RLM creates a persistent programming environment, a REPL, short for Read-Eval-Print Loop. Think of a live Python session that remembers variables. The full prompt is stored as a variable in that session. The root model only sees short metadata at first: how long the prompt is, a short prefix, the type of the data, and how to reach pieces of it. The model is then prompted, or later trained, to write code that peeks into that variable, transforms it, chunks it, searches it, and builds intermediate results into other variables. Crucially, the environment also exposes a function to call a sub-language-model or even a full sub-RLM on a new prompt the code constructs. The root model works in a loop. It writes code, the REPL runs the code, the model only sees truncated metadata about what was printed, and state lives in the REPL variables rather than in the model’s chat history. When a special final answer variable or final answer tag is set, the loop stops and that value is returned. This design means the neural network never has to hold the entire prompt. The prompt lives outside as data. The model manages it with programs.
Figure 2 shows the RLM architecture: the prompt is loaded as a variable in a REPL, the root model writes code to inspect and decompose it, and can recursively call sub-models on selected slices.
Why treating the prompt as part of the environment matters
This is the paper’s deepest conceptual move. In ordinary use, a language model’s input is tokens that flow through attention. In an RLM, the input is also an object in a symbolic computer. The model can slice strings, loop over documents, filter with regular expressions, accumulate lists of partial answers, and call itself on chosen chunks. The neural model still does the semantic work, but the organization of that work is external and programmatic. That separation is what allows effective input length far beyond K, unbounded output length in principle by stitching sub-answers in the REPL, and a semantic horizon that can grow like the size of the prompt or even like the square of the size of the prompt when the code launches that many sub-calls.
How the RLM loop works step by step
Given prompt P, initialize a REPL with a variable holding P as a string or structured object. Add a function for sub-model or sub-RLM calls. Give the root model only metadata about the environment, not the full text. In each iteration the root model produces code. The REPL executes it and updates state. Only short metadata about printed output is appended to the root history. That truncation is intentional. It forces the model to store long strings in REPL variables and to inspect them with code or sub-calls instead of pasting everything into the root window. If the model sets Final, or uses tags like FINAL or FINAL_VAR in their implementation, the system returns that answer. In principle, if each root turn is trimmed to a small constant number of tokens, the number of root iterations is limited by the root context size, but each iteration can launch arbitrarily many sub-calls. So the real compute budget can still scale with problem size.
Three design choices the authors say most scaffolds get wrong
They contrast their Algorithm 1 with a deceptively similar Algorithm 2 that has bad design choices. First bad choice: put the user prompt into the model history at the start. That immediately inherits the context limit and eventually needs compaction. Second bad choice: force the final answer to be generated directly as a Finish action from the model’s own tokens. That caps output length at the model window. Third bad choice: offer code execution and a sub-LLM as separate tools the model can only invoke a few times by verbalizing each call. That is not the same as writing a program that, inside a loop over thousands of slices, launches model calls and stores results. Without symbolic recursion, the agent can only hand off a handful of explicitly described subtasks, not systematically process all of a huge input.
How they implemented it in practice
Their implementation uses a Python REPL. Tools, including sub-LM and sub-RLM calls, appear as modules or functions the code can call. The initial prompt is a variable. The model interacts until it gives a final answer either as text or as a REPL variable. Print output is truncated so the root context does not fill too fast. For GPT-5 experiments, they often use GPT-5 as the root and GPT-5-mini for recursive calls to balance quality and cost. They study recursion depths from zero to three. Depth zero means REPL and code only, no sub-calls. Depth one allows sub-LLM calls. Depth greater than one allows nested RLMs. For Qwen models they use Fireworks pricing. For Claude Code baselines they use Claude Opus 4.1. They keep methods task-agnostic by fixing system prompts across tasks rather than handcrafting a different agent per benchmark.
What recursion depth means in plain terms
At depth zero the model is a very smart programmer staring at a huge string variable. It can search with code, count, filter, and build answers in variables, but it cannot ask another language model to interpret the meaning of a chunk. That works well when pure code access is enough, and on some code understanding settings the no-sub-call RLM was surprisingly strong. At depth one the model can write loops like for each chunk call llm_query and save the answer. That unlocks semantic labeling, summarization of pieces, and multi-hop reading without stuffing all documents into one window. At higher depths a hard sub-problem can itself become a full RLM with its own REPL. That is useful when a sub-task needs its own multi-step chunking and aggregation, not just one shot of a sub-model.
The idea that task complexity scales with length
The authors insist that you cannot talk about effective context without talking about the task. Needle-in-a-haystack keeps the amount of critical information roughly constant as you pad more junk text. Frontier models now often solve those even at million-token scales. OOLONG requires almost every line to be labeled and aggregated, so work scales roughly linearly with length. OOLONG-Pairs requires properties over pairs of entries, so work scales roughly quadratically. That is why GPT-5 can look fine on simple retrieval scaling plots and still collapse on denser tasks. An RLM’s costs and strategies also track task complexity: more dense problems tend to trigger more sub-calls and more careful decomposition.
The evaluation tasks in detail
S-NIAH is a single needle-in-a-haystack style task from RULER ideas. You must find a specific phrase or number buried in a lot of unrelated text. The thing you need is basically constant size as the haystack grows. BrowseComp-Plus with one thousand documents is a multi-hop deep research style question over an offline corpus. Gold and evidence documents are guaranteed to be among the thousand. The answer needs several documents pieced together, so it is harder than a pure needle even though the number of relevant documents is still not the whole corpus. OOLONG, specifically the trec_coarse split, asks the model to semantically label many questions and aggregate those labels into an answer. Nearly all entries matter, so it is a linear density stress test. OOLONG-Pairs is their modified version with twenty queries that require listing pairs of entries that satisfy properties, not just counting with clever math shortcuts. Because the answer is the list of pairs, you cannot dodge the pairwise nature with inclusion-exclusion style tricks. LongBench-v2 CodeQA is multiple-choice repository understanding. The model must reason over files in a codebase to pick the right statement. Context lengths in the paper range from tens of thousands of tokens up to multi-million token regimes, with BrowseComp-Plus around six to eleven million tokens.
Baselines they compare against
They compare plain base model calls, CodeAct agents that can run code in a think-act loop, CodeAct with BM25 retrieval, CodeAct with a sub-call tool but without offloading the full prompt out of the model context, compaction agents that iteratively summarize as the window fills, and coding agents like OpenCode and Claude Code both with and without context offloading to a file. The distinction with CodeAct is important. CodeAct can execute code, but if the full user prompt is still sitting in the language model context, you inherit the same window limit. RLM puts the prompt in the environment first. Compaction agents are the industry-default long-horizon strategy and are intentionally included as a strong practical baseline. For expensive GPT-5 compaction runs they often use GPT-5-nano to summarize and GPT-5 to answer, to keep cost under control.
Main results and what they mean
Across the suite, RLMs show strong performance even at ten million plus tokens and often beat other methods by large double-digit relative gains while staying in a similar cost ballpark.
Table 1 compares RLMs against baselines across four benchmarks. RLMs consistently outperform base models, compaction agents, CodeAct, and coding agents, often by double-digit margins. On BrowseComp-Plus with GPT-5, base models hit context limits or score poorly, while RLM depth one scores around ninety-one percent with average cost under a dollar, cheaper than a naive linear cost of stuffing millions of tokens into a strong model. On OOLONG, RLM depth one with GPT-5 and with Qwen3-Coder both beat their base models substantially. On OOLONG-Pairs, base GPT-5 and base Qwen3-Coder essentially fail with F1 near zero, while RLM depth one reaches about fifty-eight percent with GPT-5 and about twenty-three percent with Qwen3-Coder, and higher recursion depths with GPT-5 climb further, up to around seventy-six percent at depth three. That is one of the clearest demonstrations that programmatic recursion is doing work that pure next-token reasoning cannot. On CodeQA, offloading context into the REPL already helps a lot, and for Qwen3-Coder the depth zero RLM can even beat higher-depth variants, suggesting that for some repository tasks pure code navigation is enough and extra recursion can add noise.
Six big observations from the results section
First, RLMs scale into the multi-million token regime and outperform base models and common task-agnostic scaffolds with comparable cost. Second, the REPL is necessary for long inputs, while recursive sub-calling is especially valuable on information-dense tasks. Depth zero already beats many baselines on long inputs because the prompt is outside the window, but OOLONG-style tasks often need semantic sub-calls rather than keyword heuristics alone. Third, plain LM performance degrades as a function of both length and problem complexity, while RLM performance degrades more slowly. Fourth, RLM cost is usually comparable, and the median RLM run can even be cheaper than the median base model run, though averages are pulled up by long-tailed failure trajectories where the RLM keeps searching. Fifth, beyond long context, RLMs help long reasoning. On LongCoT-mini, a hard compositional reasoning benchmark, RLM with GPT-5.2 beats the base model, and with explicit decomposition hints it jumps further by building a graph of subproblems and solving nodes with sub-calls while traversing the graph in code. Sixth, training helps. A small model fine-tuned as an RLM improves a lot and becomes more efficient, and reinforcement learning on shorter synthetic long-context data can generalize to longer harder splits.
Why the first decomposition attempt matters
Section 5 studies how models behave as RLMs. Typical behavior is probe the data, choose a decomposition, then launch sub-calls. On BrowseComp-Plus the model often uses its prior knowledge to narrow the search space before calling sub-models. On OOLONG-Pairs it may need to stitch many sub-answers because the final list is too long or too structured for one shot. The authors ablate system prompt examples on OOLONG and find that in-context examples of RLM strategies, even unrelated ones, improve both the first decomposition and final score. A bad first plan can still be recovered from, but a good first plan helps a lot. They also find many syntax errors in Qwen3-Coder RLM trajectories, even when the final answer is correct. That helps explain why deeper recursion sometimes hurts Qwen more than GPT-5: errors propagate into nested sub-RLMs. GPT-5 is cleaner as a root controller.
Figure 4(a) shows how in-context examples of RLM strategies improve both the first decomposition and final score. Figure 4(b) shows Qwen3-Coder trajectories contain significantly more syntax errors than GPT-5, even in correct rollouts.
Training the first native RLM at small scale
They train RLM-Qwen3-8B by distillation style supervised fine-tuning. They run a large model, Qwen3-Coder-480B, as an RLM on LongBenchPro tasks, collect trajectories, filter out total failures and single-turn junk, split each root turn into training samples of history to next root action, fix common template mistakes programmatically, and fine-tune Qwen3-8B. The key training insight is that leaf sub-calls are basically normal general-purpose language model requests. The hard special skill is being a good root: manipulating the REPL, deciding when to sub-call, and managing the loop. So they focus training on root behavior rather than trying to retrain every leaf. With only about one thousand filtered samples from an unrelated domain, the 8B RLM improves by a median of roughly twenty-eight percent across the evaluation tasks and approaches vanilla GPT-5 quality on some long-context tasks. It is also faster and cheaper at inference because it makes better decisions and fewer mistakes. Separately, they use reinforcement learning with verifiable rewards on MRCRv2, training on shorter sequences with fewer needles and seeing generalization to much longer sequences with more needles. That suggests RLM control policies can length-generalize.
Figure 3(a) shows rejection fine-tuning Qwen3-8B on distilled RLM trajectories improves performance across benchmarks. Figure 3(b) shows RL training on shorter sequences generalizes to much longer ones.
Things that did not work and practical quirks
One system prompt does not fit all models. A prompt tuned for GPT-5 made Qwen3-Coder overuse sub-calls, so they had to warn it to batch aggressively. Weak coding models struggle because the whole paradigm is code-mediated. Thinking models that spend too many tokens on internal reasoning can hit per-call output limits mid-trajectory. Their sequential blocking implementation of sub-calls makes RLMs slow in wall-clock time; asynchronous calls would help a lot. Distinguishing a final answer from intermediate thoughts via FINAL tags is brittle, and models sometimes output a plan as if it were the answer. They believe native training will eventually remove the need for fragile answer tags.
Long reasoning beyond long context
LongCoT-mini has interdependent subproblems across math, chemistry, computer science, logic, and chess. A plain model tries to hold a huge chain of thought and drifts: lost intermediate values, compounding errors. An RLM can orchestrate rather than solve everything itself. With decomposition hints, the root is told to build a plan graph, memoize verified node answers in a dictionary, solve ready nodes in batches through sub-calls, verify before propagating, handle cycles carefully, and assemble the final answer by lookup rather than redoing math in Python. The paper reports that without a REPL-like isolation mechanism, even giving the base model decomposition hints can confuse it or hurt overall score, while the RLM benefits strongly. That is evidence that the environment is not just a storage hack for long documents; it is a control plane for multi-step reasoning.
Table 2 shows RLM with GPT-5.2 on LongCoT-mini, achieving 65.6% overall (vs 38.7% base) with decomposition hints, and outperforming the base model on every domain.
How this relates to other research directions
There are two broad ways people attack long context: change the base model architecture and training so it natively supports longer windows, or build scaffolds around a fixed model. RLMs sit in the second camp. They differ from lossy memory and compaction systems because the model itself decides how to manage external context rather than always summarizing away detail. They differ from many multi-agent systems because sub-calls are not fixed by a human workflow. They differ from methods that generate a program with model calls in one shot because the RLM can iterate, see execution feedback, and refine. The closest intuition is: put the prompt in the external world, then let the model program against it and recurse.
Limitations the authors admit
Harder and more natural long-context tasks still need better evaluation. Guardrails against runaway sub-call cost are underexplored. Asynchronous sandboxed REPLs could cut runtime and cost but add system complexity. Exploding costs on bad trajectories are a real risk. Their training results are small-scale and should be scaled carefully with larger models, more domains, and online on-policy rollouts. They frame RLM trajectories as a new kind of reasoning trace that future work can bootstrap and reinforce the way chain-of-thought was scaled.
The big picture in one paragraph
A language model does not have to see everything at once to use everything. If you give it a computer memory for the prompt, a loop for iterative code, and the power to call itself on pieces it selects, it can turn an unbounded document into a program of reads, transforms, and recursive questions. That program can do constant, linear, or quadratic semantic work depending on the task. Existing models can already be prompted into this behavior, and small models can be trained into better controllers. The result is a general inference paradigm that looks like a normal model API but behaves like a recursive programming agent over its own input.
What was the research trying to make possible
It was trying to make it possible for a general-purpose language model interface to accept arbitrarily long prompts and still do dense, high-quality work over that material, without waiting for bigger context windows and without relying on lossy summarization. More precisely, it wanted unbounded effective input length, effectively unbounded output length by external assembly, and a way for the model to perform an amount of semantic work that can grow with the size of the prompt or even with the square of that size when the task demands pairwise processing. It also wanted this as a task-agnostic inference method, not a custom pipeline per benchmark, and it wanted early evidence that models can be trained to be better at this recursive control style rather than only prompted into it.
What assumption does it quietly depend on
It quietly depends on the assumption that the base model is already a competent coder and planner that can use a REPL honestly: write correct enough Python, inspect variables, recover from truncated prints, and design chunking strategies without being spoon-fed the full text. It also assumes that semantic subproblems can be carved into pieces small enough for ordinary model calls, and that stitching those pieces in code preserves the answer. Another quiet assumption is that truncated stdout metadata is enough for the root to stay oriented, so the model will not need the full intermediate text in its neural context. For cost claims, it assumes sub-calls can be batched and that runaway recursion can be limited by depth caps and prompting. For the strongest dense tasks, it assumes the model will choose a decomposition that actually covers the information density of the problem rather than a shallow heuristic search. When those assumptions fail, as with weak coding models or models that over-call or syntax-error constantly, the scaffold loses its advantage.
What becomes obvious after reading it that was not obvious before
What becomes obvious is that long-context failure is often not only an attention or memory problem inside the Transformer. It is also an interface problem: we force the entire world into the token window and then invent summarizers to cope. Once the prompt is an external object, many impossible-looking tasks become ordinary programs that happen to call language models as subroutines. It also becomes obvious that context length and task complexity are coupled. A million-token needle task and a thirty-two-thousand-token pairwise aggregation task are not the same kind of hard. Another non-obvious point is that output length is part of the long-context story. If the answer itself is a huge structured object, generating it only through the root model’s tokens is a bottleneck; assembling it in the REPL is a different regime. Finally, it becomes clear that recursion depth and sub-calling are not free magic. They help most when the problem is information-dense, and they can hurt when the model is a messy coder, because errors multiply through nested calls.
Where the idea breaks if you push it outside the paper
It breaks when the model cannot code well enough to be the controller of the environment. It breaks when safety or product constraints forbid arbitrary code execution against user data. It breaks when sub-call fan-out becomes economically or latency-wise unacceptable, for example a naive quadratic strategy on a huge corpus with no pruning. It breaks when the answer depends on global structure that cannot be recovered from local chunks plus simple aggregation, and the model never invents the right global plan. It breaks if stdout truncation hides the exact failure mode the root needs to see, or if final-answer tagging fails and the system returns a plan or partial buffer. It may also break in highly interactive multimodal settings not studied here, or when the environment must include tools beyond a Python REPL and the prompt variable. Pushing to hundreds of millions of tokens without asynchronous execution, caching, and strong guardrails would likely make wall-clock time and rare catastrophic bills the dominant practical failures even if accuracy remains good on paper benchmarks. And if future base models get enormous reliable context windows with no rot, pure RLM scaffolding might become less necessary for moderate lengths, though the programmatic recursion idea could still matter for long-horizon reasoning and huge outputs.
What long-running problem did this paper move, even slightly
The long-running problem is how to get language models to do reliable work over more text and more steps than fits comfortably in one forward context. For years the field has swung between bigger windows, better positional methods, retrieval, memory modules, summarization agents, and multi-agent delegation. This paper moves that problem by reframing long context as external symbolic interaction plus recursive self-calls rather than as a pure sequence modeling capacity race. Even slightly, it shows a concrete recipe that already works with today’s frontier models, fails in informative ways that training can fix, and converts some tasks that look like context-window failures into tasks that look like algorithm design over a string variable. That is a real shift: from hoping the model remembers everything in attention, toward letting the model program how it will read, recurse, and remember outside itself.