Notes on SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
What is this paper about?
This paper from Salesforce AI Research is about teaching large language models to become what they call autonomous single-agent deep research systems. Deep research means the ability to take a complex question, search the internet for information, browse relevant web pages, write and run code to compute things, and then synthesize everything into a final answer, all without a human telling the model what to do at each step. The key word is autonomous, meaning the model figures out on its own what action to take next based on what it has learned so far. The paper focuses on starting from models that are already very good at reasoning, sometimes called thinking models, and then using reinforcement learning to teach them these agentic skills without ruining their existing reasoning abilities.
Single-agent versus multi-agent systems
There are two main ways people build deep research systems. In a multi-agent system, you have multiple different AI models that each play a specific role. One might be a planner that breaks the problem into smaller pieces. Another might be a researcher that searches the web. Another might be a coder that writes and runs code. Another might be a supervisor that checks the work. These roles are predefined and the workflow is mostly fixed in advance. In a single-agent system, there is just one model that does everything. It decides by itself whether to search, browse, code, or answer, based on whatever the situation demands. The paper argues that single-agent systems are simpler, more flexible, and more likely to generalize to new kinds of problems they have never seen before. They also point out that if you ever want to build a multi-agent system, you can always plug a strong single-agent into it as one of the specialized agents.
The three tools the agent is given
The agent gets only three basic tools to work with. The first is a search tool that takes a query and returns the top ten organic search results from the web, each with a URL, a title, and a short description. The second is a browse tool that takes a URL and a section number and returns the content of that web page converted to plain markdown text. Importantly, all hyperlinks are stripped from the page content, which means the agent cannot click on links. The only way to discover new URLs is through the search engine. This is a deliberate choice to keep things challenging and force the agent to be strategic. The third tool is a code interpreter that runs Python code on the local machine with a five minute timeout. Each execution is stateless, meaning variables are not shared between runs. The code cannot access the file system, the command line, sensitive packages, or the internet. The paper deliberately keeps the tools minimal and basic because they want the agent to learn to work hard during training. If the tools were too powerful or too easy to use, the agent would not be challenged enough to learn good strategies.
The clever trick for managing long conversations
One of the biggest practical problems with these agents is that they produce extremely long conversations. Every time the model thinks, it generates a long chain of thought that can be thousands of tokens. Every time it calls a tool, the tool result gets added to the conversation. Very quickly the context window fills up and there is no more room to work. The paper proposes a two-part solution. First, they reformulate the entire multi-turn conversation as a single-turn problem. Instead of having alternating user and assistant messages, they put everything into one giant user message that says here is the original question, here is everything that has happened so far including all tool calls and tool results, now what do you want to do next. This sounds like a small change but it makes a huge difference because models like QwQ and Qwen have been trained mostly on single-turn reasoning tasks like math problems. When you force them into long multi-turn conversations, their thinking quality degrades and they start producing repetitive or nonsensical outputs. By keeping everything in a single-turn format, the model operates much closer to what it was trained to do. Second, they give the agent a special tool called clean_memory that lets the agent delete old information from its context and replace it with a summary of what it thinks is important. When the memory gets too full, the model is told you need to clean up and it cannot do anything else until it does. This teaches the agent to manage its own attention and decide what is worth keeping and what can be discarded.
How they generated training data
Existing datasets for multi-hop question answering were too easy for these models. Even without search, the reasoning models could answer most of them correctly because the answers were already memorized during training. So the paper designed a synthetic data pipeline that generates much harder questions iteratively. They also included traditional math and code reasoning problems plus long-form report writing tasks. For reports, they had an LLM generate both a research question and a detailed grading rubric. The resulting dataset is so challenging that even OpenAI Deep Research with o3 scored less than sixty five percent on it. The paper notes that some questions required up to fifty tool calls to complete.
The reinforcement learning recipe
The core of the paper is their RL training approach. They start from models that are already optimized for reasoning and continue training them with reinforcement learning to add agentic skills. The algorithm is based on REINFORCE, a classic RL method, with a few important modifications. The most interesting modification is length-normalized advantage. When you run RL on these agentic tasks, some trajectories are very short with just a few tool calls and others are extremely long with dozens of tool calls. Without normalization, the long trajectories dominate the training signal because they contribute more steps to each batch. Even if the long trajectories are wrong and get negative rewards, they still dominate and the model starts to learn bad behaviors like making repetitive tool calls without making progress. The normalization divides the advantage by the trajectory length so that each step contributes roughly equally regardless of how long the trajectory is. This stabilizes training significantly. They also filter out invalid trajectories such as those that were truncated due to context overflow or that ended with format errors. They maintain a balanced ratio of positive to negative examples in each batch to prevent collapse. They also reuse partial rollouts. If a trajectory is interrupted partway through, they do not throw it away. Instead they treat the partial state as a new starting point and continue rolling out from there with the current policy. This exposes the model to more diverse intermediate states.
Main results
The best model, called SFR-DR-20B, is based on the gpt-oss-20b model and achieves 28.7 percent on the full text-only version of Humanity s Last Exam, a famously difficult benchmark of expert-level questions across math and science. This is a 65 percent relative improvement over the base gpt-oss-20b model which scored 17.3 percent. On FRAMES, which tests multi-hop reasoning with web browsing, SFR-DR-20B scores 82.8 percent, outperforming all other open-source single-agent and multi-agent systems of comparable size. On GAIA, which tests general assistant-like tasks, it scores 66 percent. These numbers are competitive with proprietary systems like OpenAI Deep Research with o3 even though SFR-DR-20B is a much smaller and fully open-source model. The paper also trained variants based on Qwen3-8B and QwQ-32B, and these also showed strong improvements over their base models and compared favorably to other open-source systems.
Why the gpt-oss-20b variant performed best
The paper observes that different base models behave very differently. The Qwen-family models tend to generate very long chains of thought, often four to five times longer than gpt-oss-20b per step. This makes them harder to steer with RL because there is so much thinking between each action. The gpt-oss-20b model is much more token-efficient, generating shorter thoughts and more tool calls. It also started with a higher baseline of tool usage before RL training. The paper hypothesizes that gpt-oss-20b was already better primed for agentic tasks during its original training. Another interesting observation is that RL training had opposite effects on different models. For QwQ and Qwen3, RL increased the average response length because the model learned to think more. For gpt-oss-20b, RL actually decreased the response length, making it even more efficient.
More tool calls is not always better
The paper provides a careful caveat about tool usage. Intuition might suggest that more tool calls means more effort and therefore better answers. But the paper shows that under standard RL without length normalization, the model degenerates into making repetitive identical tool calls. The trajectory gets longer but the behavior gets worse. The key is not how many tool calls the model makes but whether those calls are diverse and strategically executed. Their length normalization prevents this degenerate behavior and keeps tool usage growing at a moderate and productive rate.
The fault tolerance system
LLMs are stochastic and sometimes produce outputs in the wrong format. The paper describes a multi-level error handling system. If the model produces a tool call with a misplaced special token, the system tries to repair it deterministically. If that fails, the model receives a syntax error message and gets a chance to try again. Similar warnings appear if the model calls a non-existent tool or passes invalid parameters. This keeps the trajectory going even when the model makes small mistakes, rather than crashing the entire episode.
The infrastructure for training
Training these models required significant engineering. The paper built an in-house pipeline that aggressively parallelizes rollouts across multiple GPUs using SGLang for inference. All tool executions are local except for the search API. Tool results are cached so repeated calls do not waste time. The inference engines, verifier models, and training models are all co-located on the same GPUs. During gradient updates, the inference engines are offloaded to free up memory, and vice versa. This eliminates idle GPU time. They also implemented recovery procedures for out-of-memory errors and crashed components to reduce the need for human supervision during long training runs.
Contamination prevention
Since all the benchmarks are publicly available on the internet, there is a risk that the agent will cheat by visiting the exact page where the answer is published. The paper blocks certain domain names like huggingface.co and returns an unavailable error if the agent tries to access them. This is an important methodological detail because recent work has shown that up to 3.4 percent of usable samples on Humanity s Last Exam can be contaminated in this way. The paper notes that most open-source baselines do not mention any such precautions, which may inflate their reported numbers.
Chase Questions
What was the research trying to make possible?
The research was trying to make it possible for a single AI model, not a team of specialized models, to autonomously research and answer very hard questions by searching the web, reading pages, and writing code, all without human guidance at intermediate steps. The deeper goal was to show that you can take an existing model that is already good at reasoning and use reinforcement learning to turn it into an effective research agent without losing its reasoning ability. The paper wanted to demonstrate a simple, reproducible recipe that works across different open-source models and produces results competitive with proprietary systems.
What assumption does it quietly depend on?
The paper quietly depends on the assumption that the base reasoning models already have most of the necessary knowledge and reasoning capabilities inside them, and that the main thing holding them back from being good research agents is the lack of training on how to use tools strategically. In other words, it assumes the models already know enough about the world and just need to learn when to search, what to search for, and how to integrate what they find. This assumption is visible in the choice to start from reasoning-optimized models rather than base models, and in the choice to keep tools minimal so that the agent has to rely on its own reasoning rather than on tool capabilities. The paper also assumes that synthetic data generated by another LLM is sufficient to teach these skills, which quietly assumes that the generating LLM is capable enough to create suitably challenging and diverse training examples.
What becomes obvious after reading it that was not obvious before?
What becomes obvious is that the format of the conversation matters enormously more than one might intuitively expect. The simple trick of reformatting a multi-turn conversation as a single-turn contextual question produced a ten percent absolute improvement on FRAMES for the 32B model, all without any training at all. This makes clear that many of the difficulties in building long-horizon agents are not necessarily due to the model lacking capability but due to the input format being far from what the model was trained to handle. It also becomes obvious that more tool calls does not mean better reasoning and that naive RL training actively encourages degenerate repetitive behavior if not carefully constrained. The length normalization insight is something that seems obvious in retrospect but is easy to miss when focused on more sophisticated algorithmic improvements.
Where does the idea break if you push it outside the paper?
The idea starts to break in several places if pushed too far. First, the entire approach depends on having a strong reasoning model as a starting point. If the base model is not already capable of complex reasoning, the RL recipe will not magically create that capability. Second, the tool set is deliberately minimal and the browse tool strips all hyperlinks, which makes the agent fundamentally dependent on the search engine for discovering new information. If the search engine is poor or the information is not indexed, the agent cannot find it. In a real-world setting where information is spread across internal databases, PDFs, images, and other non-web sources, this minimal tool set would be insufficient. Third, the training data is entirely synthetic and generated by another LLM. This creates a ceiling on what the agent can learn because it cannot surpass the capabilities or knowledge of the teacher model. Fourth, the approach requires significant engineering infrastructure for RL training with parallel rollouts, caching, co-located GPUs, and recovery mechanisms. Smaller labs without such resources would struggle to reproduce the results. Fifth, the paper only evaluates on benchmarks that are essentially about fact-finding and reasoning on the public web. It does not test scenarios where the agent needs to interact with APIs, manipulate databases, fill out forms, authenticate to services, or operate in a continuously changing environment. In those settings, the stateless code interpreter and the stripped-page browse tool would be serious limitations.
What long-running problem did this paper move, even slightly?
The paper moved the problem of training autonomous single-agent systems from something that was mostly done with multi-agent scaffolding or cold-start training from base models to something that can be done with continual RL on reasoning-optimized models. It showed that a relatively simple recipe with synthetic data and careful normalization can produce a 20 billion parameter model that competes with much larger proprietary systems. This is a small but meaningful step toward democratizing deep research capabilities. The paper also advanced the understanding of a specific practical problem: how to stop RL training from degenerating into repetitive tool-calling behavior. The length normalization insight and the analysis of how long trajectories dominate the loss even when penalized are contributions that future work in agentic RL can build upon. The paper also demonstrated that inference-time formatting tricks can unlock significant gains without any training, which may influence how future reasoning models are designed to handle multi-turn interactions natively.