DeepSeekMath Paper Notes
DeepSeekMath is a 7 billion parameter language model created by DeepSeek AI in collaboration with Tsinghua University and Peking University. It is designed specifically to solve mathematical problems by reasoning step by step. The model achieves 51.7 percent accuracy on the MATH benchmark, which is a collection of very hard competition level math problems. This score is close to what GPT-4 and Gemini Ultra achieve, even though those are much larger closed source models that nobody can inspect or modify. DeepSeekMath is fully open source, meaning anyone can download it, study it, and build on top of it. The paper shows that you do not need a giant model with hundreds of billions of parameters to do well at math. What you need instead is high quality training data and a smart way to train the model using reinforcement learning.
How They Built the Training Data
The researchers started with the observation that there is a huge amount of mathematical content already available on the public internet, specifically in a massive web archive called Common Crawl. The challenge is that Common Crawl contains trillions of web pages and only a tiny fraction of them are about math. So they built a pipeline to automatically find math related pages. They began with a small seed dataset called OpenWebMath, which is a collection of high quality math web pages that were already curated by other researchers. Using this seed, they trained a fastText classifier, which is a lightweight text classification algorithm, to recognize math content. The classifier was trained on 500,000 math pages as positive examples and 500,000 random web pages as negative examples. This classifier was then run against the entire deduplicated Common Crawl to find pages that look like math.
The first pass collected about 40 billion tokens of math content. But the researchers noticed that many math pages were being missed because the initial seed dataset was not diverse enough. So they added a second step. They organized the entire Common Crawl into domains, which are groups of pages sharing the same base URL. For each domain they checked what percentage of its pages were already collected. Domains where more than 10 percent of pages were collected were classified as math related domains, such as mathoverflow dot net. Human annotators then looked at these domains and identified specific URL patterns that contain math, like mathoverflow dot net questions. Pages matching these URL patterns that were missed in the first pass were added back to the seed corpus. The fastText classifier was then retrained on this enriched seed and the whole process was repeated. After four such iterations they ended up with 35.5 million math web pages totaling 120 billion tokens. This is about 7 times larger than the math data used by Google’s Minerva model and 9 times larger than OpenWebMath. After the fourth iteration nearly 98 percent of collectible math pages were already captured, so they stopped.
The data also includes multilingual content, primarily English and Chinese. This is different from previous math datasets which were almost entirely English. The researchers also took care to remove benchmark contamination. They filtered out any web page that contained a 10 gram substring that matches any substring from the evaluation benchmarks like GSM8K and MATH, to ensure that the model was not simply memorizing test answers.
Why Code Training Helps Math
An interesting finding in this paper is that training a model on computer code before training it on mathematics significantly improves its math ability. The researchers ran controlled experiments using a smaller 1.3 billion parameter model. They compared two stage training where the model first sees 400 billion tokens of code and then 150 billion tokens of math versus two stage training where the first stage uses general text instead of code versus one stage training where code and math are mixed together. They found that code pre training enhances the model’s ability to solve math problems both with and without using programming tools. Even without allowing the model to write and run code, the model that was pre trained on code performed better at pure math reasoning. This provides a partial answer to a long standing debate in the AI community about whether code training improves reasoning. The paper suggests it does, at least for mathematical reasoning. However, when code and math are mixed in a single stage of training, the improvement in pure reasoning is less pronounced, possibly because the small 1.3 billion parameter model lacks the capacity to absorb both types of data at once. The 7 billion parameter version does not seem to have this limitation.
ArXiv Papers Do Not Help
Another counterintuitive finding is that training on academic papers from arXiv, which is a common practice in math focused models, does not improve mathematical reasoning. The researchers tested this on two model sizes and using different arXiv datasets. They tried MathPile, which is over 85 percent arXiv papers, and ArXiv RedPajama, which is the full collection of arXiv LaTeX files with preambles and comments removed. Neither dataset produced any meaningful improvement on GSM8K, MATH, MMLU STEM, or miniF2F benchmarks. In some cases the arXiv trained models even performed worse than having no math training at all. The paper is careful to note some limitations of this finding. It is possible that arXiv papers help with other math tasks not tested in this study, such as converting formal proofs into informal language. It is also possible that arXiv papers become beneficial at larger model scales or when mixed with other types of data. But for the 7 billion parameter scale and for the benchmarks considered, arXiv papers do not move the needle. This is an important result because many previous projects invested heavily in processing arXiv data, and this paper suggests that effort might be better spent on web data.
The Base Model Performance
The final base model DeepSeekMath Base 7B was initialized from DeepSeek Coder Base v1.5 7B, which is a model already trained extensively on code. It was then further trained on 500 billion tokens, of which 56 percent came from the DeepSeekMath Corpus, 4 percent from AlgebraicStack which is math related code, 10 percent from arXiv, 20 percent from general GitHub code, and 10 percent from general web text in English and Chinese. The result is a base model that achieves 64.2 percent on GSM8K, which is grade school math problems, and 36.2 percent on MATH, which is competition level problems. This outperforms Minerva 540B, a closed source model that is 77 times larger. It also outperforms all open source base models including Llemma 34B, a model that is nearly 5 times larger. On Chinese math benchmarks the lead is even larger because the DeepSeekMath Corpus includes Chinese content while prior datasets are almost entirely English. Beyond math, the base model also shows improvements on general reasoning benchmarks like MMLU and BBH, and maintains strong coding performance. This shows that math training does not narrow the model’s capabilities but actually broadens its reasoning ability.
Supervised Fine Tuning
After pre training the base model, the researchers applied supervised fine tuning, which means they showed the model many examples of math problems with step by step solutions written in natural language, in Python code, or using a mix of both. They created 776,000 training examples covering English and Chinese problems across many fields of mathematics including algebra, probability, number theory, calculus, and geometry. The resulting model, DeepSeekMath Instruct 7B, achieves 82.9 percent on GSM8K and 46.8 percent on MATH using chain of thought reasoning, which is the approach where the model writes out its reasoning steps before giving the final answer. This surpasses all open source instruction tuned models and comes close to proprietary models. When allowed to use Python tools, the model reaches 57.4 percent on MATH. The fine tuned model is competitive with models 10 times its size, such as DeepSeek LLM Chat 67B.
Group Relative Policy Optimization GRPO
The most novel technical contribution of this paper is a reinforcement learning algorithm called Group Relative Policy Optimization, or GRPO. To understand GRPO, it helps to first understand PPO, which is the standard reinforcement learning algorithm used to fine tune language models. In PPO, there are four models that need to be kept in memory simultaneously. There is the policy model, which is the language model being trained. There is the reward model, which scores how good an output is. There is the reference model, which is a frozen copy of the policy from before training started and is used to prevent the policy from drifting too far. And there is the value model or critic model, which is a separate neural network of similar size to the policy model that tries to predict how good a given state or token position is. The value model is needed in PPO to calculate the advantage, which measures how much better or worse an action is compared to the average expectation. Training and storing this value model doubles the memory requirements and adds significant computational cost.
GRPO eliminates the value model entirely. Instead of training a separate network to estimate the baseline, GRPO samples a group of outputs from the policy model for the same question and uses the average reward of the group as the baseline. For each question, the model generates multiple answers, typically 64 of them. Each answer is scored by the reward model. The rewards are then normalized by subtracting the group mean and dividing by the group standard deviation. Answers that score above the group average receive a positive advantage and are reinforced, meaning the model is encouraged to produce similar outputs. Answers that score below the group average receive a negative advantage and are penalized. This group relative comparison is conceptually clean because reward models themselves are typically trained on comparative data, meaning they learn to tell which of two answers is better rather than assigning absolute scores. So using group comparisons aligns naturally with how reward models work. The KL divergence between the current policy and the reference policy is added directly to the loss function rather than being folded into the reward calculation, which keeps the math simpler.
GRPO can be used with either outcome supervision or process supervision. Outcome supervision gives a single reward at the end of the entire answer. Process supervision gives a reward at the end of each reasoning step, which provides more fine grained feedback. The paper experiments with both and finds that process supervision leads to better results. GRPO can also be run iteratively, where the reward model is periodically retrained on new data sampled from the latest policy model. This iterative approach yields further improvements.
How GRPO Improves Performance
When GRPO is applied on top of DeepSeekMath Instruct 7B, GSM8K accuracy goes from 82.9 percent to 88.2 percent and MATH accuracy goes from 46.8 percent to 51.7 percent. Remarkably, the reinforcement learning was done using only the chain of thought instruction tuning data from GSM8K and MATH, which is about 144,000 questions. Despite being trained only on these two datasets in CoT format, the model shows improvements on out of domain benchmarks like CMATH, a Chinese math benchmark, and MGSM zh, a multilingual version of GSM8K. This suggests that reinforcement learning does not just make the model better at the specific training tasks but improves its general mathematical reasoning ability.
The paper investigates why RL works by examining two metrics. Pass at K measures whether the correct answer appears anywhere among K sampled outputs. Maj at K measures whether the majority vote among K sampled outputs gives the correct answer. The Instruct model and the RL model have similar Pass at K scores, meaning the RL model does not generate correct answers that the Instruct model could not already generate. But the RL model has significantly higher Maj at K scores, meaning the correct answer appears more consistently across multiple samples. The improvement comes from making the output distribution more robust. The model becomes more reliable at producing the correct answer rather than occasionally stumbling upon it by chance. In other words, RL cleans up the probability distribution by suppressing incorrect reasoning paths and amplifying correct ones, without necessarily expanding the set of problems the model can solve.
The Unified Paradigm
The paper provides a unified way of thinking about different training methods including supervised fine tuning, rejection sampling fine tuning, direct preference optimization, PPO, and GRPO. All these methods can be described in terms of three components. First is the data source, which determines where the training examples come from. Some methods use offline data sampled once from a fixed model, while others use online data sampled fresh from the model being trained. Second is the reward function, which determines the quality signal. This can be a simple rule like whether the final answer is correct, or it can be a learned neural reward model that gives nuanced scores. Third is the algorithm, which processes the data and reward signal to produce a gradient coefficient that determines how much each token’s probability should be increased or decreased. The paper shows mathematically how each method maps to this framework and draws key insights. Online training, where data is sampled from the current model rather than a fixed dataset, performs better than offline training because the data distribution stays relevant as the model improves. Model based rewards outperform rule based rewards because they provide differentiated signals rather than just a binary correct incorrect judgment. GRPO with process supervision, which gives step level gradient coefficients, outperforms GRPO with outcome supervision, which gives only a single coefficient per answer.
Limitations of the Paper
The paper acknowledges several limitations. DeepSeekMath is weaker than closed models like GPT-4 on geometry problems and theorem proving. The model struggles with problems involving triangles and ellipses, which suggests a data selection bias. The paper notes that during a dry run the model could not handle such problems at all. Additionally, due to the small 7 billion parameter scale, the model does not benefit from few shot prompting as much as GPT-4 does. GPT-4 can improve its performance when given a few examples in the prompt, but DeepSeekMath performs similarly whether it is given examples or not. The model also has limitations in its reinforcement learning pipeline. The RL only uses questions from the instruction tuning data, so it may not generalize to truly novel problem types. The paper suggests future work should explore out of distribution questions, tree search decoding strategies, and more efficient inference techniques to improve exploration during RL.
Chase Questions
What was the research trying to make possible
The research was trying to make advanced mathematical reasoning capabilities available to everyone by creating an open source language model that could compete with proprietary systems like GPT-4 and Gemini Ultra on hard math problems. Before this paper, the best open source models lagged far behind closed models on the MATH benchmark. The gap was around 20 to 30 percentage points. DeepSeekMath closed most of this gap using only 7 billion parameters, which is small enough to run on consumer hardware. The paper also made the data collection pipeline and the reinforcement learning algorithm publicly available, so that other researchers can build better math models without needing access to proprietary data or expensive compute.
What assumption does it quietly depend on
The paper quietly depends on the assumption that web text contains sufficient mathematical knowledge to train a state of the art model, and that the main bottleneck is finding and filtering that text rather than generating new knowledge or using structured sources like textbooks. It also assumes that the fastText classifier approach with iterative human annotation is an adequate substitute for deeper understanding of what makes math content useful. There is an implicit assumption that more math tokens are always better, up to 120 billion tokens. The paper does not prove that all 120 billion tokens are equally valuable or that a smaller more carefully curated set would not work just as well. Another quiet assumption is that correctness of the final answer is a sufficient signal for training, both in the supervised data and as the basis for the reward model used in reinforcement learning. This assumes that getting the right answer means the reasoning was correct, which is not always true. A model can arrive at the right answer through flawed reasoning or lucky guesses.
What becomes obvious after reading it that was not obvious before
It becomes obvious that data quality and data processing strategy matter far more than model size for mathematical reasoning. A 7 billion parameter model trained on carefully selected web data can beat a 540 billion parameter model trained on less carefully selected data. This is a striking result that shifts focus from scaling parameters to scaling data quality. It also becomes obvious that code pre training is a powerful and perhaps necessary ingredient for mathematical reasoning, and that arXiv papers, despite their apparent relevance, are surprisingly ineffective. Another thing that becomes obvious is that reinforcement learning improves consistency rather than capability. The model does not learn to solve new types of problems through RL. It just becomes more reliable at producing the correct answer for problems it could already solve. This reframes what RL actually does in the math domain.
Where does the idea break if you push it outside the paper
If you push the DeepSeekMath approach outside the paper into other domains, it might break in several ways. The iterative data collection pipeline relies on having a reasonable seed dataset to start with and having math content that is discoverable through URL patterns and fastText classification. For domains where good seed data does not exist or where the content is not clustered in predictable URL patterns, the pipeline would not work. The approach also assumes that the domain has abundant public web data, which is true for math but might not be true for specialized scientific fields that are behind paywalls or discussed mainly in private forums. GRPO assumes that you can get a reliable reward signal from comparing outputs. For tasks where correctness is subjective, such as creative writing or translation, the group relative approach might not provide meaningful advantages because there is no objective ground truth to anchor the comparison. The finding that arXiv papers are useless might not generalize to other model sizes or other domains. At larger scales or for more advanced mathematical topics, formal papers might become essential. The finding might also be specific to the benchmarks tested, which are mostly K 12 and undergraduate level competition problems. For research level mathematics, arXiv papers might be the only available data source.
What long running problem did this paper move even slightly
This paper moved the problem of democratizing advanced AI reasoning. Before DeepSeekMath, the narrative was that only massive proprietary models could achieve strong math performance, and that open source would always lag far behind. The paper showed that with clever data engineering and an efficient RL algorithm, a small open model can close most of the gap. This shifts the conversation from how many parameters you need to how well you select and process your data. The paper also moved the problem of reinforcement learning efficiency. By eliminating the value model in PPO, GRPO reduces the memory footprint of RL fine tuning by roughly half. This makes RL training accessible to teams with limited GPU resources. The unified paradigm for comparing training methods is another small but meaningful contribution. It gives researchers a common language to discuss what is really different between SFT, RFT, DPO, PPO, and GRPO, which should lead to more systematic progress rather than ad hoc experimentation. Finally, the surprising finding about arXiv papers being ineffective might save future research teams months of effort spent processing academic papers that do not actually help.