This paper is a technical preview of DeepSeek-V4, a new family of large language models from DeepSeek-AI. The benchmark scores are good, but that is not the interesting part. The interesting part is that the models try to make million-token context practical. The model can read and work with extremely long inputs, such as huge documents, long agent conversations, long chains of tool use, and long internal reasoning, without the usual explosion in cost and memory. The paper presents two models. DeepSeek-V4-Pro is the larger one, with about 1.6 trillion total parameters but only about 49 billion active for each token. DeepSeek-V4-Flash is the smaller and cheaper one, with about 284 billion total parameters and about 13 billion active per token. Both are Mixture-of-Experts models, which means they contain many expert sub-networks and only a few are switched on for any given token. Both claim native support for a context length of one million tokens. The authors also introduce a maximum reasoning effort mode called DeepSeek-V4-Pro-Max, the strongest setting of the Pro model and the one they compare most aggressively against frontier closed models.
Figure (left) shows DeepSeek-V4-Pro-Max competing with frontier models on benchmarks. (right) shows inference FLOPs and KV cache size versus DeepSeek-V3.2. V4-Pro uses only ~27% of the FLOPs and ~10% of the KV cache at 1M context.
Why long context was the real bottleneck
Modern large language models got a big boost from test-time scaling, which means letting the model think longer, produce more reasoning tokens, call more tools, and explore more paths before answering. That approach works, but it hits a hard wall. Standard attention, the core mechanism that lets every token look at every earlier token, grows roughly with the square of sequence length. If the sequence becomes ten times longer, the attention work can become about one hundred times heavier. The model must also store a Key-Value cache of past tokens so that generation does not recompute everything from scratch. That cache grows with sequence length and becomes a memory bottleneck. Ultra-long reasoning, multi-document analysis, and long-horizon agents stay expensive even if the model is smart. DeepSeek-V4 tries to break that efficiency barrier so that million-token contexts stop being rare research demos and become ordinary infrastructure for training, serving, and agent systems.
The high-level architecture picture
DeepSeek-V4 still looks like a Transformer. Tokens go through embeddings, then many Transformer blocks, then a prediction head. It also keeps Multi-Token Prediction modules, which means the model predicts multiple future tokens, not just the next one. That idea was already used in DeepSeek-V3 and is kept here because it worked. The feed-forward parts still use DeepSeekMoE, meaning shared experts plus many routed experts. Three things are new. First, residual connections between layers are upgraded with Manifold-Constrained Hyper-Connections, often abbreviated mHC. Second, attention is no longer ordinary full attention. Instead the model uses a hybrid of Compressed Sparse Attention and Heavily Compressed Attention, often abbreviated CSA and HCA. Third, most parameters are trained with the Muon optimizer rather than only AdamW. The model keeps the proven MoE and multi-token prediction backbone, then upgrades the residual highway, redesigns attention for long sequences, and changes the optimizer so training converges faster and more stably.
Figure 2 shows how CSA and HCA attention layers, DeepSeekMoE feed-forward layers, and mHC residual connections fit together in the DeepSeek-V4 architecture.
What was inherited from DeepSeek-V3
DeepSeek-V4 deliberately reuses a lot of DeepSeek-V3. The MoE design still uses fine-grained experts: one shared expert that always runs, plus many routed experts of which only a few fire for each token. Load balancing still uses an auxiliary-loss-free strategy, with only a light sequence-wise balance loss to stop extreme imbalance inside one sequence. Multi-Token Prediction is kept with the same basic setup as V3. Tokenization still uses a 128K vocabulary, with a few special tokens added for context construction. Fill-in-Middle training and token-splitting ideas also continue. But there are meaningful tweaks. The function that scores how well a token matches an expert changes from a simple sigmoid to a square-root softplus form. The old hard limit on how many routing target nodes can be used is removed, and parallelism is redesigned around that freedom. The first few Transformer blocks no longer use dense feed-forward layers. They use MoE with hash routing, a simple routing rule that assigns tokens to experts based on a fixed hash of the token ID rather than a learned soft score. That is cheaper and more stable early in the network, where the model is still mostly lifting raw tokens into a richer representation.
Manifold-Constrained Hyper-Connections in simple words
In a normal Transformer, each layer takes an input, transforms it, and adds the original input back through a residual connection. That residual path is one of the main reasons deep networks train at all, because it gives information and gradients a highway through the stack. Hyper-Connections go further by widening the residual stream. Instead of one residual vector of hidden size, the residual stream becomes several parallel residual channels, often written as an expansion factor times the hidden size. The actual layer still sees a normal-sized input, so the inner computations do not suddenly become huge. The residual highway itself becomes wider and more expressive. That is attractive because it gives another scaling axis with relatively low compute cost. The problem is that unconstrained Hyper-Connections can become numerically unstable when stacked deeply. Signals can explode, cancel, or drift in messy ways.
Manifold-Constrained Hyper-Connections try to keep the expressivity while forcing the residual mixing matrix to live on a safer mathematical set. The residual transformation matrix is constrained to be doubly stochastic, meaning every row and every column sums to one, and all entries are non-negative. Matrices like that behave gently. Their spectral norm is at most one, so they do not expand vectors in a wild way. The set is also closed under multiplication, which matters when many residual transformations are stacked layer after layer. Input and output maps into and out of the residual stream are also forced to be non-negative and bounded through sigmoid-style transforms, so channels cannot cancel each other through large negative weights. The parameters of these maps are dynamic, meaning they depend on the current residual state, but they also have static learned biases and small gating factors. After the raw maps are generated, the residual matrix is projected onto the doubly stochastic manifold with the Sinkhorn-Knopp algorithm. That means exponentiate for positivity, then repeatedly normalize rows and columns, usually for about twenty iterations. The result is a residual highway that is wider and more flexible than a plain skip connection, but still controlled enough to train at large depth and scale. Engineering-wise, mHC costs extra activation memory and pipeline communication, so the authors use fused kernels, selective recomputation, and pipeline adjustments. With those tricks, wall-time overhead is only about 6.7 percent of an overlapped pipeline stage.
Why attention needed a redesign
Even with a better residual path, attention is still the long-context villain. In ordinary attention, every query token looks at every previous key and value. For one million tokens that is brutal both in floating-point operations and in cache size. DeepSeek-V4 therefore builds a hybrid attention system with two complementary styles, CSA and HCA, interleaved through the layers. The shared philosophy is compression. Instead of storing and attending over every raw token, the model compresses groups of Key-Value entries into fewer summary entries. CSA compresses moderately and then adds sparsity on top. HCA compresses much more aggressively but keeps dense attention over the heavily compressed stream. Together they cut both compute and memory enough that million-token context becomes practical, especially when combined with low-precision storage.
Compressed Sparse Attention explained carefully
Compressed Sparse Attention works in stages. First it builds token-level Key-Value-like entries and also learns compression weights. Then it compresses every block of tokens into one compressed Key-Value entry. In the paper the compression rate for CSA is four, meaning roughly four original tokens become one compressed entry. The compression is weighted and soft. Softmax over a local set of entries produces mixture weights, and the compressed entry is a weighted combination of those entries plus learned positional biases. There is also a small overlap idea in the indexing of neighboring blocks so that the sequence is effectively shortened by the compression factor in a smooth way. After compression, CSA does not let every query attend to every compressed entry. That would still be expensive. Instead it uses a lightning indexer. The indexer is a cheap scoring network that ranks compressed blocks for each query and keeps only the top-k most relevant ones. For Flash, top-k is 512 compressed entries. For Pro, top-k is 1024. Those selected compressed entries then enter the main attention as shared keys and values in a Multi-Query Attention style. Multi-Query Attention means many query heads share the same key and value stream, which saves cache and compute.
There are more details that matter for quality and cost. Queries for both the indexer and the main attention are produced in a low-rank way. The hidden state is first projected down into a compressed latent, then expanded into many heads. Because the number of query heads is large, dumping all head outputs into one giant projection would be expensive, so CSA uses grouped output projection. Heads are split into groups, each group is projected to a smaller intermediate output, and then those intermediates are projected to the final attention output. The model also adds a sliding-window branch of recent uncompressed Key-Value entries so local neighbors are never lost. That matters because pure block compression can hide tokens inside the current incomplete block, and language modeling often depends strongly on recent local context. For CSA, window size is 128. There is also query and Key-Value normalization with RMSNorm just before core attention to stop attention logits from exploding. Rotary positional embeddings are applied only partially, specifically to the last 64 dimensions of the relevant vectors, and a reverse rotary is applied on the attention output so the model ends up with relative position information rather than sticky absolute position artifacts. Finally, attention sink logits are added so each head can choose not to put all of its probability mass on real tokens. A head can decide that none of the candidates deserve full attention.
Figure 3 shows how CSA compresses KV entries (4:1 ratio), uses a lightning indexer for sparse top-k selection, and combines selected compressed entries with a sliding-window branch for local detail.
Heavily Compressed Attention explained carefully
Heavily Compressed Attention is the more extreme sibling. It compresses every large block of tokens into one entry, with a much larger compression rate. In both Flash and Pro, that rate is 128, so one hundred twenty-eight tokens collapse into one heavily compressed Key-Value entry. Unlike CSA, HCA does not use sparse top-k selection over compressed blocks. After heavy compression, the compressed stream is short enough that dense Multi-Query Attention over it is acceptable. HCA still uses the same family of ideas for low-rank query generation, grouped output projection, sliding-window local attention, partial rotary embeddings, RMSNorm before attention, and attention sinks. The difference in philosophy is clear. CSA keeps more fine-grained compressed memory and then sparsely picks the useful pieces. HCA throws away most fine grain early, keeps a coarse global memory that is cheap to attend over densely, and relies on the sliding window plus other layers for local detail. By interleaving CSA and HCA across depth, the model gets both sparse selective memory and cheap global memory.
Figure 4 shows HCA compressing 128 tokens into one KV entry and attending densely over the compressed stream, plus a sliding window for local context.
How efficient this hybrid attention really is
The efficiency gains come from several stacked choices, not one trick alone. Compression reduces how many Key-Value entries exist. Sparse top-k in CSA further reduces how many of those entries each query actually touches. Multi-Query Attention reduces key and value fan-out. Grouped output projection reduces projection cost. Mixed precision storage stores rotary dimensions in BF16 and the rest in FP8, nearly cutting Key-Value cache size in half relative to pure BF16. Indexer attention can run in FP4. Relative to DeepSeek-V3.2, the paper claims that in the one-million-token setting DeepSeek-V4-Pro needs only about 27 percent of the single-token inference FLOPs and about 10 percent of the Key-Value cache. DeepSeek-V4-Flash is even leaner, around 10 percent of the FLOPs and 7 percent of the cache of V3.2. Against a classical BF16 grouped-query attention baseline with eight groups and head dimension 128, the Key-Value cache of DeepSeek-V4 can shrink to roughly 2 percent in the million-token regime. That is why the authors claim million-token context is no longer a special mode but something they can support routinely.
The Muon optimizer
Training such a model also needs a strong optimizer. Most modules in DeepSeek-V4 use Muon. AdamW is kept only for embeddings, prediction heads, RMSNorm weights, and some static mHC biases and gates. Muon works on whole weight matrices rather than elementwise like Adam. It accumulates momentum, applies a Nesterov-style trick, orthogonalizes the update with Newton-Schulz iterations, rescales the update so its root-mean-square matches a target, then applies weight decay and the step. DeepSeek-V4 uses a hybrid Newton-Schulz schedule. Eight iterations use aggressive coefficients that push singular values near one, then two final iterations use gentler coefficients that settle them exactly at one. Because the attention design already normalizes queries and keys, the authors do not need the QK-Clip trick that some Muon users rely on to stop exploding attention logits. The practical promise of Muon here is faster convergence and better stability on huge matrix-shaped parameters.
Infrastructure: making MoE communication hide under compute
A giant MoE model is a systems problem as much as a math problem. Expert Parallelism means different experts live on different devices, so tokens must be dispatched to the right devices and later combined back. That communication can dominate if done naively. DeepSeek-V4 uses a fine-grained scheme that fuses communication and computation into one pipelined mega-kernel. An MoE layer has dispatch, first linear, activation, second linear, and combine. Profiling showed that total communication time is less than total computation time inside a layer, so communication can be hidden under compute if scheduled carefully. The authors split experts into waves. As soon as one wave finishes communication, it starts computing while the next wave is still transferring data and earlier waves are combining results. That fine pipeline keeps both network and compute busy and is especially helpful for small-batch latency-sensitive cases such as reinforcement learning rollouts. They report roughly 1.5 to 1.73 times speedup in general inference and up to about 1.96 times in some latency-sensitive settings. They also share hardware co-design lessons. Once interconnect bandwidth is high enough relative to compute, more bandwidth stops helping much. Power headroom matters when compute, memory, and network all peak together. Pull-based remote reads avoid notification latency. And cheaper activations than SwiGLU could free the matrix-multiply pipeline from waiting on expensive nonlinearities.
TileLang and the cost of too many tiny operators
The architecture would create hundreds of small operators if written naively in ordinary framework ops. DeepSeek therefore uses TileLang, a domain-specific language for writing fused GPU kernels that are both productive to develop and fast enough for production. TileLang generates both device code and lightweight host launchers so that Python-side validation no longer burns tens or hundreds of microseconds per call. It uses an SMT solver for formal integer reasoning during compilation, which helps with layout inference, bounds checks, vectorization, and barrier insertion. Numerical accuracy is treated seriously. Fast-math is off by default, approximate math is opt-in, and IEEE-style operations with explicit rounding are available when needed. They also aim for bitwise reproducibility against hand-written CUDA baselines by controlling algebraic simplification and layout choices.
Batch-invariant and deterministic kernels
DeepSeek-V4 puts unusual effort into making kernels batch-invariant and deterministic. Batch-invariant means the output for a token does not change just because the batch around it changed. Deterministic means rerunning the same computation gives bit-identical results. That is important for debugging, comparing training stages, and making post-training behavior stable. Attention is hard because popular split-KV load balancing can change accumulation order. Their solution is a dual-kernel strategy. One kernel computes a whole sequence on one streaming multiprocessor for full waves, another uses multiple processors for the leftover partial wave but carefully preserves the same accumulation order. Matrix multiplies abandon many split-k shortcuts that break invariance, then recover performance with other optimizations. Determinism also requires eliminating atomicAdd races in sparse attention backward and carefully ordering MoE backward writes. Even tiny mHC matrix multiplies that must use split-k get a deterministic reduction kernel afterward.
Training framework upgrades for Muon, mHC, and compressed attention
Muon does not play nicely with classic ZeRO optimizer sharding because it needs full gradient matrices, not only shards of elementwise state. The authors therefore use a hybrid ZeRO strategy. Dense matrices are packed into balanced buckets with limited ZeRO width and some padding. MoE parameters are flattened carefully without splitting a logical matrix, then sharded across ranks. Matrices of the same shape can be batched through Newton-Schulz. MoE gradients for communication can be stochastically rounded to BF16 to cut traffic, while final local summation stays in FP32 for robustness. For mHC, fused kernels, selective recomputation of cheaper intermediates, and DualPipe adjustments keep the overhead small. For long-context attention under context parallelism, sequence packing and compression create awkward boundaries. Their two-stage fix first sends trailing uncompressed entries to the next rank so compression blocks that straddle ranks can finish, then all-gathers compressed entries and pads them into a regular layout. Sparse top-k visibility is handled by explicit indices, while HCA and the CSA indexer can use deterministic visibility rules. They also extend automatic differentiation so developers can mark individual tensors for checkpointing rather than whole modules. The system traces the graph, builds minimal recomputation subgraphs, and reuses storage pointers without extra copies.
Inference Key-Value cache management
Serving hybrid attention is harder than serving ordinary Transformers because different layers and branches have different cache shapes and lifetimes. There are compressed CSA entries, compressed HCA entries, indexer keys, sliding-window uncompressed entries, and temporary uncompressed tail states waiting until a full compression block is ready. PagedAttention-style systems usually assume more uniform cache policies. DeepSeek-V4 therefore uses a heterogeneous layout with two main parts. A state cache holds per-request fixed-size state for sliding-window Key-Value entries and for uncompressed tails not yet ready to compress. A classical Key-Value cache holds the compressed CSA and HCA blocks. Cache blocks are sized around the least common multiple of the two compression rates so both CSA and HCA layers can share a sensible block geometry. Sparse attention kernels are co-designed with this layout, including padding for cache-line friendliness. For shared-prefix reuse, compressed CSA and HCA entries can be stored on disk and reloaded, while incomplete tail blocks still need recomputation. Sliding-window cache is bulkier, so they offer three policies. Store all of it for zero recompute. Checkpoint it periodically and recompute tails. Store none of it and recompute enough recent layers from compressed caches. Different deployments can choose different storage-versus-compute tradeoffs.
Pre-training data and schedule
The models are trained on more than 32 trillion tokens for Flash and about 33 trillion for Pro. The data continues from the DeepSeek-V3 corpus but is cleaned and expanded. Web data is filtered harder to remove auto-generated and templated sludge that can cause model collapse. Math and code remain central, and agentic data is added during mid-training to strengthen tool-using behavior. Multilingual data is expanded for long-tail cultural knowledge. Long documents such as scientific papers and technical reports are emphasized because the model is meant to handle long contexts. Training starts at 4K sequence length and gradually grows to 16K, 64K, and finally 1M. Sparse attention is not used from the first step. The model first warms up with denser attention, then introduces sparsity after enough tokens, with a short indexer warmup stage. Batch size ramps up to tens of millions of tokens. Learning rate warms up, holds, then cosine-decays. Multi-token prediction loss weight starts higher and later drops. Flash and Pro share the same qualitative recipe, with Pro using more layers, larger hidden size, more experts, larger top-k, and a slightly different early-layer attention choice.
Concrete model sizes and layer choices
DeepSeek-V4-Flash has 43 Transformer layers and hidden size 4096. The first two layers are pure sliding-window attention. Later layers interleave CSA and HCA. CSA compression rate is 4, HCA compression rate is 128, sliding window is 128, query heads are 64, main query heads are 64, and attention top-k is 512. Each MoE layer has 1 shared expert and 256 routed experts, of which 6 are active, with expert intermediate size 2048. mHC expansion is 4. That yields 284B total and 13B active parameters. DeepSeek-V4-Pro has 61 layers and hidden size 7168. The first two layers use HCA rather than pure sliding window, then CSA and HCA interleave. CSA top-k rises to 1024, query heads are 64, main query heads are 128, query dimension remains 128, head dimension is 512, query compression dimension is 1536, and output projection uses 16 groups. MoE uses 1 shared expert and 384 routed experts with intermediate size 3072, still activating 6 routed experts per token. Total parameters are 1.6T with 49B active. These numbers matter because they show the efficiency claim is not only about clever attention. The Flash model especially proves that a smaller active footprint can still beat a previous larger base model when architecture and data improve.
How they stabilized training of a trillion-scale MoE
Training was not smooth. Loss spikes kept appearing. Simple rollbacks helped temporarily but did not solve the root pattern. Empirically, spikes linked to outliers in MoE layers, and routing seemed to amplify them. Two practical remedies helped. The first is Anticipatory Routing. At a training step, feature computation uses current parameters, but routing decisions use older parameters from some steps earlier. That breaks a tight feedback loop in which bad routing and bad activations reinforce each other. To avoid loading parameters twice, they precompute and cache routing indices ahead of time. Infrastructure work keeps the extra wall-clock cost around 20 percent when the mode is on, and an automatic detector turns it on only around spikes, then later turns it off. The second remedy is SwiGLU clamping. The linear part of SwiGLU is clamped to about negative ten to ten, and the gate upper bound is capped at ten. That kills extreme activations without hurting final quality, according to the authors. They admit the deeper theory is still incomplete, but the recipes worked for both Flash and Pro.
What the base models already show after pre-training
Even before post-training, the base models already look strong. DeepSeek-V4-Flash-Base, despite fewer active and total parameters than DeepSeek-V3.2-Base, beats it on many world-knowledge and long-context evaluations. That is an important efficiency story. Better architecture and data can outweigh raw parameter count. DeepSeek-V4-Pro-Base then jumps further and becomes the strongest DeepSeek foundation model across knowledge, reasoning, coding, and long context in their internal comparisons. LongBench-V2 scores rise clearly from V3.2-Base to Flash-Base to Pro-Base. Knowledge tasks such as Simple-QA verified and FACTS Parametric improve dramatically for Pro-Base. Coding and math are more mixed for Flash versus V3.2, but Pro generally leads. The pre-training recipe already delivers both capability and long-context competence before any instruction or agent finetuning.
Table 1 compares DeepSeek-V3.2-Base, V4-Flash-Base, and V4-Pro-Base across knowledge, reasoning, coding, and long-context tasks. V4-Flash-Base beats V3.2-Base despite fewer active parameters, and V4-Pro-Base leads across the board.
Post-training philosophy: specialists first, then on-policy distillation
After pre-training, DeepSeek-V4 does not simply mix all skills in one big reinforcement learning soup. It first grows specialists. For each domain such as math, coding, agents, and instruction following, a separate expert model is trained with supervised fine-tuning and then reinforcement learning using Group Relative Policy Optimization. Reward signals are domain-specific. For hard-to-verify tasks, they avoid classic scalar reward models trained from huge human preference sets. Instead they use a Generative Reward Model that judges trajectories with rubrics and can itself be improved with reinforcement learning. The same actor can generate and judge, which fuses reasoning ability into evaluation and reduces the need for massive annotation. After many specialists exist, a single student model is trained with multi-teacher On-Policy Distillation. The student generates its own trajectories, then learns by reverse KL matching against the relevant teachers on those on-policy samples. Full-vocabulary logits are used rather than cheap token-level KL estimates, because full distributions give more stable gradients. More than ten teachers can be involved. Engineering makes this possible by offloading teacher weights, caching last-layer teacher hidden states instead of materializing huge vocab logits for every teacher, loading one teacher head at a time, and computing exact KL with a specialized kernel.
Reasoning modes and how thinking is managed
Both Flash and Pro support three reasoning effort modes. Non-think is fast and intuitive, meant for simple daily tasks. Think High is conscious step-by-step reasoning for harder problems. Think Max pushes reasoning as far as practical, with longer context windows, weaker length penalties during RL, and a special system prompt that demands thorough deliberation, edge-case stress testing, and explicit writing of intermediate hypotheses. Thinking is wrapped in think tags. Tool calling uses a new XML-like schema with a special DSML token, which reduced escaping failures and tool-format errors. Interleaved thinking is refined for agents. In tool-calling workflows, reasoning traces are preserved across the whole conversation, including across user turns, so the model does not forget its plan every time a human message arrives. In ordinary chat, older thinking is still discarded on new user turns to keep context clean. Because the context window is huge, preserving long thinking traces for agents becomes much more viable than before.
Quick Instruction as a latency trick
Chatbots often need helper decisions before answering. Should we search the web? What title should this chat get? What queries should we issue? How authoritative must sources be? What domain is this? Should this URL be fetched? Usually a separate small model does those jobs, which means redundant prefilling and extra engineering. Quick Instruction instead appends special tokens to the same input and reuses the already-built Key-Value cache of the main model. Several helper tasks can even run in parallel. That cuts time to first token and removes the need to maintain a separate helper model.
Post-training infrastructure for million-token RL and reliable rollouts
Post-training for million-token agents is brutal if the stack is fragile. They apply FP4 quantization-aware training to MoE expert weights and to the CSA indexer query-key path so the model adapts to low precision before deployment. Index scores can also be reduced from FP32 to BF16 with almost no recall loss, speeding top-k selection. During RL sampling they use real FP4 weights, not only simulated quantization, so sampled behavior matches production. The rollout service is preemptible and fault-tolerant. Every newly generated token is written to a token-level write-ahead log. On preemption, unfinished Key-Value cache is saved and later resumed. On hard failure, the log can rebuild the cache by re-prefilling saved tokens. Regenerating unfinished requests from scratch would bias the model toward shorter answers, because short completions are more likely to finish before interruption. The write-ahead design avoids that correctness bug. For million-token training data movement, they separate lightweight metadata from heavy per-token fields, shuffle using metadata, and stream heavy fields through shared memory, releasing them mini-batch by mini-batch.
Sandbox infrastructure for agent training
Agent training needs safe places to run code, shells, browsers, and software projects. DeepSeek built DeepSeek Elastic Compute, a large sandbox platform that can manage hundreds of thousands of concurrent sandboxes. One Python interface covers four execution backends: fast prewarmed function calls, Docker-compatible containers, Firecracker microVMs for denser isolation, and full QEMU VMs for arbitrary guest systems. Images load quickly through layered on-demand storage on a distributed filesystem. Memory and lock optimizations raise packing density. Every sandbox keeps an ordered trajectory log of commands and results so preempted training jobs can fast-forward through already completed steps, preserve provenance, and deterministically replay sessions. This is the unglamorous backbone that makes serious coding-agent and tool-agent reinforcement learning possible at scale.
How strong the final models are on public-style evaluations
DeepSeek-V4-Pro-Max becomes the headline open model in the paper. On broad world knowledge, especially SimpleQA-style factuality, it strongly outperforms prior open models and closes much of the gap to the best proprietary systems, though Gemini-class models still lead on some knowledge suites. On educational and hard reasoning sets such as MMLU-Pro, GPQA, and Humanity’s Last Exam, it is competitive with or better than strong open rivals and close to, but not fully above, the newest closed frontier. On coding competitions and math contests it is especially impressive. The paper reports Codeforces ratings that put DeepSeek-V4-Pro-Max among the best systems they evaluated, even ranking high against human contestants in their setup, and says this is the first time an open model matched closed models on that style of coding evaluation. Formal math with Lean tools also looks very strong, especially when informal reasoning is used to explore and formal verification is used to certify. Agent benchmarks such as software engineering repair, terminal tasks, browsing, and multi-tool suites show DeepSeek-V4-Pro roughly on par with leading open models and still a bit behind or mixed against the best closed models, depending on the task. The Flash model is weaker on pure knowledge, as expected from fewer parameters, but with a large thinking budget it can approach Pro on many reasoning tasks and remains highly cost-effective.
Table 6 compares DeepSeek-V4-Pro-Max against frontier closed and open models across knowledge, reasoning, coding, math, and agent tasks. Pro-Max is competitive with the best closed models on reasoning and coding, and leads on several math and coding benchmarks.
Long-context evaluation and the meaning of one million tokens
Because the whole point of the architecture is long context, the paper evaluates retrieval and long-document tasks at large windows. On MRCR-style multi-needle retrieval, Pro-Max stays strong out to very large inputs. Performance is especially stable up to around 128K and remains competitive even at 1M tokens, though some degradation appears after 128K. On CorpusQA-like more realistic long-document question answering, Pro also beats Gemini-3.1-Pro in their re-evaluation setup while still trailing Claude Opus 4.6 on some retrieval metrics. The important qualitative point is not that the model is perfect at one million tokens. The important point is that quality remains usable while cost and memory stay far below older attention designs. That is what turns million-token context from a marketing number into something you can actually serve.
Figure 9 shows DeepSeek-V4 series retrieval performance on MRCR. Performance is highly stable within 128K and remains competitive even at 1M tokens. Pro-Max beats Gemini-3.1-Pro at most lengths but trails Claude Opus 4.6 on some retrieval metrics.
Reasoning effort really changes the model
Table-style comparisons in the paper show that Non-think, High, and Max are not cosmetic labels. Hard benchmarks jump sharply when more thinking budget is allowed. Easy knowledge can improve too, but the biggest gains appear on difficult reasoning, contest math, contest coding, and long agent trajectories. Compared with DeepSeek-V3.2, the V4 series gets more out of extra test-time tokens on tasks like Humanity’s Last Exam and terminal-bench style agent work. The architecture is cheaper at long context and a better way to do test-time scaling, because longer thoughts and longer tool traces no longer punish the system as severely.
Figure 10 shows how DeepSeek-V4-Pro and V4-Flash scale with reasoning effort on Humanity’s Last Exam and Terminal Bench 2.0. The Max mode substantially outperforms Non-think and High modes, and V4 series show larger gains from test-time compute than DeepSeek-V3.2.
Real-world product evaluations beyond leaderboard scores
The authors argue that public benchmarks miss much of real user experience, so they also report internal product-oriented studies. For Chinese functional and creative writing, DeepSeek-V4-Pro beats Gemini-3.1-Pro overall in their pairwise tests, especially on writing quality and on actually following user constraints rather than imposing a model house style. On the hardest multi-turn or high-constraint writing prompts, Claude Opus can still win. For search, the non-think mode uses retrieval-augmented generation while thinking mode uses agentic multi-step search. V4-Pro improves substantially over V3.2 on search QA, especially precise fact lookup and planning, though comparison and recommendation tasks improve less. Agentic search beats plain RAG on complex questions while staying only slightly more expensive. For Chinese white-collar professional tasks across many industries, human judges preferred DeepSeek-V4-Pro-Max over Opus-4.6-Max more often than not, with strengths in task completion, depth, professional tone, and long-form coherence. Weak spots include strict instruction following on formatting details, concise summarization of huge inputs, and aesthetic slide-like formatting. For internal real R&D coding agents across PyTorch, CUDA, Rust, and C++ tasks, V4-Pro substantially beats Claude Sonnet 4.5 and approaches Opus 4.5. In a survey of DeepSeek’s own developers, most said V4-Pro is ready or nearly ready to be their default coding model, while still noting occasional trivial mistakes, weak handling of vague prompts, and overthinking.
Limitations and what the authors themselves want to fix next
The authors are candid that the architecture is bold and somewhat complicated. To reduce risk they kept many previously validated tricks, which made the system less elegant. Future versions aim to distill the design down to fewer essential parts without losing quality. Anticipatory Routing and SwiGLU clamping work, but their theory is incomplete, so they want better foundations and better internal metrics for predicting instability before it happens. They also want more kinds of sparsity beyond MoE and sparse attention, including sparser embeddings, plus lower-latency system techniques so long-context interaction feels snappy rather than merely possible. Long-horizon multi-round agents remain a major target. Multimodal capabilities are planned. Better data curation and synthesis are treated as continuing work, not a finished story. V4 is framed as a preview that proves million-token open models can be efficient and strong, not as the final clean architecture.
The conceptual heart of the paper
If you strip away all the product names and benchmark tables, the paper is really arguing for a new default assumption in language-model design. The old assumption was that context length is a scarce resource, so models should stay short-context by default and treat long context as a special expensive mode. The new assumption is that long context should be cheap enough to be ordinary. Once that is true, several other research directions unlock at once. Test-time scaling can use longer thoughts. Agents can keep longer memories, traces, and tool histories. Multi-document scientific and enterprise workflows become first-class. Even future ideas like online learning become more plausible because the model can condition on much larger personal or session state. DeepSeek-V4 tries to make that assumption real through compressed hybrid attention, safer wide residual connections, a matrix-aware optimizer, aggressive systems engineering, and a post-training pipeline that first builds specialists and then distills them into one model that can think at multiple budgets.
How the pieces fit together as one system
The design works as stacked layers of the same efficiency idea. At the representation level, MoE activates only a few experts per token so total parameters can be huge while compute per token stays moderate. At the depth-connection level, mHC widens the residual highway without letting it become unstable. At the sequence level, CSA and HCA compress and sparsify memory so attention no longer owns the whole cost curve. At the optimizer level, Muon updates large matrices more effectively. At the cluster level, fused expert kernels hide communication, TileLang fuses operators, and deterministic batch-invariant kernels keep experiments trustworthy. At the serving level, heterogeneous Key-Value management and on-disk prefix caches make long shared contexts reusable. At the post-training level, FP4-aware training, full-vocabulary on-policy distillation, fault-tolerant rollouts, and massive sandboxes turn the base model into a practical reasoner and agent. No single idea is the whole story. The paper’s strength is that the architecture, the training recipe, and the infrastructure all point at the same target, long-context intelligence that is actually affordable.
What remains uncertain or not fully settled
A careful reader should also notice open questions. Compression always risks losing rare but critical local details, and sliding windows and hybrid layers mitigate that, but they do not guarantee perfect recall of every fact in a million-token haystack. Sparse top-k selection can miss relevant blocks if the indexer is wrong. The architecture is complex, so reproducibility and community adoption may be harder than for cleaner designs. Some of the strongest claims depend on internal evaluation harnesses, internal contest reconstructions, and internal product pairwise tests, which are informative but not as easy for outsiders to audit as fully public leaderboards. The gap to the best proprietary models is narrowed but not erased, especially on some knowledge and agent suites. And while million-token support is native, quality is not flat across the whole window. There is still degradation at the extreme end. None of that undoes the main achievement, but it keeps the result in the category of major progress rather than final victory.
Closing synthesis before the chase questions
DeepSeek-V4 is best understood as an efficiency-first frontier open model family. It preserves the MoE and multi-token prediction heritage of DeepSeek-V3, then redesigns residual connections, attention, optimization, and infrastructure so that one-million-token contexts become routine rather than heroic. Pro is the high-capacity model. Flash is the cost-efficient sibling. Max is the deepest reasoning mode. Across knowledge, contest reasoning, formal math, software agents, Chinese writing, search, and professional workflows, the paper presents evidence that open models can sit close to closed frontier systems while being dramatically cheaper to run at long context. The lasting contribution is a concrete demonstration that the long-context bottleneck can be moved by co-designing algorithms and systems instead of waiting for hardware to make quadratic attention affordable.
Chase questions
What was the research trying to make possible?
The research was trying to make highly capable open language models that can use one-million-token contexts efficiently enough for real training, real serving, and real long-horizon agents. It was trying to remove the practical barrier that stopped test-time scaling and multi-document or multi-step agent workflows from going much longer, not just raise benchmark scores. By combining compressed sparse attention, heavily compressed attention, better residual connections, Muon optimization, and heavy systems work, the authors wanted million-token intelligence to become ordinary infrastructure rather than a rare expensive demo. They also wanted a post-training path that could grow domain specialists and then merge them into one model with controllable reasoning effort, so the same base architecture could serve both cheap fast answers and deep expensive reasoning.
What becomes obvious after reading it that was not obvious before?
After reading the paper carefully, it becomes obvious that long-context progress is no longer mainly a data or prompt trick. It is an end-to-end co-design problem. Attention compression alone is not enough. You also need residual pathways that remain stable when the network is deep and expressive, optimizers that train giant matrix parameters cleanly, kernels that hide expert communication, cache layouts that understand hybrid memory types, disk strategies for shared prefixes, deterministic numerics so long RL runs stay debuggable, and post-training machinery that can handle million-token rollouts without collapsing under preemption and memory pressure. It also becomes obvious that model size and active compute are different stories. Flash can beat a larger older base model because architecture and data quality matter as much as parameter count. Another non-obvious point is that specialist-then-distill post-training can replace mixed reinforcement learning as the main unification step, and that preserving full teacher vocab distributions, not just token-level shortcuts, is treated as important for stable distillation. It also becomes clear that “supports 1M context” only becomes meaningful when both quality and cost remain acceptable. The paper’s real claim is that efficiency improvements are large enough to change what people will actually deploy.
What long-running problem did this paper move, even slightly?
The long-running problem is the conflict between the rising value of long context and the brutal cost of vanilla attention. For years, the field has known that agents, retrieval over large corpora, multi-file software work, extended chain-of-thought, and memory-rich assistants all want longer windows, while quadratic attention and growing Key-Value caches punish every extra token. Many papers chipped at pieces of this problem with sparsity, compression, better serving kernels, or longer-context finetuning. DeepSeek-V4 moves the problem by showing a full-stack open-model recipe in which hybrid compressed attention, model architecture, optimizer, training schedule, inference cache design, and post-training systems all point at the same goal and deliver order-of-magnitude efficiency gains at million-token scale while remaining competitive on hard general intelligence benchmarks. It does not abolish the long-context problem. Compression, sparsity, and retrieval over long windows still leave quality and complexity challenges. But it shifts the frontier from “can we support long context at all” toward “how cleanly, cheaply, and reliably can long-context intelligence become the default.” That is real movement on a problem the field has been stuck on for a long time.