Trendings on Speculative Decoding

Speculative decoding has quietly become the default way to make large language models fast. The premise is simple: a small, cheap drafter guesses several tokens ahead, and the big target model verifies them in a single parallel forward pass — accepting the guesses it agrees with and correcting the first one it doesn't. Because rejected drafts are thrown away, the output stays bit-for-bit identical to plain autoregressive decoding. It's a pure efficiency win, not a quality trade.

In my own writing on the LLM space — from my LLM Year in Review to my MLSys 2026 recap — I've watched this technique go from a clever trick to a production necessity. What's striking about 2026 is how the whole field converged on the same answer in the span of two months — and from three very different angles: DeepSeek's server-scale DSpark, Google's open Gemma 4 drafters, and Google's on-device Gemini Nano frozen Multi-Token Prediction. This post walks that timeline, summarizes each, lays them side by side — and keeps a running ledger of where they land in production.

The 2026 Wave, at a Glance

Two timelines, one story — both running newest to earliest. First the speculator releases: three launches, eight weeks, one idea — from the open-weights drafters that put speculative decoding in everyone's hands, to the edge, to hyperscale serving. Then the speculator trainers: the open-source training libraries that made those drafters reproducible, a lineage spanning exactly one year, July 2025 to July 2026 (each is covered in detail below).

Speculator Releases — three launches, eight weeks · newest first

The 2026 speculative-decoding wave, newest first — three launches and the production adoption that followed.

Speculator Trainers — exactly one year, Jul 2025 – Jul 2026 · newest first

A year of training-library releases, newest first — the tooling that turns speculative decoding from a paper into a pipeline.

Three names show up everywhere in these announcements. Here's all three explained side by side — plain English first, then the technical details — using the linked sources:

So the clean mental model: EAGLE and MTP are both autoregressive drafters (they differ in where the drafter lives — an extra feature-conditioned layer vs. extra heads), while DFlash is the odd one out: it's parallel, generating the whole block in one shot via diffusion. That single difference — serial vs. one-pass drafting — is the axis the rest of this post turns on.

Bonus term — MTP-1: just "MTP with a single extra head," i.e. one predicted-token-ahead. DeepSeek-V4 ships with MTP-1 built in (2 tokens per forward pass); DSpark then stacks its semi-parallel, confidence-scheduled verification on top of that baseline.

Autoregressive Decoding vs. Parallel Drafting — the axis every method lives on

Underneath all three releases is a single tension. A drafter can produce its guesses autoregressively (one token at a time, each guess conditioned on the last) or by parallel drafting (predicting a whole block of tokens in one shot). They fail in opposite ways — and the 2026 systems are essentially different bets on how to combine them.

This is exactly the gap P-EAGLE ("Parallel-Drafting EAGLE") was built to close: it transforms EAGLE from autoregressive into parallel multi-token prediction via a learnable shared hidden state, removing the serial-pass bottleneck while keeping EAGLE's feature-grounded accuracy (arXiv:2602.01469; see also the vLLM and Amazon SageMaker AI write-ups).

And it's why DSpark's author, Dmytro Dzhulgakov — co-founder & CTO of Fireworks AI (and a longtime PyTorch core maintainer) — frames DSpark as the fusion of both styles:

"DSpark ≈ EAGLE + MTP. DFlash is stronger at initial positions; EAGLE-3 is more coherent on long drafting but has lower-quality guesses in the beginning. DSpark combines both parallel-block and autoregressive ideas, beating either approach." — @dzhulgakov

DSpark is not a new model — it's an inference-time optimization layered on top of existing DeepSeek-V4 checkpoints. As Dmytro Dzhulgakov put it in his thread, it "ingeniously integrates many speculative decoding ideas to achieve 1.5× to 5× higher throughput in a real production system." The key word is integrates: DSpark is less a single trick than a careful assembly of the field's best ideas, tuned for serving traffic.

V4 already ships with one built-in MTP-1 head, so the base model emits two tokens per forward pass before DSpark even enters the picture. DSpark builds on that with what DeepSeek calls a "semi-parallel" method: it speculatively generates multiple candidate tokens, then applies adaptive / confidence-scheduled verification — only spending verifier compute on the promising guesses instead of rigidly checking one token at a time. The reported effect is the elimination of "computational waste from invalid checks."

• Per-user speed: ~60–85% faster on DeepSeek-V4 Flash; ~57–78% on the Pro variant (secondary reporting).
• Throughput: 1.5×–5× higher depending on concurrency (the thread's headline; reporting cites 51%–400%).
• No retraining of the base model — it reuses deployed V4 weights, so the gain is essentially free engineering on top of MTP-1.
• Released as a full paper (33 authors from Peking University & DeepSeek-AI, led by Xin Cheng et al., with Wenfeng Liang as senior author) plus the open DeepSpec training repo and public checkpoints (deepseek-ai/dspark_qwen3_4b_block7, etc.).

The enthusiasm wasn't confined to DeepSeek's research org. Tianyi Cui (崔添翼) — the former Jane Street quant who joined DeepSeek in March 2026 to lead its Harness team building DeepSeek's agent stack — spotlighted the open-source DeepSpec repo and DSpark on Weibo. It's a small but telling signal of how broadly the company is leaning on speculative decoding — not just to serve V4 cheaply, but as table-stakes infrastructure for the latency-sensitive agent products the Harness team is racing to ship. And the adoption didn't stop at DeepSeek's own walls: a month later, optimized DSpark drafters turned up as the single largest slice of Intent Lab's autonomous 6.3× TensorRT-LLM speedup (see the adoption ledger below).

DSpark in Ten Ideas — Dmytro's thread, walked through

Dmytro Dzhulgakov (Fireworks AI) explained DSpark as a "10 ideas" thread that builds up from the very basics of LLM decoding to the full DSpark design. Here is that thread walked through step by step, each with its original diagram. Every card links back to the source tweet.

Gemma 4 is where this trend went open. Alongside the main lineup, Google announced a companion line of autoregressive drafter models plus an MTP head — officially branded the "Gemma 4 Assistant" and published as the -assistant checkpoints (e.g. google/gemma-4-E4B-it-assistant; the Transformers model type is gemma4_assistant, loaded as the assistant_model). Under the same Apache-2.0 license, it delivers up to 3× faster inference "with zero degradation in output quality or reasoning accuracy." Crucially, it shipped with day-one support across Transformers, MLX, vLLM, SGLang, and Ollama, so the speedup was usable immediately rather than a research artifact.

Per the official docs, Gemma 4's MTP is more than a bolt-on drafter. Three enhancements make the draft tokens cheaper and more accurate:

• Shared input embeddings. The draft model reuses the target's embedding table rather than carrying its own.
• Target activations. The drafter takes the target's last-layer activations, concatenates them with token embeddings, and down-projects to the drafter's dimension — so it isn't blind to the backbone's state.
• Efficient embedder. To avoid scoring the whole vocabulary, tokens are grouped into clusters; the head first picks likely clusters, then restricts final calculations to those (E2B/E4B variants).

One honest caveat from the docs: for the MoE model (Gemma 4 26B A4B), verifying drafted tokens can require loading additional expert weights, which can offset the gains at batch size 1 on hardware without good parallelism. Higher batch sizes recover the win as expert activations overlap. This Gemma 4 release is the direct ancestor of the Gemini Nano work that follows — Google's own blog notes the frozen-MTP effort builds on "accelerating Gemma 4 with MTP."

Gemini Nano Frozen MTP — June 26: the same idea, at the edge

Seven weeks after the Gemma 4 launch, Google Research took the same idea to the far end of the deployment spectrum: not a datacenter, but a phone. Mobile inference lives under hard RAM and energy budgets, and a standard speculative-decoding setup wants a separate drafter model (e.g. 128M params) that competes for memory and is "blind" to the main model's internal state. Their answer, detailed on the Google Research blog, is to retrofit Multi-Token Prediction onto a frozen Gemini Nano v3 — already deployed on Pixel 9 and 10.

The design rests on three moves:

• Late-exit MTP head. Instead of a standalone drafter, a lightweight Transformer head is appended to the final layers of the main model and predicts future tokens from the backbone's high-dimensional hidden states.
• Frozen backbone. The base model's weights are frozen; only the head is trained. This guarantees zero degradation in capability or safety alignment, and lets efficiency updates roll out with full backward compatibility — output stays bit-for-bit identical.
• Zero-copy architecture. The head cross-attends directly to the main model's frozen KV cache instead of maintaining its own. This eliminates drafter prefill latency and saves ~130 MB per instance versus a standalone drafter.

In production workloads like AI Notification Summaries and Proofread, MTP correctly predicts nearly two additional tokens per pass, yielding 50%+ speedups on Pixel 9 and up to 55% better token acceptance on highly structured tasks (e.g. smart replies) — while cutting energy use by waking heavy processors less often. The work explicitly builds on the EAGLE framework and CALM (Confident Adaptive Language Modeling), and points ahead to branching/parallel decoding and "verification leniency."

The throughline: all three keep output lossless, all three move away from standalone drafters toward heads that reuse the backbone's own state (embeddings, activations, or KV cache), and all three are converging on the same next frontier — branching drafts and relaxed verification. The difference is purely environmental: Gemma 4 optimizes for reach, Gemini Nano for memory and energy, and DSpark for throughput under load.

The real test of any inference technique is production traffic. The deployments below shipped with the speculators covered above — each entry records what ran, where it ran, and the numbers reported, newest first.

Redrawn from Intent Lab's "Breakdown of performance gains" chart — speculative decoding dwarfs every other category combined.

The 2026 releases stand on a well-defined chain of research. The foundational and directly-cited papers, grouped by theme:

1 · Foundations

2 · EAGLE family

3 · Parallel & diffusion drafting

4 · DeepSeek & DSpark

5 · MTP for speculative decoding

6 · Retrospective

Model checkpoints. The systems above are not just papers — they ship weights you can run: DSpark on DeepSeek-V4-Flash (deepseek-ai/DeepSeek-V4-Flash-DSpark), the open Gemma 4 Assistant MTP drafters (google/gemma-4-E4B-it-assistant), and the EAGLE/EAGLE-3 drafter checkpoints used across vLLM and SGLang. DSpark's thread author and explainer is Dmytro Dzhulgakov (@dzhulgakov), co-founder & CTO of Fireworks AI and a longtime PyTorch core maintainer.

Every system above ships a trained drafter — an EAGLE head, an MTP head, a DFlash block. But who trains them, and on what infrastructure? The quieter half of the 2026 story is the rise of dedicated open-source training libraries that turn drafting from a research artifact into a repeatable pipeline. They all wrestle with the same core obstacle — funneling the target model's hidden states into the draft model cheaply, at scale — and each attacks it from a different angle. DeepSpec, the repo behind DSpark above, and Tencent's brand-new AngelSpec are the latest entries in a line that runs back through 2025 — the Speculator Trainers rail at the top of this post traces the full one-year lineage, July 2025 to July 2026. Here is each library in detail:

The libraries, in detail

Two months, two labs, three releases, one converging idea — and beneath them, six open-source training stacks (from SGLang, vLLM/Red Hat, AWS, PyTorch×Mooncake, DeepSeek, and Tencent Hunyuan) turning those drafters into a reproducible pipeline. If 2024–2025 was about training bigger and cheaper models, 2026 is shaping up to be the year speculative decoding became the standard way to serve them — on a phone, in a datacenter, and everywhere in between.