Speculative Decoding: The Art of Guessing Well

Large Language Models spend most of their decoding time not computing, but waiting: every new token requires reading all of the model weights from memory, only to multiply them by a single vector. What if the model could check several tokens for the price of one?

In this post we’ll derive speculative decoding from first principles, build a general framework that covers every modern drafter, and walk through the architectures that became the industry standard, from EAGLE-3 and multi-token prediction to block-diffusion drafters like DFlash and DSpark. We’ll finish with the question of how to train a drafter, and why minimizing KL divergence is not quite what we want.

This post is a sequel to Transformers Inference Optimization Toolset, whose notation, memory bandwidth arithmetic and KV cache calculations we’ll reuse, so if “decoding is memory-bound” sounds unfamiliar, it’s worth skimming its first sections. We’ll focus on drafters attached to the target model: small networks that read the target’s hidden states and run next to it on the same GPU. A separate small LLM appears only as a baseline, drafting by string matching (n-grams, prompt lookup, suffix trees) goes to a footnote¹, and serving systems appear only where they shape how drafters are built and used.

Why decoding is slow

Let’s recall that text generation runs in two stages. Prefill ingests the whole prompt in one compute-bound pass. Decoding produces one token per forward pass, and the arithmetic intensity of each pass is far below the ops:byte ratio of the GPU: at batch size 1, every weight is read from HBM, used for one multiply-add with the hidden state of a single token and left alone until the next step.

A real world example: take an 8B model in bf16 on H100 GPU. Just reading its weights once takes

\[\underset{\text{weights, bf16}}{16 \text{ GB}} \ / \ \underset{\text{H100 HBM bandwidth}}{3.35 \text{ TB/s}} \approx 4.8 \text{ ms} \ \Rightarrow \ \le 208 \ \tfrac{\text{tokens}}{\text{s}} \ \text{ for an 8B model at batch size 1},\]

no matter how fast the tensor cores are, while the math for one token, $2 \cdot 8 \cdot 10^9 = 16$ GFLOPs (ignoring attention over the KV cache), takes about $0.016$ ms at H100’s $989$ TFLOPs of dense bf16 compute. And here’s the punchline: 8 tokens instead of one cost 8 times more math and still nearly the same 4.8 ms, because the weights are read once either way. The pass stays memory-bound until the number of tokens in it reaches the ops:byte ratio, $989 / 3.35 \approx 295$.

Time of one forward pass of an 8B model in bf16 on H100 versus the number of tokens $n$ scored in that pass: $\max \big( \frac{16 \text{ GB}}{3.35 \text{ TB/s}}, \frac{2 \cdot 8 \cdot 10^9 \cdot n}{989 \text{ TFLOPs}} \big)$, ignoring KV cache reads. Until $n \approx 295$ the pass is bound by reading the weights, so extra tokens are almost free.

Batching requests fills this gap, but it improves throughput, not latency: every user still gets one token per 4.8 ms. To make a single sequence faster, the extra tokens have to come from the same sequence, and we don’t know them yet. Unless we guess them.

Draft and verify

Suppose we have a cheap drafter $q$ that guesses the next tokens much faster than the target model $p$ we actually want to sample from. With greedy decoding, one round of speculative decoding looks like this:

1. The drafter proposes $\gamma$ tokens $x_{t+1}, \dots, x_{t+\gamma}$ that continue the verified context $x_{\le t}$.
2. The target runs once on the context extended with all $\gamma$ guesses. Thanks to causal attention, this pass gives its next-token prediction at each of the $\gamma + 1$ positions.
3. We keep the longest prefix of guesses on which the target’s argmax agrees with the drafter and append the target’s own token at the first disagreement. If all $\gamma$ guesses agree, the target’s prediction after the last one comes for free.

The output is exactly what greedy decoding would have produced, it just arrives faster: a round costs one target pass and yields between 1 and $\gamma + 1$ tokens. The verification pass has already written the KV cache entries of accepted tokens, and those of rejected tokens are dropped.

One greedy round with $\gamma = 4$. The drafter proposes tokens one by one (light blue), then the target scores all of them in a single pass (orange). The longest agreeing prefix is accepted (green), the rest is rejected (red), and the target’s own token at the first mismatch is appended: 3 new tokens for the price of one target pass.

Speculative sampling

With temperature sampling the target has a distribution $p$ rather than one correct answer, and the output must follow exactly that distribution. Speculative sampling (Leviathan et al., 2023; Chen et al., 2023) generalizes the matching rule:

\[\text{accept } x \sim q \ \text{ w.p. } \min\Big(1, \frac{p(x)}{q(x)}\Big), \qquad \text{otherwise sample from } \ \frac{(p - q)_+}{\sum_x (p(x) - q(x))_+},\]

where $(\cdot)_+ = \max(\cdot, 0)$. Tokens the drafter underestimates, $q(x) \le p(x)$, are always kept, tokens it overestimates are thinned out, and the residual tops up exactly the mass the drafter was missing. Averaging the acceptance probability over the drafter’s samples gives the acceptance rate

\[\alpha = \mathbb{E}_{x \sim q}\Big[\min\Big(1, \frac{p(x)}{q(x)}\Big)\Big] = \sum_x \min\big(p(x), q(x)\big),\]

the overlap of the two distributions. Its complement, the probability that a drafted token gets rejected, is the total variation distance, the largest difference in probability the two distributions can assign to any set of tokens:

\[1 - \alpha = \operatorname{TV}(p, q) = \frac{1}{2}\sum_x \vert p(x) - q(x) \vert.\]

Leviathan et al. denote it $D_{LK}$. The closer the drafter is to the target in TV, the more of its guesses survive. The residual’s normalizer is $\sum_x (p(x) - q(x))_+ = 1 - \alpha$, the probability of a rejection, so the output follows $p$:

\[\begin{aligned} \Pr(\text{output} = x) &= q(x)\min\Big(1, \frac{p(x)}{q(x)}\Big) + (1 - \alpha)\,\frac{(p(x) - q(x))_+}{1 - \alpha} \\ &= \min\big(p(x), q(x)\big) + \big(p(x) - q(x)\big)_+ \\ &= p(x). \end{aligned}\]

Whatever the drafter does, the output follows the target exactly, and a bad drafter can only make us slower. Greedy matching is the special case of one-hot $p$ and $q$. For a draft of $\gamma$ tokens the rule is applied left to right and stops at the first rejection. When all $\gamma$ tokens are accepted, treating the missing drafter distribution as zero turns the residual into the target’s own distribution for the bonus token:

@jit
def verify(key, p, q, draft):
    """p: [γ+1, V] target probs, q: [γ, V] drafter probs, draft: [γ] tokens."""
    gamma = draft.shape[0]
    key_u, key_r = jax.random.split(key)
    p_x = p[jnp.arange(gamma), draft]
    q_x = q[jnp.arange(gamma), draft]
    accepted = jax.random.uniform(key_u, (gamma,)) < jnp.minimum(1.0, p_x / q_x)
    n = jnp.argmin(jnp.append(accepted, False))                     # first rejection
    q_n = jnp.where(n < gamma, q[jnp.minimum(n, gamma - 1)], 0.0)   # zero after γ
    residual = jnp.maximum(p[n] - q_n, 0.0)
    token = jax.random.categorical(key_r, jnp.log(residual))
    return n, token  # keep draft[:n], then append token

How many tokens does a round produce? If every drafted token were accepted independently with probability $\alpha$, the number of accepted draft tokens would follow a geometric distribution truncated at $\gamma$, and with the bonus token a round would yield

\[\mathbb{E}[\#\text{tokens}] = \frac{1 - \alpha^{\gamma + 1}}{1 - \alpha}.\]

In practice the drafter gets worse the further it guesses. If $\alpha_k$ is the probability of accepting the $k$-th token given that all previous ones were accepted, the expected number of tokens per round, the acceptance length, is

\[\tau \approx 1 + \sum_{k=1}^{\gamma} \prod_{i=1}^{k} \alpha_i.\]

This is the number papers report most often, and the one every drafter in this post tries to raise.

Sampling with shared noise

Speculative sampling has one inconvenient property: with the same random seed, a different drafter makes different accept/reject decisions, so the text changes, even though its distribution doesn’t. For an alternative, let’s look at how sampling is implemented. Sampling from $\operatorname{softmax}(\ell)$ over a vocabulary of $V$ tokens, where $\ell$ are the logits, textbook-style means normalizing, computing a cumulative sum and inverting it with one uniform number. GPUs prefer the Gumbel-max trick, one elementwise pass and an argmax instead of a prefix sum over the vocabulary: add independent Gumbel noise to the logits and take the argmax,

\[x = \arg\max_i \big(\ell_i + g_i\big), \qquad g_i = -\log(-\log u_i), \quad u_i \sim U(0, 1).\]

Why does this work? $\xi_i = e^{-g_i} = -\log u_i$ is exponential with rate 1, and maximizing $\ell_i + g_i$ is the same as minimizing $\xi_i e^{-\ell_i}$, exponential with rate $e^{\ell_i}$. Among independent exponential clocks, clock $i$ rings first with probability equal to its rate over the sum of all rates:

\[\Pr(x = i) = \Pr\Big(\xi_i e^{-\ell_i} < \min_{j \neq i} \xi_j e^{-\ell_j}\Big) = \frac{e^{\ell_i}}{\sum_j e^{\ell_j}} = \operatorname{softmax}(\ell)_i.\]

vLLM’s sampler, for example, runs exactly this race, taking the argmax of $p_i / \xi_i$, because torch.multinomial would force the GPU to synchronize with the CPU. For us the trick matters for another reason: once the noise $g$ is fixed, sampling is a deterministic argmax, just like greedy decoding.

So let the drafter and the target share the noise. At draft position $k$ the drafter proposes $x_{t+k} = \arg\max_i \big(\log q_k(i) + g_{k,i}\big)$, the target computes its own $\arg\max_i \big(\log p_k(i) + g_{k,i}\big)$ in the verification pass, and verification becomes exact matching, as in the greedy case. This is Gumbel coupling (Daliri et al., 2024), and its output is exactly what the target would have sampled with that noise on its own. It’s drafter-invariant: for a fixed seed every drafter produces the same text, only faster or slower, which makes speculative decoding reproducible and much easier to test and debug.

Gumbel-max sampling for the token after “The cat sat on the”. In each row, the orange dot is the target’s score $\log p + g$ and the blue dot the drafter’s; the rightmost dots are the target’s sample and the draft, accepted when they coincide. With shared noise both dots of a row move together and the argmaxes mostly agree; with independent noise they rarely do. Click the titles to switch, and draw new noise to watch the match rate converge.

The price is a lower acceptance rate. Speculative sampling reaches the best possible agreement, $1 - \operatorname{TV}(p, q)$, because the verifier sees both distributions. Two models that share only the noise agree with probability

\[\Pr\big[\text{match}\big] = \sum_j \frac{1}{\sum_i \max\big(\frac{p(i)}{p(j)}, \frac{q(i)}{q(j)}\big)} \ \geq \ \frac{1 - \operatorname{TV}(p, q)}{1 + \operatorname{TV}(p, q)},\]

and no scheme without communication can guarantee more than the right-hand side in the worst case. With independent noise the match probability would drop to $\sum_i p(i)\, q(i)$. In the example above that’s 0.679 with shared noise and 0.181 with independent noise, against $\alpha = 0.70$ for speculative sampling, and on real LLM distributions the gap to $1 - \operatorname{TV}$ is often just as small. In JAX the whole draft-and-verify step is a few argmaxes over the same noise:

@jit
def gumbel_verify(key, log_p, log_q):
    """log_p: [γ+1, V] target log-probs, log_q: [γ, V] drafter log-probs along the draft.
    The same key, i.e. the same noise, is used by the drafter and the target."""
    g = jax.random.gumbel(key, log_p.shape)
    draft = jnp.argmax(log_q + g[:-1], axis=-1)              # what the drafter proposes
    target = jnp.argmax(log_p + g, axis=-1)                  # what the target samples
    n = jnp.argmin(jnp.append(draft == target[:-1], False))  # first mismatch
    return draft, n, target[n]  # keep draft[:n], then append target[n]

The verifier’s side

Before we get to drafters, let’s look at the verifier. It can accept more of the same chain, check several continuations at once, and it stops being free once the batch grows.

Block verification

Speculative sampling stops at the first rejection, which is a myopic decision. If the drafter overestimates $x_{t+1}$ but underestimates $x_{t+2}$, the pair as a whole may be as likely under the target as under the drafter, yet token-wise verification often throws both away. Block verification (Sun et al., 2024) decides on the whole draft jointly². With $p_k$ and $q_k$ the target’s and the drafter’s distributions at position $k$, the verifier keeps a running weight of how much of each prefix the target still covers,

\[w_k = \min\Big(1,\ w_{k-1}\frac{p_k(x_{t+k})}{q_k(x_{t+k})}\Big), \qquad w_0 = 1,\]

so a ratio below one on one token can be compensated by ratios above one later on (note that $\min(1, ab) \geq \min(1, a) \min(1, b)$). Each prefix survives with probability

\[\Pr\big(x_{t+1:t+k} \text{ survives}\big) = \frac{\sum_x \big(w_k\, p_{k+1}(x) - q_{k+1}(x)\big)_+}{\sum_x \big(w_k\, p_{k+1}(x) - q_{k+1}(x)\big)_+ + 1 - w_k},\]

the longest surviving prefix, of length $n$, is accepted, and the next token is sampled from the normalized residual $\big(w_n\, p_{n+1} - q_{n+1}\big)_+$. With $q_{\gamma+1} \equiv 0$ the bonus token falls out for free again, since then the whole draft survives with probability $w_\gamma$. Compared to verify, only the prefix length and the residual change:

@jit
def block_verify(key, p, q, draft):
    """Same inputs as verify(), but the prefix is accepted jointly."""
    gamma = draft.shape[0]
    key_u, key_r = jax.random.split(key)
    ratio = p[jnp.arange(gamma), draft] / q[jnp.arange(gamma), draft]

    def step(w, ratio_k):
        w = jnp.minimum(w * ratio_k, 1.0)
        return w, w

    _, w = jax.lax.scan(step, 1.0, ratio)
    w = jnp.concatenate([jnp.ones(1), w])                          # w_0, ..., w_γ
    mass = jnp.maximum(w[:-1, None] * p[:-1] - q, 0.0).sum(-1)     # residual mass
    mass = jnp.append(mass, w[-1])                                  # zero draft after γ
    survival = jnp.where(w < 1.0, mass / (mass + 1.0 - w), 1.0)
    survived = jax.random.uniform(key_u, (gamma,)) <= survival[1:]
    n = jnp.max(jnp.where(survived, jnp.arange(1, gamma + 1), 0))  # longest prefix
    q_n = jnp.where(n < gamma, q[jnp.minimum(n, gamma - 1)], 0.0)
    residual = jnp.maximum(w[n] * p[n] - q_n, 0.0)
    token = jax.random.categorical(key_r, jnp.log(residual))
    return n, token

Sun et al. prove that for a given draft block verification accepts the most tokens in expectation among all lossless rules, so it never loses to the token-wise rule. In a toy example with $p(a) = \frac{1}{3}$, $p(b) = \frac{2}{3}$, $q(a) = \frac{2}{3}$, $q(b) = \frac{1}{3}$ at every position and $\gamma = 2$, token-wise verification accepts $\frac{10}{9}$ tokens per round, block verification $\frac{11}{9}$. With PaLM-2 models the gain was about 8% more tokens per round and 5–8% in walltime, for a few extra vector operations and with any drafter.

Token trees

A chain dies at the first wrong token, but the target can verify several continuations in the same pass. We can draft a token tree whose branches share prefixes, flatten it into one sequence and give the target a tree attention mask, in which every node attends only to the context and to its own ancestors (Miao et al., 2023), with positional ids following the node depth. Verification walks down from the root, moving at every node to the child the target accepts, and returns the longest root-to-node path the target agrees with. The tree adds nodes to the verification pass, but no extra passes, and its expected acceptance length is

\[\tau(\mathcal{T}) = 1 + \sum_{v \in \mathcal{T}} \Pr\big(\text{path root} \to v \text{ accepted}\big),\]

the tree version of $\sum_k \prod_{i \le k} \alpha_i$. Given a budget of nodes, the best tree keeps the nodes with the highest path probabilities, and that’s what practical tree builders try to approximate.

A token tree of 7 drafted nodes continuing “…sat on the” (left) and its attention mask (right), where each row is a node and colored cells mark what it attends to. Numbers are the drafter’s path probabilities; if calibrated, their sum plus one is the expected acceptance length. The target accepts “mat”, then “,”, the drafter’s second choice at that depth, and appends its own “then”: 3 tokens per pass, where the chain of top choices “mat and purred” would have stopped after “mat”.

Building trees without breaking exactness

Multi-draft versions of speculative sampling keep trees lossless: SpecInfer’s multi-round rule tries the children one by one with the usual ratio, passing the target’s residual on to the next sibling after each rejection, and SpecTr (Sun et al., 2023) derives the optimal rule from optimal transport. Both assume that the children are sampled independently from $q$: the ratio $p(x) / q(x)$ corrects for how often the drafter proposes $x$, so it’s exact only if $q(x)$ really is that frequency.

Practical trees are rarely built this way. Trees grown along the drafter’s most confident branches, assembled best-first from the most likely paths or pruned by a confidence threshold all take the top candidates deterministically: the top candidate is proposed with probability one, a token outside the top-$k$ with probability zero, and plugging $q(x)$ into the rule biases the output. Other proposal schemes need their own rules, like sampling without replacement through the Gumbel-top-$k$ trick in Recursive Speculative Decoding (Jeon et al., 2024). For a deterministic tree each child’s honest proposal is a point mass, and the rule degenerates into token matching: sample the target’s own token at the node and accept it if it’s among the children. Popular implementations do exactly this, setting $q(x) = 1$ for every child and renormalizing the target’s distribution after each rejected sibling. A node is then accepted with probability $\sum_{x \in \text{children}} p(x)$: the drafter’s probabilities shape the tree, never the acceptance. The same goes for anything that reshapes the drafter’s distribution after the fact, such as top-$k$ truncation: the verifier must receive exactly the distribution the draft was sampled from. Traversal Verification (Weng et al., 2025) brings block verification to trees with sampled children, checking root-to-leaf sequences from the leaves up instead of discarding a subtree as soon as its root is rejected.

Trees and hybrid targets

The tree mask works because softmax attention treats its keys as a set, so every node simply picks out its ancestors. But frontier targets are no longer pure softmax attention: Kimi-K3 interleaves 69 layers of Kimi Delta Attention, a linear attention with a recurrent state, with 24 MLA layers (Kimi Team, 2026), Qwen3.5 uses Gated DeltaNet in most of its layers, and DeepSeek-V4 compresses the context into blocks. A recurrent layer updates one state token by token, so a tree needs a separate state per branch, which no mask can express, and with compressed attention branches diverging from a shared prefix compress into different blocks. STree (Wu et al., 2025) verifies trees in state space and hybrid models by scanning them with accumulated state transitions. For DeepSeek-V4-Flash, Liu and Li (2026) combine branch-aware causal verification, temporary per-branch state discarded after the pass, and a refresh of the compressed state along the accepted path. With 8 drafted nodes their trees accept about 20% more tokens per round than chains, but throughput grows by at most 18.5% and stops growing with larger budgets even as the acceptance length keeps rising.

Even with softmax attention only, trees are harder for the serving engine than chains:

• Attention kernels. The ancestor-only mask is neither causal nor a simple window, so it misses the fast paths of kernels such as FlashAttention-2, and tree verification often falls back to slower generic attention.
• KV cache bookkeeping. The verification pass writes keys and values for every node, but only the accepted path survives, so its entries have to be compacted or remapped in a paged cache. A rejected chain is simply truncated.
• Static shapes. Engines capture decoding steps as CUDA graphs and overlap CPU scheduling with GPU work, and both want the same shapes at every step. SGLang, for example, requires a chain (--speculative-eagle-topk 1) when its overlap scheduler is on, and that’s the default.

Verification bottlenecks

So far verification cost one decoding step. The first figure is the best case, though: a roofline where reading the weights and computing overlap perfectly, so every token below the ops:byte ratio is free. Real passes overlap the two only partly, and a pessimistic model simply adds them, $T(n) = t_{\text{mem}} + t_{\text{comp}}(n)$, so that 8 tokens cost 2.4% more than one and 295 tokens twice as much. With $B$ requests and $N$ drafted nodes per request (a chain has $N = \gamma$), the pass scores $B(N + 1)$ tokens, and the speedup over plain decoding at the same batch size is

\[\text{speedup}(N, B) = \frac{\tau(N) \cdot T(B)}{T\big(B(N + 1)\big) + T_{\text{draft}}},\]

where $T_{\text{draft}}$ is the time spent drafting. The numerator grows slowly: even with an optimal tree, the acceptance length grows only roughly logarithmically with the tree size, as observed in Sequoia (Chen et al., 2024). The denominator grows with every scored token, and steeply once $B(N+1)$ passes the knee, so the best tree shrinks as the batch grows:

Idealized speedup of tree speculation for an 8B model on H100 versus the number of drafted nodes per request, for four batch sizes. The tree is built best-first from a drafter whose top candidate matches the target with probability $\alpha$ and whose $j$-th candidate matches with probability $\alpha(1-\alpha)^j$; verification time adds weight reads and compute, $T(n) = t_{\text{mem}} + t_{\text{comp}}(n)$, and drafting costs $0.05$ of a decoding step. Dots mark the best tree for each batch size; move the slider to see them shift with $\alpha$.

With $\alpha = 0.8$, a single request is best served by a tree of about 80 nodes, eight requests by 16, 32 requests by 6 and 128 requests by just 3, while a 64-node tree there makes decoding almost three times slower than no speculation. Better drafters can afford bigger trees: from $\alpha = 0.6$ to $0.9$ the best tree grows from 65 to 82 nodes for one request and from 2 to 5 nodes for 128. That’s why Sequoia picks the tree size per hardware, 64 to 128 nodes on an A100 but 768 when the weights are offloaded and streamed to an L40, where the flat part of the curve becomes enormous. In serving, TurboSpec (Liu et al., 2024) adapts speculation to the load by tracking goodput, the rate of tokens that actually get accepted, since naive speculation can make a busy server slower. Together with their engineering costs, this is why trees pay off most where latency matters and the batch is small, while high-throughput serving tends to use chains with a draft length that adapts to the load.

Anatomy of a drafter

Drafters look very different on the surface, but every drafter in this post is one conditional distribution

\[q_\phi\big(x_{t+k} \mid x_{\le t},\; \mathbf{H},\; x_{t+1:t+k-1}\big), \qquad k = 1, \dots, \gamma,\]

where $x_{\le t}$ is the verified context, $\mathbf{H}$ is whatever the drafter reads from the target (from its last hidden state to several of its layers or even its KV cache), and $x_{t+1:t+k-1}$ are its own earlier guesses in this round. The most consequential decision is how a draft token depends on the guesses before it, since that decides how many sequential passes a round takes. There are three basic factorizations:

\[\begin{aligned} \text{independent:} \quad & q(x_{t+1:t+\gamma}) = \textstyle\prod_{k} q_k(x_{t+k} \mid x_{\le t}, \mathbf{H}) \\ \text{autoregressive:} \quad & q(x_{t+1:t+\gamma}) = \textstyle\prod_{k} q(x_{t+k} \mid x_{\le t}, \mathbf{H}, x_{t+1:t+k-1}) \\ \text{parallel + causal head:} \quad & q(x_{t+1:t+\gamma}) = \textstyle\prod_{k} \operatorname{head}\big(\mathbf{z}_k, x_{t+1:t+k-1}\big)(x_{t+k}) \end{aligned}\]

The independent line costs one drafter pass, but no position knows what the others guessed. The autoregressive line is how an LLM generates text: full dependency for $\gamma$ sequential passes. The last line splits the work: a parallel network computes states $\mathbf{z}_{1:\gamma} = f_\phi(x_{\le t}, \mathbf{H})$ for all positions in one pass, and only a tiny head runs token by token, looking at the previous guess or at all of them. The next sections follow these lines family by family.

Autoregressive drafters

The original speculative decoding papers drafted with a separate small LLM from the target’s family, for example a 4B model for a 70B Chinchilla in Chen et al. Any smaller model with the same tokenizer will do, but it shares nothing with the target: it recomputes representations the target has already built and keeps its own full KV cache. The drafters in this section stay autoregressive but read the target instead.

EAGLE

EAGLE (Li et al., 2024) drafts autoregressively at the level of hidden states, which it calls features. Each of the target’s last hidden states $\mathbf{h}_i$, right before the LM head, is concatenated with the embedding $e(x_{i+1})$ of the token actually sampled from it, an FC layer reduces the pair back to the hidden size, and a single transformer layer predicts the next hidden state:

\[\mathbf{a}_{t+1} = \operatorname{Layer}_\phi\Big(\operatorname{FC}\big[\mathbf{h}_i;\, e(x_{i+1})\big]_{i \le t}\Big).\]

The target’s LM head turns $\mathbf{a}_{t+1}$ into the drafter’s distribution $q := q(\cdot \mid \mathbf{a}_{t+1})$ over $x_{t+2}$, and the drafter learns both to match the target’s next hidden state and to predict that token:

\[\mathcal{L} = \operatorname{SmoothL}_1\big(\mathbf{a}_{t+1}, \mathbf{h}_{t+1}\big) + w_{\text{cls}} \cdot \mathcal{L}_{\operatorname{CE}}\big(q,\, x_{t+2}\big),\]

with $w_{\text{cls}} = 0.1$. The shifted tokens matter, since $\mathbf{h}_t$ alone doesn’t tell which token was sampled from it. At later draft steps the target’s hidden states don’t exist yet, so the drafter feeds back its own predictions $\mathbf{a}$. EAGLE-2 (Li et al., 2024b) kept the drafter and replaced the static token tree with a dynamic one, grown wherever the drafter is confident, since its confidence turned out to be well calibrated with acceptance.

EAGLE-3 and multi-token prediction

EAGLE-3 (Li et al., 2025) keeps EAGLE’s input, target features concatenated with the embedding of the next token, and changes two things. It fuses low-, mid- and high-level hidden states of the target, $\mathbf{H} = \mathbf{W}_{\text{fuse}}\,\big[\mathbf{h}^{(l_1)};\, \mathbf{h}^{(l_2)};\, \mathbf{h}^{(l_3)}\big]$, instead of the last one, and it drops the hidden-state regression, training on tokens only. Its hidden states $\mathbf{a}$ then no longer have to look like the target’s, so later draft steps must be part of training. The training-time test unrolls the drafter for several steps: $\mathbf{H}$ is available only for the verified context, and at every later position the drafter consumes its own hidden states from the previous step, exactly as at inference, while the tokens come from the training text:

\[\mathcal{L}_{\text{TTT}} = \sum_{k=1}^{\gamma} \mathcal{L}_{\operatorname{CE}}\Big(q_\phi\big(\cdot \mid x_{< t+k},\ \mathbf{H}_{\le t},\ \mathbf{a}_{t+1:t+k-1}\big),\; x_{t+k}\Big).\]

With these changes the drafter finally scaled with training data, and EAGLE-3 reported speedups of up to 6.5×.

The same structure was trained into targets from the start. Multi-token prediction (MTP) modules, popularized by DeepSeek-V3 (DeepSeek-AI, 2024) as an auxiliary pretraining objective that densifies the training signal, are $D$ extra layers applied in sequence. The $k$-th one normalizes the hidden state $\mathbf{a}^{(k-1)}_i$ from the previous depth, starting from $\mathbf{a}^{(0)}_i = \mathbf{h}_i$, concatenates it with the normalized embedding of the token $k$ steps ahead, and passes the pair through its own FC layer and transformer layer, just like an EAGLE drafter:

\[\mathbf{a}^{(k)}_t = \operatorname{Layer}_k\Big(\operatorname{FC}_k\big[\operatorname{RMSNorm}(\mathbf{a}^{(k-1)}_i);\, \operatorname{RMSNorm}(e(x_{i+k}))\big]_{i \le t}\Big).\]

The shared LM head turns $\mathbf{a}^{(k)}_t$ into $q_k := q(\cdot \mid \mathbf{a}^{(k)}_t)$ over token $x_{t+k+1}$, so the $k$-th module looks $k+1$ tokens ahead, and the cross-entropies of all depths are averaged and added to the usual next-token loss with a small weight:

\[\mathcal{L} = \mathcal{L}_{\operatorname{CE}}\big(p,\, x_{t+1}\big) + \frac{w_{\text{MTP}}}{D} \sum_{k=1}^{D} \mathcal{L}_{\operatorname{CE}}\big(q_k,\, x_{t+k+1}\big).\]

At inference the modules become a drafter that comes with the model for free: DeepSeek-V4 serves with MTP-1, a single module drafting one token per round, and Qwen3.5 has native MTP layers. Kimi-K3 shows how close the two families are: it pre-trains one MTP layer and fine-tunes it into an EAGLE-3-style drafter that fuses the outputs of its 1st, 4th and final AttnRes blocks (Kimi Team, 2026).

Both remain sequential: a round costs $\gamma$ drafter passes, and when verification isn’t free this overhead can eat the whole gain. In vLLM’s throughput tests on AMD GPUs, where plain decoding of Qwen3-8B already ran at 3,530 tokens/s, i.e. with a large batch, EAGLE-3 on MATH500 stayed slower than plain decoding for every draft length from 1 to 7. The 6.5× and this slowdown come from opposite ends of the batch size range.

Parallel drafters

How much does running the drafter once per token cost? Let’s measure time in target passes, with $c$ the cost of one drafter pass, and recall from the first figure that verifying a short draft costs about one decoding step. With $n_{\text{draft}}$ drafter passes per round, the speedup over plain decoding is

\[\text{speedup} = \frac{\tau}{n_{\text{draft}} \cdot c + 1}, \qquad n_{\text{draft}} = \begin{cases} \gamma & \text{autoregressive drafter} \\ 1 & \text{parallel drafter} \end{cases}\]

A real world example: a decent drafter accepts $\alpha = 0.8$ of tokens, drafts $\gamma = 5$ of them, and costs $c = 0.05$ of a target pass. Then

\[\tau = \frac{1 - 0.8^6}{0.2} \approx 3.69, \qquad \text{speedup} = \frac{3.69}{5 \cdot 0.05 + 1} \approx 2.95\times,\]

so a fifth of every round goes to drafting. A parallel drafter proposes all $\gamma$ tokens in a single pass and reaches $\frac{3.69}{0.05 + 1} \approx 3.51\times$, almost 20% faster at the same acceptance rate and draft length. The gap grows with $\gamma$: an autoregressive drafter has an optimal draft length beyond which extra guesses cost more than they bring, while for a parallel drafter longer drafts are nearly free, at least while verification stays memory-bound. The idea is as old as speculative decoding itself¹, but for a long time parallel guesses were too weak to compete.

Papers differ in benchmarks, block sizes, temperatures and checkpoints, so absolute acceptance lengths vary a lot between them: for one and the same drafter and target, published numbers range from about 4 to 7.8 tokens per round. From here on we compare each method only with baselines from its own paper, and only where the gain is substantial.

Medusa

Medusa (Cai et al., 2024) puts $\gamma$ cheap heads on the target’s last hidden state $\mathbf{h}_t$, the $k$-th guessing the token $k$ steps ahead:

\[q_k(\cdot \mid \mathbf{h}_t) = \operatorname{softmax}\Big(\mathbf{W}_2^{(k)}\big(\operatorname{SiLU}(\mathbf{W}_1^{(k)} \mathbf{h}_t) + \mathbf{h}_t\big)\Big), \quad k = 1, \dots, \gamma,\]

where $\mathbf{W}_2^{(k)}$ is initialized from the target’s LM head. The heads run on top of the target’s own forward pass, so drafting is almost free, and their candidates form a static token tree. The catch is the first line of our factorization: head $k$ never sees what heads $1, \dots, k-1$ guessed, and a single hidden state is little context to guess far ahead. If the first head hesitates between “New” and “Los”, the second can only spread its mass between “York” and “Angeles”, and $\alpha_k$ decays quickly with $k$³.

P-EAGLE

The most direct way to a strong parallel drafter is to take a good autoregressive one and stop feeding it its own guesses. P-EAGLE (Hui et al., 2026) does this to EAGLE-3, filling every position after the first with two learned vectors: a mask token embedding for the unknown token and a shared hidden state for the missing one. The harder part is training: each position of a sequence of length $L$ now carries $\gamma$ prediction depths, and attention over $L\gamma$ positions needs $\mathcal{O}\big((L\gamma)^2\big)$ memory, prohibitive for the long outputs of reasoning models. P-EAGLE precomputes the depth-aware attention mask once and slices it per example, and splits long sequences into segments with gradient accumulation inside a sequence. Its acceptance length stays on par with EAGLE-3, but with one drafter pass instead of five it is 1.10–1.36× faster than EAGLE-3 in vLLM on gpt-oss-120B, gpt-oss-20B and Qwen3-Coder-30B. PARD (An et al., 2025) applies the same trick to a separate small LM that fills several placeholder positions per pass, so that one drafter can serve a whole family of targets.

DFlash

Filling placeholder positions in one pass is what masked diffusion language models do, refining their guesses over several such steps at generation time, and early work drafted with whole discrete diffusion models (Christopher et al., 2024). DFlash (Chen et al., 2026) calls itself a block diffusion drafter, but the name needs care: it’s trained like a masked diffusion model, yet at inference one forward pass over the last token the target produced and a block of mask tokens predicts all masked positions, with no refinement. In effect it’s a masked block predictor.

The block is embedded with the target’s embedding table, which DFlash shares and keeps frozen together with the LM head, and its positions attend to each other bidirectionally. To predict well from masks, the drafter needs a strong grip on the context, so DFlash takes hidden states from five target layers, spread uniformly between the second and the third-to-last, fuses them into $\mathbf{H}$ as EAGLE-3 does, with an extra RMSNorm, and injects them as extra keys and values into every draft layer:

\[\mathbf{z}^{(j+1)} = \operatorname{Attn}\Big(\mathbf{Q} = \mathbf{W}_Q^{(j)} \mathbf{z}^{(j)},\; \mathbf{K} = \big[\mathbf{W}_K^{(j)} \mathbf{H}_{\le t};\, \mathbf{W}_K^{(j)} \mathbf{z}^{(j)}\big],\; \mathbf{V} = \big[\mathbf{W}_V^{(j)} \mathbf{H}_{\le t};\, \mathbf{W}_V^{(j)} \mathbf{z}^{(j)}\big]\Big),\]

where $\mathbf{z}^{(j)}$ are the hidden states of the block at layer $j$, starting from the embeddings. EAGLE-3 sees the fused states only at its input, so their influence fades with depth; in DFlash every layer attends to the target’s context directly, which lets the drafter be deeper (5 layers) and still cheap, because it runs once. Training samples random positions in target-generated text as anchors, the last verified token of an imagined round, masks the block after each anchor, and applies cross-entropy with weights decaying exponentially along the block.

The most telling numbers come from a controlled comparison by LMSYS: 5-layer EAGLE-3 and DFlash drafters for Qwen3-4B trained on the same data both reach $\tau = 4.2$ on GSM8K, yet the speedups are 2.1× and 3.3×. Only $n_{\text{draft}}$ moved. These drafters are weaker than the released DFlash checkpoints, but they were trained identically. Adoption followed quickly: the DFlash drafter for Qwen3.5-397B-A17B beats the model’s native MTP in throughput in every setting LMSYS benchmarked, by up to 1.5×.

DDTree

A parallel drafter gives us the full distribution at every position of the block at once. DDTree (Ringel and Romano, 2026) turns these per-position marginals into a token tree without any training. A path $\rho$ scores its log-probability under the drafter, $\log q(\rho) = \sum_i \log q_i(\rho_i)$, which simply adds up over independent marginals, so the classic best-first enumeration finds the top paths for a node budget. Each popped path pushes two successors, its best child one level deeper and its next-best sibling:

import heapq

def build_tree(log_q, budget):
    """log_q: [γ, V] per-position log-probs from the parallel drafter.
    Returns `budget` paths, each a tuple of per-depth candidate ranks."""
    log_q = np.asarray(log_q)
    top = np.argsort(-log_q, axis=-1)[:, :budget]   # candidate tokens per depth
    lq = np.take_along_axis(log_q, top, axis=-1)    # and their log-probs
    heap = [(-lq[0, 0], (0,))]                      # (-score, ranks along path)
    tree = []
    while heap and len(tree) < budget:
        neg, path = heapq.heappop(heap)
        tree.append(path)
        d = len(path)
        if d < lq.shape[0]:                         # child: best token one level deeper
            heapq.heappush(heap, (neg - lq[d, 0], path + (0,)))
        r = path[-1] + 1
        if r < lq.shape[1]:                         # sibling: next-best token, same depth
            heapq.heappush(heap, (neg + lq[d - 1, r - 1] - lq[d - 1, r], path[:-1] + (r,)))
    return tree

Scores only decrease along both kinds of edges, so paths leave the heap in order and every parent before its children. The tree on the verifier’s side was built exactly this way, and here is how it grows with the budget:

Best-first token tree built from the per-position marginals of a parallel drafter, as in DDTree. Right: the top candidates $q_k$ at each block position $k$ from one drafter pass, shaded if they made it into the tree. Left: the tree, with nodes numbered in the order they leave the heap. Move the slider to change the budget: the tree grows deeper along confident paths and wider where the drafter hesitates.

The tree is verified in one target pass with an ancestor-only mask, which FlashAttention-2 doesn’t support, so DDTree falls back to PyTorch’s generic scaled dot-product attention, and since the tree is deterministic, sampling is verified by token matching. On Qwen3-8B it lifts DFlash’s acceptance length by almost 40% and its speedup from 5.6× to 7.5×.

Restoring dependency in the block

DFlash has the same catch as Medusa, one level up. Every position of the block is a marginal, so nothing ties the token at position $t+k$ to the one before it, and when the target is unsure between two continuations, independent sampling happily mixes them:

A drafter unsure between “New York” and “Los Angeles” after “I flew from Paris to”. Left: the probability of each pair of first two tokens, green if coherent, red if mixed up. Right: four sampled drafts. With independent marginals half of them mix the cities, get rejected at the second token and lose the rest of the block (faded); a causal head conditions each token on the previous one and keeps every draft coherent. Click the titles to switch.

Marginals fail in a more mundane way too: nothing tells position $k$ that position $k-1$ already took a token, so neighbours often predict the same token twice. Chen et al. (2026) call it the repetition trap, and blocks that fall into it get shorter accepted prefixes.

How much does the missing dependency cost? Qiang et al. (2026) call it parallel blindness and split rejection in two. The information floor is the rejection no drafter can avoid with the information it has: seeing the context and only the last $m$ realized tokens before position $k$, the best it can do is

\[F_k^{(m)} = \mathbb{E}\Big[\min_{q}\ \mathbb{E}\big[\operatorname{TV}(p_k, q) \mid x_{\le t},\, x_{t+k-m:t+k-1}\big]\Big],\]

where the inner expectation averages over the unseen earlier tokens of the block and the best $q$ is chosen separately for every information state. A fully parallel drafter has $m = 0$, and with the whole prefix the floor vanishes. Everything above the floor is the drafter’s model gap. The floors are estimated by sampling hundreds of target continuations per anchor and solving the inner minimization exactly.

On Qwen3-4B the all-parallel floor grows from 0.078 at the second position of a 7-token block to 0.286 at the seventh, capping any parallel drafter, however large, at about 71% acceptance there. But a single realized predecessor removes 86–100% of the floor, and an independent mutual information estimate agrees. So a cheap head that sees the previous token is almost enough, and that’s what the drafters in this section add to the parallel backbone. Yet real drafters sit far above their floors: across four targets, 43–64% of what the released DFlash checkpoints reject at the last position is model gap, not parallel blindness.

DSpark

DSpark (Cheng et al., 2026) from DeepSeek puts a light causal head on a DFlash-style backbone. Its default Markov head adds a low-rank transition bias to the backbone’s logits $\boldsymbol{\ell}_k$, so that position $k$ depends on the token drafted just before it:

\[q_k(\cdot \mid x_{\le t+k-1}) = \operatorname{softmax}\big(\boldsymbol{\ell}_k + \mathbf{W}_1[x_{t+k-1}]\,\mathbf{W}_2\big),\]

where the lookup table $\mathbf{W}_1 \in \mathbb{R}^{V \times 256}$ and $\mathbf{W}_2 \in \mathbb{R}^{256 \times V}$ form a rank-256 transition matrix. The bias ignores the context: “York” gets the same boost after “New” whether the text is about a city or a newspaper, and everything else has to come from the backbone. The backbone runs once, tokens are then sampled left to right, each picking its row of the bias, and the block’s logits for verification are one extra matmul:

def markov_logits(z, prev_tok, W_o, W1, W2):
    """z: [γ, d] backbone states, prev_tok: [γ] previous draft tokens,
    W1: [V, 256], W2: [256, V] - low-rank token transition."""
    return z @ W_o + W1[prev_tok] @ W2

Here W_o is the target’s LM head. The sequential part is so light that scaling the draft from 4 to 16 tokens adds only 0.2–1.3% of latency. The alternative RNN head carries a gated recurrent state over the drafted tokens (the last line of our factorization) but gives only marginal gains. Here are the parallel architectures so far next to EAGLE-3:

Four drafters in one schema: the target on the left, what the drafter reads from it in the middle, the drafter on the right with inputs at the bottom and draft tokens at the top. Medusa puts independent heads on the last hidden state next to the target’s LM head. EAGLE-3 feeds a low, a middle and a high layer into one decoder layer and drafts a token per pass. DFlash injects five fused layers as keys and values into every draft layer and drafts a masked block in one pass. DSpark adds a Markov head that samples the block left to right. Click the titles to switch.

The second half of DSpark is about the verification budget. A confidence head $\pi_k = \sigma\big(\mathbf{w}^T[\mathbf{z}_k; \mathbf{W}_1[x_{t+k-1}]]\big)$ predicts the acceptance rate of position $k$ given that its prefix survived, trained towards $\pi_k^\ast = 1 - \operatorname{TV}(p_k, q_k)$. The product $\prod_{i \le k} \pi_{b,i}$ estimates the chance that the first $k$ tokens of request $b$ survive, and a scheduler picks the draft lengths $\gamma_b$ of the whole batch to maximize accepted tokens per second:

\[\lbrace \gamma_b \rbrace^{*} = \arg\max_{\lbrace \gamma_b \rbrace} \ \underbrace{\sum_b \Big(1 + \sum_{k \le \gamma_b} \prod_{i \le k} \pi_{b,i}\Big)}_{\text{expected tokens per round}} \cdot \ \underbrace{\operatorname{SPS}\Big(\sum_b (\gamma_b + 1)\Big)}_{\text{rounds per second}},\]

where $\operatorname{SPS}$ (steps per second) is the engine’s profiled number of rounds per second for a given number of tokens in the batch. Under light load every request drafts long; under heavy load verification becomes compute-bound and only confident prefixes get budget. The problem is solved greedily by sorting the survival probabilities of all candidate positions.

DSpark lifts DFlash’s acceptance length by 16–18%. In Qiang et al.’s terms its Markov head removes almost the whole floor: given the true previous token, nearly all of what DSpark still rejects is model gap. In live DeepSeek-V4 serving with $\gamma \le 5$, it made per-user generation 60–85% faster than MTP-1 at matched throughput, the first parallel drafter proven in frontier production.

Domino

Domino (Huang et al., 2026) gives the backbone a head with full-prefix memory, the last line of our factorization: a small GRU summarizes the embeddings of all tokens drafted so far into $\mathbf{s}_{k-1}$, and a rank-256 correction $\mathbf{W}_2\sigma(\mathbf{W}_1[\mathbf{z}_k; \mathbf{s}_{k-1}])$ is added to the logits $\boldsymbol{\ell}_k$. A head that sees the prefix tempts the optimizer to neglect the backbone, so Domino’s base-anchored curriculum trains the backbone’s own distribution $q_k^{\text{base}} := \operatorname{softmax}(\boldsymbol{\ell}_k)$ first and gradually shifts the loss towards the corrected $q_k$:

\[\mathcal{L} = (1 - \beta)\, \mathcal{L}_{\operatorname{CE}}\big(q_k,\, x_{t+k}\big) + \beta\, \mathcal{L}_{\operatorname{CE}}\big(q_k^{\text{base}},\, x_{t+k}\big),\]

with $\beta$ annealed linearly from 1 to 0. It raises DFlash’s acceptance length and speedup by about 18%.

DominoTree

DDTree assumes the distribution at each depth doesn’t depend on the path, which breaks with a causal head. DominoTree (Lin and Jang, 2026) is a training-free best-first tree that scores each node with Domino’s correction recomputed along its own path, restricted to the top-$M$ candidates per node. Applying the correction at all brings most of the gain, about 10% in acceptance length, and recomputing it per path adds another 5%. Like DDTree, it builds the tree deterministically and verifies by matching.

DFlash2

DFlash2 (Inco AI, 2026) selects among the candidates the backbone already proposes instead of rewriting each position’s full distribution. Its diagnosis, on Inco’s own five-layer DFlash drafter for Qwen3-4B on GSM8K at temperature 0: the correct token is among the top 16 candidates 99.5% of the time at the first position but only 87.8% at the seventh; the share of attention a position spends on the rest of the block falls from 30% in the first layer to 8% in the fifth; and an oracle picking the best path through the top-16 candidates would reach an acceptance length of 6.79 instead of 4.27. So DFlash2 changes two things, both easy to read in the vLLM implementation: how positions exchange information inside the backbone, and how the draft is read off its output.

One draft pass of DFlash and DFlash2 after “I flew from Paris to”, when the target continues with “Los Angeles last week” (norms and residual connections left out). Both read the fused target states as keys and values in every layer. DFlash decodes every position on its own and mixes the two cities. DFlash2 wraps every attention and MLP sublayer in two-tap convolutions (pink) and walks through the top candidates from the last verified token. Click the titles to switch.

Convolutions inside the backbone. Since the block barely uses attention to look at itself, DFlash2 adds a hard-wired channel to the left neighbour, a two-tap depthwise dynamic convolution

\[\operatorname{Conv}(\mathbf{z})_k = \big(\mathbf{b}_0 + \boldsymbol{\delta}_{k,0}\big) \odot \mathbf{z}_k + \big(\mathbf{b}_1 + \boldsymbol{\delta}_{k,1}\big) \odot \mathbf{z}_{k-1},\]

where $\mathbf{b}_0, \mathbf{b}_1$ are learned per-channel kernels and $\boldsymbol{\delta}_k$ is a linear projection of $\mathbf{z}_k$, one number per group of 16 channels. Each attention and MLP sublayer is wrapped in two of them, on its normalized input and on its output before the residual connection, with the coefficients of both predicted from the sublayer’s input. Twenty convolutions in a row carry a signal across the whole block. They run over the block only, with the tap zeroed at its boundary, so the first mask position reads the last verified token, while the target’s context still arrives as keys and values. For 3% more parameters they recover much of the gain of a 15-layer drafter at late positions, and the local exchange moves from attention to the convolutions.

A selector instead of an argmax. DFlash2 keeps the top $K = 16$ candidates $C_k$ at each position with their logits $\boldsymbol{\ell}_k[b]$ and scores every adjacent pair $a \in C_{k-1}$, $b \in C_k$:

\[S_k(a, b) = \boldsymbol{\ell}_k[b] + \big\langle \mathbf{A}[a] \odot \mathbf{P}\mathbf{z}_k,\ \mathbf{B}[b] \big\rangle,\]

where $\mathbf{A}$ and $\mathbf{B}$ are 256-dimensional token codebooks for the predecessor and the successor and $\mathbf{P}$ projects $\mathbf{z}_k$ to 256 dimensions. One batched product yields a $16 \times 16$ matrix per position, with no extra backbone or LM head pass. The draft is a walk through this lattice: starting from the last verified token, each position takes the row $S_k(a, \cdot)$ of the token chosen before and picks its argmax, or samples from its softmax at the sampling temperature. In vLLM the walk is one small Triton program per request, and lattice plus walk take 0.02 ms. The verifier receives exactly this softmax over the 16 candidates, zero elsewhere, as $q_k$, so the truncation is part of the proposal and the output stays exact.

Unlike DSpark’s head, the transition is gated by the context through $\mathbf{z}_k$, so “York” after “New” can be boosted for a city and not for a newspaper, and it’s evaluated on $16 \times 16$ pairs instead of the full vocabulary. In Inco’s ablation the selector has almost 40 times fewer parameters than DSpark’s correction and a sixteenth of its latency, yet accepts more tokens. The price is that no path can leave the top 16, one more reason for the convolutions to keep recall high.

Altogether it’s a parallel backbone with a head that sees only the previous token, order one in Qiang et al.’s terms, for about 1.3% of round latency. On Qwen3.5-4B it accepts 21% more tokens per round than DFlash and 9% more than DSpark.

Qiang et al. push the selector further. At order one there is almost no floor left, so what remains is model gap, and they argue that a context-blind transition added to the logits, as in DSpark, limits what such a head can express. Their prefix-attention head builds a query from the predecessor’s embedding and the drafter state, attends over the committed prefix and maps the result through a low-rank projection to predecessor-dependent logits, on top of DFlash2’s lattice and convolutions. Its floor is DSpark’s, but its model gap at the last position is a quarter lower, which gives a few percent more accepted tokens than the released DSpark checkpoint.

Looping and editing the block

A parallel drafter is cheap enough to run twice: with $c = 0.05$, two passes still cost a tenth of a target pass. So instead of adding a head, we can loop the drafter over its own block, the Jacobi-style refinement¹ done by a drafter rather than by the target. Such drafters don’t fit a single line of our factorization, since every extra pass conditions the whole block on the previous guess.

D-Loop (Chen et al., 2026), from the authors who named the repetition trap, runs the DFlash drafter twice: the first pass proposes a block, the second conditions on a selected prefix of it and regenerates the suffix in parallel. The block becomes two semi-autoregressive halves without a single new parameter, and one drafter learns both roles with a prefix–suffix objective. It beats DFlash and DSpark trained in the same setting, most of all when sampling: at temperature 1 it accepts 37% more tokens per round than DSpark on Qwen3-8B.

DEdit (Yu et al., 2026) lets the DFlash drafter edit its whole draft: after the first unmasking pass, each editing pass takes the current draft instead of masks and re-predicts every position at once, with bidirectional attention inside the block, so later guesses can repair earlier errors. The hard part is repairing without breaking correct tokens. ProposalMix training builds editing inputs by mixing the drafter’s first-pass predictions with ground-truth tokens according to its confidence, which halves the edits that shorten the accepted prefix. With two editing passes over a block of 32, DEdit accepts 10% more tokens per round than DSpark trained on the same data. Every editing pass costs a drafter pass, so the speedup grows less than the acceptance length, and a causal editor accepts less: the future context is what makes editing work.

Beyond a single round

So far every round has been self-contained: draft, verify once, throw away the rejected part and start from scratch. The newest drafters break these habits.

Drafting past the verifier

With strong drafters the target often accepts every drafted token, so verifying after each drafting stage is often unnecessary. Adaptive draft lengths decide how much to draft before verifying: SpecDec++ (Huang et al., 2024) trains an acceptance prediction head for an autoregressive drafter and stops once the predicted chance of a rejection crosses a threshold, and DSpark’s scheduler chooses lengths for a whole batch, but only within one block. Going beyond a block is harder for a parallel drafter, whose next block needs target hidden states for tokens that aren’t verified yet. DLoop (Gu et al., 2026), not to be confused with D-Loop above, keeps drafting while the drafter stays confident, feeding it its own hidden states in place of the missing target ones, and verifies all accumulated tokens in one pass. Loop-aware training exposes the drafter to these self-generated states so that it stays reliable in later stages. DLoop trades a few cheap drafter passes for fewer expensive verifications and improves walltime by 5 to 41% across EAGLE-3, DFlash, Domino, DSpark and MTP drafters. PEARL (Liu et al., 2024) attacks the same idle time from the other side, drafting the next tokens while the target is still verifying the current ones, so that neither model waits for the other.

Recycling rejected work

Verification also computes target hidden states for rejected tokens, and the rejected part of a draft often still contains the right tokens a little later. Token Recycling (Luo et al., 2024) was a training-free version of this idea, storing candidate tokens from earlier steps in an adjacency matrix and growing the next draft tree from them. Two recent methods recycle much richer state into parallel drafters.

ReTrace (Lin et al., 2026) shows that rejected positions of a DFlash draft often still align with the target’s continuation, and conditions the next block on the rejected suffix instead of fresh mask tokens. It keeps the suffix’s hidden representations, aligns them with the new block, corrects them with signals from the same verification pass and fuses them into the drafter’s input embeddings through a gated residual connection, without an extra forward pass.

Carryover drafting (Koo et al., 2026) recycles the target’s side: hidden states the verifier computed for rejected tokens become temporary KV context for the drafter, marked with a single learned embedding and replaced every round, so their size stays bounded by one block. Standard parallel training never produces rejected states like the ones at inference, so Carryover’s draft–verify–draft training produces them while keeping all positions in parallel. With DFlash and a DSpark-style drafter it raises the vLLM speedup by up to 14%, and by up to 29% on translation.

Both stay lossless, because rejected tokens are never committed, they only inform the next draft. And both change what a drafter keeps between rounds, which brings us to memory.

Drafters and the KV cache

So far we’ve counted only the weights a drafter reads. With long contexts and large batches the KV cache takes over: MagicDec (Sadhukhan et al., 2024) showed that decoding then becomes bound by reading the KV cache, which grows with both the sequence length and the batch size, and that speculative decoding keeps paying off even at batch sizes from 32 to 256, as long as the drafter’s own KV cache stays small.

DFlash and the drafters built on it keep their own KV cache: keys and values projected from $\mathbf{H}$ in every draft layer, for the whole context. A real world example: a DFlash-style drafter for Qwen3-8B has 5 layers with the target’s layout of 8 KV heads of dimension 128, so one request with 32k tokens of context in bf16 needs

\[\underset{\mathbf{K/V}}{2} \cdot \underset{\text{bf16}}{2} \cdot \underset{\text{sequence length}}{32{,}768} \cdot \underset{\text{draft layers}}{5} \cdot \underset{\text{KV heads}}{8} \cdot \underset{\text{head dim}}{128} = \underset{\text{bytes per request}}{671{,}088{,}640}.\]

That’s 0.67 GB, only 14% on top of the target’s own 36-layer cache, but 128 such requests need 86 GB for the drafter alone, more than the whole HBM of an H100. Hybrid targets fare worse: Qwen3.5-397B-A17B uses linear attention in most layers, so its own cache is small, and a drafter cache inflates the total by 1.8×.

Memory taken by the KV cache of a DFlash-style drafter (5 layers, 8 KV heads of dimension 128, bf16) for Qwen3-8B at 32k context versus the number of concurrent requests. Attending to the full context, it overtakes the target’s weights at 24 requests and the whole HBM of an H100 at 119. With a 2,048-token sliding window (dashed) it stays at 5.4 GB even with 128 requests.

Sliding windows

The most straightforward fix is the one long-context LLMs use themselves: a sliding window. If each draft layer attends only to the last $W$ positions, the cache stops growing with the context, and with $W = 2{,}048$ our drafter needs

\[\underset{\mathbf{K/V}}{2} \cdot \underset{\text{bf16}}{2} \cdot \underset{\text{window}}{2{,}048} \cdot \underset{\text{draft layers}}{5} \cdot \underset{\text{KV heads}}{8} \cdot \underset{\text{head dim}}{128} \approx \underset{\text{bytes per request}}{42 \text{ MB}},\]

16 times less at 32k tokens and 500 times less at 1M. MagicDec itself drafted this way, running the target over a StreamingLLM-style window of recent tokens and a few attention sinks, and parallel drafters have converged on it too: z-lab trains its recent DFlash drafters, such as the one for Qwen3.5-27B, with sliding-window attention “for improved long-context performance”, Red Hat’s DFlash drafter for DeepSeek-V4-Flash uses 2,048-token windows in all five layers, and a recent comparison of parallel drafters (Jiang et al., 2026) runs all of them with the same window.

The window costs less acceptance than one might expect. The keys and values in it are projected from target hidden states that have already attended over the whole context, and drafters mostly look at recent tokens anyway: Eldenk et al. (2026) observe that the attention of EAGLE-3 and MTP drafters drifts from the prompt towards the tokens they’ve just drafted. Windowed-MTP (Valliappan, 2026) puts a 4K window with a few sink tokens on the native MTP drafters of hybrid Qwen models at 1M tokens of context: the acceptance length even grew slightly over full-context drafting while the time per output token fell by a third, and only a 1K window started to hurt.

So for memory alone a window is mostly enough. It doesn’t remove the drafter’s cache as a separate structure: every new token still has to be projected into keys and values in every draft layer and written into a second cache pool with its own layout, which the engine has to allocate, page and roll back (Windowed-MTP needs a ring buffer and a remapping step to actually reclaim the memory). On Qwen3-8B this injection step gets 4.7× slower going from 4 to 128 concurrent requests. And it leaves open another question: could the drafter get more out of the target’s own cache than out of the few hidden states it projects into its own?

Other ways to save memory

Before windows became the default for parallel drafters, three other ways to keep the drafter’s memory in check appeared.

Shrink the drafter’s cache. TriForce (Sun et al., 2024) drafts with the target itself over a small, retrieval-based selection of its KV cache, SpecExtend (Cha et al., 2025) uses the target’s attention scores to decide which part of the context a small drafter keeps, without retraining, and QuantSpec (Tiwari et al., 2025) drafts with the target over a hierarchical 4-bit quantized KV cache and 4-bit weights, accepting over 90% of drafted tokens. Yuan et al. (2026) give a strong drafter a compressed memory that a small adaptor builds and updates incrementally, cutting drafter-side memory by over 70% at 32k contexts.

Reuse the target’s cache. Self-speculative decoding drafts with a part of the target: Draft & Verify (Zhang et al., 2023) skips some intermediate layers, and LayerSkip (Elhoushi et al., 2024) trains the target with early exits and drafts with its first layers, whose cache and activations verification then reuses. LongSpec (Yang et al., 2025) gives a small drafter self-attention over a 512-token sliding window and cross-attention to the target’s KV cache, so its own cache stays constant however long the context. Gemma 4’s MTP assistants go all the way: their decoder layers have only query projections and read keys and values straight from the target’s cache.

Keep a recurrent state. ReDrafter (Cheng et al., 2024) drafts with a small RNN conditioned on the target’s last hidden state, and Mamba drafters (Choi et al., 2025) replace the drafter’s transformer with a state space model. OWL (Lee et al., 2025) drafts with an LSTM conditioned only on the last token’s state, and on long inputs, where EAGLE-3 slows decoding down to 0.81×, reaches about 5 times EAGLE-3’s acceptance length.

The target’s cache as a source of information

The target’s KV cache holds keys and values from every layer for every position, far more than the three or five hidden states a drafter usually reads, and some drafters read it purely for acceptance. GliDe (Du et al., 2024) keeps a small drafter with its own cache and adds a cross-attention layer that “catches a glimpse” of the target’s cache so the drafter behaves more like the target, raising the acceptance rate by up to 23.5%. Mixture of Attentions (Zimmer et al., 2024) blames EAGLE’s limits on partial observability, since the last hidden states are a lossy function of the target’s full state. Its Layer Self-Attention attends across the keys and values of all target layers for each verified token and averages them into one vector per token for the drafter to cross-attend to; the full model accepts 25% more tokens per round than EAGLE-2, though the layer attention itself contributes only a modest part. Both are autoregressive and use the target’s cache on top of what they already had. A parallel drafter that drops its own cache must rely on the target’s cache instead, a stronger bet.

H-Spec

H-Spec (Jiang et al., 2026) makes that bet, combining two of the older answers: it reads the target’s keys and values in place and keeps a recurrent state rebuilt every round, and it gains acceptance rather than paying for the memory with it. It starts from the obvious fix: let each DFlash layer attend to one target layer’s keys and values directly, like GliDe. The first two positions of the block stayed as good as before, but conditional acceptance at later ones fell by up to 16.5 percentage points. Keys and values are made for retrieval, each describing one position for future queries, and none summarizes the context as a whole, as the projected hidden states did.

A constant-size summary is already there: under causal attention the target’s last hidden state $\mathbf{h}_t$ has seen the whole context, and every verification pass computes it. To consume one vector per round, H-Spec turns to state space models. The recurrent form of linear attention from the toolset post folds the whole prefix into one matrix, $\mathbf{U}_i = \mathbf{U}_{i-1} + \phi(\mathbf{k}_i)\mathbf{v}_i^T$, and Mamba-2 (Dao and Gu, 2024) adds a learned, input-dependent forgetting factor $a_i \in (0, 1)$:

\[\mathbf{U}_i = a_i\, \mathbf{U}_{i-1} + \mathbf{k}_i \mathbf{v}_i^T, \qquad \mathbf{o}_i = \mathbf{U}_i^T \mathbf{q}_i,\]

where $a_i$, $\mathbf{q}_i$, $\mathbf{k}_i$ and $\mathbf{v}_i$ are computed from the current input (Mamba calls them $\bar{A}$, $C$, $B$ and $x$). Where attention keeps and reads a key and a value for every past token, a Mamba layer keeps only $\mathbf{U}$, whose size doesn’t depend on the context, and decides what to forget. That’s how hybrid targets like Qwen3.5 get their small caches, and why they still keep some full attention layers: the summary is lossy and can’t retrieve an arbitrary token exactly.

H-Spec gives these two jobs to two modules, both fed by the target for free. Each of its four layers starts with a Mamba-2 module, and three follow it with attention. The Mamba modules start from a state $\mathbf{U}_0$ projected from the fused last-position hidden states of five target layers and scan only the block, in one parallel pass since the recurrence is linear; identical mask tokens still get different states at different positions. The attention modules match the target’s KV layout and read three target layers’ keys and values in place, through a 2,048-token window, for the token-level detail a summary can’t hold. DSpark’s Markov head sits on top. For Qwen3-8B the Mamba state holds 48 heads × 64 × 4 = 12,288 numbers per layer and is rebuilt every round, so nothing grows with the context and nothing has to be injected.

How a parallel drafter conditions on a context of $L$ tokens. DFlash writes the target’s projected hidden states into its own cache for every new token, a cache that grows with the context unless windowed. H-Spec’s attention reads the target’s KV cache in place, and its Mamba layers start from a fixed-size state built from $\mathbf{h}_t$ and scan the block in one parallel pass. Move the slider to grow the context, and click the titles to switch.

Both halves matter: dropping the target’s keys and values cuts the acceptance length by a quarter or more, and dropping the summary by about a tenth. Against P-EAGLE, DFlash and DSpark retrained with the same recipe and the same 2,048-token window, H-Spec accepts 5–13% more tokens per round from 1k to 32k tokens of context, which is what the target’s cache and summary add as information, and in vLLM it reaches up to 17% higher peak throughput, with an advantage growing with concurrency. The costs are a deeper computational graph, which drafts slower than DFlash below about 32 concurrent requests or 512 tokens of input, and attention layers that must copy the target’s KV layout.

Serving engines have converged on a handful of these families: vLLM’s benchmark on AMD GPUs compares exactly four method values of --speculative-config, namely mtp, eagle3, dflash and dspark. Across all of them the architecture has been moving the same way, from drafting step by step towards a parallel backbone, a small causal head and deep conditioning on the target, built with the serving engine in mind⁴. What’s left is the objective.

Training the drafter

Recall that the speedup is governed by the acceptance length, and the acceptance length by $\alpha = 1 - \operatorname{TV}(p, q)$. Yet EAGLE-3, MTP, DFlash and Domino are all trained with cross-entropy, i.e. towards a KL objective, and newer drafters are moving away from it: DSpark and H-Spec put 90% of their loss on a total variation term, and Kimi-K3 drops KL altogether.

KL is only a proxy

The usual recipe is cross-entropy on target-generated tokens, or forward KL to the target’s full distribution, the same objective with a denser signal. KL reaches zero exactly when $q = p$, where $\alpha = 1$, but this only helps if the drafter family $\mathcal{Q}$ contains $p$. A drafter typically has 1–5% of the target’s parameters and sees it only through a few hidden states, so in general it can’t represent $p$ exactly, and then the two objectives can pick different solutions:

\[q_{\text{KL}} = \arg\min_{q \in \mathcal{Q}} \operatorname{KL}(p \,\Vert\, q) \ \ne \ q_{\alpha} = \arg\max_{q \in \mathcal{Q}} \sum_x \min\big(p(x), q(x)\big).\]

Pinsker’s inequality,

\[1 - \alpha \le \sqrt{\tfrac{1}{2} \operatorname{KL}(p \,\Vert\, q)},\]

guarantees a high $\alpha$ for a low KL, but not which $q \in \mathcal{Q}$ has the highest $\alpha$. The objectives disagree on modes they can’t fit. Forward KL is mass-covering: it punishes $q(x) \approx 0$ wherever $p(x) > 0$, so a drafter with limited capacity spreads over all modes of the target. Reverse KL, $\operatorname{KL}(q \,\Vert\, p)$, is mode-seeking: it punishes mass where $p$ has little, so the drafter collapses onto a single mode. The acceptance rate only rewards overlap. That TV, not KL, is the divergence tied to acceptance isn’t new: DistillSpec (Zhou et al., 2023) already compared it with KL variants for distilling drafters. Here’s a toy example with a mixture of three Gaussians as the target and a single Gaussian as the drafter:

A Gaussian drafter $q = \mathcal{N}(\mu, \sigma^2)$ (blue) fitted to a target (orange) that mixes $\mathcal{N}(-4, 0.8^2)$, $\mathcal{N}(-0.5, 0.4^2)$ and $\mathcal{N}(6, 0.4^2)$ with weights 0.4, 0.3 and 0.3, discretized into 241 tokens. The green area is the overlap $\alpha$. Move the sliders, or click the labels on the right to jump to the optimum of each objective.

The forward KL optimum spreads over all three modes and accepts 42% of tokens, the reverse KL optimum sits on the widest mode and accepts 40%, and the $\alpha$-optimal drafter partially covers the two left modes and accepts 54%. Same family, same capacity, and neither KL finds it.

So why not maximize $\alpha$ directly? Look at the gradients with respect to the drafter’s logits $\ell$:

\[\frac{\partial \alpha}{\partial \ell_y} = \frac{q(y)}{2}\Big(s(y) - \sum_x q(x)\, s(x)\Big), \qquad \frac{\partial \operatorname{KL}(p \,\Vert\, q)}{\partial \ell_y} = q(y) - p(y),\]

where $s(x) = \operatorname{sign}\big(p(x) - q(x)\big)$. The gradient of $\alpha$ is scaled by $q(y)$, so the tokens where the drafter puts little mass, exactly the ones it needs to learn, barely get any push, while KL pulls wherever $p$ has mass and $q$ doesn’t. The figure starts in this regime: $q$ sits between two modes, $\alpha \approx 0.005$ and small moves of $\mu$ or $\sigma$ barely change it, while KL reacts to every move. A real drafter starts spread over $10^5$ tokens or more, where the TV gradient is tiny, and in our experiments drafters trained on TV alone ended up well behind KL.

LK losses

In our recent paper we introduced LK losses (Samarin et al., 2026), which target the acceptance rate directly; the name comes from Leviathan et al.’s $D_{LK}$. The first is a hybrid whose weight moves from KL to TV as the drafter improves:

\[\mathcal{L}_{\text{LK}}^{\lambda} = \lambda\, \operatorname{KL}(p \,\Vert\, q_\phi) + (1 - \lambda)\, \operatorname{TV}(p, q_\phi), \qquad \lambda = \exp\big(-\eta \cdot \operatorname{sg}[\alpha]\big),\]

where $\operatorname{sg}$ is a stop-gradient and $\alpha$ is the acceptance rate at the given draft position, averaged over the batch and the sequence. While the drafter is poor, $\lambda \approx 1$ and it gets KL’s dense gradients; as $\alpha$ grows, the weight shifts to the acceptance rate itself. The second is the likelihood of acceptance:

\[\mathcal{L}_{\text{LK}}^{\alpha} = -\log \alpha = -\log \sum_x \min\big(p(x), q_\phi(x)\big), \qquad \nabla \mathcal{L}_{\text{LK}}^{\alpha} = \frac{1}{\alpha} \nabla \operatorname{TV}(p, q_\phi).\]

It has the same optimum as TV, but the factor $\frac{1}{\alpha}$ amplifies the gradient exactly when acceptance is low. Both losses come from the same two logit tensors as KL:

@jit
def drafter_losses(target_logits, draft_logits, eta=3.0):
    """Logits: [N, γ, V], N = batch × sequence positions."""
    log_p = jax.nn.log_softmax(target_logits, axis=-1)
    log_q = jax.nn.log_softmax(draft_logits, axis=-1)
    p, q = jnp.exp(log_p), jnp.exp(log_q)
    kl = (p * (log_p - log_q)).sum(-1)
    alpha = jnp.minimum(p, q).sum(-1)
    lam = jnp.exp(-eta * jax.lax.stop_gradient(alpha.mean(0)))  # one λ per draft position
    lk_hybrid = lam * kl + (1.0 - lam) * (1.0 - alpha)
    lk_likelihood = -jnp.log(alpha)
    return kl.mean(), lk_hybrid.mean(), lk_likelihood.mean()

Across drafter architectures and target sizes, both losses improved the acceptance length over KL, most of all where acceptance is lowest: at temperature 1 and at late draft positions. DSpark’s and H-Spec’s fixed blend of 0.1 cross-entropy and 0.9 TV is a hybrid with a constant weight; the adaptive weight lets the drafter lean on KL early in training and on acceptance later.

Kimi-K3

Kimi-K3’s technical report fine-tunes its pre-trained MTP layer into an EAGLE-3-style drafter, unrolled for seven steps following the training-time test, with $\mathbf{W}_{\text{fuse}}$ initialized as $[\mathbf{0}\ \mathbf{0}\ \mathbf{I}]$ so that training starts from the final hidden state the MTP layer was pre-trained on. Since minimizing KL doesn’t guarantee maximizing the acceptance rate of a capacity-limited drafter, they directly optimize $\mathcal{L}_{\text{LK}}^{\alpha} = -\log \alpha$, at temperature 1 and with no auxiliary cross-entropy term.

Beyond per-token objectives

KL, TV and LK losses are all per-token, but the acceptance length is a sum of products:

\[\tau \approx 1 + \sum_{k=1}^{\gamma} \prod_{i=1}^{k} \alpha_i, \qquad \frac{\partial \tau}{\partial \alpha_k} = \frac{1}{\alpha_k} \sum_{j=k}^{\gamma} \prod_{i=1}^{j} \alpha_i.\]

Position $k$ only pays off if every position before it was accepted, so early positions are worth more; for a constant $\alpha$ the weight of position $k$ decays roughly as $\alpha^{k-1}$. Parallel drafters already use such weights: DFlash multiplies the cross-entropy at position $k$ by $\exp\big(-\frac{k-1}{\kappa}\big)$, with $\kappa = 7$ for blocks of 16, and Domino uses the same exponential decay. That’s the decay $\tau$ prescribes at $\alpha \approx e^{-1/7} \approx 0.87$, the per-position acceptance behind an acceptance length of 6–7 out of 16, about what DFlash reaches:

Weights that the acceptance length assigns to the acceptance rate of each position in a block of 16 when every position has the same $\alpha$ (bars, normalized to the first position), against DFlash’s fixed loss weights (dashed). Move the slider to change $\alpha$.

A fixed schedule is only right for one acceptance rate. D-PACE (Wu et al., 2026) computes adaptive weights from a differentiable surrogate of the acceptance length built from the drafter’s own confidences. The E2E-TV loss (Li et al., 2026) drops the per-token form and maximizes the expected number of accepted draft tokens directly, with gradients flowing through the products:

\[\mathcal{L}_{\text{E2E}} = -\sum_{k=1}^{\gamma} \prod_{i=1}^{k} \big(1 - \operatorname{TV}(p_i, q_i)\big).\]

This product is the survival probability of token-wise verification. Under block verification a surplus at one token can pay for a deficit at the next, so drafters with the same TV at every position can accept different numbers of tokens. BV loss (Kim et al., 2026) derives the objective from block verification itself: for a fixed drafted prefix, averaged over the rest of the draft and the verifier’s coins, it accepts at least the first $k$ tokens with probability exactly $w_k$, the running weight from the verifier’s side, so the expected number of accepted draft tokens is $\sum_k \mathbb{E}_{x \sim q}[w_k]$. Training text comes from the target, and importance weighting turns the clipped running weight into a running minimum of likelihood ratios:

\[\mathbb{E}_{x \sim q}\big[w_k\big] = \mathbb{E}_{x \sim p}\big[\min(1, r_1, \dots, r_k)\big], \qquad r_k = \prod_{i=1}^{k} \frac{q_i(x_{t+i})}{p_i(x_{t+i})}.\]

As in our likelihood loss, a logarithm keeps the gradients alive when acceptance is low:

\[\mathcal{L}_{\text{BV}} = -\log \sum_{k=1}^{\gamma} \min\big(1, r_1, \dots, r_k\big).\]

In practice the last token of each prefix is integrated over the vocabulary, which brings in the target’s full distribution and makes the single-token case exactly $\mathcal{L}^{\alpha}_{\text{LK}}$. DFlash and DSpark drafters trained from scratch with BV loss for Qwen3-4B and Qwen3-8B accept 13–21% more tokens per round under block verification than with cross-entropy and 4–16% more than with LK losses, and the gains carry over to token-wise verification and greedy decoding. Drafters trained on TV alone reached only 1.0–2.2 tokens per round, the vanishing gradient from above. And the cross-entropy drafter ended up matching the target’s tokens more often than the BV drafter, 54% against 47%, yet the BV drafter kept 21% more draft tokens per round: like KL, token accuracy is only a proxy.

A second gap is the context. Losses are computed on target-generated text, while at inference the drafter continues its own guesses, from anchors that depend on earlier acceptances. EAGLE-3’s training-time test closes the first part by unrolling the drafter. Draft-OPD (Lei et al., 2026) goes further with on-policy distillation, replaying draft proposals from the anchors where rounds actually start at inference and weighting the distillation by acceptance. DLoop’s loop-aware training and Carryover’s draft–verify–draft training apply the same idea to states that exist only across rounds: the drafter’s own hidden states for unverified tokens and the target’s states for rejected ones.

The third gap is between rounds. For a parallel drafter the distribution at position $t$ depends on where its round started, which depends on how many tokens the drafter got accepted before, so the states it is evaluated on depend on the drafter itself. EDR (Zhao and Cai, 2026) captures this with a Markov reward process conditioned on a target rollout of length $L$. A state $(t_0, t)$ means that the current round started at position $t_0$ and has reached $t$. With probability $A^{q}_{t_0,t}$ the token is accepted and the process moves to $(t_0, t+1)$; otherwise a new round starts at $(t, t+1)$, and every rejection costs one round. The expected number of rounds is then exactly

\[\mathbb{E}[\#\text{rounds}] = 1 + \sum_{t=1}^{L} \sum_{t_0=0}^{t-1} \omega^{q}_{t_0,t}\, \epsilon^{q}_{t_0,t},\]

where $\omega^{q}_{t_0,t}$ is the probability that decoding visits state $(t_0, t)$ (its occupancy) and $\epsilon^{q}_{t_0,t}$ is the expected local rejection cost there, a TV distance between the drafter and the target.

For an autoregressive drafter, whose prediction at position $t$ doesn’t depend on where the round started, the occupancies sum to one at every position and the formula collapses to $1 + \mathbb{E}\big[\sum_t \epsilon_t\big]$, the total TV distance along target rollouts, so a per-token acceptance objective is exactly right. For parallel drafters it isn’t, and even E2E’s product of $1 - \operatorname{TV}$ terms only approximates the survival probability inside a round, the same $\approx$ as in our formula for $\tau$; Qiang et al. found that for DFlash and DSpark in serving acceptances along a block are positively correlated, so the true survival is higher than the product. EDR weights the local costs by occupancies that depend on $q$, derives an exact temporal-difference gradient that can be estimated without bias from target rollouts, and has no hyperparameters. The same machinery gives an exact offline evaluator of the number of rounds, so two drafters can be compared on shared target rollouts without running speculative decoding. Fine-tuning DSpark and DFly⁵ with EDR gives small but consistent gains over both the released checkpoints and E2E-TV across nine math, code and chat benchmarks.

Per-token, per-block and per-sequence objectives form a ladder: LK losses optimize the acceptance of each token, E2E and BV losses the acceptance length of each round, under token-wise and block verification respectively, and EDR the number of rounds for the whole sequence.

Conclusion

Decoding is memory-bound: one forward pass reads all the weights, so verifying a handful of tokens costs about as much as generating one. Speculative sampling makes guessing lossless, shared Gumbel noise makes it reproducible, block verification and token trees squeeze more out of every draft, and as long as verification stays memory-bound the speedup reduces to a single formula, $\tau / (n_{\text{draft}} \cdot c + 1)$. Every drafter is a different answer to two questions: what should the guesser see, and how should it be trained?

Architecture decides what $q$ can express. We went from a separate small LLM to autoregressive drafting over target hidden states, to parallel drafters with target states injected into every layer, to parallel drafters that restore dependency with a light causal head or a second pass over their own block, and finally to drafters that carry rejected work into the next round and read the target’s own cache instead of keeping one. Information floors show that a head needs to see only a token or two back to undo most of parallel blindness, so what remains is the drafter’s own model gap, and since a sliding window already bounds the drafter’s memory, reading the target’s cache pays off as information more than as saved memory. The objective decides which $q$ we actually get, and since drafters are small, the KL optimum is not the acceptance optimum.

Plenty of questions remain open: drafters for long-context and agentic workloads, where prefill and memory dominate and tree drafting gains shrink; acceptance-aware training for parallel drafters, which have the most to gain from it; and drafters for diffusion language models, which already generate whole blocks. Whatever comes next, it will still be the art of guessing well.

---

1. Jacobi decoding (Santilli et al., 2023) and Lookahead decoding (Fu et al., 2024) let the target refine a whole window of guesses in parallel, and CLLMs (Kou et al., 2024) fine-tune the model to converge in fewer iterations. Drafting without any model, by copying n-grams from the prompt (prompt lookup) or from earlier outputs (suffix decoding), is surprisingly strong for code editing and agent loops, where outputs repeat their inputs. ↩ ↩² ↩³

2. Why block verification is lossless, in short. By induction $q(x_{t+1:t+k})\, w_k \le p(x_{t+1:t+k})$, where $q(x_{t+1:t+k})$ and $p(x_{t+1:t+k})$ are products of per-position probabilities, so $w_k$ is the share of the drafted prefix that the target can still pay for. A kept prefix is extended either by the next drafted token $y$, with weight $\min\big(q_{k+1}(y), w_k\, p_{k+1}(y)\big)$, or by a sample from the residual $\big(w_k\, p_{k+1}(y) - q_{k+1}(y)\big)_+$, and the two add up to exactly $w_k\, p_{k+1}(y)$: the single-token argument of speculative sampling, scaled by $w_k$. The survival probabilities keep this bookkeeping exact even though the verifier decides only after seeing the whole draft. Sun et al. (2024) give the full proof. ↩

3. Blockwise parallel decoding (Stern et al., 2018) did this first. Hydra (Ankner et al., 2024) made Medusa heads sequentially dependent, and Medusa-2’s typical acceptance trades exactness for speed by accepting any plausible token instead of using speculative sampling. ↩

4. Serving engines matter as much as drafters. SGLang’s Spec V2 overlaps host-side scheduling with GPU work and raises throughput by over 33%, from about 11.4k to 15.3k tokens/s, for Qwen3-8B on a single B200 at concurrency 32 (LMSYS). ↩

5. DFly (Liu et al., 2026) comes from the AngelSpec framework. It’s another block-diffusion drafter with a predecessor-conditioned autoregressive head, i.e. the same “parallel + causal head” family as DSpark and Domino. ↩