Researchers Introduce Osprey to Accelerate LLM Speculative Decoding

Researchers have released a scientific preprint introducing Osprey, a framework designed to improve speculative decoding for faster LLM inference. Traditional drafter models are typically trained for a specific target LLM, making their acceptance rates vulnerable to workload distribution shifts. Osprey solves this by bootstrapping drafters from off-the-shelf pretrained small language models as reusable, target-agnostic assets.

To meet latency requirements, Osprey prunes the small language model into a shallow backbone and restores its modeling capabilities via target-agnostic pre-training. It then adapts the backbone to specific target models through vocabulary alignment, zero-initialized QKV expansion, and distillation from the target's output distribution.

Empirical results demonstrate that a single Osprey backbone improved mean acceptance length by 16.1% on Qwen3-8B, 21.2% on Llama-3.3-70B-Instruct, and 22.7% on the 229B MiniMax-M2.5, achieving 17.5% higher tokens per second. The code has been made publicly available on GitHub.