Researchers Introduce X-CoSD for Communication-Efficient LLM Decoding

A preprint published on arXiv introduces X-CoSD, a novel collaborative speculative decoding (CoSD) framework designed for distributed large language model (LLM) inference. In traditional CoSD architectures, an on-device small language model (SLM) drafts tokens while a server-side LLM verifies them. However, standard methods require a shared vocabulary and lead to high network overhead due to token distribution exchanges.

To overcome these constraints, X-CoSD enables collaborative decoding between models with heterogeneous vocabularies. The framework uses hybrid resampling (HR), which splits the residual resampling process so that distribution transmissions are restricted only to the overlapping vocabulary segments.

The authors also proposed X-CoSD-E, an advanced variant where the server only transmits replacement candidates and probabilities for on-device verification. Experimental results indicate that both methods significantly boost generation speeds while fully preserving the output quality of the server LLM.