NCP-ArchPreview Report Introduces Next Concept Prediction for LLMs

A technical report released on arXiv introduces NCP-ArchPreview, a latent-space language model that extends autoregressive pretraining beyond standard Next-Token Prediction (NTP). Alongside NTP, the architecture leverages Next Concept Prediction (NCP) to predict discrete concepts spanning multiple tokens, which are then used to guide subsequent generation.

Scaled to 8.9 billion parameters and trained on 5.73 trillion tokens from the Dolma-3 dataset, NCP-ArchPreview represents the largest demonstration of a latent-space language model to date. It achieved the final pretraining loss of OLMo-3-7B while consuming only 51.3% of the total training tokens. Following full pretraining, it outperformed OLMo-3-7B by 2.45 points on downstream macro-averages, including a 5.99-point gain on GSM8K.

Using only 85% of standard computation, the architecture also approaches the training loss of a strictly parameter-aligned 8.9B baseline. Additionally, researchers noted that updating a lightweight 17M-parameter VQ module enables effective domain adaptation, while integrating concept representations into a DFlash2 drafter improves mean accepted length by 4.17%.