Augmenting Continuous Diffusion Language Models with Discrete Token Guidance

Hui Ren¹, Zihan Li¹, Chang Liu¹, Huidong Liu², Alexander Schwing^1
1University of Illinois Urbana-Champaign    ²Amazon.com, Inc.

HC-DLM augments continuous diffusion language models with discrete token guidance:
a shared latent plans every token jointly, while tokens read out at each step keep it anchored to valid text.

_{The latent plans, the tokens guide, at every step (simulated example). Interactive version on the project page.}

Each family of diffusion language models has a blind spot:

• Discrete diffusion decodes tokens in parallel, but samples each one independently from its marginal.
• Continuous diffusion plans all tokens in one shared latent, but never checks that plan against real tokens until the very end.

HC-DLM keeps both strengths: tokens are planned jointly in the latent, and the latent is guided by tokens at every step. The continuous latent is the only persistent generative state; at every step the model reads a token draft $k_t$ out of the latent $x_t$, and that draft guides the next latent update.

$$ p_{\theta,\phi}(k_{0:T},x_{0:T})=p(x_T),p_\theta(k_0\mid x_0)\prod_{t=1}^{T}\underbrace{p_\theta(k_t\mid x_t)}_{\text{read out tokens}};\underbrace{p_\phi(x_{t-1}\mid x_t,k_t)}_{\text{token-guided latent denoising}} $$

_{One reverse step, four designs. Discrete diffusion updates tokens directly, one marginal at a time. Continuous diffusion denoises a latent that is blind to tokens and decodes only at the end. Hybrid models attach a continuous signal to a self-contained discrete chain. In HC-DLM the two levels talk at every step: tokens are read out of the latent, then guide its next update.}

Noise corrupts tokens and latent independently, so training stays simple. A single variational bound on the token likelihood splits into three terms and trains everything end to end: reconstruction for the token readout, token-guided flow matching for the latent denoiser, and an entropy term that keeps the encoder from collapsing.

_{One principled objective, trained end to end. An encoder maps clean tokens to a latent, independent forward kernels produce the noisy pair, the token-conditioned denoiser predicts the clean latent, and the token predictor decodes it back to tokens.}

From Sudoku and Countdown to open-domain text, HC-DLM leads discrete, continuous and hybrid diffusion baselines of the same size on Hard Sudoku, Countdown and LM1B. Parameter counts exclude token embeddings.

_{Left: solutions emerge early: accuracy of the intermediate prediction decoded at each step, vs. latent diffusion without token guidance. Right: holds up with fewer steps: Hard Sudoku accuracy against step count; MDM degrades sharply as steps shrink.}

The code and the artifacts are coming soon. Stay tuned!

If you find this work useful, please consider citing:

@misc{ren2026hcdlm,
      title={Hierarchical Continuous Diffusion Language Models}, 
      author={Hui Ren and Zihan Li and Chang Liu and Huidong Liu and Alexander Schwing},
      year={2026},
      eprint={2610.02193},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.02193}, 
}