The answer is yes and the key idea is a mixture of distributions.

Instead of trying to explicitly model all possible 64-token continuations, we introduce a latent variable ε sampled from a simple latent space.

Then the model generates tokens independently conditioned on ε.

At first glance, this looks too simple: independent tokens cannot capture complex text structure, right?

But the important part is that tokens are independent only after conditioning on ε.

The final text distribution is obtained by averaging over all latent variables (see the first attached image).

So the model is actually a mixture of many factorized distributions.

And this is powerful: the generator Gθ can encode global structure, style, topic, dependencies, and correlations between tokens thorugh the latent variable ε. As a result, the marginal distribution over text can still be highly expressive, even though each conditional distribution is factorized.

This direction has already been explored in recent works such as Di4C (ICML 2025) and VADD (ICLR 2026).

The last image is a great illustration from VADD. Without a latent variable, a factorized model fails to capture dependencies, this is what happens with MDLM. But with a latent variable, VADD can recover structured distributions like checkerboards and spirals.

However, there is still a major problem.

These mixtures in VADD is trained with a VAE-style objective. And in practice, VAE losses can be fragile: they require balancing reconstruction quality against the regularization term. If this balance is not right, the model can learn poor latent representations and produce weak samples.

So the real question becomes:

Can we design a better loss function for training mixtures of distributions?

🚀 How do we train this mixture of distributions?

To do that, we use the idea of Inverse Distillation for Discrete Diffusion Models, introduced in our previous work IBMD, ICML 2025.

The idea is simple.

Usually, diffusion models are trained in the forward direction:

we have samples from the real data distribution p*,
and we train a diffusion model f* by minimizing some loss:

f* = argmin_f L(f, p*)

This is the standard diffusion training pipeline.

Data → train diffusion model.

But in Inverse Distillation, we flip the problem.

Instead of starting from data, we start from a pretrained diffusion model f*.

Now we want to learn a generator distribution pθ such that:

if we trained a diffusion model on samples from pθ,
the optimal diffusion model would be the given pretrained model f*.

Formally:

f* = argmin_f L(f, pθ)

So we are asking:

What distribution could have produced this pretrained diffusion model?

In the ideal case, the generator Gθ should produce samples from the same distribution that the teacher diffusion model was originally trained on.

In other words:

pθ = p*

This is the core intuition behind Inverse Distillation.

The generator does not just imitate individual samples.
It learns to produce samples whose induced diffusion training process matches the teacher model.

But there is a problem.

We cannot directly optimize this objective, because it contains an inner optimization:

argmin_f L(f, pθ)

This “train a diffusion model to optimality” step is not something we can directly differentiate through in practice.

So we introduce a practical loss: IDLM loss (3-rd image).

The intuition is the following:

The first term measures how well the pretrained teacher diffusion model f* fits the samples generated by pθ.

The second term asks:
what is the best possible diffusion model f̂ we could train on these generated samples?

Then the generator minimizes the gap between them.

So Gθ is trained to produce samples for which the teacher model f* is already optimal.

🧪 Experiments: how much can we accelerate diffusion language models?

All models were trained on the OWT dataset.

For evaluation, we tracked two metrics:

GenPPL — generation perplexity
Entropy — diversity of generated text

The goal was simple:

⚡️ reduce the number of diffusion steps as much as possible
while keeping GenPPL and Entropy almost unchanged.

In other words, we wanted acceleration without sacrificing sample quality or diversity.

And here is what we got 👇

1. Masked diffusion distillation

For masked diffusion, our IDLM-MDLM model achieved:

Result: 🔥 64× acceleration

with no noticeable degradation in GenPPL and Entropy.

(See the first image)

2. Uniform diffusion distillation

For uniform diffusion, our IDLM-DCD model achieved:

Result: 🚀 256× acceleration

again, without sacrificing the measured metrics.

(See the second image)

So the takeaway is:

Inverse Distillation allows us to compress many diffusion steps into just a few, while preserving the behavior of the original model.

This is exactly what we need for practical parallel generation:
fewer steps, faster sampling, same quality signals.

P.S. Feel free to destroy the whole GenPPL + Entropy evaluation protocol in the comments 😄