Disclosure: Amplify is an investor in Inception.
At Stanford there is a course called Deep Generative Models. Were you to enroll, you would learn about the whole spectrum of algorithms and probabilistic methods for generating text, images, audio, and more. The professors would start with the simplest, most inefficient algorithms and architectures, gradually moving towards more complex, efficient solutions as the semester went on. Sort of like a textbook.
Since 2018, when professors Stefano Ermon and Aditya Grover taught the first iteration of this course, the world has exploded with incredibly powerful LLMs. Each month (or week) yields a new release from a megalab beating benchmarks and so on and so forth. Given how successful these models have become, you’d reasonably guess that they make use of rather complex, advanced techniques from this “textbook” of algorithms for generative models. Surely underlying these incredible machines is incredible algorithmic depth.
But you’d have guessed wrong! Despite autoregression being perhaps the most inefficient possible way to generate text, it has formed the foundation of all of today’s most popular and widely adopted LLMs. Nearly all SOTA models today generate tokens one at a time, the most basic and poorly optimized way to do their job. Curious indeed.
Now if all of this is true, a reader might naturally wonder…what’s on the other end of the spectrum? What’s the most efficient, cutting-edge way to generate data? That would be diffusion. Instead of generating tokens sequentially, the model works its way to the final answer all at once, like a digital sculptor from a block of marble embeddings. Diffusion is how basically all image and video models work today (like Runway), and is starting to finally make its way into text too.
The promise of diffusion – this most efficient but complex method of generation – is bonkers speed. And indeed, the leading diffusion LLM, Mercury 2, is undoubtedly the world’s fastest reasoning language model. It produces 1,009 tokens/sec on NVIDIA Blackwell GPUs at quality that’s competitive with leading speed-optimized models. This is 10x faster than Haiku 4.5, so fast that as a user you start to feel like the model’s responses are almost instantaneous.
Behind this beast of a model is Inception, cofounded by none other than Stefano (CEO) and Aditya (CTO), plus their longtime collaborator Volodymyr Kuleshov (Chief Scientist). This all-star academic team has their fingerprints on much of the history of diffusion models, and had to solve several major challenges to get diffusion to work with the discrete medium of text.
Based on extensive interviews with the Inception team, this post will tell the story of diffusion’s origins, how it made its way to language, and what it took to get it production-ready for Mercury and Mercury 2:
- A brief (academic) history of diffusion models
- Bringing diffusion models to text
- Taking text diffusion models from academia to production
- Inception, Mercury 2, and dLLMs of the future
Click on any section to navigate straight here if you’re into the whole brevity thing.
A brief (academic) history of diffusion models
To understand how we got to diffusion models for text, you need to understand how we got to diffusion models for images.
You could reasonably argue that this all started in 2015 with Sohl-Dickstein et al.’s Deep Unsupervised Learning using Nonequilibrium ThermodynamicsDeep Unsupervised Learning using Nonequilibrium ThermodynamicsA central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves both flexibility and tractability. The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data. This approach allows us to rapidly learn, sample from, and evaluate probabilities in deep generative models with thousands of layers or time steps, as well as to compute conditional and posterior probabilities under the learned model. We additionally release an open source reference implementation of the algorithm.arXiv:1503.03585v8View paper. The paper explores a new method for training a generative model through diffusion:
*“The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data. This approach allows us to rapidly learn, sample from, and evaluate probabilities in deep generative models with thousands of layers or time steps, as well as to compute conditional and posterior probabilities under the learned model.”*
This was probably the birth of diffusion as far as deep learning is concerned, but it was viewed mostly as an impractical, if academically interesting, idea. At the time, state of the art for generating images was GANs, or Generative Adversarial Networks. They were promising but painfully slow and hard to work with.
Around this time is also when our protagonists first met. In 2016, Stefano was running a lab at Stanford. It is also when Aditya started his PhD at said lab to start exploring the open space around generative models and probabilistic modeling. Meanwhile, Volodymyr was also a PhD student at Stanford, but in another lab working mostly on computational bio. He eventually joined Stefano’s lab as a postdoc in 2017…which means this team has been working together for close to 10 years. Ideas around diffusion and generative models were circulating, but it was ultimately a niche discipline with a small community around it.
Niche as it was, the community continued to make progress. In 2019, Generative Modeling by Estimating Gradients of the Data DistributionGenerative Modeling by Estimating Gradients of the Data DistributionWe introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.arXiv:1907.05600v3View paper from Stefano and Yang Song introduced the idea of score-based generative modeling. Instead of trying to model the probability distribution of your data directly, the key idea was to learn the score function: the direction in which data becomes more likely. This paper was one of the first to seriously challenge the dominance of GANs, showing that an alternative generative paradigm could produce highly competitive results. The idea was rapidly picked up by the research community and became one of the key conceptual foundations for modern diffusion methods.
The way these new types of generative models worked is through the idea of noise. To train a model to generate an image, you essentially teach it in reverse. You start with an image, and then gradually add Gaussian noise to it in the form of a random small number added to each pixel. This is done repeatedly over some N number of steps – called a forward process – until the image is basically complete noise and every pixel is a random number. You then train a model to undo this noise, and predict what the original image was, which is the direction in which data becomes more likely (the score function).
Early diffusion-style models were an exciting proof of concept, but they were way too slow to work in practice. Sampling could require hundreds or even thousands of steps, which meant generating a single image could take minutes. That same year, Stefano’s lab authored another paper introducing DDIMDenoising Diffusion Implicit ModelsDenoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a Markovian diffusion process. We construct a class of non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. We empirically demonstrate that DDIMs can produce high quality samples $10 \times$ to $50 \times$ faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, and can perform semantically meaningful image interpolation directly in the latent space.arXiv:2010.02502v4View paper, which showed how to reduce the number of sampling steps dramatically, bringing it down toward 20 to 50 while preserving quality. This was one of the key breakthroughs that made diffusion start to look practical.
But diffusion for images really broke into the mainstream when Rombach et al. introducedHigh-Resolution Image Synthesis with Latent Diffusion ModelsBy decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs. Code is available at https://github.com/CompVis/latent-diffusion .arXiv:2112.10752v2View paper Latent Diffusion Models. This paper introduced two critical ideas:
First, they figured out how to compress the space that these diffusion models were operating in. Instead of working through pixels – a massive mathematical space that could take hundreds of GPU days to train on – they proposed a method to instead use the latent space generated by a class of pre-trained autoencoders. These autoencoders were trained to find an optimal balance between reducing the complexity of the pixel space and preserving as much detail as possible. The result was 8x compression of the original pixel space and a model that could run on a consumer GPU.
Second, they introduced text labels into the model training process so that you could prompt the model to produce the image you wanted. Via layers of cross-attention trained on text-image pairs, the models could learn not just how to generate an image, but how to generate a particular image.
These advances were so meaningful that they culminated in the launch of a consumer-facing model that you’ve probably heard of: Stable Diffusion. It was the first of its kind, a diffusion model finally fast and powerful enough to actually ship. And as you can now see, it represented almost 10 years in the making. Research can often feel like it was dropped from the sky, but it’s actually an origin story of how one thing led to another, and at some point you have something that works.
Today’s diffusion models for images and video follow this basic architecture. During pretraining, every training image is paired with a text description (🙏 CLIP). The training objective is given this noisy image and this text embedding, predict the noise. Then, to improve model adherence to prompt, 10-20% of training time the prompt is randomly dropped, forcing the network to also learn random generation (called classifier-free guidance). At inference time, you generate an image with a prompt, an image without a prompt, and take the difference.
Bringing diffusion models to text
Image generation was a natural first beachhead for diffusion models, because images exist in a continuous pixel space that’s easy to add Gaussian noise to (literally, basic addition).
Unfortunately, text does not enjoy this same luxury. Text is discrete. You cannot simply add or subtract a letter to a word and expect that the meaning is “roughly the same.” A pixel plus a small random number is pretty similar to the original pixel; a word plus or minus a letter is nonsense. Just ask my editor.
As such, it took several years of additional research to figure out how to get diffusion models working for text. And this is undoubtedly part of why autoregressive models have become the standard, despite their naive simplicity. All things considered, we would prefer nuanced, efficient methods like diffusion…but when ChatGPT was originally released in 2022, nobody knew how to make them work.
Token masks to approximate Gaussian noise
In fact, one year prior to the ChatGPT release (2021) was when early inklings of a solution for text diffusion were forming. In Structured Denoising Diffusion Models in Discrete State-SpacesStructured Denoising Diffusion Models in Discrete State-SpacesDenoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matrices that introduce absorbing states. The third allows us to draw a connection between diffusion models and autoregressive and mask-based generative models. We show that the choice of transition matrix is an important design decision that leads to improved results in image and text domains. We also introduce a new loss function that combines the variational lower bound with an auxiliary cross entropy loss. For text, this model class achieves strong results on character-level text generation while scaling to large vocabularies on LM1B. On the image dataset CIFAR-10, our models approach the sample quality and exceed the log-likelihood of the continuous-space DDPM model.arXiv:2107.03006v3View paper, Austin et al. proposed an interesting idea: what if you could add noise to discrete spaces through masks? A mask simply obscures an individual token in a sequence, thereby messing with the semantic meaning of the overall sequence.
In my head at least – this explanation is not researcher-approved – I think of this solution as adding noise at the sequence level instead of the token level. Yes, you can’t just change a word, but you can remove a word in a sentence, which changes the semantic meaning of said sentence.
SEDD and a loss function for discrete diffusion
With the mask idea in place, further work could progress on making discrete diffusion possible. One major outstanding issue was figuring out the right loss function. Diffusion models operating in continuous spaces made use of score matching. In a pixel space, you’d train the model to figure out which “direction” to go in to get to a higher fidelity, less noisy pixel. You can think of diffusion models as learning a vector field over the data space at every noise level that points toward higher probability regions. At inference, you're integrating that vector field from pure noise back to the data manifold.
Unfortunately, this doesn’t really work for discrete spaces. If the score points you in the right “nearby direction” then you need the data to live on in a continuous space so that "nearby" and "direction" actually mean something. Discrete tokens don’t live in that space, there is no continuity between “cat” and “dog” (sorry 90’s kids), and you cannot nudge a token slightly in a direction.
Thankfully in 2023, a paperDiscrete Diffusion Modeling by Estimating the Ratios of the Data DistributionDespite their groundbreaking performance for many generative modeling tasks, diffusion models have fallen short on discrete data domains such as natural language. Crucially, standard diffusion models rely on the well-established theory of score matching, but efforts to generalize this to discrete structures have not yielded the same empirical gains. In this work, we bridge this gap by proposing score entropy, a novel loss that naturally extends score matching to discrete spaces, integrates seamlessly to build discrete diffusion models, and significantly boosts performance. Experimentally, we test our Score Entropy Discrete Diffusion models (SEDD) on standard language modeling tasks. For comparable model sizes, SEDD beats existing language diffusion paradigms (reducing perplexity by $25$-$75$\%) and is competitive with autoregressive models, in particular outperforming GPT-2. Furthermore, compared to autoregressive mdoels, SEDD generates faithful text without requiring distribution annealing techniques like temperature scaling (around $6$-$8\times$ better generative perplexity than un-annealed GPT-2), can trade compute and quality (similar quality with $32\times$ fewer network evaluations), and enables controllable infilling (matching nucleus sampling quality while enabling other strategies besides left to right prompting).arXiv:2310.16834v3View paper (SEDD) from Stefano’s lab made a breakthrough
by proposing score entropy, a novel loss that naturally extends score matching to discrete spaces, integrates seamlessly to build discrete diffusion models, and significantly boosts performance.
SEDD's contribution was defining a score-like quantity that's native to discrete spaces — the ratio of probabilities between states rather than a gradient. Instead of “which direction increases probability,” you ask “how much more probable is token A than token B at this position.” It won best paper at ICML 2024.
With SEDD, the field was finally in a place where diffusion for text was becoming possible. So Stefano, Aditya (now a professor at UCLA), and Volodymyr (now a professor at Cornell) got together in 2024 to discuss what was next. They believed that dLLMs were the future, so the next natural step was to try and train one at a competitive size to existing LLMs…but how? Despite Stanford being incredibly well funded for a university, the amount of compute required to train a meaningfully-sized dLLM was not within reach (the largest they had gotten to was 1B parameters). So they decided to start a company.
Inception announced their $50M seed in 2025 along with their flagship model, Mercury. Thanks to diffusion, Mercury was up to 10X faster and more efficient than similar quality SOTA LLMs.
When speed is speed but also speed is quality
It’s worth taking a bit of a detour to explain why speed matters so much for LLMs. Yea, 10x speedup is a fancy headline, and as a user you kind of do want your models to feel at least semi-instantaneous. But there is another important consumer of LLM responses, and that is LLMs themselves; particularly in an RL context.
Increasing amounts, perhaps even the majority, of progress in LLMs over the last couple of years is likely due to RL in post-training. Last year I wrote about how RL has unique infrastructure challenges (avoiding idle GPU cycles) since the model is doing training and inference at the same time. Today’s RL training pipelines spend many more times the cycles on inference (generation) as they do on training. If you can flip that equation, thanks to incredible speed from a dLLM, you have completely changed the RL scaling law. It’s a strange thing to say, but in a way speed becomes quality.
Alongside Inception, open source and academic work continues in the dLLM space. Early last year, a working group across several Chinese universities released LLaDALarge Language Diffusion ModelsThe capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml-gsai.github.io/LLaDA-demo/.arXiv:2502.09992v3View paper, a dLLM trained from scratch implementing all of the above concepts (masking, etc.). A few months later came Dream 7BDream 7B: Diffusion Large Language ModelsWe introduce Dream 7B, the most powerful open diffusion large language model to date. Unlike autoregressive (AR) models that generate tokens sequentially, Dream 7B employs discrete diffusion modeling to refine sequences in parallel through iterative denoising. Our model consistently outperforms existing diffusion language models on general, mathematical, and coding tasks. Dream 7B demonstrates superior planning abilities and inference flexibility, including arbitrary-order generation, infilling capabilities, and tunable quality-speed trade-offs. These results are achieved through simple yet effective training techniques, including AR-based LLM initialization and context-adaptive token-level noise rescheduling. We release both Dream-Base and Dream-Instruct to facilitate further research in diffusion-based language modeling.arXiv:2508.15487v1View paper, which wasn’t quite trained from scratch but instead was built on an existing AR LLM foundation (similar work was done to create DiffuGPT and DiffuLLaMAScaling Diffusion Language Models via Adaptation from Autoregressive ModelsDiffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale compared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR language models, we propose adapting these models to build text diffusion models. We demonstrate connections between AR and diffusion modeling objectives and introduce a simple continual pre-training approach for training diffusion models. Through systematic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts. We release a suite of DLMs (127M-355M-7B) capable of generating fluent text, performing in-context learning, filling in the middle without prompt re-ordering, and following instructions https://github.com/HKUNLP/DiffuLLaMA.arXiv:2410.17891v3View paper). These models are promising starts but undoubtedly not ready or in use for real production use cases.
Taking text diffusion models from academic to production
There is, as many readers know first hand, a long long space between ideas working in papers and ideas working in production. dLLMs had many years of catching up to do with autoregressive models, where thousands of researchers had been pounding away on improving every single part of the training and inference process for 5 years now. In particular there were 3 areas that needed improving before these things were ready for prod:
1) A caching system for diffusion models
Despite autoregressive models being relatively architecturally primitive, they can get pretty fast thanks to a nice property of AR: the ability to cache. Specifically, the KV cache allows you to generate each token once and then store it for efficient retrieval; without it, AR models would need to regenerate the entire token sequence on each generation step and thus be painfully slow.
Because diffusion models have bidirectional attention (i.e. every token can attend to all others in the sequence, you get past and future context), developing an efficient caching system is nontrivial. There are a few ideas in flight:
- Block DiffusionBlock Diffusion: Interpolating Between Autoregressive and Diffusion Language ModelsDiffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lmsarXiv:2503.09573v3View paper, which Volodymyr was part of, takes pieces from both diffusion and AR models and allows for a KV cache (with the associated efficiencies).
- Fast-dLLMFast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel DecodingDiffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities. However, the practical inference speed of open-sourced Diffusion LLMs often lags behind autoregressive models due to the lack of Key-Value (KV) Cache and quality degradation when decoding multiple tokens simultaneously. To bridge this gap, we introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop. Additionally, we identify the root cause of generation quality degradation in parallel decoding as the disruption of token dependencies under the conditional independence assumption. To address this, we propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality. Experimental results on LLaDA and Dream models across multiple LLM benchmarks demonstrate up to \textbf{27.6$\times$ throughput} improvement with minimal accuracy loss, closing the performance gap with autoregressive models and paving the way for practical deployment of Diffusion LLMs.arXiv:2505.22618v3View paper builds on this hybrid AR / diffusion method where blocks of text are cached, but each individual block is generated via diffusion.
- DPadDPad: Efficient Diffusion Language Models with Suffix DropoutDiffusion-based Large Language Models (dLLMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens at each step while retaining only a small fraction. We propose Diffusion Scratchpad (DPad), a training-free method that restricts attention to a small set of nearby suffix tokens, preserving fidelity while eliminating redundancy. DPad integrates two strategies: (i) a sliding window, which maintains a fixed-length suffix window, and (ii) distance-decay dropout, which deterministically removes distant suffix tokens before attention computation. This simple design is compatible with existing optimizations such as prefix caching and can be implemented with only a few lines of code. Comprehensive evaluations across multiple benchmarks on LLaDA-1.5 and Dream models demonstrate that DPad delivers up to $\mathbf{61.4\times}$ speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference. Our code is available at https://github.com/Crys-Chen/DPad.arXiv:2508.14148v2View paper proposes a diffusion scratchpad that restricts attention to a small set of nearby suffix tokens via a sliding window and distance-decay dropout.
- dKV-CachedKV-Cache: The Cache for Diffusion Language ModelsDiffusion Language Models (DLMs) have been seen as a promising competitor for autoregressive language models. However, diffusion language models have long been constrained by slow inference. A core challenge is that their non-autoregressive architecture and bidirectional attention preclude the key-value cache that accelerates decoding. We address this bottleneck by proposing a KV-cache-like mechanism, delayed KV-Cache, for the denoising process of DLMs. Our approach is motivated by the observation that different tokens have distinct representation dynamics throughout the diffusion process. Accordingly, we propose a delayed and conditioned caching strategy for key and value states. We design two complementary variants to cache key and value step-by-step: (1) dKV-Cache-Decode, which provides almost lossless acceleration, and even improves performance on long sequences, suggesting that existing DLMs may under-utilise contextual information during inference. (2) dKV-Cache-Greedy, which has aggressive caching with reduced lifespan, achieving higher speed-ups with quadratic time complexity at the cost of some performance degradation. dKV-Cache, in final, achieves from 2-10x speedup in inference, largely narrowing the gap between ARs and DLMs. We evaluate our dKV-Cache on several benchmarks, delivering acceleration across general language understanding, mathematical, and code-generation benchmarks. Experiments demonstrate that cache can also be used in DLMs, even in a training-free manner from current DLMs.arXiv:2505.15781v1View paper develops a couple of ideas, primary of which is a delayed caching strategy, wherein only the key and value states of decoded tokens gets cached.
Of these Fast-dLLM is the most widely adopted. A follow paper (Fast-dLLM v2Fast-dLLM v2: Efficient Block-Diffusion LLMAutoregressive (AR) large language models (LLMs) have achieved remarkable performance across a wide range of natural language tasks, yet their inherent sequential decoding limits inference efficiency. In this work, we propose Fast-dLLM v2, a carefully designed block diffusion language model (dLLM) that efficiently adapts pretrained AR models into dLLMs for parallel text generation, requiring only approximately 1B tokens of fine-tuning. This represents a 500x reduction in training data compared to full-attention diffusion LLMs such as Dream (580B tokens), while preserving the original model's performance. Our approach introduces a novel training recipe that combines a block diffusion mechanism with a complementary attention mask, enabling blockwise bidirectional context modeling without sacrificing AR training objectives. To further accelerate decoding, we design a hierarchical caching mechanism: a block-level cache that stores historical context representations across blocks, and a sub-block cache that enables efficient parallel generation within partially decoded blocks. Coupled with our parallel decoding pipeline, Fast-dLLM v2 achieves up to 2.5x speedup over standard AR decoding without compromising generation quality. Extensive experiments across diverse benchmarks demonstrate that Fast-dLLM v2 matches or surpasses AR baselines in accuracy, while delivering state-of-the-art efficiency among dLLMs - marking a significant step toward the practical deployment of fast and accurate LLMs. Code and model will be publicly released.arXiv:2509.26328v1View paper) was released late last year that turns this idea into a full scale model, adapting an existing open source AR LLM into a diffusion model using this block caching strategy.
2) Parallel decoding
In order to have performance gains of AR models in practice, diffusion models must unmask more than one token per step (duh). But tokens are not statistically independent, and thus historically parallel generation of multiple tokens at once has significantly degraded quality.
At NeurIPS in 2025, Aditya’s new group at UCLA proposed a framework for balancing these constraints called APD (adaptive parallel decoding). APD dynamically adjusts the number of tokens sampled in parallel based on generated probabilities from the model and a small “sidecar” autoregressive model that can compute the likelihood of sequences in parallel.
Another approach is DAWNDAWN: Dependency-Aware Fast Inference for Diffusion LLMsDiffusion large language models (dLLMs) have shown advantages in text generation, particularly due to their inherent ability for parallel decoding. However, constrained by the quality--speed trade-off, existing inference solutions adopt conservative parallel strategies, leaving substantial efficiency potential underexplored. A core challenge is that parallel decoding assumes each position can be filled independently, but tokens are often semantically coupled. Thus, the correct choice at one position constrains valid choices at others. Without modeling these inter-token dependencies, parallel strategies produce deteriorated outputs. Motivated by this insight, we propose DAWN, a training-free, dependency-aware decoding method for fast dLLM inference. DAWN extracts token dependencies and leverages two key motivations: (1) positions dependent on unmasked certain positions become more reliable, (2) simultaneously unmasking strongly coupled uncertain positions induces errors. Given those findings, DAWN leverages a dependency graph to select more reliable unmasking positions at each iteration, achieving high parallelism with negligible loss in generation quality. Extensive experiments across multiple models and datasets demonstrate that DAWN speedups the inference by 1.80-8.06x over baselines while preserving the generation quality. Code is released at https://github.com/lizhuo-luo/DAWN.arXiv:2602.06953v1View paper from Luo et al. This idea explicitly builds an inter-token dependency graph that allows the model to select more reliable tokens to unmask in parallel.
3) Reasoning and test time scaling
Pre-trained base models are promising, but not consumer-ready: to actually get used dLLMs will need to be post-trained like their AR counterparts. Do dLLMs respond to techniques developed for AR like SFT and RL? Can dLLMs reason the way SOTA LLMs today can?
Mid-last year Aditya’s lab at UCLA released an interesting paperd1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement LearningRecent large language models (LLMs) have demonstrated strong reasoning capabilities that benefits from online reinforcement learning (RL). These capabilities have primarily been demonstrated within the left-to-right autoregressive (AR) generation paradigm. In contrast, non-autoregressive paradigms based on diffusion generate text in a coarse-to-fine manner. Although recent diffusion-based large language models (dLLMs) have achieved competitive language modeling performance compared to their AR counterparts, it remains unclear if dLLMs can also leverage recent advances in LLM reasoning. To this end, we propose d1, a framework to adapt pre-trained masked dLLMs into reasoning models via a combination of supervised finetuning (SFT) and RL. Specifically, we develop and extend techniques to improve reasoning in pretrained dLLMs: (a) we utilize a masked SFT technique to distill knowledge and instill self-improvement behavior directly from existing datasets, and (b) we introduce a novel critic-free, policy-gradient based RL algorithm called diffu-GRPO, the first integration of policy gradient methods to masked dLLMs. Through empirical studies, we investigate the performance of different post-training recipes on multiple mathematical and planning benchmarks. We find that d1 yields the best performance and significantly improves performance of a state-of-the-art dLLM. Our code is released at https://dllm-reasoning.github.io/.arXiv:2504.12216v2View paper in this space proposing d1, a framework for bringing the post-training techniques we know and love from AR models to dLLMs. There are two major innovations here:
- They take an existing SFT dataset (S1ks1: Simple test-time scalingTest-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset s1K of 1,000 questions paired with reasoning traces relying on three criteria we validate through ablations: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time compute by forcefully terminating the model's thinking process or lengthening it by appending "Wait" multiple times to the model's generation when it tries to end. This can lead the model to double-check its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1-32B exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). Further, scaling s1-32B with budget forcing allows extrapolating beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at https://github.com/simplescaling/s1arXiv:2501.19393v3View paper) and adapt it to use masking, ergo allowing it to be used as part of dLLM training.
- They develop diffu-GRPO, a policy-gradient RL algorithm applied specifically to masked dLLMs.
Both of these techniques show promising results over a LLaDA base model, independently and in tandem (indicating a synergistic effect). Interestingly, despite the fact that the diffu-GRPO training process used a fixed sequence length of 256 tokens, they observed performance gains at other lengths too (128 and 512), which suggests that the model isn’t just overfitting, it actually learned some sort of general reasoning.
Another important trend that has driven gains for AR LLMs is inference-time scaling. Longer reasoning chains have demonstrated better performance and are now standard in models like Opus 4.6. Can diffusion models benefit from inference-time scaling as well?
Also out of Aditya’s lab, Reflect-DiTReflect-DiT: Inference-Time Scaling for Text-to-Image Diffusion Transformers via In-Context ReflectionThe predominant approach to advancing text-to-image generation has been training-time scaling, where larger models are trained on more data using greater computational resources. While effective, this approach is computationally expensive, leading to growing interest in inference-time scaling to improve performance. Currently, inference-time scaling for text-to-image diffusion models is largely limited to best-of-N sampling, where multiple images are generated per prompt and a selection model chooses the best output. Inspired by the recent success of reasoning models like DeepSeek-R1 in the language domain, we introduce an alternative to naive best-of-N sampling by equipping text-to-image Diffusion Transformers with in-context reflection capabilities. We propose Reflect-DiT, a method that enables Diffusion Transformers to refine their generations using in-context examples of previously generated images alongside textual feedback describing necessary improvements. Instead of passively relying on random sampling and hoping for a better result in a future generation, Reflect-DiT explicitly tailors its generations to address specific aspects requiring enhancement. Experimental results demonstrate that Reflect-DiT improves performance on the GenEval benchmark (+0.19) using SANA-1.0-1.6B as a base model. Additionally, it achieves a new state-of-the-art score of 0.81 on GenEval while generating only 20 samples per prompt, surpassing the previous best score of 0.80, which was obtained using a significantly larger model (SANA-1.5-4.8B) with 2048 samples under the best-of-N approach.arXiv:2503.12271v1View paper explores how to scale inference compute for diffusion models (particularly with images in this context). The idea is that instead of generating N images and picking the best, we can equip diffusion transformers with in-context reflection: the model sees its previous attempts alongside textual feedback and refines its generation accordingly, inspired by DeepSeek-R1's reasoning paradigm.
Inception, Mercury 2, and dLLMs of the future
For now, Mercury 2 is closed source, so we don’t know exactly what’s going on behind the scenes to make it work. To achieve the results it has (and be commercially viable) there is evidently much more going on than just a large pre-trained dLLM base model. It’s possible the company will release more information in the future, but it’s a black box for now.
But the funny thing about companies founded by academics is that for at least the first few years, even black boxes aren’t necessarily that opaque. Publishing leaves a paper trail (literally). And given how prolific the Inception cofounders have been on diffusion topics, I’d venture we can guess that many of their ideas over the last 5 years have probably made it into Mercury: d1, APD, block diffusion, etc.
And then there’s the inference question. Any company providing blazing fast inference must have their kernel writer(s) or at least the labor equivalent. The Mercury paper talks about dynamic batching and paging, which are their own can of worms to figure out for diffusion models. Note that Mercury itself is still a Transformer architecture, despite the fact that it was trained via diffusion.
What we do know is that Mercury 2 is fast as hell and that if there was ever a team that was going to figure out dLLMs, it’s probably this one. You can try Mercury 2 for free here, and get in touch with Sales if you’re interested for your company. And if this work is interesting to you, Inception is hiring.
