WeDLM (wedlm.github.io, developed by Tencent) is an open-source diffusion language model (dLLM) framework. Unlike traditional diffusion language models that rely on bidirectional attention (which breaks key-value caching and fails to beat optimized autoregressive engines), WeDLM reconciles diffusion generation with standard causal attention. By introducing Topological Reordering and Streaming Parallel Decoding, it provides full compatibility with standard LLM serving optimizations (FlashAttention, PagedAttention, vLLM, and native KV caching) while achieving up to 3–6× wall-clock speedups over vLLM-optimized autoregressive baselines without compromising output quality.

Official WebsiteHugging FaceGitHubWeDLM-8B-Instruct

Diffusion Language Models (dLLMs) hold the promise of parallel token generation, but their conventional reliance on bidirectional attention prevents efficient prefix caching and causes costly pipeline “stop-and-wait” bottlenecks. As a result, theoretical parallelization gains have rarely translated into real-world wall-clock speedups over mature autoregressive (AR) serving engines like vLLM. WeDLM resolves this structural dilemma by performing parallel mask recovery entirely under a standard causal mask.

Core Technical Architecture & Profile at a Glance

Feature / Dimension	WeDLM Technical Specification	Core Serving & Inference Advantage
Attention Mechanism	Standard Causal Attention (Lower-Triangular Mask)	Enables native prefix KV caching and avoids bidirectional attention overhead.
Architectural Bridging	Topological Reordering (Physical vs. Logical Decoupling)	Moves clean observed tokens to the physical prefix while maintaining RoPE logical positions.
Decoding Strategy	Streaming Parallel Decoding (Dynamic Sliding Window)	Continuously commits confident tokens without “stop-and-wait” block bubbles.
Speedup vs. vLLM	3–6× on Math/Code, up to 10× on sequential tasks	Outperforms production-grade vLLM autoregressive baselines in real wall-clock latency.
Base Model Lineage	Initialized from pre-trained AR checkpoints (e.g., Qwen)	Seamless continued pre-training directly from mature open-weight models.
Inference Stack Support	FlashAttention, PagedAttention, CUDA Graphs, chatllm.cpp	Drops directly into production inference infrastructure without custom kernels.

Key Capabilities and Technical Innovations

• Topological Reordering: Decouples logical semantic positions (handled by Rotary Position Embeddings / RoPE) from physical token order in memory. Observed/clean tokens are placed in the physical prefix, allowing masked noisy tokens to attend causally to known context while preserving strict causal attention properties.

• Native Prefix-Cache Compatibility: Because KV states depend strictly on committed causal prefixes, newly resolved tokens are instantly cached into standard KV memory buffers without recomputation or invalidating previous tokens.

• Streaming Parallel Decoding: Replaces rigid block-by-block diffusion with a dynamic sliding window. Confident tokens are asynchronously committed to the growing prefix while the model continuously refills new masks, maintaining maximum GPU compute saturation with zero pipeline bubbles.

• Low-Entropy Task Acceleration: Delivers significant acceleration on structured, low-entropy tasks such as mathematical reasoning (GSM8K, MATH) and code generation, while maintaining high generation quality across standard instruction benchmarks.