Mixture of Experts: The Architecture Reshaping Open-Source LLMs

For years, scaling language models meant scaling compute linearly with parameters. A 70B dense model activates all 70 billion weights on every token. That worked until it didn't. Training and serving frontier-scale dense models became prohibitively expensive, and the open-source community started looking for a different trade-off: keep total capacity high, but activate only a fraction of the network per forward pass. That is the promise of Mixture of Experts (MoE). In 2024 and 2025, MoE stopped being a research curiosity and became the default architecture for serious open-weight releases — Mixtral, DeepSeek-V2/V3, OLMoE, Qwen-MoE, and Llama 4 Maverick all route tokens through sparse expert pools instead of monolithic feed-forward blocks. This post walks through how MoE actually works, how the architecture evolved from Google research papers to production open models, and what builders should understand before picking an MoE checkpoint for inference or fine-tuning.

The core idea: decouple capacity from compute

A standard Transformer block has two main pieces: multi-head self-attention and a feed-forward network (FFN). In a dense model, every token passes through the same FFN at every layer. MoE replaces that single FFN with a pool of expert FFNs — smaller, parallel sub-networks — plus a router (also called a gating network) that decides which experts each token should visit.

The router computes a score for each expert given the current hidden state, selects the top-k highest-scoring experts, runs the token through only those k experts, and combines their outputs (typically a weighted sum based on router softmax scores). The rest of the experts sit idle for that token. Total parameter count grows with the number of experts, but active parameters — and therefore FLOPs per token — stay roughly proportional to k, not to the full expert count.

• Total parameters: the sum of all expert weights plus shared layers (attention, embeddings, sometimes shared experts)
• Active parameters: only the weights touched during a single forward pass for one token
• Sparsity ratio: active / total — DeepSeek-V3 activates 37B of 671B (~5.5%); Mixtral 8x7B activates ~13B of ~47B (~28%)
• Expert parallelism: different experts can live on different GPUs, which is how trillion-parameter MoEs get trained and served at all

The mental model that finally clicked for me: MoE is not 'a smaller model that cheats.' It is a large model that only wakes up the sub-networks relevant to the current token. You pay memory to store all experts, but you pay compute like a much smaller dense model.

A brief history: from Sparsely-Gated MoE to Switch Transformers

MoE predates the current LLM wave. Shazeer et al. (2017) introduced Sparsely-Gated Mixture-of-Experts layers, showing that conditional computation could scale model capacity without proportional compute cost. The idea sat largely in research until Google applied it at scale.

GShard (Lepikhin et al., 2021) brought MoE to large Transformer training with top-2 routing and automatic sharding across TPU pods, scaling multilingual translation models beyond 600B parameters. Switch Transformers (Fedus et al., 2021) pushed further with top-1 routing — each token sees exactly one expert — arguing that simpler routing scales better and remains competitive. ST-MoE (Zoph et al., 2022) refined training stability with router z-loss and other regularizers. These papers established the vocabulary we still use today: auxiliary load-balancing loss, expert capacity, token dropping, and the routing collapse problem where a few experts hoard all tokens.

What changed in 2023–2024 is that open-weight labs stopped treating MoE as an exotic scaling trick and started shipping MoE as the primary product architecture. Mistral's Mixtral 8x7B (Jiang et al., 2024) under Apache 2.0 was the inflection point: a model that matched or beat Llama 2 70B on many benchmarks while activating roughly 13B parameters per token, with weights you could actually download and run.

How a modern MoE layer works, step by step

Most open-source MoE LLMs keep the attention stack dense and sparsify only the FFN. Here is the forward pass at one MoE layer for a single token x with hidden dimension d:

1. Attention produces hidden state h (same as dense Transformer)
2. Router computes logits s = W_r · h for each of N experts
3. Softmax over s yields gate probabilities; top-k experts are selected
4. Selected experts each apply their SwiGLU FFN: E_i(h) = SwiGLU_i(h)
5. Outputs are combined: y = Σ g_i · E_i(h), where g_i are normalized gate weights for selected experts
6. Residual connection: output = h + y (layer norm placement varies by model family)

Batching complicates this picture. In a batch of B tokens, each token may route to different experts. Efficient implementations (Megablocks, grouped GEMM kernels in vLLM, DeepSpeed-MoE) group tokens by destination expert so each expert processes a contiguous sub-batch rather than one token at a time. Without these kernels, MoE inference is memory-bandwidth bound and slower than dense models despite fewer active FLOPs.

Architectural innovations in open-source MoE models

1. Fine-grained expert segmentation (DeepSeekMoE)

Classic MoE designs use a handful of large experts — Mixtral has 8 experts per layer, top-2 active. DeepSeekMoE (Dai et al., ACL 2024) argues this limits specialization: each expert is still a big FFN that tends to learn overlapping knowledge. DeepSeek's fix is fine-grained segmentation: split what would be one large FFN into many smaller experts (e.g., 64 routed experts with 8 active), each expert being a fraction of a standard FFN width. With more, smaller experts, the combinatorial space of active expert sets grows, and individual experts can specialize on narrower patterns — syntax, math tokens, code identifiers, multilingual subspaces.

2. Shared experts

DeepSeekMoE also introduces always-on shared experts — one or more FFNs that process every token regardless of routing. Shared experts capture common knowledge (basic grammar, frequent function words, universal representations) so routed experts do not duplicate it. DeepSeek-V2 and V3 extend this pattern; FLAME-MoE (CMU, 2025) uses 2 shared experts alongside 64 routed experts with top-8 gating, explicitly mirroring production DeepSeek and OLMoE designs for reproducible research.

3. Auxiliary-loss-free load balancing (DeepSeek-V3)

Load balancing is the hardest practical problem in MoE training. Without intervention, routers collapse: a few experts get all tokens, others starve, and GPU utilization becomes uneven. The standard fix since Switch Transformers is an auxiliary loss that penalizes imbalanced expert usage. The trade-off is well documented — auxiliary losses stabilize routing but can hurt model quality by adding gradient noise unrelated to the language modeling objective.

DeepSeek-V3 (DeepSeek-AI, 2024) pioneers an auxiliary-loss-free strategy (Wang et al., 2024): per-expert bias terms are added to routing scores and updated based on observed load, outside the main backpropagation path. Experts that receive too few tokens get their bias increased; overloaded experts get bias decreased. DeepSeek reports stable training over 14.8T tokens without irrecoverable loss spikes — a meaningful result for anyone who has watched MoE training runs collapse at scale.

4. Multi-head Latent Attention (MLA)

MoE addresses FFN compute; MLA (introduced in DeepSeek-V2) addresses KV-cache memory. Standard multi-head attention stores full key and value tensors per token per layer, which dominates inference memory at long context. MLA compresses keys and values into a low-rank latent space, dramatically shrinking the KV cache. DeepSeek-V2 reports 93.3% KV cache reduction versus dense baselines. For MoE models targeting 128K+ context, MLA is as important as sparse FFNs for making inference economically viable.

Open-source MoE models worth knowing

Mixtral 8x7B and 8x22B (Mistral AI, 2024)

Mixtral 8x7B (Jiang et al., arXiv:2401.04088) remains the reference implementation for 'classic' sparse MoE: 8 SwiGLU experts per layer, top-2 routing, 32K context, Apache 2.0 license. ~47B total parameters, ~13B active per token. It outperformed Llama 2 70B on math, code, and multilingual benchmarks at a fraction of inference cost. Mixtral 8x22B scaled the same pattern. Mistral also contributed Megablocks integration to vLLM, which mattered as much as the weights for making MoE runnable on consumer and cloud GPU stacks.

DeepSeek-V2, DeepSeek-V3, and DeepSeek-R1 (DeepSeek-AI, 2024–2025)

DeepSeek-V2 (236B total, 21B active, 128K context) combined MLA with DeepSeekMoE fine-grained routing. DeepSeek-V3 pushed to 671B total with 37B active, 256 experts per layer, 8 active, plus shared experts and auxiliary-loss-free balancing — trained for roughly $5.6M in reported GPU cost, a watershed moment for open-weight efficiency. DeepSeek-R1 applied reinforcement learning on top of the V3 base for reasoning, demonstrating that MoE bases fine-tune and RL-scale as well as dense models. These models are MIT licensed and reshaped the competitive landscape for coding and math benchmarks.

OLMoE (Allen AI, 2024)

OLMoE-1B-7B (Muennighoff et al., arXiv:2409.02060) is the most fully open MoE release to date: weights, training data, code, and logs. 7B total parameters, 1.3B active, 64 experts with top-8 routing, trained on 5.1T tokens. OLMoE outperformed all open 1B-active models and matched Llama 2 13B at ~10× lower inference cost. Their controlled ablations are essential reading: dropless token-based routing beats expert-based routing; fine-grained experts beat coarse ones; MoE trains ~2× faster than dense models with equivalent active parameters at the same FLOP budget.

FLAME-MoE (CMU, 2025)

FLAME-MoE (arXiv:2505.20225) fills the gap between one flagship model release and reproducible science. Seven decoder-only models from 38M to 1.7B active parameters, all with 64 experts, top-8 routing, and 2 shared experts. Full training traces, routing logs, and checkpoints enable studies of expert specialization dynamics — the paper shows experts increasingly specialize on distinct token subsets, co-activation matrices stay sparse, and routing stabilizes early in training. If you are researching MoE rather than just deploying it, start here.

Qwen-MoE, Llama 4, and the 2025–2026 wave

Alibaba's Qwen series adopted MoE variants (Qwen1.5-MoE, Qwen2-MoE, Qwen3-MoE) with fine-grained routing and shared experts, competitive with dense Qwen models at lower active-parameter counts. Meta's Llama 4 Maverick uses MoE for its flagship open-weight release. By 2025, the pattern is consistent across Chinese and Western labs: MoE for frontier-scale open models, dense models for smaller edge deployments. The architectural convergence — 64–256 experts, top-4 to top-8 routing, 1–2 shared experts, auxiliary or bias-based load balancing — is striking.

Training dynamics: what the papers teach us

Expert specialization and routing phases

OLMoE and FLAME-MoE both document that experts do not start specialized — they diverge during training. Early training shows high router entropy (experts used roughly evenly); mid-training shows increasing specialization; late training stabilizes co-activation patterns. Mixtral routing analysis reveals temporal locality: consecutive tokens in deeper layers often share the same expert assignment, reflecting syntactic structure. This locality helps batch efficiency but can concentrate load on specific experts for structured inputs.

Load balancing strategies across generations

• Switch Transformer (Fedus et al., 2021): auxiliary load-balancing loss + token dropping when experts exceed capacity
• GShard (Lepikhin et al., 2021): top-2 routing with auxiliary loss; more stable than top-1 at scale
• ST-MoE (Zoph et al., 2022): router z-loss to prevent logit blow-up and training instability
• Mixtral (Jiang et al., 2024): auxiliary loss + dynamic token redistribution + Megablocks sparse kernels
• DeepSeek-V3 (2024): bias-based balancing without auxiliary loss on the main objective
• Expert Choice routing (Zhou et al., 2022): invert the problem — experts choose tokens instead of tokens choosing experts

Scaling laws for MoE vs dense

Recent scaling-law work (Clark et al., 2024; Bi et al., 2024) confirms that MoE and dense models follow different compute-optimal frontiers. At a fixed training FLOP budget, MoE models with more total but fewer active parameters can match or beat dense models — but only when routing stays healthy and expert count is tuned to the target active-parameter regime. OLMoE's finding that MoE trains ~2× faster than dense at equivalent active params is the practical headline; the caveat is you need more total GPU memory to hold all experts and expert-parallel infrastructure to train efficiently.

Inference: where MoE helps and where it hurts

MoE inference has a split personality. On FLOPs, you win: DeepSeek-V3's 37B active params compute like a 37B dense model. On memory, you lose: you must store all 671B parameters (or shard them across GPUs). On latency, results depend on batch size, kernel quality, and expert placement. Small batches with high routing variance underutilize GPUs; large concurrent batches amortize routing overhead.

• Single-user, low-latency serving: dense models often win unless you have optimized MoE kernels (vLLM, TensorRT-LLM, SGLang) and enough GPU memory for expert sharding
• High-throughput API serving: MoE shines — large batches group tokens by expert efficiently
• Fine-tuning: LoRA on MoE is supported (PEFT libraries target expert layers), but full fine-tuning requires expert-parallel training infrastructure
• Quantization: INT4/FP8 on MoE is active research; quantizing 671B total weights is harder than quantizing 37B active compute

For teams deciding between Mixtral 8x7B and a dense Llama 3 8B: if you have the GPU memory for 47B total weights and a serving stack with MoE support, Mixtral delivers 70B-class quality at 13B-class compute. If you are deploying on a single 24GB GPU with llama.cpp, the dense 8B model is the pragmatic choice. MoE is an architecture for scale, not a free lunch on a laptop.

What builders should take away

• MoE decouples model capacity from per-token compute — the defining architectural shift in open-source LLMs since 2024
• The open-source stack is mature: Mixtral (Apache 2.0), DeepSeek (MIT), OLMoE (Apache 2.0), and FLAME-MoE (research) give you weights, papers, and training logs
• Fine-grained experts + shared experts + improved load balancing (DeepSeekMoE lineage) beat classic 8-expert top-2 designs at the same active-parameter budget
• Training MoE requires routing stability tooling; inference requires sparse kernel support — do not treat an MoE checkpoint like a dense model in your infra
• For research, OLMoE and FLAME-MoE are the papers and platforms to cite; for production, DeepSeek-V3 and Mixtral set the current open-weight benchmarks

We deploy both dense SLMs and larger MoE models depending on the job: classify and extract on small dense models in-network; route to a MoE API when the task needs frontier reasoning. The architecture choice is not ideological — it is a memory, latency, and quality trade-off that MoE makes explicit for the first time at open-source scale.

Research papers and further reading

Foundational MoE: Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — arXiv:1701.06538. GShard: Lepikhin et al. (2021), Scaling Giant Models with Conditional Computation and Automatic Sharding — arXiv:2006.16668. Switch Transformers: Fedus et al. (2021), Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — JMLR / arXiv:2101.03961. ST-MoE: Zoph et al. (2022), ST-MoE: Designing Stable and Transferable Sparse Expert Models — arXiv:2202.08906. Expert Choice: Zhou et al. (2022), Mixture-of-Experts with Expert Choice Routing — arXiv:2202.09368.

Modern open-source MoE: Mixtral of Experts — Jiang et al. (2024), arXiv:2401.04088. DeepSeekMoE — Dai et al. (2024), ACL 2024, arXiv:2401.06066, DOI: 10.18653/v1/2024.acl-long.70. DeepSeek-V2 — DeepSeek-AI (2024), arXiv:2405.04434. DeepSeek-V3 — DeepSeek-AI (2024), arXiv:2412.19437. OLMoE — Muennighoff et al. (2024), arXiv:2409.02060. FLAME-MoE — CMU (2025), arXiv:2505.20225.

Load balancing and training stability: Wang et al. (2024), Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts — arXiv:2408.15664. Hugging Face review: NormalUhr (2024), A Review on the Evolvement of Load Balancing Strategy in MoE LLMs — huggingface.co/blog/NormalUhr/moe-balance. Scaling comparisons: Clark et al. (2024), Revisiting MoE and Dense Speed-Accuracy Comparisons — arXiv. Bi et al. (2024), Scaling Laws Across Model Architectures: Dense and MoE Models — arXiv.

Practical guides: Hugging Face (2023), Mixture of Experts Explained — huggingface.co/blog/moe. NVIDIA (2024), Applying Mixture of Experts in LLM Architectures — developer.nvidia.com/blog. Maarten Grootendorst (2024), A Visual Guide to Mixture of Experts — maartengrootendorst.com/blog/moe.