Researchers introduce HOPE (Higher-Order Pruning of Experts), a pruning algorithm designed for Mixture-of-Experts (MoE) language models that explicitly accounts for pairwise cooperation between experts. The authors argue that first-order methods, which score experts independently, can break cooperative groups and suffer at high pruning rates.
Why expert interactions matter
In MoE layers each token is routed to a small subset of experts (typical K=8). The authors observe stable co-activation patterns: experts tend to form cooperative clusters that are consistently selected together. Removing experts based only on individual importance can dismantle these clusters. The HOPE objective introduces pairwise terms that capture average gate-weighted output products between co-activated experts, encoded in an F matrix collected over a calibration set.
HOPE method and reported results
HOPE minimizes a quadratic surrogate that upper-bounds pruning error and yields a binary quadratic program over experts. The F matrix has diagonals that recover single-expert importance (closely matching REAP rankings; Spearman ρ=0.988 reported by the authors) and off-diagonals that capture cooperative strength. The QP is solved via a continuous relaxation and rounded to a prune-set.
Evaluations covered Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, with two calibration datasets and benchmarks including Tulu3 and SWE-Bench Pro. Across conditions HOPE achieved the best average rank (2.07 out of 5 methods) and the highest share of top finishes (39% top-1, 67% top-2). The authors report that HOPE’s advantage grows at aggressive pruning: at 50% pruning the average rank improved to 1.58 versus 2.42 for the next-best method, REAP. On SWE-Bench Pro with Qwen3.5-35B-A3B at 50% pruning, HOPE scored 42.3, the unpruned model scored 41.8, and REAP dropped to 36.2.
Practical use and limitations
The authors provide a three-step package: calibrate (one forward pass to collect F), solve (QP to produce a prune-set), and prune (create the smaller checkpoint). They report that calibration dominates runtime and that solving the QP takes a few seconds per layer. The code was tested on Qwen3, Qwen3-Next, Qwen3.5 and GLM-4.5.
The authors note limitations: HOPE’s benefit is most pronounced at 40–50% pruning; it prunes layers independently and captures pairwise but not higher-order interactions. The paper and code are released by the authors (paper listed as arXiv:2609.18916) for replication and further study.
Original source: Hugging Face Blog