DataSculpting.AI presents Profile-Guided MoE Compilation (PGMC)

Joint Trajectory-Aware Compilation of Frozen Mixture-of-Experts Models

Abstract

1. Introduction

Sparse MoE architectures increase parameter capacity while evaluating only a subset of experts for each token. A conventional routed layer can be simplified as

where \(h_l\) is the hidden representation entering layer \(l\), \(g_l\) is the router, \(E_{l,e}\) is expert \(e\), and \(p_{l,e}\) is the router weight assigned to that expert. Models such as Mixtral and DeepSeekMoE illustrate the practical importance of sparse expert routing, while OLMoE and OpenMoE provide evidence that expert routing exhibits measurable specialization and structure.

Sparse arithmetic does not eliminate the cost of moving sparse parameters. When all experts cannot remain resident in fast device memory, a token's route may induce CPU–GPU transfers, NVMe reads, inter-GPU communication, cache evictions, synchronization, or some combination of these. As a result, the systems problem is not simply “which expert should run?” but also “where should that expert live, when should it be transferred, what should remain cached, and how should future likely paths affect those choices?”

Existing work has addressed many of these questions individually or in partial combinations. PGMC does not claim those mechanisms as new. Its proposed contribution is to treat the frozen model's observed routing trajectories as a shared optimization substrate for all of them.

The decisive benchmark is therefore not PGMC versus an unoptimized MoE. It is PGMC versus the strongest practical composition of the same constituent techniques when those techniques are optimized independently.

2. Related Work

2.1 Sparse MoE architectures and routing structure

Mixtral 8x7B uses eight feed-forward experts per layer and routes each token to two experts. DeepSeekMoE pursues finer-grained expert specialization and separates shared experts from routed experts. OLMoE provides an open MoE model, code, data, logs, and analysis reporting high expert specialization. OpenMoE reports token-related routing regularities, including context-independent specialization and routing patterns that stabilize early in training. These systems establish that routing is neither arbitrary nor uniformly distributed, creating the possibility of exploitable structure in frozen-model execution traces.

2.2 Predictive expert selection and prefetch

SiDA-MoE uses data-aware serving to exploit sparse activation under constrained GPU memory. Read-ME decouples routing from the MoE backbone to enable pre-computation and lookahead scheduling for expert-aware batching and caching. ExpertFlow uses predictive routing paths, expert caching, and token scheduling to reduce expert-transfer overhead. A later ExpertFlow variant further studies adaptive prediction horizons and cache-aware prefetch behavior. These systems demonstrate that future expert usage can be predicted or anticipated sufficiently well to improve serving performance in at least some regimes.

2.3 Expert placement and cross-layer dependence

MoETuner formulates expert placement as an optimization problem that jointly considers token load, communication, and computation cost. Its design explicitly exploits routing dependency across adjacent layers. This is close to one component of PGMC and materially limits any novelty claim based solely on cross-layer expert affinity or placement optimization.

2.4 Profile-guided physical reorganization

The open-source mbolt project directly applies profile-guided optimization to MoE model files: it records expert-routing traces, reorganizes expert bytes so frequently co-activated experts are physically colocated, and uses merged reads to reduce storage operations while preserving the model's logical weights. This is direct prior art for profile-guided physical expert reordering. PGMC therefore does not claim physical permutation of frozen experts as a standalone contribution.

2.5 Hardware-aware expert caching

Apple's SpecMD studies speculative expert prefetching and caching under realistic hardware constraints. Its reported results show that common temporal-locality assumptions such as LRU and LFU can be poor models of MoE expert access, and it introduces a Least-Stale eviction policy based on predictable expert behavior. This supports PGMC's premise that expert management should be optimized against actual routing patterns and hardware cost rather than generic cache heuristics.

2.6 Novelty boundary

Taken individually, the following ideas are established or have close prior art:

Accordingly, PGMC should not be presented as the first profile-guided MoE optimizer or the first expert predictor. Its publishable claim depends on demonstrating an additional systems benefit from joint trajectory-aware compilation.

3. Design

3.1 Frozen models as executable programs

Let the frozen MoE be \(M\). For token \(t\), define its routing trajectory as

where

is the expert set selected at layer \(l\). For top-1 routing this reduces to

PGMC treats the frozen checkpoint as the semantic program and the routing trace as profile data describing how that program is exercised. The checkpoint remains the reference implementation; the compiled representation reorganizes execution around observed behavior.

3.2 Profiling

Let \(D_p\) be a representative profiling corpus. For each routed token, PGMC records at least

Where useful, it also records a compact routing signature derived from the hidden state:

with

Candidate projections include random projection, PCA, product quantization, a small learned projection, or a compact classifier trained only from routing traces. The routing signature does not attempt to preserve the full hidden representation; it only needs to retain information useful for distinguishing routing regimes.

3.3 Compile once, execute many times

Given model \(M\), profile \(D_p\), and target hardware description \(H\), define the compiler:

where \(I_H\) is a hardware-specific compiled inference image containing:

• the original expert weights, possibly physically permuted;
• logical-to-physical expert maps;
• a routing intermediate representation;
• trajectory-cluster metadata;
• candidate-route policies;
• cache/residency guidance;
• prefetch schedules or rules;
• hardware-specific transfer groupings.

The same checkpoint may therefore compile into different images for NVMe-offloaded inference, CPU–GPU systems, single-node multi-GPU systems, or distributed accelerator clusters.

4. Routing Intermediate Representation

PGMC's proposed central artifact is the Routing Intermediate Representation (RIR):

Here \(G\) is an expert transition graph, \(\mathcal{T}\) is a set of routing trajectories or trajectory clusters, \(\mathcal{C}\) represents contextual routing regimes, and \(H\) is the target hardware cost model.

4.1 Expert transition graph

Represent expert instance \(e\) at layer \(l\) as node

For consecutive layers, create an edge

weighted by

A context-conditioned form is

4.2 Multi-layer trajectories

Pairwise transitions may lose structure contained in longer paths. PGMC therefore treats

as a first-class profiling object. Observed paths may be clustered:

This allows the compiler to recognize shared prefixes, branches, recurring route motifs, and candidate bundles spanning multiple layers.

4.3 Contextual routing regimes

The same token can route differently in different contexts. PGMC therefore estimates a trajectory distribution closer to

than to a token-only map. At runtime it may estimate

and use that distribution to prepare likely resources. The native router remains authoritative.

4.4 Hardware cost annotations

For the target hardware \(H\), define

as the relevant cost of moving or preparing resource \(b\) after resource \(a\). Depending on deployment, \(d_H\) may include contiguous-storage distance, PCIe transfer, NVLink communication, NUMA effects, remote-node traffic, device-memory pressure, or transfer concurrency. The RIR therefore describes both likely execution and the cost of serving that execution.

5. Optimization

5.1 Joint objective

Let

denote expert physical placement,

the route-prediction/candidate policy,

the prefetch policy, and

the cache/residency policy. PGMC seeks

A practical cost function may be

where:

• \(T\): time per output token;
• \(B\): bytes transferred;
• \(N\): inter-device or inter-node communication;
• \(W\): wasted speculative transfer;
• \(M\): memory overhead of the compiled metadata and prepared expert set.

5.2 Physical expert permutation

Transformer layers themselves are not generally reorderable. Experts within an MoE layer can often be physically permuted while preserving logical identity. Let

be a permutation of expert storage slots in layer \(l\). The router's logical indices are remapped accordingly. The intended invariant is

within the numerical behavior of the backend. Physical arrangement can then be optimized without intentionally changing model semantics.

5.3 Cost-aware candidate selection

A conventional predictor might choose the most probable future experts:

PGMC instead permits candidate selection to depend on systems cost:

This permits a slightly lower-probability candidate to be included when it is nearly free to fetch because it shares a physical transfer block with higher-probability experts. Conversely, a high-probability expert may be deferred when fetching it early would consume disproportionate bandwidth or evict more valuable state.

5.4 Trajectory-aware placement

A pairwise placement objective is

PGMC proposes the stronger trajectory-aware objective

The distinction matters when several experts form a recurrent path or branch structure whose total transfer behavior cannot be captured well by independent pairwise adjacency.

5.5 Correctness architecture

The initial PGMC runtime is speculative but lossless. The native router computes

PGMC only prepares resources predicted to be useful. A correct prediction hides latency; a partial prediction reduces some latency; an incorrect prediction falls back to the ordinary fetch path. The guiding rule is:

6. Experimental Methodology

6.1 Models

The initial proof of concept should use OLMoE because its weights, code, data, logs, and routing analysis are openly available. A second-stage evaluation should include at least one materially different architecture such as Mixtral and one architecture with finer expert specialization such as DeepSeekMoE.

6.2 Workloads

The profiling corpus \(D_p\) should include distinct workload classes: natural prose, code, mathematics, multilingual text, dialogue, factual questions, and long-context material. A separate \(D_{test}\) must remain unseen during compilation. Evaluation should report both in-distribution and held-out-domain performance.

6.3 First go/no-go experiment: routing predictability

For lookahead distance \(k\), measure

Compare increasingly informative predictors:

If trajectory history and compact state information do not materially improve prediction beyond simple token or previous-expert baselines, the more complex RIR may not be justified.

6.4 Second go/no-go experiment: oracle prefetch ceiling

Record the actual future route, then provide an oracle prefetcher with perfect future expert knowledge. Measure the best achievable reduction in latency and transfer stalls under the target hardware constraints. If perfect foresight yields negligible gain, then route prediction is not the relevant bottleneck and the project should stop before implementing a full compiler.

6.5 Baselines and ablations

The core research claim requires:

under the same model, hardware, memory budget, workload, and correctness constraints. Beating P0 alone is insufficient.

6.6 Primary metrics

Time per output token:

Time to first token:

Throughput:

Expert bytes per token:

Prepared-expert hit rate:

Speculation waste:

Compilation metadata overhead:

6.7 Correctness tests

For the lossless implementation, verify for every token and routed layer:

Compare logits:

Differences should be zero or within the expected numerical variation of the execution backend. Under deterministic execution, generated token sequences should also be compared directly.

6.8 Pre-registered hypotheses

1. H1 — Trajectory information: routing history and compact contextual state predict future experts better than token-only and previous-expert baselines.
2. H2 — Oracle opportunity: perfect future expert knowledge materially improves serving performance in memory-constrained regimes.
3. H3 — Cost-aware prediction: candidate sets optimized for expected execution cost outperform candidate sets optimized only for expert-prediction accuracy.
4. H4 — Joint optimization: PGMC reduces TPOT, BPT, or both relative to P5 under equal resource constraints.
5. H5 — Cross-workload transfer: a profile compiled on \(D_p\) retains measurable benefit on held-out \(D_{test}\).
6. H6 — Semantic preservation: speculative PGMC preserves authoritative expert selections and the output distribution of the reference implementation.

7. Threats to Validity

7.1 Routing may not be predictable far enough ahead

Adjacent-layer dependence does not imply useful long-horizon predictability. If prediction accuracy collapses after one or two layers, there may be insufficient time to hide transfer latency. The proposed \(A(k)\) experiment explicitly measures this.

7.2 Perfect prediction may not matter

The target runtime may already hide most expert-transfer latency through batching, overlap, resident capacity, or bandwidth. The oracle-prefetch ceiling is therefore necessary before investing in a complex predictor.

7.3 Profile overfitting

A compiled image may fit the profiling workload while degrading on different domains. PGMC must therefore report held-out routing distributions, performance on unseen workload classes, and sensitivity to profile size and composition.

7.4 Metadata and wasted prefetch can erase the gain

The RIR, trajectory model, candidate sets, and speculative transfers consume memory and bandwidth. Any reported speedup must include these costs rather than treating the compiler structures as free.

7.5 Hardware dependence

An optimization that helps NVMe-offloaded inference may not help a large multi-GPU system with high-bandwidth interconnect. PGMC should therefore be understood as a hardware-aware compiler, not as a universal static layout.

7.6 Existing systems may already capture most of the available benefit

ExpertFlow, MoETuner, SpecMD, mbolt, and related systems already exploit substantial pieces of the same routing structure. The key risk is that the remaining benefit from joint optimization is too small to justify compiler complexity. That is why P5—the independently optimized composition—is the critical baseline.

7.7 Rapidly evolving prior art

MoE inference optimization is moving quickly. This draft does not claim an exhaustive novelty search. Before submission, related work should be re-audited for new systems combining placement, routing prediction, caching, prefetch, and hardware-aware compilation.

8. Discussion

8.1 Why the compiler framing matters

A trained checkpoint answers one question:

Which learned parameters implement the desired model?

Deployment answers another:

Given that the function is now fixed, what representation of those parameters is cheapest to execute on this machine?

Those are different optimization problems. Compilers routinely transform an executable representation while preserving semantics. PGMC proposes applying the same principle to sparse neural networks.

Under this view:

• the frozen checkpoint is the semantic source program;
• routing traces are profile-guided optimization data;
• the RIR is the compiler intermediate representation;
• expert permutation and placement are layout passes;
• candidate-route prediction is a speculative execution pass;
• cache residency and prefetch are memory-scheduling passes;
• the hardware model supplies target-specific cost.

8.2 Why prediction accuracy is not the final objective

Suppose three future experts have probabilities

A predictor optimized only for accuracy ranks these independently. A compiler can consider that the three may be physically colocated or share a transfer unit. The correct systems objective is therefore not necessarily

It is closer to

This coupling between probabilistic prediction and physical layout is one of the principal areas where PGMC may differ from a pipeline of separately tuned components.

8.3 Future extensions

If the lossless system succeeds, later work could investigate route bundles, dynamic re-compilation from production traces, multiple hardware-specific compiled images, or approximate subgraph execution. The latter should be treated separately because it changes the correctness criterion. One possible bound would be

The first paper should avoid that complication. A lossless systems result is easier to interpret and easier to compare fairly.

8.4 Publication criterion

If yes, PGMC supports the claim that the frozen routing process contains a useful compiler-level structure not fully captured by independent optimizers. If no, the result is still informative: it establishes that the apparent common structure does not yield sufficient additional benefit when optimized jointly.

9. References

[1] Jiang, A. Q., et al. Mixtral of Experts. arXiv:2401.04088, 2024. https://arxiv.org/abs/2401.04088

[2] Dai, D., et al. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. ACL 2024, pp. 1280–1297. https://aclanthology.org/2024.acl-long.70/

[3] Muennighoff, N., et al. OLMoE: Open Mixture-of-Experts Language Models. ICLR 2025. ICLR paper page

[4] Xue, F., et al. OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models. arXiv:2402.01739, 2024. https://arxiv.org/abs/2402.01739

[5] Du, Z., et al. SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models. arXiv:2310.18859, 2023. https://arxiv.org/abs/2310.18859

[6] Cai, R., et al. Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design. NeurIPS 2024. NeurIPS paper page

[7] He, X., et al. ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling. DAC 2026. DOI: 10.1145/3770743.3804292. Implementation and citation

[8] Shen, Z., et al. ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference. arXiv:2510.26730, 2025. https://arxiv.org/abs/2510.26730

[9] Go, S., and Mahajan, D. MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing. arXiv:2502.06643, 2025. https://arxiv.org/abs/2502.06643

[10] Hoang, D., Jaiswal, A., Samragh, M., and Cho, M. SpecMD: A Comprehensive Study on Speculative Expert Prefetching. Apple Machine Learning Research / arXiv:2602.03921, 2026. Apple research page

[11] mbolt — profile-guided layout optimization for MoE model files. Open-source project describing routing-trace-driven physical expert reordering and merged reads. https://github.com/doramirdor/mbolt