For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_router_gate_mixed_gemv
struct Struct_router_gate_mixed_gemv
MOGG wrapper for the mixed-input router-gate GEMV.
Computes c = a @ b^T for a bf16 activation a and an fp32 or bf16
router weight b, returning fp32 c. M is runtime-dynamic (one graph
serves both decode and prefill), so the op branches on runtime M:
M == 0(graph-capture warmup): no launch.- tiny
M(M <= 64on MI355X,M <= 128on SM100; decode): the fused mixed GEMV —ais dotted against the weightbin a SINGLE launch, fusing away the standalone bf16->fp32 activation cast that otherwise precedes the router GEMV. With an fp32bthe activation widens to fp32 in registers; with a bf16bboth operands feed the fused bf16 pair-dot (v_dot2_f32_bf16), which accumulates in fp32. - large
M(prefill): the fused GEMV is catastrophically slow. On SM100 fromM = 8704the op runsrouter_gate_tf32_gemm, one bf16 tensor-core GEMM that readsaonce; elsewhere it falls back to the baseline chain — with a bf16b, one bf16 x bf16 -> fp32 matmul; with an fp32b, castato fp32 then an ordinary fp32 matmul (two launches, matching baseline).precise_max_mkeeps that matmul on the IEEE-fp32 split-K GEMV up to that row count instead of crossing to the TF32 tile GEMM above 64 rows, so a caller can make the launch fp32-precise up to the batch it actually serves and leave larger (prefill) batches on the baseline's speed.
Because bf16->fp32 widening is lossless, every route is fp32 arithmetic
over exactly the widened bf16 values: at the same launch config the fused
GEMV is bit-identical to the gemv_split_k of the same operand dtypes
over a pre-cast activation. The MI355X decode bands with act_wide_k
reorder the K summation (each lane folds 8 consecutive K through a SIMD
accumulator),
so those bands match the fp32 GEMV only to fp32 rounding, not
bit-for-bit. On SM100 the baseline's fp32 matmul switches to TF32 tensor
cores above M = 64, so the fused GEMV's IEEE-fp32 result there is the
more precise of the two, not the same value.
A caller whose own fp32 chain keeps the split-K GEMV above M = 64 is
unaffected, because both routes reduce identically. A caller that needs
the fused route to stay bit-identical to the chain above that edge must
cap it with max_m: Mistral Large 4's router is such a caller, and passes
the 64 its chain's crossover sits at.
A caller that needs one consistent precision up to some row count — not
just bit-identity with a chain that itself changes precision at 64 rows —
sets precise_max_m. Up to that count the fused window then only chooses
between one launch and two, both of which compute the same IEEE-fp32
product, so a row's result no longer depends on how many rows share the
launch. That is Mistral Large 4's router: with speculative decoding a
verify step and a context step run different row counts, and a precision
change between them made a served answer depend on the scheduler's step
mix. Past that count the route passes matmul its default
use_tf32=True: the baseline's own call, at every row count. At 64 rows and
below both routes run the IEEE split-K GEMV regardless of use_tf32, which
only separates TF32 from IEEE above 64 rows, and there the route and the
baseline cross together. The route is also fp32-weight only: a bf16 weight
takes the bf16 matmul arm below before this one is reached, so a
precise_max_m set for it has no effect at any row count.
This is not part of generic matmul/GEMV dispatch. The MiniMax-M3 router emits this explicit graph op on MI355X and SM100, and the Kimi K3 router on gfx950 (MI355X); other architectures retain the standard router path.
N (expert count) and K (hidden size) are read from the weight b's
static [N, K] layout at compile time — the op carries no redundant N/K
custom-op parameters. N still selects the kernel's check_bounds_n guard.
Implemented traits
Methods
execute
static def execute[max_m: Int = Int(128), precise_max_m: Int = Int(0)](c: T, a: T, b: T, context: DeviceContext)
Executes the mixed bf16-A router-gate GEMV with an M-based route to the baseline matmuls for large (prefill) M.
Constraints:
The weight b must have a static [N, K] shape (both dims known at
compile time).
Parameters:
- max_m (
Int): The caller's cap on the fused launch's row count -- the largest M the fused path serves, sincerouter_gate_use_mixed_gemvtakes it as the window's last row. The effective window is the smaller of this and the target's tuned one (64 on MI355X, 128 on SM100), so the default of 128 leaves every target at its own window. A graph staged before this parameter existed carries nomax_mattribute and binds to that same default. - precise_max_m (
Int): The last row count whose above-window fp32 matmul stays on the IEEE-fp32 split-K GEMV instead of truncating its operands to TF32 on tensor cores.0, the default, is the baseline at every row count. A positive value bounds the precise route: the row count is runtime-dynamic, so both matmuls are staged andMpicks one per launch. It requires the shape that dispatch supports (Nat most 256,Kat least 2048 and a multiple of the fp32 SIMD width) and SM100's tcgen05 tensor cores; the op is a compile error otherwise. It also requires an fp32 weight: the op's bf16 matmul arm runs first for a bf16 weight and leaves this value unused.
Args:
- c (
T): Output[M, N]fp32 tensor. - a (
T): Activation[M, K]bf16 tensor. - b (
T): Weight[N, K]fp32 or bf16 tensor (transpose_b layout). - context (
DeviceContext): The device context.