IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_router_gate_mixed_gemv

struct Struct_router_gate_mixed_gemv

MOGG wrapper for the mixed-input router-gate GEMV.

Computes c = a @ b^T for a bf16 activation a and an fp32 or bf16 router weight b, returning fp32 c. M is runtime-dynamic (one graph serves both decode and prefill), so the op branches on runtime M:

  • M == 0 (graph-capture warmup): no launch.
  • tiny M (M <= 64 on MI355X, M <= 128 on SM100; decode): the fused mixed GEMV — a is dotted against the weight b in a SINGLE launch, fusing away the standalone bf16->fp32 activation cast that otherwise precedes the router GEMV. With an fp32 b the activation widens to fp32 in registers; with a bf16 b both operands feed the fused bf16 pair-dot (v_dot2_f32_bf16), which accumulates in fp32.
  • large M (prefill): the fused GEMV is catastrophically slow. On SM100 from M = 8704 the op runs router_gate_tf32_gemm, one bf16 tensor-core GEMM that reads a once; elsewhere it falls back to the baseline chain — with a bf16 b, one bf16 x bf16 -> fp32 matmul; with an fp32 b, cast a to fp32 then an ordinary fp32 matmul (two launches, matching baseline). precise_max_m keeps that matmul on the IEEE-fp32 split-K GEMV up to that row count instead of crossing to the TF32 tile GEMM above 64 rows, so a caller can make the launch fp32-precise up to the batch it actually serves and leave larger (prefill) batches on the baseline's speed.

Because bf16->fp32 widening is lossless, every route is fp32 arithmetic over exactly the widened bf16 values: at the same launch config the fused GEMV is bit-identical to the gemv_split_k of the same operand dtypes over a pre-cast activation. The MI355X decode bands with act_wide_k reorder the K summation (each lane folds 8 consecutive K through a SIMD accumulator), so those bands match the fp32 GEMV only to fp32 rounding, not bit-for-bit. On SM100 the baseline's fp32 matmul switches to TF32 tensor cores above M = 64, so the fused GEMV's IEEE-fp32 result there is the more precise of the two, not the same value.

A caller whose own fp32 chain keeps the split-K GEMV above M = 64 is unaffected, because both routes reduce identically. A caller that needs the fused route to stay bit-identical to the chain above that edge must cap it with max_m: Mistral Large 4's router is such a caller, and passes the 64 its chain's crossover sits at.

A caller that needs one consistent precision up to some row count — not just bit-identity with a chain that itself changes precision at 64 rows — sets precise_max_m. Up to that count the fused window then only chooses between one launch and two, both of which compute the same IEEE-fp32 product, so a row's result no longer depends on how many rows share the launch. That is Mistral Large 4's router: with speculative decoding a verify step and a context step run different row counts, and a precision change between them made a served answer depend on the scheduler's step mix. Past that count the route passes matmul its default use_tf32=True: the baseline's own call, at every row count. At 64 rows and below both routes run the IEEE split-K GEMV regardless of use_tf32, which only separates TF32 from IEEE above 64 rows, and there the route and the baseline cross together. The route is also fp32-weight only: a bf16 weight takes the bf16 matmul arm below before this one is reached, so a precise_max_m set for it has no effect at any row count.

This is not part of generic matmul/GEMV dispatch. The MiniMax-M3 router emits this explicit graph op on MI355X and SM100, and the Kimi K3 router on gfx950 (MI355X); other architectures retain the standard router path.

N (expert count) and K (hidden size) are read from the weight b's static [N, K] layout at compile time — the op carries no redundant N/K custom-op parameters. N still selects the kernel's check_bounds_n guard.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[max_m: Int = Int(128), precise_max_m: Int = Int(0)](c: T, a: T, b: T, context: DeviceContext)

Executes the mixed bf16-A router-gate GEMV with an M-based route to the baseline matmuls for large (prefill) M.

Constraints:

The weight b must have a static [N, K] shape (both dims known at compile time).

Parameters:

  • ​max_m (Int): The caller's cap on the fused launch's row count -- the largest M the fused path serves, since router_gate_use_mixed_gemv takes it as the window's last row. The effective window is the smaller of this and the target's tuned one (64 on MI355X, 128 on SM100), so the default of 128 leaves every target at its own window. A graph staged before this parameter existed carries no max_m attribute and binds to that same default.
  • ​precise_max_m (Int): The last row count whose above-window fp32 matmul stays on the IEEE-fp32 split-K GEMV instead of truncating its operands to TF32 on tensor cores. 0, the default, is the baseline at every row count. A positive value bounds the precise route: the row count is runtime-dynamic, so both matmuls are staged and M picks one per launch. It requires the shape that dispatch supports (N at most 256, K at least 2048 and a multiple of the fp32 SIMD width) and SM100's tcgen05 tensor cores; the op is a compile error otherwise. It also requires an fp32 weight: the op's bf16 matmul arm runs first for a bf16 weight and leaves this value unused.

Args:

  • ​c (T): Output [M, N] fp32 tensor.
  • ​a (T): Activation [M, K] bf16 tensor.
  • ​b (T): Weight [N, K] fp32 or bf16 tensor (transpose_b layout).
  • ​context (DeviceContext): The device context.

Was this page helpful?