IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

router_gate_mixed_gemv

def router_gate_mixed_gemv[b_type: DType, static_N: Int, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine](c: TileTensor[.float32, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.bfloat16, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine], m: Int, n: Int, k: Int, ctx: DeviceContext)

Launches the mixed bf16-activation router-gate GEMV.

Fuses the standalone bf16→fp32 activation cast into the router GEMV: a is loaded as bf16 and widened to fp32 in registers, then dotted against the weight b (c = a @ b^T) in a single launch. b is fp32 (MiniMax-M3 on MI355X and SM100; the Kimi K3 router's legacy fp32 gate) or bf16 (the K3 checkpoint's own gate storage). MiniMax-M3 uses this path on MI355X and on SM100, and the Kimi K3 router on gfx950.

Because bf16→fp32 widening is lossless and the reduction structure matches gemv_split_k, the result is bit-identical to the gemv_split_k of the same operand dtypes at the same launch config over a pre-cast activation -- except on act_wide_k bands (see the [896, 7168] table above), where each lane owns 8 consecutive K and the row folds a SIMD accumulator once: still fp32 arithmetic over exactly the widened bf16, but a different summation order from the plain GEMV.

The MI355X (gfx950 / CDNA4) launch config is the cache-busting sweep winner for the N=128, K=6144 router-gate shape: simd_width=4, tile_m=1, 128 threads, unroll=2, with tile_n=2 at M<=16 and tile_n=4 above, and the weight kept cache-resident (weight_non_temporal=False) so the per-row-block weight rereads hit L2 — the same cache policy the merged KERN-3219 fp32 router path selects. The wide [896, 7168] router gate instead dispatches through the cold-weight band table _amd_router_gate_896x7168_config: its 92 x 25.7 MB fp32 weights are never cache-resident at decode, so the per-band winners differ (streaming loads at m == 1, static-K staging above).

The SM100 config keeps the 256-thread, simd_width=4 K split of the standard fp32 router dispatch, so every row is bit-identical to that path at the same M <= 64. Above 64 this launch keeps IEEE fp32 arithmetic, while the standard dispatch uses TF32 tensor cores. Larger tile_m * tile_n reuses each weight and activation load across more outputs, which cuts the L2 traffic that bounds this launch (the weight is re-read once per row block, the activation once per weight-row group), at the price of a longer warp-reduce tail; the tiling is the B200 sweep winner per gate shape and M bucket, held in _SM100_ROUTER_GATE_ROWS.

On SM100, b must be initialized before execution and remain unchanged: the kernel prefetches its first weight tile before waiting for dependent grids. A weight produced by a PDL predecessor is unsupported.

Parameters:

  • ​b_type (DType): Weight element type (fp32 or bf16).
  • ​static_N (Int): Static output width (weight rows / expert count). Selects the check_bounds_n guard at compile time.
  • ​c_layout (TensorLayout): Layout of the fp32 output tensor.
  • ​a_layout (TensorLayout): Layout of the bf16 activation tensor.
  • ​b_layout (TensorLayout): Layout of the weight tensor.
  • ​c_engine (TensorEngine): Engine of the fp32 output tensor.
  • ​a_engine (TensorEngine): Engine of the bf16 activation tensor.
  • ​b_engine (TensorEngine): Engine of the weight tensor.

Args:

Was this page helpful?