For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
router_gate_mixed_gemv
def router_gate_mixed_gemv[b_type: DType, static_N: Int, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine](c: TileTensor[.float32, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.bfloat16, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine], m: Int, n: Int, k: Int, ctx: DeviceContext)
Launches the mixed bf16-activation router-gate GEMV.
Fuses the standalone bf16→fp32 activation cast into the router GEMV: a is
loaded as bf16 and widened to fp32 in registers, then dotted against the
weight b (c = a @ b^T) in a single launch. b is fp32 (MiniMax-M3 on
MI355X and SM100; the Kimi K3 router's legacy fp32 gate) or bf16 (the K3
checkpoint's own gate storage). MiniMax-M3 uses this path on MI355X and on
SM100, and the Kimi K3 router on gfx950.
Because bf16→fp32 widening is lossless and the reduction structure matches
gemv_split_k, the result is bit-identical to the gemv_split_k of the
same operand dtypes at the same launch config over a pre-cast activation
-- except on act_wide_k bands (see the [896, 7168] table above), where
each lane owns 8 consecutive K and the row folds a SIMD accumulator once:
still fp32 arithmetic over exactly the widened bf16, but a different
summation order from the plain GEMV.
The MI355X (gfx950 / CDNA4) launch config is the cache-busting sweep winner
for the N=128, K=6144 router-gate shape: simd_width=4, tile_m=1, 128 threads, unroll=2, with tile_n=2 at M<=16 and tile_n=4 above,
and the weight kept cache-resident (weight_non_temporal=False) so the
per-row-block weight rereads hit L2 — the same cache policy the merged
KERN-3219 fp32 router path selects. The wide [896, 7168] router gate
instead dispatches through the cold-weight band table
_amd_router_gate_896x7168_config: its 92 x 25.7 MB fp32 weights are
never cache-resident at decode, so the per-band winners differ
(streaming loads at m == 1, static-K staging above).
The SM100 config keeps the 256-thread, simd_width=4 K split of the
standard fp32 router dispatch, so every row is bit-identical to that path
at the same M <= 64. Above 64 this launch keeps IEEE fp32 arithmetic,
while the standard dispatch uses TF32 tensor cores. Larger tile_m * tile_n
reuses each weight and activation load across more outputs, which cuts the
L2 traffic that bounds this launch (the weight is re-read once per row
block, the activation once per weight-row group), at the price of a longer
warp-reduce tail; the tiling is the B200 sweep winner per gate shape and
M bucket, held in _SM100_ROUTER_GATE_ROWS.
On SM100, b must be initialized before execution and remain unchanged:
the kernel prefetches its first weight tile before waiting for dependent
grids. A weight produced by a PDL predecessor is unsupported.
Parameters:
- b_type (
DType): Weight element type (fp32 or bf16). - static_N (
Int): Static output width (weight rows / expert count). Selects thecheck_bounds_nguard at compile time. - c_layout (
TensorLayout): Layout of the fp32 output tensor. - a_layout (
TensorLayout): Layout of the bf16 activation tensor. - b_layout (
TensorLayout): Layout of the weight tensor. - c_engine (
TensorEngine): Engine of the fp32 output tensor. - a_engine (
TensorEngine): Engine of the bf16 activation tensor. - b_engine (
TensorEngine): Engine of the weight tensor.
Args:
- c (
TileTensor[.float32, c_layout, MutAnyOrigin, Engine=c_engine]): Output[M, N]fp32 tensor. - a (
TileTensor[.bfloat16, a_layout, ImmutAnyOrigin, Engine=a_engine]): Activation[M, K]bf16 tensor. - b (
TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine]): Weight[N, K]fp32 or bf16 tensor (transpose_b layout). - m (
Int): Runtime row count (M). The router window isM<=64on MI355X andM<=128on SM100. - n (
Int): Output width (N). - k (
Int): Contraction dim (K). - ctx (
DeviceContext): The device context.