IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_mixed_gemv_epilogue

struct Struct_mixed_gemv_epilogue

MOGG wrapper for the mixed GEMV whose store carries the graph's epilogue.

Computes c = a @ b^T in fp32 for a bf16 activation a and an fp32 weight b, storing every output element through the epilogue the graph fused onto c. moe.router.gate.mixed.gemv does the same arithmetic but writes its result plainly, so it cannot serve a caller whose projection carries elementwise work: that work would become a launch of its own and the cast the fused form removes would be paid for by the chain it leaves behind.

M is runtime-dynamic -- one graph serves both decode and prefill -- so the op branches on runtime M:

  • M == 0 (graph-capture warmup): no launch.
  • tiny M (M <= max_m, at most SM100's window of 64; decode): one fused mixed GEMV, the activation read as bf16 and widened in registers, written through the epilogue.
  • large M (prefill): the chain the caller would otherwise have staged -- the activation widened in a launch of its own, then the fp32 matmul, both storing through that same epilogue. Those two launches and the kernel the matmul picks at that row count are the baseline's.

Because bf16->fp32 widening is lossless, every route is fp32 arithmetic over exactly the widened bf16, so at the default window the result is bit-identical to today's cast + fp32 gemv_split_k_epilogue_fn chain at every row count: inside the window the fused GEMV is that kernel with the cast folded into the load, and above it the op stages that chain. A caller narrowing max_m below the chain's 64-row crossover only shortens the fused window, and the rows it gives up take that staged chain.

Only SM100 (Blackwell) is served, and the gate below says exactly that. Its launch table and the fp32 GEMV this launch must reproduce are that target's, so every other target -- AMD, and NVIDIA targets below SM100, whose fp32 matmuls take a dispatch of their own -- gets an empty window and the staged chain below: today's launches, unchanged.

N and K are read from the weight b's static [N, K] layout at compile time -- the op carries no redundant N/K custom-op parameters.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[max_m: Int = Int(64)](c: T, a: T, b: T, context: DeviceContext)

Executes the mixed bf16-A x fp32-B GEMV through c's epilogue.

Constraints:

The weight b must have a static [N, K] shape (both dims known at compile time).

Parameters:

  • ​max_m (Int): The last row count the fused launch serves. SM100's own window is _MIXED_GEMV_EPILOGUE_MAX_M (64), the row count at which an fp32 matmul of this shape crosses to the TF32 tile GEMM, so the effective window is the smaller of this and 64.

Args:

  • ​c (T): Output [M, N] fp32 tensor; its fused epilogue owns the stores.
  • ​a (T): Activation [M, K] bf16 tensor.
  • ​b (T): Weight [N, K] fp32 tensor (transpose_b layout).
  • ​context (DeviceContext): The device context.

Was this page helpful?