For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_mixed_gemv_epilogue
struct Struct_mixed_gemv_epilogue
MOGG wrapper for the mixed GEMV whose store carries the graph's epilogue.
Computes c = a @ b^T in fp32 for a bf16 activation a and an fp32 weight
b, storing every output element through the epilogue the graph fused onto
c. moe.router.gate.mixed.gemv does the same arithmetic but writes its
result plainly, so it cannot serve a caller whose projection carries
elementwise work: that work would become a launch of its own and the cast
the fused form removes would be paid for by the chain it leaves behind.
M is runtime-dynamic -- one graph serves both decode and prefill -- so the
op branches on runtime M:
M == 0(graph-capture warmup): no launch.- tiny
M(M <= max_m, at most SM100's window of 64; decode): one fused mixed GEMV, the activation read as bf16 and widened in registers, written through the epilogue. - large
M(prefill): the chain the caller would otherwise have staged -- the activation widened in a launch of its own, then the fp32 matmul, both storing through that same epilogue. Those two launches and the kernel the matmul picks at that row count are the baseline's.
Because bf16->fp32 widening is lossless, every route is fp32 arithmetic over
exactly the widened bf16, so at the default window the result is
bit-identical to today's cast + fp32 gemv_split_k_epilogue_fn chain at
every row count: inside the window the fused GEMV is that kernel with the
cast folded into the load, and above it the op stages that chain. A caller
narrowing max_m below the chain's 64-row crossover only shortens the fused
window, and the rows it gives up take that staged chain.
Only SM100 (Blackwell) is served, and the gate below says exactly that. Its launch table and the fp32 GEMV this launch must reproduce are that target's, so every other target -- AMD, and NVIDIA targets below SM100, whose fp32 matmuls take a dispatch of their own -- gets an empty window and the staged chain below: today's launches, unchanged.
N and K are read from the weight b's static [N, K] layout at compile
time -- the op carries no redundant N/K custom-op parameters.
Implemented traits
Methods
execute
static def execute[max_m: Int = Int(64)](c: T, a: T, b: T, context: DeviceContext)
Executes the mixed bf16-A x fp32-B GEMV through c's epilogue.
Constraints:
The weight b must have a static [N, K] shape (both dims known at
compile time).
Parameters:
- max_m (
Int): The last row count the fused launch serves. SM100's own window is_MIXED_GEMV_EPILOGUE_MAX_M(64), the row count at which an fp32 matmul of this shape crosses to the TF32 tile GEMM, so the effective window is the smaller of this and 64.
Args:
- c (
T): Output[M, N]fp32 tensor; its fused epilogue owns the stores. - a (
T): Activation[M, K]bf16 tensor. - b (
T): Weight[N, K]fp32 tensor (transpose_b layout). - context (
DeviceContext): The device context.