For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mixed_gemv_store_epilogue
def mixed_gemv_store_epilogue[static_N: Int, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, EpilogueFnType: ElementwiseEpilogueFn](c: TileTensor[.float32, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.bfloat16, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[.float32, b_layout, ImmutAnyOrigin, Engine=b_engine], m: Int, n: Int, k: Int, epilogue_fn: EpilogueFnType, ctx: DeviceContext)
Launches the mixed GEMV with its output stored through epilogue_fn.
The bf16 activation is widened to fp32 in registers and dotted against the
unchanged fp32 weight in one launch, as router_gate_mixed_gemv does, and
every output element goes through epilogue_fn instead of a plain store.
That is what lets a caller whose projection carries elementwise work -- a
scale chain the graph would otherwise stage as a launch of its own -- keep
one launch for the whole projection. gemv_split_k_epilogue_fn is the same
kernel gemv_split_k is, so an epilogue that stores the untouched value
reproduces the plain launch bit for bit, and the fp32 form of the same
kernel over a pre-cast activation is the reference both are compared to.
The B200 bands are router_gate_mixed_gemv's: tile_m and tile_n move
with the row count, and neither reorders the K reduction. Only the SM100
launch is served here -- the [896, 7168] cold-weight table and the gfx950
[128, 6144] sweep are expressed through gemv_split_k parameters
(small_m_static_k, small_m_k_chunk, expert_major_grid, act_wide_k)
that this kernel variant does not take, and a non-SM100 NVIDIA target's fp32
matmuls take a dispatch of its own -- so a caller on another target stages
its own chain instead.
Parameters:
- static_N (
Int): Static output width (weight rows); selects thecheck_bounds_nguard at compile time. - c_layout (
TensorLayout): Layout of the fp32 output tensor. Only its shape is read: the epilogue owns every store. - a_layout (
TensorLayout): Layout of the bf16 activation tensor. - b_layout (
TensorLayout): Layout of the fp32 weight tensor. - c_engine (
TensorEngine): Engine of the fp32 output tensor. - a_engine (
TensorEngine): Engine of the bf16 activation tensor. - b_engine (
TensorEngine): Engine of the fp32 weight tensor. - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn(inferred).
Args:
- c (
TileTensor[.float32, c_layout, MutAnyOrigin, Engine=c_engine]): Output[M, N]fp32 tensor, read for its shape alone. - a (
TileTensor[.bfloat16, a_layout, ImmutAnyOrigin, Engine=a_engine]): Activation[M, K]bf16 tensor. - b (
TileTensor[.float32, b_layout, ImmutAnyOrigin, Engine=b_engine]): Weight[N, K]fp32 tensor (transpose_b layout). - m (
Int): Runtime row count (M). - n (
Int): Output width (N). - k (
Int): Contraction dim (K). - epilogue_fn (
EpilogueFnType): Stores each output element, carrying the graph's epilogue. - ctx (
DeviceContext): The device context.