For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

grouped_gemv_kernel

def grouped_gemv_kernel[c_type: DType, a_type: DType, b_type: DType, N: Int, K: Int, rows_per_sg: Int, tokens_per_pass: Int, num_sg: Int, acc_width: Int, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, ao_layout: TensorLayout, ei_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, ao_engine: TensorEngine, ei_engine: TensorEngine](c: TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine], a_offsets: TileTensor[.uint32, ao_layout, ImmUnsafeAnyOrigin, Engine=ao_engine], expert_ids: TileTensor[.int32, ei_layout, ImmUnsafeAnyOrigin, Engine=ei_engine])

Grouped GEMV: one simdgroup per rows_per_sg output columns of a group.

Grid (ceil(N / (rows_per_sg * num_sg)), 1, num_active_experts). Tokens of the group are processed tokens_per_pass at a time; the weight chunk is loaded once per step and reused by every token of the pass. expert_ids[group] == -1 writes zeros, matching naive_grouped_matmul_kernel.