For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
dispatch_gemv
def dispatch_gemv[c_type: DType, a_type: DType, b_type: DType, EpilogueFnType: ElementwiseEpilogueFn, ComputeFnType: ElementwiseComputeFn, //, transpose_b: Bool = False, pdl_level: PDLLevel = PDLLevel(), has_epilogue_fn: Bool = False, has_compute_fn: Bool = True](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], epilogue_fn: EpilogueFnType, compute_fn: ComputeFnType, ctx: DeviceContext)
Dispatch M=1 (or N=1) matmul to GEMV or SM100 GEMM based on (N, K).
For most M=1 shapes GEMV is preferred, but for certain large (N, K)
combinations the SM100 GEMM kernel achieves higher throughput. Add new
(N, K) pairs to SM100_GEMV_SHAPES as they are identified through benchmarking.
N=1 always routes to GEMV: SM100 TMA requires N * sizeof(c_type) % 16 == 0.
Parameters:
- c_type (
DType): Output element type (inferred). - a_type (
DType): Element type of the LHS operanda(inferred). - b_type (
DType): Element type of the RHS operandb(inferred). - EpilogueFnType (
ElementwiseEpilogueFn): Type of the elementwise epilogue closure (inferred). - ComputeFnType (
ElementwiseComputeFn): Type of the compute epilogue closure (inferred). - transpose_b (
Bool): Whetherbis stored transposed (defaults toFalse). - pdl_level (
PDLLevel): Programmatic dependent launch level for the dispatched kernel (defaults toPDLLevel.OVERLAP_AT_BEGINNING). - has_epilogue_fn (
Bool): Whetherepilogue_fnstores the output (defaults toFalse). - has_compute_fn (
Bool): Whether to applycompute_fn(defaults toTrue).
Args:
- c (
TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Output matrix as a rank-2 mutableTileTensorof shape[M, N]. - a (
TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): LHS input matrix as a rank-2TileTensorof shape[M, K]. - b (
TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): RHS input matrix as a rank-2TileTensorof shape[K, N], or[N, K]whentranspose_bis set. - epilogue_fn (
EpilogueFnType): Stores each output value. Ignored unlesshas_epilogue_fn. - compute_fn (
ComputeFnType): Compute epilogue, for example a static scale, applied to each output value before it is stored. Ignored unlesshas_compute_fn. - ctx (
DeviceContext): Device context used to enqueue the selected kernel.