IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

dispatch_gemv

def dispatch_gemv[c_type: DType, a_type: DType, b_type: DType, EpilogueFnType: ElementwiseEpilogueFn, ComputeFnType: ElementwiseComputeFn, //, transpose_b: Bool = False, pdl_level: PDLLevel = PDLLevel(), has_epilogue_fn: Bool = False, has_compute_fn: Bool = True](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], epilogue_fn: EpilogueFnType, compute_fn: ComputeFnType, ctx: DeviceContext)

Dispatch M=1 (or N=1) matmul to GEMV or SM100 GEMM based on (N, K).

For most M=1 shapes GEMV is preferred, but for certain large (N, K) combinations the SM100 GEMM kernel achieves higher throughput. Add new (N, K) pairs to SM100_GEMV_SHAPES as they are identified through benchmarking.

N=1 always routes to GEMV: SM100 TMA requires N * sizeof(c_type) % 16 == 0.

Parameters:

  • ​c_type (DType): Output element type (inferred).
  • ​a_type (DType): Element type of the LHS operand a (inferred).
  • ​b_type (DType): Element type of the RHS operand b (inferred).
  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of the elementwise epilogue closure (inferred).
  • ​ComputeFnType (ElementwiseComputeFn): Type of the compute epilogue closure (inferred).
  • ​transpose_b (Bool): Whether b is stored transposed (defaults to False).
  • ​pdl_level (PDLLevel): Programmatic dependent launch level for the dispatched kernel (defaults to PDLLevel.OVERLAP_AT_BEGINNING).
  • ​has_epilogue_fn (Bool): Whether epilogue_fn stores the output (defaults to False).
  • ​has_compute_fn (Bool): Whether to apply compute_fn (defaults to True).

Args:

Was this page helpful?