IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

batched_matmul

def batched_matmul[*, transpose_a: Bool = False, transpose_b: Bool = False, saturated_vnni: Bool = False, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](c_buf: TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type], a_buf: TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type], b_buf: TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type], *, context: Optional[DeviceContext[target]] = DeviceContext[_].NoneValue)

TileTensor primary implementation of batched_matmul.

Parameters:

  • ​transpose_a (Bool): Whether the LHS input is transposed (defaults to False; not yet supported).
  • ​transpose_b (Bool): Whether the RHS input is transposed (defaults to False).
  • ​saturated_vnni (Bool): Whether to use saturated VNNI accumulation on CPU (defaults to False; not applicable on GPU).
  • ​pdl_level (PDLLevel): Programmatic dependent launch level for the SM100 GPU kernels; other targets ignore it. OVERLAP_AT_BEGINNING releases dependent grids at kernel entry, so every dependent must wait before reading the output. The SM100 plain matmul may prefetch its B operand ahead of that wait, because B is a weight read straight from the checkpoint by contract. A value the graph computes each call (a cast, concat, derived parameter or sharded view) is not one, so this op's output must not feed it as B (defaults to PDLLevel.OVERLAP_AT_BEGINNING).

Args:

def batched_matmul[EpilogueFnType: ElementwiseEpilogueFn, //, *, transpose_a: Bool = False, transpose_b: Bool = False, saturated_vnni: Bool = False, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](c_buf: TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type], a_buf: TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type], b_buf: TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type], epilogue_fn: EpilogueFnType, *, context: Optional[DeviceContext[target]] = DeviceContext[_].NoneValue)

Same as the plain overload, storing the output through epilogue_fn.

Parameters:

  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn (inferred).
  • ​transpose_a (Bool): Whether the LHS input is transposed (defaults to False; not yet supported).
  • ​transpose_b (Bool): Whether the RHS input is transposed (defaults to False).
  • ​saturated_vnni (Bool): Whether to use saturated VNNI accumulation on CPU (defaults to False; not applicable on GPU).
  • ​pdl_level (PDLLevel): Programmatic dependent launch level for the SM100 GPU kernels; other targets ignore it. OVERLAP_AT_BEGINNING releases dependent grids at kernel entry, so every dependent must wait before reading the output. The SM100 plain matmul may prefetch its B operand ahead of that wait, because B is a weight read straight from the checkpoint by contract. A value the graph computes each call (a cast, concat, derived parameter or sharded view) is not one, so this op's output must not feed it as B (defaults to PDLLevel.OVERLAP_AT_BEGINNING).

Args:

def batched_matmul[ComputeFnType: ElementwiseComputeFn, //, *, transpose_a: Bool = False, transpose_b: Bool = False, saturated_vnni: Bool = False, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](c_buf: TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type], a_buf: TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type], b_buf: TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type], compute_fn: ComputeFnType, *, context: Optional[DeviceContext[target]] = DeviceContext[_].NoneValue)

Same as the plain overload, storing compute_fn(idx, value) into c_buf.

Parameters:

  • ​ComputeFnType (ElementwiseComputeFn): Type of compute_fn (inferred).
  • ​transpose_a (Bool): Whether the LHS input is transposed (defaults to False; not yet supported).
  • ​transpose_b (Bool): Whether the RHS input is transposed (defaults to False).
  • ​saturated_vnni (Bool): Whether to use saturated VNNI accumulation on CPU (defaults to False; not applicable on GPU).
  • ​pdl_level (PDLLevel): Programmatic dependent launch level for the SM100 GPU kernels; other targets ignore it. OVERLAP_AT_BEGINNING releases dependent grids at kernel entry, so every dependent must wait before reading the output. The SM100 plain matmul may prefetch its B operand ahead of that wait, because B is a weight read straight from the checkpoint by contract. A value the graph computes each call (a cast, concat, derived parameter or sharded view) is not one, so this op's output must not feed it as B (defaults to PDLLevel.OVERLAP_AT_BEGINNING).

Args:

Was this page helpful?