For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
batched_matmul
def batched_matmul[*, transpose_a: Bool = False, transpose_b: Bool = False, saturated_vnni: Bool = False, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](c_buf: TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type], a_buf: TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type], b_buf: TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type], *, context: Optional[DeviceContext[target]] = DeviceContext[_].NoneValue)
TileTensor primary implementation of batched_matmul.
Parameters:
- transpose_a (
Bool): Whether the LHS input is transposed (defaults toFalse; not yet supported). - transpose_b (
Bool): Whether the RHS input is transposed (defaults toFalse). - saturated_vnni (
Bool): Whether to use saturated VNNI accumulation on CPU (defaults toFalse; not applicable on GPU). - pdl_level (
PDLLevel): Programmatic dependent launch level for the SM100 GPU kernels; other targets ignore it.OVERLAP_AT_BEGINNINGreleases dependent grids at kernel entry, so every dependent must wait before reading the output. The SM100 plain matmul may prefetch its B operand ahead of that wait, because B is a weight read straight from the checkpoint by contract. A value the graph computes each call (a cast, concat, derived parameter or sharded view) is not one, so this op's output must not feed it as B (defaults toPDLLevel.OVERLAP_AT_BEGINNING).
Args:
- c_buf (
TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type]): Output tensor of shape(..., m, n); rank must be at least 2 and match the inputs. - a_buf (
TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type]): LHS input tensor of shape(..., m, k). - b_buf (
TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type]): RHS input tensor of shape(..., k, n), or(..., n, k)whentranspose_bis set. - context (
Optional[DeviceContext[target]]): Optional device context used for dispatch and parallelism (defaults toNone).
def batched_matmul[EpilogueFnType: ElementwiseEpilogueFn, //, *, transpose_a: Bool = False, transpose_b: Bool = False, saturated_vnni: Bool = False, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](c_buf: TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type], a_buf: TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type], b_buf: TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type], epilogue_fn: EpilogueFnType, *, context: Optional[DeviceContext[target]] = DeviceContext[_].NoneValue)
Same as the plain overload, storing the output through epilogue_fn.
Parameters:
- EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn(inferred). - transpose_a (
Bool): Whether the LHS input is transposed (defaults toFalse; not yet supported). - transpose_b (
Bool): Whether the RHS input is transposed (defaults toFalse). - saturated_vnni (
Bool): Whether to use saturated VNNI accumulation on CPU (defaults toFalse; not applicable on GPU). - pdl_level (
PDLLevel): Programmatic dependent launch level for the SM100 GPU kernels; other targets ignore it.OVERLAP_AT_BEGINNINGreleases dependent grids at kernel entry, so every dependent must wait before reading the output. The SM100 plain matmul may prefetch its B operand ahead of that wait, because B is a weight read straight from the checkpoint by contract. A value the graph computes each call (a cast, concat, derived parameter or sharded view) is not one, so this op's output must not feed it as B (defaults toPDLLevel.OVERLAP_AT_BEGINNING).
Args:
- c_buf (
TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type]): Tensor of shape(..., m, n)giving the output shape; rank must be at least 2 and match the inputs. Some dispatch paths use it as scratch for the product. - a_buf (
TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type]): LHS input tensor of shape(..., m, k). - b_buf (
TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type]): RHS input tensor of shape(..., k, n), or(..., n, k)whentranspose_bis set. - epilogue_fn (
EpilogueFnType): Stores each output chunk. It receives coordinates of the same rank asc_buf, not the collapsed rank-3 coordinates the kernels compute over. - context (
Optional[DeviceContext[target]]): Optional device context used for dispatch and parallelism (defaults toNone).
def batched_matmul[ComputeFnType: ElementwiseComputeFn, //, *, transpose_a: Bool = False, transpose_b: Bool = False, saturated_vnni: Bool = False, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](c_buf: TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type], a_buf: TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type], b_buf: TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type], compute_fn: ComputeFnType, *, context: Optional[DeviceContext[target]] = DeviceContext[_].NoneValue)
Same as the plain overload, storing compute_fn(idx, value) into c_buf.
Parameters:
- ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn(inferred). - transpose_a (
Bool): Whether the LHS input is transposed (defaults toFalse; not yet supported). - transpose_b (
Bool): Whether the RHS input is transposed (defaults toFalse). - saturated_vnni (
Bool): Whether to use saturated VNNI accumulation on CPU (defaults toFalse; not applicable on GPU). - pdl_level (
PDLLevel): Programmatic dependent launch level for the SM100 GPU kernels; other targets ignore it.OVERLAP_AT_BEGINNINGreleases dependent grids at kernel entry, so every dependent must wait before reading the output. The SM100 plain matmul may prefetch its B operand ahead of that wait, because B is a weight read straight from the checkpoint by contract. A value the graph computes each call (a cast, concat, derived parameter or sharded view) is not one, so this op's output must not feed it as B (defaults toPDLLevel.OVERLAP_AT_BEGINNING).
Args:
- c_buf (
TileTensor[Engine=c_buf.Engine, linear_idx_type=c_buf.linear_idx_type]): Output tensor of shape(..., m, n); rank must be at least 2 and match the inputs. - a_buf (
TileTensor[Engine=a_buf.Engine, linear_idx_type=a_buf.linear_idx_type]): LHS input tensor of shape(..., m, k). - b_buf (
TileTensor[Engine=b_buf.Engine, linear_idx_type=b_buf.linear_idx_type]): RHS input tensor of shape(..., k, n), or(..., n, k)whentranspose_bis set. - compute_fn (
ComputeFnType): Maps each output chunk to the value to store. It receives coordinates of the same rank asc_buf, not the collapsed rank-3 coordinates the kernels compute over. - context (
Optional[DeviceContext[target]]): Optional device context used for dispatch and parallelism (defaults toNone).