For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
heuristic_and_outliers_dispatch
def heuristic_and_outliers_dispatch[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, //, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel()](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext) -> Int
Dispatches an SM100 block-scaled matmul by selecting a tuning config from per-format outlier tables for specific M ranges, falling back to a small-BN config for GEMVs (m == 1) and a heuristic config table for the remaining cases. Returns DISPATCH_HIT when a matching config is found and launched, or DISPATCH_MISS when no config matches.
Parameters:
- c_type (
DType): Element type of the output tensorc(inferred). - a_type (
DType): Element type of the LHS input tensora(inferred). - b_type (
DType): Element type of the RHS input tensorb(inferred). - scales_dtype (
DType): Element type of the per-block scale tensorsa_scalesandb_scales(inferred). - SF_VECTOR_SIZE (
Int): Number of elements each scale factor covers. Must match the format: 16 for NVFP4, 32 for MXFP4, or 32 for MXFP8. - transpose_b (
Bool): Whetherbis stored transposed. Must beTrue(defaults toTrue). - pdl_level (
PDLLevel): Programmatic Dependent Launch scheduling level for overlapping this kernel with prior GPU work (defaults toPDLLevel()).
Args:
- c (
TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Output TileTensor accumulating the matmul result. - a (
TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): LHS input TileTensor. - b (
TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): RHS input TileTensor (must be transposed). - a_scales (
TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type]): Per-block scales fora. - b_scales (
TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type]): Per-block scales forb. - tensor_sf (
Float32): Global tensor scaling factor applied asalpha. - ctx (
DeviceContext): Device context used to launch the kernel.
Returns:
Int: DISPATCH_HIT if a config was selected and the kernel launched, otherwise DISPATCH_MISS.
def heuristic_and_outliers_dispatch[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, EpilogueFnType: ElementwiseEpilogueFn, //, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel()](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], tensor_sf: Float32, epilogue_fn: EpilogueFnType, ctx: DeviceContext) -> Int
Dispatches an SM100 block-scaled matmul (see the other overload), delivering each output value to an epilogue closure instead of writing c.
Parameters:
- c_type (
DType): Element type of the output tensorc(inferred). - a_type (
DType): Element type of the LHS input tensora(inferred). - b_type (
DType): Element type of the RHS input tensorb(inferred). - scales_dtype (
DType): Element type of the per-block scale tensors (inferred). - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn(inferred). - SF_VECTOR_SIZE (
Int): Number of elements each scale factor covers. - transpose_b (
Bool): Whetherbis stored transposed. Must beTrue. - pdl_level (
PDLLevel): Programmatic Dependent Launch scheduling level.
Args:
- c (
TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Output TileTensor; only its shape is read. - a (
TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): LHS input TileTensor. - b (
TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): RHS input TileTensor (must be transposed). - a_scales (
TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type]): Per-block scales fora. - b_scales (
TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type]): Per-block scales forb. - tensor_sf (
Float32): Global tensor scaling factor applied asalpha. - epilogue_fn (
EpilogueFnType): Stores each output value at its global(m, n). - ctx (
DeviceContext): Device context used to launch the kernel.
Returns:
Int: DISPATCH_HIT if a config was selected and the kernel launched, otherwise DISPATCH_MISS.
def heuristic_and_outliers_dispatch[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, ComputeFnType: ElementwiseComputeFn, //, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel(), compute_fn_needs_vectors: Bool = False, compute_fn_col_period: Int = Int(0), compute_fn_col_start: Int = Int(0)](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], tensor_sf: Float32, compute_fn: ComputeFnType, ctx: DeviceContext) -> Int
Dispatches an SM100 block-scaled matmul (see the other overload), storing compute_fn(idx, value) into c.
Parameters:
- c_type (
DType): Element type of the output tensorc(inferred). - a_type (
DType): Element type of the LHS input tensora(inferred). - b_type (
DType): Element type of the RHS input tensorb(inferred). - scales_dtype (
DType): Element type of the per-block scale tensors (inferred). - ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn(inferred). - SF_VECTOR_SIZE (
Int): Number of elements each scale factor covers. - transpose_b (
Bool): Whetherbis stored transposed. Must beTrue. - pdl_level (
PDLLevel): Programmatic Dependent Launch scheduling level. - compute_fn_needs_vectors (
Bool): Whethercompute_fnreads each value with its neighbors in the row, so it must get consecutive columns rather than one element at a time. - compute_fn_col_period (
Int): When nonzero,compute_fnreturns its input unchanged at every column whose index modulo this period is belowcompute_fn_col_start, so an unfused pass visits only the remaining columns. - compute_fn_col_start (
Int): The first column of each period thatcompute_fnmay change.
Args:
- c (
TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Output TileTensor accumulating the matmul result. - a (
TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): LHS input TileTensor. - b (
TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): RHS input TileTensor (must be transposed). - a_scales (
TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type]): Per-block scales fora. - b_scales (
TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type]): Per-block scales forb. - tensor_sf (
Float32): Global tensor scaling factor applied asalpha. - compute_fn (
ComputeFnType): Transforms each output value before it is stored. - ctx (
DeviceContext): Device context used to launch the kernel.
Returns:
Int: DISPATCH_HIT if a config was selected and the kernel launched, otherwise DISPATCH_MISS.