IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

heuristic_and_outliers_dispatch

def heuristic_and_outliers_dispatch[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, //, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel()](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext) -> Int

Dispatches an SM100 block-scaled matmul by selecting a tuning config from per-format outlier tables for specific M ranges, falling back to a small-BN config for GEMVs (m == 1) and a heuristic config table for the remaining cases. Returns DISPATCH_HIT when a matching config is found and launched, or DISPATCH_MISS when no config matches.

Parameters:

  • ​c_type (DType): Element type of the output tensor c (inferred).
  • ​a_type (DType): Element type of the LHS input tensor a (inferred).
  • ​b_type (DType): Element type of the RHS input tensor b (inferred).
  • ​scales_dtype (DType): Element type of the per-block scale tensors a_scales and b_scales (inferred).
  • ​SF_VECTOR_SIZE (Int): Number of elements each scale factor covers. Must match the format: 16 for NVFP4, 32 for MXFP4, or 32 for MXFP8.
  • ​transpose_b (Bool): Whether b is stored transposed. Must be True (defaults to True).
  • ​pdl_level (PDLLevel): Programmatic Dependent Launch scheduling level for overlapping this kernel with prior GPU work (defaults to PDLLevel()).

Args:

Returns:

Int: DISPATCH_HIT if a config was selected and the kernel launched, otherwise DISPATCH_MISS.

def heuristic_and_outliers_dispatch[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, EpilogueFnType: ElementwiseEpilogueFn, //, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel()](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], tensor_sf: Float32, epilogue_fn: EpilogueFnType, ctx: DeviceContext) -> Int

Dispatches an SM100 block-scaled matmul (see the other overload), delivering each output value to an epilogue closure instead of writing c.

Parameters:

  • ​c_type (DType): Element type of the output tensor c (inferred).
  • ​a_type (DType): Element type of the LHS input tensor a (inferred).
  • ​b_type (DType): Element type of the RHS input tensor b (inferred).
  • ​scales_dtype (DType): Element type of the per-block scale tensors (inferred).
  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn (inferred).
  • ​SF_VECTOR_SIZE (Int): Number of elements each scale factor covers.
  • ​transpose_b (Bool): Whether b is stored transposed. Must be True.
  • ​pdl_level (PDLLevel): Programmatic Dependent Launch scheduling level.

Args:

Returns:

Int: DISPATCH_HIT if a config was selected and the kernel launched, otherwise DISPATCH_MISS.

def heuristic_and_outliers_dispatch[c_type: DType, a_type: DType, b_type: DType, scales_dtype: DType, ComputeFnType: ElementwiseComputeFn, //, SF_VECTOR_SIZE: Int, transpose_b: Bool = True, pdl_level: PDLLevel = PDLLevel(), compute_fn_needs_vectors: Bool = False, compute_fn_col_period: Int = Int(0), compute_fn_col_start: Int = Int(0)](c: TileTensor[c_type, Engine=c.Engine, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Engine=a.Engine, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Engine=b.Engine, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[scales_dtype, Engine=a_scales.Engine, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[scales_dtype, Engine=b_scales.Engine, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], tensor_sf: Float32, compute_fn: ComputeFnType, ctx: DeviceContext) -> Int

Dispatches an SM100 block-scaled matmul (see the other overload), storing compute_fn(idx, value) into c.

Parameters:

  • ​c_type (DType): Element type of the output tensor c (inferred).
  • ​a_type (DType): Element type of the LHS input tensor a (inferred).
  • ​b_type (DType): Element type of the RHS input tensor b (inferred).
  • ​scales_dtype (DType): Element type of the per-block scale tensors (inferred).
  • ​ComputeFnType (ElementwiseComputeFn): Type of compute_fn (inferred).
  • ​SF_VECTOR_SIZE (Int): Number of elements each scale factor covers.
  • ​transpose_b (Bool): Whether b is stored transposed. Must be True.
  • ​pdl_level (PDLLevel): Programmatic Dependent Launch scheduling level.
  • ​compute_fn_needs_vectors (Bool): Whether compute_fn reads each value with its neighbors in the row, so it must get consecutive columns rather than one element at a time.
  • ​compute_fn_col_period (Int): When nonzero, compute_fn returns its input unchanged at every column whose index modulo this period is below compute_fn_col_start, so an unfused pass visits only the remaining columns.
  • ​compute_fn_col_start (Int): The first column of each period that compute_fn may change.

Args:

Returns:

Int: DISPATCH_HIT if a config was selected and the kernel launched, otherwise DISPATCH_MISS.

Was this page helpful?