IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

matmul_gpu_qint4

def matmul_gpu_qint4[c_type: DType, a_type: DType, //, group_size: Int](c_tt: TileTensor[c_type, Engine=c_tt.Engine, linear_idx_type=c_tt.linear_idx_type], a_tt: TileTensor[a_type, Engine=a_tt.Engine, linear_idx_type=a_tt.linear_idx_type], b_tt: TileTensor[.uint8, Engine=b_tt.Engine, linear_idx_type=b_tt.linear_idx_type], ctx: DeviceContext) where (c_type == DType.bfloat16) if (a_type == DType.bfloat16) else (a_type == DType.bfloat16)

Launches a GPU int4 quantized matrix multiplication for the given tile tensors.

Constraints:

Requires an NVIDIA GPU target. a_type and c_type must both be bfloat16.

Parameters:

  • ​c_type (DType): The dtype of the output matrix.
  • ​a_type (DType): The dtype of the A matrix elements.
  • ​group_size (Int): The number of K elements sharing a single scale.

Args:

Raises:

An error if the input tensors are not rank-2 or the target is not a GPU.

def matmul_gpu_qint4[c_type: DType, a_type: DType, EpilogueFnType: ElementwiseEpilogueFn, //, group_size: Int](c_tt: TileTensor[c_type, Engine=c_tt.Engine, linear_idx_type=c_tt.linear_idx_type], a_tt: TileTensor[a_type, Engine=a_tt.Engine, linear_idx_type=a_tt.linear_idx_type], b_tt: TileTensor[.uint8, Engine=b_tt.Engine, linear_idx_type=b_tt.linear_idx_type], epilogue_fn: EpilogueFnType, ctx: DeviceContext) where (c_type == DType.bfloat16) if (a_type == DType.bfloat16) else (a_type == DType.bfloat16)

Same as the plain overload, storing the output through epilogue_fn.

Constraints:

Requires an NVIDIA GPU target. a_type and c_type must both be bfloat16.

Parameters:

  • ​c_type (DType): The dtype of the output matrix.
  • ​a_type (DType): The dtype of the A matrix elements.
  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn (inferred).
  • ​group_size (Int): The number of K elements sharing a single scale.

Args:

Raises:

An error if the input tensors are not rank-2 or the target is not a GPU.

Was this page helpful?