For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
matmul_gpu_qint4
def matmul_gpu_qint4[c_type: DType, a_type: DType, //, group_size: Int](c_tt: TileTensor[c_type, Engine=c_tt.Engine, linear_idx_type=c_tt.linear_idx_type], a_tt: TileTensor[a_type, Engine=a_tt.Engine, linear_idx_type=a_tt.linear_idx_type], b_tt: TileTensor[.uint8, Engine=b_tt.Engine, linear_idx_type=b_tt.linear_idx_type], ctx: DeviceContext) where (c_type == DType.bfloat16) if (a_type == DType.bfloat16) else (a_type == DType.bfloat16)
Launches a GPU int4 quantized matrix multiplication for the given tile tensors.
Constraints:
Requires an NVIDIA GPU target. a_type and c_type must both be
bfloat16.
Parameters:
- c_type (
DType): The dtype of the output matrix. - a_type (
DType): The dtype of the A matrix elements. - group_size (
Int): The number of K elements sharing a single scale.
Args:
- c_tt (
TileTensor[c_type, Engine=c_tt.Engine, linear_idx_type=c_tt.linear_idx_type]): The output tile tensor in global memory. - a_tt (
TileTensor[a_type, Engine=a_tt.Engine, linear_idx_type=a_tt.linear_idx_type]): The left-hand (activation) tile tensor in global memory. - b_tt (
TileTensor[.uint8, Engine=b_tt.Engine, linear_idx_type=b_tt.linear_idx_type]): The packed quantized weight tile tensor in global memory. - ctx (
DeviceContext): The device context used to enqueue the kernel.
Raises:
An error if the input tensors are not rank-2 or the target is not a GPU.
def matmul_gpu_qint4[c_type: DType, a_type: DType, EpilogueFnType: ElementwiseEpilogueFn, //, group_size: Int](c_tt: TileTensor[c_type, Engine=c_tt.Engine, linear_idx_type=c_tt.linear_idx_type], a_tt: TileTensor[a_type, Engine=a_tt.Engine, linear_idx_type=a_tt.linear_idx_type], b_tt: TileTensor[.uint8, Engine=b_tt.Engine, linear_idx_type=b_tt.linear_idx_type], epilogue_fn: EpilogueFnType, ctx: DeviceContext) where (c_type == DType.bfloat16) if (a_type == DType.bfloat16) else (a_type == DType.bfloat16)
Same as the plain overload, storing the output through epilogue_fn.
Constraints:
Requires an NVIDIA GPU target. a_type and c_type must both be
bfloat16.
Parameters:
- c_type (
DType): The dtype of the output matrix. - a_type (
DType): The dtype of the A matrix elements. - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn(inferred). - group_size (
Int): The number of K elements sharing a single scale.
Args:
- c_tt (
TileTensor[c_type, Engine=c_tt.Engine, linear_idx_type=c_tt.linear_idx_type]): The output tile tensor; the kernel does not write it. - a_tt (
TileTensor[a_type, Engine=a_tt.Engine, linear_idx_type=a_tt.linear_idx_type]): The left-hand (activation) tile tensor in global memory. - b_tt (
TileTensor[.uint8, Engine=b_tt.Engine, linear_idx_type=b_tt.linear_idx_type]): The packed quantized weight tile tensor in global memory. - epilogue_fn (
EpilogueFnType): Epilogue that stores each output element. - ctx (
DeviceContext): The device context used to enqueue the kernel.
Raises:
An error if the input tensors are not rank-2 or the target is not a GPU.