IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

matmul_qint4

def matmul_qint4[group_size: Int](a: TileTensor[.float32, Engine=a.Engine, linear_idx_type=a.linear_idx_type], b: TileTensor[.uint8, Engine=b.Engine, linear_idx_type=b.linear_idx_type], c: TileTensor[.float32, Engine=c.Engine, linear_idx_type=c.linear_idx_type], ctx: Optional[DeviceContext[CompilationTarget()]] = None)

Computes a matrix multiply of a float32 A matrix against block-wise quantized int4 B weights, producing a float32 result.

Dispatches to an architecture-specific kernel (VNNI, AVX2, NEON i8mm, or NEON dotprod) at compile time.

Parameters:

  • ​group_size (Int): Number of elements per quantization group.

Args:

def matmul_qint4[EpilogueFnType: ElementwiseEpilogueFn, //, group_size: Int](a: TileTensor[.float32, Engine=a.Engine, linear_idx_type=a.linear_idx_type], b: TileTensor[.uint8, Engine=b.Engine, linear_idx_type=b.linear_idx_type], c: TileTensor[.float32, Engine=c.Engine, linear_idx_type=c.linear_idx_type], epilogue_fn: EpilogueFnType, ctx: Optional[DeviceContext[CompilationTarget()]] = None)

Same as the plain overload, also passing each output element to epilogue_fn.

The kernel uses c as its accumulator, so it is written either way.

Parameters:

  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn (inferred).
  • ​group_size (Int): Number of elements per quantization group.

Args:

Was this page helpful?