For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

gemv

Apple M5 bf16/fp16 decode GEMV: out = x @ W^T for a few activation rows.

At batch-1 (and small-batch) decode a Linear is a matrix-vector product: the weight W[N, K] is read once and there is no activation matrix to feed the simdgroup MMA, so the kernel is bound by the weight read. This is the 16-bit sibling of fp8_gemv.mojo: register-resident, no threadgroup memory, no barrier(), no MMA.

Each warp owns rows_per_warp consecutive rows of W (output columns). Its 32 lanes stride down K in tile_k-element chunks, so adjacent lanes read adjacent 16-byte runs of every row. Per chunk a lane issues rows_per_warp weight loads and tile_m activation loads, and the K loop is unrolled unroll chunks deep so each lane keeps rows_per_warp * unroll weight loads in flight; the load count in flight is what reaches DRAM bandwidth when N is small. Products accumulate in fp32 vectors that are reduced once, after the K loop, by a warp.sum per output.

The weight is read once for all tile_m activation rows, so the kernel also covers small-batch decode (M <= tile_m) at the cost of one extra fp32 accumulator set per row.

comptime values

APPLE_GEMV_MAX_M

comptime APPLE_GEMV_MAX_M = 8

Largest M enqueue_apple_gemv serves. At M = 16 the per-row fp32 accumulators spill and the kernel falls below the tiled matmul.

Functions

• ​apple_gemv_kernel: Computes rows_per_warp columns of c[:m] = a[:m] @ weight^T per warp.
• ​apple_gemv_small_batch_supported: Returns whether _matmul_gpu routes an [m, k] activation here.
• ​enqueue_apple_gemv: Enqueues c = a @ weight^T for 1 <= M <= APPLE_GEMV_MAX_M.
• ​enqueue_apple_gemv_config: Enqueues apple_gemv_kernel with an explicit launch configuration.