For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
ihc_mix_gemv_block
def ihc_mix_gemv_block[dtype: DType, num_mix: Int, num_warps: Int, poison: Int = Int(0)](partials: TileTensor[.float32, Engine=partials.Engine, address_space=partials.address_space, linear_idx_type=partials.linear_idx_type], streams: TileTensor[dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type], weight: TileTensor[.float32, Engine=weight.Engine, address_space=weight.address_space, linear_idx_type=weight.linear_idx_type], num_rows: Int, row_block: Int, split: Int, tid: Int)
Projects _GEMV_TILES 16-row tiles onto one K split on tensor cores.
One CTA's work: tid is the thread in a block of num_warps warps that
takes rows row_block * 64 ... and split split of the columns. Warp
w owns _GEMV_WARP_K columns of the flattened streams and runs
m16n8k16 MMAs over them for every row tile. The float32 weight is split
into a bfloat16 hi and lo = weight - hi, two MMAs per tile, which keeps
about 16 bits of it; the bfloat16 streams are exact.
K is a reduction axis, so it can be permuted freely as long as A and B
agree. Lane (g, t) = (lane // 4, lane % 4) loads 8 contiguous columns
8t .. 8t+7 of each 32-column chunk for rows g and g + 8; the first
four are its fragment of the chunk's first MMA, the last four of the
second, and it loads row g of the weight at the same columns.
Parameters:
- dtype (
DType): The stream dtype. - num_mix (
Int): Number ofweightrows. - num_warps (
Int): Warps per block. - poison (
Int): A planted fault (IHC_POISONS), 0 for none.
Args:
- partials (
TileTensor[.float32, Engine=partials.Engine, address_space=partials.address_space, linear_idx_type=partials.linear_idx_type]): Splits's share of each row's projections, then of its sum of squares. Shape:[splits, num_rows, num_mix + 1]. - streams (
TileTensor[dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type]): The flattened streams. Shape:[num_rows, k]. - weight (
TileTensor[.float32, Engine=weight.Engine, address_space=weight.address_space, linear_idx_type=weight.linear_idx_type]): The projection weight. Shape:[num_mix, k]. - num_rows (
Int): Number of rows. - row_block (
Int): The block of_GEMV_TILES * 16rows. - split (
Int): The block ofnum_warps * _GEMV_WARP_Kcolumns. - tid (
Int): The thread in the block.