For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
ihc_collapse_rms_norm_block
def ihc_collapse_rms_norm_block[dtype: DType, hc_mult: Int, num_mix: Int, num_splits: Int, hidden: Int, block_size: Int, poison: Int = Int(0)](output: TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], gates: TileTensor[.float32, Engine=gates.Engine, address_space=gates.address_space, linear_idx_type=gates.linear_idx_type], streams: TileTensor[dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type], partials: TileTensor[.float32, Engine=partials.Engine, address_space=partials.address_space, linear_idx_type=partials.linear_idx_type], base: TileTensor[.float32, Engine=base.Engine, address_space=base.address_space, linear_idx_type=base.linear_idx_type], scale: TileTensor[.float32, Engine=scale.Engine, address_space=scale.address_space, linear_idx_type=scale.linear_idx_type], gamma: TileTensor[dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], hc_magnitude: Float32, hc_eps: Float32, streams_eps: Float32, norm_eps: Float32, row: Int, tid: Int)
Collapses one token's streams and RMS-normalizes the result.
One CTA's work: tid is the thread in a block of block_size that takes
token row. The gates come from partials alone, so the streams are read
once, straight into the collapse.
Parameters:
- dtype (
DType): The stream, output andgammadtype. - hc_mult (
Int): Number of residual streams. - num_mix (
Int): Gate logits per token:hc_mult(preonly) or2 * hc_mult(prethenpost). - num_splits (
Int): Leading dimension ofpartials. - hidden (
Int): Width of one stream. - block_size (
Int): Threads per block. - poison (
Int): A planted fault (IHC_POISONS), 0 for none.
Args:
- output (
TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]):rms_norm(sum_j pre_j * streams[:, j]), the collapse rounded todtypefirst. Shape:[num_rows, hidden]. - gates (
TileTensor[.float32, Engine=gates.Engine, address_space=gates.address_space, linear_idx_type=gates.linear_idx_type]):pre, thenpostwhennum_mix == 2 * hc_mult. Shape:[num_rows, num_mix]. - streams (
TileTensor[dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type]): Shape:[num_rows, hc_mult, hidden]. - partials (
TileTensor[.float32, Engine=partials.Engine, address_space=partials.address_space, linear_idx_type=partials.linear_idx_type]):ihc_mix_gemv's output. Shape:[num_splits, num_rows, num_mix + 1]. - base (
TileTensor[.float32, Engine=base.Engine, address_space=base.address_space, linear_idx_type=base.linear_idx_type]): Per-logit bias. Shape:[num_mix]. - scale (
TileTensor[.float32, Engine=scale.Engine, address_space=scale.address_space, linear_idx_type=scale.linear_idx_type]): One scale per band. Shape:[num_mix // hc_mult]. - gamma (
TileTensor[dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): The RMSNorm weight. Shape:[hidden]. - hc_magnitude (
Float32): Upper bound ofpost. - hc_eps (
Float32): Added to every gate. - streams_eps (
Float32): Epsilon of the streams' RMS. - norm_eps (
Float32): Epsilon of the output RMSNorm. - row (
Int): The token. - tid (
Int): The thread in the block.