For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mtp_eh_norm_kernel
def mtp_eh_norm_kernel[dtype: DType, OutLayoutType: TensorLayout, out_origin: MutOrigin, EmbedLayoutType: TensorLayout, embed_origin: ImmOrigin, PrevLayoutType: TensorLayout, prev_origin: ImmOrigin, EWLayoutType: TensorLayout, ew_origin: ImmOrigin, HWLayoutType: TensorLayout, hw_origin: ImmOrigin, hidden_size: Int, block_threads: Int, max_warps_per_block: Int, OutEngine: TensorEngine, EmbedEngine: TensorEngine, PrevEngine: TensorEngine, EWEngine: TensorEngine, HWEngine: TensorEngine](out_buf: TileTensor[dtype, OutLayoutType, out_origin, Engine=OutEngine], embed: TileTensor[dtype, EmbedLayoutType, embed_origin, Engine=EmbedEngine], prev: TileTensor[dtype, PrevLayoutType, prev_origin, Engine=PrevEngine], enorm_weight: TileTensor[dtype, EWLayoutType, ew_origin, Engine=EWEngine], hnorm_weight: TileTensor[dtype, HWLayoutType, hw_origin, Engine=HWEngine], epsilon: Float32, num_tokens: Int32)
Normalizes both inputs for one token and writes them side by side.
One pass, 16 bytes at a time: each thread holds two 8-channel pieces of
both input rows in registers (four 16-byte loads), reduces their squares
through block_reduce_dual_sum, and stores both halves from those
registers, so each element of a row is read once and written once. The
arithmetic matches ops.rms_norm in its Llama-style configuration --
f32 accumulation, rsqrt(mean + eps), cast before the weight multiply --
with one bf16 rounding per step.
The register path needs hidden_size == block_threads * 16 at compile
time. Any other shape takes a strided body that walks the row with 2-byte
accesses, so the ragged sizes the tests use keep working and every shape
gets the same arithmetic either way.
Parameters:
- dtype (
DType): Element type of the inputs, the weights and the output. - OutLayoutType (
TensorLayout): Layout ofout_buf. - out_origin (
MutOrigin): Origin ofout_buf. - EmbedLayoutType (
TensorLayout): Layout ofembed. - embed_origin (
ImmOrigin): Origin ofembed. - PrevLayoutType (
TensorLayout): Layout ofprev. - prev_origin (
ImmOrigin): Origin ofprev. - EWLayoutType (
TensorLayout): Layout ofenorm_weight. - ew_origin (
ImmOrigin): Origin ofenorm_weight. - HWLayoutType (
TensorLayout): Layout ofhnorm_weight. - hw_origin (
ImmOrigin): Origin ofhnorm_weight. - hidden_size (
Int): Channels per input; the output row is twice this. - block_threads (
Int): The launch's block width. The register path runs whenhidden_size == block_threads * 16, and asserts that the launch matches it; every other shape takes the strided body, which needs a whole number of warps. - max_warps_per_block (
Int): Power-of-two ceiling on the block's warp count, not the launch's actual warp count.block_reduce_dual_sumreduces over a power-of-two lane group; slots past the real warp count hold zero. - OutEngine (
TensorEngine): Engine policy ofout_buf. - EmbedEngine (
TensorEngine): Engine policy ofembed. - PrevEngine (
TensorEngine): Engine policy ofprev. - EWEngine (
TensorEngine): Engine policy ofenorm_weight. - HWEngine (
TensorEngine): Engine policy ofhnorm_weight.
Args:
- out_buf (
TileTensor[dtype, OutLayoutType, out_origin, Engine=OutEngine]): Output[num_tokens, 2 * hidden_size]. - embed (
TileTensor[dtype, EmbedLayoutType, embed_origin, Engine=EmbedEngine]): Token embeddings[num_tokens, hidden_size]. - prev (
TileTensor[dtype, PrevLayoutType, prev_origin, Engine=PrevEngine]): Target hidden states[num_tokens, hidden_size]. - enorm_weight (
TileTensor[dtype, EWLayoutType, ew_origin, Engine=EWEngine]): Embedding norm weight[hidden_size]. - hnorm_weight (
TileTensor[dtype, HWLayoutType, hw_origin, Engine=HWEngine]): Hidden-state norm weight[hidden_size]. - epsilon (
Float32): Added inside the square root, matchingops.rms_norm. - num_tokens (
Int32): Rows to process.