IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

nvfp4_kv_write_row

def nvfp4_kv_write_row[cache_t: KVCacheT, gamma_layout: TensorLayout, gamma_origin: ImmOrigin, gamma_engine: TensorEngine, freqs_layout: TensorLayout, freqs_origin: ImmOrigin, freqs_engine: TensorEngine, offsets_layout: TensorLayout, offsets_origin: ImmOrigin, offsets_engine: TensorEngine, dtype: DType, freq_dtype: DType, InputFnType: def[width: Int, alignment: Int](token: Int, head: Int, col: Int) -> SIMD[dtype, width] & ImplicitlyCopyable, //, head_dim: Int, simd_width: Int, warps_per_block: Int, multiply_before_cast: Bool, interleaved: Bool, has_nope_prefix: Bool, rope_dim: Int](norm_rope: Bool, global_token_idx: Int, head_idx: Int, cache: cache_t, gamma: TileTensor[dtype, gamma_layout, gamma_origin, Engine=gamma_engine], freqs_cis: TileTensor[freq_dtype, freqs_layout, freqs_origin, Engine=freqs_engine], epsilon: Float32, weight_offset: Scalar[dtype], input_fn: InputFnType, input_row_offsets: TileTensor[.uint32, offsets_layout, offsets_origin, Engine=offsets_engine])

Writes one (token, head) K or V row from staging into a paged cache.

The row is read through input_fn. With norm_rope (a K row) it is RMS-normed and roped with the arithmetic of _fused_qk_rms_norm_rope_process_row; without it (a V row) it is stored as read. A uint8 cache stores the row as NVFP4: packed E2M1 at head_dim / 2 bytes per row plus one E4M3 scale per 16 elements, quantized from the row rounded to dtype. Any other cache stores the row cast to its dtype.

norm_rope must be uniform across the block, because the norm and the RoPE staging both synchronize the block.

Was this page helpful?