For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
nvfp4_kv_write_row
def nvfp4_kv_write_row[cache_t: KVCacheT, gamma_layout: TensorLayout, gamma_origin: ImmOrigin, gamma_engine: TensorEngine, freqs_layout: TensorLayout, freqs_origin: ImmOrigin, freqs_engine: TensorEngine, offsets_layout: TensorLayout, offsets_origin: ImmOrigin, offsets_engine: TensorEngine, dtype: DType, freq_dtype: DType, InputFnType: def[width: Int, alignment: Int](token: Int, head: Int, col: Int) -> SIMD[dtype, width] & ImplicitlyCopyable, //, head_dim: Int, simd_width: Int, warps_per_block: Int, multiply_before_cast: Bool, interleaved: Bool, has_nope_prefix: Bool, rope_dim: Int](norm_rope: Bool, global_token_idx: Int, head_idx: Int, cache: cache_t, gamma: TileTensor[dtype, gamma_layout, gamma_origin, Engine=gamma_engine], freqs_cis: TileTensor[freq_dtype, freqs_layout, freqs_origin, Engine=freqs_engine], epsilon: Float32, weight_offset: Scalar[dtype], input_fn: InputFnType, input_row_offsets: TileTensor[.uint32, offsets_layout, offsets_origin, Engine=offsets_engine])
Writes one (token, head) K or V row from staging into a paged cache.
The row is read through input_fn. With norm_rope (a K row) it is
RMS-normed and roped with the arithmetic of
_fused_qk_rms_norm_rope_process_row; without it (a V row) it is stored
as read. A uint8 cache stores the row as NVFP4: packed E2M1 at
head_dim / 2 bytes per row plus one E4M3 scale per 16 elements,
quantized from the row rounded to dtype. Any other cache stores the row
cast to its dtype.
norm_rope must be uniform across the block, because the norm and the
RoPE staging both synchronize the block.