For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_store_ragged_paged
def mla_store_ragged_paged[freq_dtype: DType, gamma_dtype: DType, collection_t: KVCollectionT, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](num_tokens: Int, input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[freq_dtype, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type], gamma: TileTensor[gamma_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, epsilon: Float32, kv_input_fn: KVInputFnType, ctx: DeviceContext, dcp_rank: Int = Int(0))
Stores raw MLA latent rows into the paged cache, normed and roped.
The store half of mla_prefill_decode_graph_bf16 without the attention:
the fused RoPE + RMSNorm + quantization kernel runs over every row the
caller supplies through kv_input_fn, RMSNorming the first kv_norm_dim
elements against gamma, roping the trailing rope slice at each row's
absolute position cache_length + token_idx, and writing the row to the
cache in the cache dtype. Rows a drafter stores but never queries need
this path; the attention graph always attends every query row, so it
cannot stand in for the store.
Parameters:
- freq_dtype (
DType): Data type of frequency cosine/sine values. - gamma_dtype (
DType): Data type of RMSNorm gamma weights. - collection_t (
KVCollectionT): Type of the KV cache collection (inferred). - KVInputFnType (
def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred). - dcp_size (
Int): Ranks of the context-parallel group that shares each request's KV positions; 1 stores every row. - dcp_interleave (
Int): Consecutive positions a rank owns before the next rank's turn (seeContextParallelLayout).
Args:
- num_tokens (
Int): Total rows to store; the caller's row count. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type]): Row offsets indicating request boundaries. Shape: [num_batches + 1]. - freqs_cis (
TileTensor[freq_dtype, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type]): Precomputed interleaved RoPE frequency values. Shape: [max_seq_len, rope_dim]; rowcache_length + token_idxropes a row. - gamma (
TileTensor[gamma_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): RMSNorm gamma weights. Shape: [kv_norm_dim]. - kv_collection (
collection_t): Paged KV cache collection. - layer_idx (
UInt32): Index of the current transformer layer. - epsilon (
Float32): Small constant for numerical stability in RMSNorm. - kv_input_fn (
KVInputFnType): Closure that loads the raw KV latent values. Shape: [num_tokens, cache_head_dim], where cache_head_dim = kv_norm_dim- rope_dim. It runs on the device, so it must copy what it captures.
- ctx (
DeviceContext): Device context for kernel execution. - dcp_rank (
Int): This rank's index in the context-parallel group. Outside a group (dcp_size == 1) the layout never reads it.