IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_store_ragged_paged

def mla_store_ragged_paged[freq_dtype: DType, gamma_dtype: DType, collection_t: KVCollectionT, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](num_tokens: Int, input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[freq_dtype, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type], gamma: TileTensor[gamma_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, epsilon: Float32, kv_input_fn: KVInputFnType, ctx: DeviceContext, dcp_rank: Int = Int(0))

Stores raw MLA latent rows into the paged cache, normed and roped.

The store half of mla_prefill_decode_graph_bf16 without the attention: the fused RoPE + RMSNorm + quantization kernel runs over every row the caller supplies through kv_input_fn, RMSNorming the first kv_norm_dim elements against gamma, roping the trailing rope slice at each row's absolute position cache_length + token_idx, and writing the row to the cache in the cache dtype. Rows a drafter stores but never queries need this path; the attention graph always attends every query row, so it cannot stand in for the store.

Parameters:

  • ​freq_dtype (DType): Data type of frequency cosine/sine values.
  • ​gamma_dtype (DType): Data type of RMSNorm gamma weights.
  • ​collection_t (KVCollectionT): Type of the KV cache collection (inferred).
  • ​KVInputFnType (def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred).
  • ​dcp_size (Int): Ranks of the context-parallel group that shares each request's KV positions; 1 stores every row.
  • ​dcp_interleave (Int): Consecutive positions a rank owns before the next rank's turn (see ContextParallelLayout).

Args:

Was this page helpful?