For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Struct_mla_store_ragged_paged
struct Struct_mla_store_ragged_paged
Registers the attention.mla.store.ragged.paged graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](kv: T, input_row_offsets: T, freqs_cis: T, kv_norm_gamma: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, epsilon: Float32, dcp_rank: Int32, context: DeviceContext)
Stores raw MLA latent rows into the paged cache, normed and roped.
The store half of attention.mla.graph.prefill.decode.paged without the
attention: each row of kv -- a request's token_idx-th row, found
via input_row_offsets -- is RMSNormed on its first kv_lora_rank
elements against kv_norm_gamma, roped on its trailing rope slice at
its absolute position cache_length + token_idx through freqs_cis,
cast to the cache dtype, and written to kv_blocks. Rows a
speculative drafter stores but never queries need this; the attention
op always attends every query row, so it cannot stand in for the
store.
Args:
- kv (
T): Raw (un-normalized) latent rows,[total_seq_len, kv_lora_rank + qk_rope_head_dim], bfloat16. - input_row_offsets (
T): Row offsets per request,[batch + 1]. - freqs_cis (
T): Interleaved RoPE table,[max_position, qk_rope_head_dim]; rowcache_length + token_idxropes a row. - kv_norm_gamma (
T): RMSNorm gamma,[kv_lora_rank]. - kv_blocks (
T): The paged cache's blocks,[pages, 1, layers, page_size, 1, kv_lora_rank + qk_rope_head_dim]. - page_stride (
T): Page-to-page distance in elements; -1 when packed. - cache_lengths (
T): Cached tokens per request,[batch]. - kv_lookup_table (
T): Page lookup table,[batch, pages_per_seq]. - max_prompt_length (
T): The batch's max prompt length,[1]. - max_cache_length (
T): The batch's max cache length,[1]. - layer_idx (
UInt32): Layer of the cache to write. - epsilon (
Float32): RMSNorm epsilon. - dcp_rank (
Int32): This rank's index in the context-parallel group, a host scalar. Outside a group (dcp_size == 1) it is 0 and the layout never reads it. - context (
DeviceContext): Device context for GPU execution.