IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_mla_store_ragged_paged

struct Struct_mla_store_ragged_paged

Registers the attention.mla.store.ragged.paged graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](kv: T, input_row_offsets: T, freqs_cis: T, kv_norm_gamma: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, epsilon: Float32, dcp_rank: Int32, context: DeviceContext)

Stores raw MLA latent rows into the paged cache, normed and roped.

The store half of attention.mla.graph.prefill.decode.paged without the attention: each row of kv -- a request's token_idx-th row, found via input_row_offsets -- is RMSNormed on its first kv_lora_rank elements against kv_norm_gamma, roped on its trailing rope slice at its absolute position cache_length + token_idx through freqs_cis, cast to the cache dtype, and written to kv_blocks. Rows a speculative drafter stores but never queries need this; the attention op always attends every query row, so it cannot stand in for the store.

Args:

  • ​kv (T): Raw (un-normalized) latent rows, [total_seq_len, kv_lora_rank + qk_rope_head_dim], bfloat16.
  • ​input_row_offsets (T): Row offsets per request, [batch + 1].
  • ​freqs_cis (T): Interleaved RoPE table, [max_position, qk_rope_head_dim]; row cache_length + token_idx ropes a row.
  • ​kv_norm_gamma (T): RMSNorm gamma, [kv_lora_rank].
  • ​kv_blocks (T): The paged cache's blocks, [pages, 1, layers, page_size, 1, kv_lora_rank + qk_rope_head_dim].
  • ​page_stride (T): Page-to-page distance in elements; -1 when packed.
  • ​cache_lengths (T): Cached tokens per request, [batch].
  • ​kv_lookup_table (T): Page lookup table, [batch, pages_per_seq].
  • ​max_prompt_length (T): The batch's max prompt length, [1].
  • ​max_cache_length (T): The batch's max cache length, [1].
  • ​layer_idx (UInt32): Layer of the cache to write.
  • ​epsilon (Float32): RMSNorm epsilon.
  • ​dcp_rank (Int32): This rank's index in the context-parallel group, a host scalar. Outside a group (dcp_size == 1) it is 0 and the layout never reads it.
  • ​context (DeviceContext): Device context for GPU execution.

Was this page helpful?