IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_mla_store_ragged_paged_nvfp4

struct Struct_mla_store_ragged_paged_nvfp4

Registers the attention.mla.store.ragged.paged.nvfp4 graph op with the graph compiler.

The scaled counterpart of attention.mla.store.ragged.paged. A packed NVFP4 latent carries a scale pool beside its values, and the unscaled registration has no operands to take one, so the two are separate registrations rather than one with optional inputs. The cache operands arrive in the order flatten_without_attention_dispatch_metadata emits for quantized cache params.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](kv: T, input_row_offsets: T, freqs_cis: T, kv_norm_gamma: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, kv_scales: T, scales_page_stride: T, scales_lookup_table: T, layer_idx: UInt32, epsilon: Float32, dcp_rank: Int32, context: DeviceContext)

Stores raw MLA latent rows into a packed NVFP4 paged cache.

Behaves as attention.mla.store.ragged.paged, writing each row of kv normed, roped and quantized at its absolute position cache_length + token_idx, but against a cache whose rows are packed e2m1 bytes with a per-group scale. The row's trailing rope dims stay unscaled FP8, so a row's logical width is twice its scaled prefix plus that tail.

Args:

  • ​kv (T): Raw (un-normalized) latent rows, [total_seq_len, kv_lora_rank + qk_rope_head_dim], bfloat16.
  • ​input_row_offsets (T): Row offsets per request, [batch + 1].
  • ​freqs_cis (T): Interleaved RoPE table, [max_position, qk_rope_head_dim]; row cache_length + token_idx ropes a row.
  • ​kv_norm_gamma (T): RMSNorm gamma, [kv_lora_rank].
  • ​kv_blocks (T): The paged cache's packed blocks, [pages, 1, layers, page_size, 1, packed_row_bytes].
  • ​page_stride (T): Page-to-page distance in elements; -1 when packed.
  • ​cache_lengths (T): Cached tokens per request, [batch].
  • ​kv_lookup_table (T): Page lookup table, [batch, pages_per_seq].
  • ​max_prompt_length (T): The batch's max prompt length, [1].
  • ​max_cache_length (T): The batch's max cache length, [1].
  • ​kv_scales (T): The scale pool, kv_blocks's shape with its last dimension the scales per row.
  • ​scales_page_stride (T): Page-to-page distance of kv_scales, which pads independently of the values.
  • ​scales_lookup_table (T): Page lookup table of the scales.
  • ​layer_idx (UInt32): Layer of the cache to write.
  • ​epsilon (Float32): RMSNorm epsilon.
  • ​dcp_rank (Int32): This rank's index in the context-parallel group, a host scalar. Outside a group (dcp_size == 1) it is 0 and the layout never reads it.
  • ​context (DeviceContext): Device context for GPU execution.

Was this page helpful?