IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

MLAKPoolAppendPaged

struct MLAKPoolAppendPaged

Registers the attention.mla.kpool.append.paged graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[*, head_dim: Int, kpool: Int, quantization_granularity: Int](tail: T, k: T, gate: T, ape: T, input_row_offsets: T, slot_idx: T, scale_ub: Float32, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, k_scales_blocks: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, ctx: DeviceContext)

Pools, FP8-quantizes and caches every k-pool this call completes.

One launch for what attention.mla.kpool.ring_close, attention.mla.kpool.compress, quantize_dynamic_scaled_float8 and the paged K and K-scale stores do together; see kpool_append_ragged_paged.

Parameters:

  • ​head_dim (Int): Channels per key; also the kernel's block width.
  • ​kpool (Int): Tokens per pool.
  • ​quantization_granularity (Int): Cache values per scale.

Args:

  • ​tail (T): Persistent slot-indexed ring [max_slots, 2, kpool, head_dim]. Read only.
  • ​k (T): This call's layer-normed keys [total_tokens, head_dim].
  • ​gate (T): This call's gate scores [total_tokens, head_dim].
  • ​ape (T): Within-pool position embedding [kpool, head_dim], float32.
  • ​input_row_offsets (T): Token row offsets per request [batch + 1].
  • ​slot_idx (T): Ring slot per request [batch].
  • ​scale_ub (Float32): Upper bound on a pool's max-abs before its scale.
  • ​kv_blocks (T): The pooled key cache's pages.
  • ​page_stride (T): Page stride of kv_blocks.
  • ​cache_lengths (T): Cached-prefix length per request [batch].
  • ​kv_lookup_table (T): Page table.
  • ​max_prompt_length (T): Longest new-token count in the batch.
  • ​max_cache_length (T): Longest cached prefix in the batch.
  • ​k_scales_blocks (T): The pooled key cache's scale pages.
  • ​scales_page_stride (T): Page stride of k_scales_blocks.
  • ​k_scales_lookup_table (T): Scale page table.
  • ​layer_idx (UInt32): The layer's index in the cache.
  • ​ctx (DeviceContext): Device context for GPU execution.

Was this page helpful?