For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MLAKPoolAppendPaged
struct MLAKPoolAppendPaged
Registers the attention.mla.kpool.append.paged graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[*, head_dim: Int, kpool: Int, quantization_granularity: Int](tail: T, k: T, gate: T, ape: T, input_row_offsets: T, slot_idx: T, scale_ub: Float32, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, k_scales_blocks: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, ctx: DeviceContext)
Pools, FP8-quantizes and caches every k-pool this call completes.
One launch for what attention.mla.kpool.ring_close,
attention.mla.kpool.compress, quantize_dynamic_scaled_float8 and
the paged K and K-scale stores do together; see kpool_append_ragged_paged.
Parameters:
- head_dim (
Int): Channels per key; also the kernel's block width. - kpool (
Int): Tokens per pool. - quantization_granularity (
Int): Cache values per scale.
Args:
- tail (
T): Persistent slot-indexed ring[max_slots, 2, kpool, head_dim]. Read only. - k (
T): This call's layer-normed keys[total_tokens, head_dim]. - gate (
T): This call's gate scores[total_tokens, head_dim]. - ape (
T): Within-pool position embedding[kpool, head_dim], float32. - input_row_offsets (
T): Token row offsets per request[batch + 1]. - slot_idx (
T): Ring slot per request[batch]. - scale_ub (
Float32): Upper bound on a pool's max-abs before its scale. - kv_blocks (
T): The pooled key cache's pages. - page_stride (
T): Page stride ofkv_blocks. - cache_lengths (
T): Cached-prefix length per request[batch]. - kv_lookup_table (
T): Page table. - max_prompt_length (
T): Longest new-token count in the batch. - max_cache_length (
T): Longest cached prefix in the batch. - k_scales_blocks (
T): The pooled key cache's scale pages. - scales_page_stride (
T): Page stride ofk_scales_blocks. - k_scales_lookup_table (
T): Scale page table. - layer_idx (
UInt32): The layer's index in the cache. - ctx (
DeviceContext): Device context for GPU execution.