For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_flare_mla_decompress_k_cache_ragged_paged
def generic_flare_mla_decompress_k_cache_ragged_paged[dtype: DType](buffer_row_offsets_1d: TileTensor[.uint32, Engine=buffer_row_offsets_1d.Engine, linear_idx_type=buffer_row_offsets_1d.linear_idx_type], cache_offsets_1d: TileTensor[.uint32, Engine=cache_offsets_1d.Engine, linear_idx_type=cache_offsets_1d.linear_idx_type], buffer_length: Int32, weight: TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type], kv_collection: PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_], layer_idx: UInt32, k_latent_buffer: TileTensor[dtype, Engine=k_latent_buffer.Engine, linear_idx_type=k_latent_buffer.linear_idx_type], k_buffer: TileTensor[dtype, Engine=k_buffer.Engine, linear_idx_type=k_buffer.linear_idx_type], context: DeviceContext)
Decompresses MLA latent K cache rows into a full K buffer via a matmul.
Gathers compressed latent K vectors from the paged KV cache into k_latent_buffer using buffer_row_offsets and cache_offsets, then multiplies by the down-projection weight to produce the decompressed K buffer.
Parameters:
- dtype (
DType): Data type of the weight and K buffers (inferred).
Args:
- buffer_row_offsets_1d (
TileTensor[.uint32, Engine=buffer_row_offsets_1d.Engine, linear_idx_type=buffer_row_offsets_1d.linear_idx_type]): Tensor denoting the start and end position of each K entry in the ragged buffer. - cache_offsets_1d (
TileTensor[.uint32, Engine=cache_offsets_1d.Engine, linear_idx_type=cache_offsets_1d.linear_idx_type]): Tensor denoting the start position of each K entry in the PagedKVCacheCollection. - buffer_length (
Int32): Number of K rows to decompress. - weight (
TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type]): Down-projection weight tensor applied to the latent K vectors. - kv_collection (
PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_]): The collection storing the KVCache entries for this layer, retrieved via layer_idx. - layer_idx (
UInt32): The index of the layer being executed, used to retrieve the KVCache objects from kv_collection. - k_latent_buffer (
TileTensor[dtype, Engine=k_latent_buffer.Engine, linear_idx_type=k_latent_buffer.linear_idx_type]): Mutable output buffer receiving the gathered latent K vectors. - k_buffer (
TileTensor[dtype, Engine=k_buffer.Engine, linear_idx_type=k_buffer.linear_idx_type]): Mutable output buffer receiving the decompressed K vectors. - context (
DeviceContext): The call context pointer, passed by the graph compiler.