For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
latent_sparse_attention_ragged_paged
def latent_sparse_attention_ragged_paged[q_type: DType, out_type: DType, //, window: Int, split_k: Bool = True, portable_p_bf16: Bool = False](output: TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], comp_indices: TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type], attn_sink: TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type], swa_cache: PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_], comp_cache: PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_], scale: Float32, ctx: DeviceContext)
Attends every query row to its window keys and its listed compressed entries.
Parameters:
- q_type (
DType): Query element type (inferred). - out_type (
DType): Output element type (inferred). - window (
Int): Sliding window length in tokens; a query atpossees positionsmax(0, pos - window + 1) .. pos. - split_k (
Bool): Whether the SM100 route may split a row's keys across CTAs. Split partial outputs are rounded to the output dtype before the combine;Falseruns one CTA per row and rounds the output once, as the portable kernel does. - portable_p_bf16 (
Bool): Diagnostic. Runs the portable GPU kernel even on SM100, with each attention weight rounded to bf16 before it multiplies its value row, the one rounding the SM100 decode makes and the portable kernel does not.
Args:
- output (
TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type]):[num_rows, num_heads, head_dim]. - q (
TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type]):[num_rows, num_heads, head_dim], the last axis contiguous. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]):[batch + 1]ragged row offsets ofq. - comp_indices (
TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type]):[num_rows, num_comp]entry indices into the compressed leaf;-1marks an unused slot. - attn_sink (
TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type]):[num_heads]per-head sink logits. - swa_cache (
PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_]): This layer's sliding-window leaf. - comp_cache (
PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_]): This layer's compressed leaf. - scale (
Float32): Softmax scale applied to the scores. - ctx (
DeviceContext): Device context used to enqueue the GPU kernel.