IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

latent_sparse_attention_ragged_paged

def latent_sparse_attention_ragged_paged[q_type: DType, out_type: DType, //, target: StringSpan[ImmStaticOrigin], window: Int, split_k: Bool = True, portable_p_bf16: Bool = False](output: TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], comp_indices: TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type], attn_sink: TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type], swa_cache: PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_], comp_cache: PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_], scale: Float32, ctx: DeviceContextBase)

Attends every query row to its window keys and its listed compressed entries.

Parameters:

  • ​q_type (DType): Query element type (inferred).
  • ​out_type (DType): Output element type (inferred).
  • ​target (StringSpan[ImmStaticOrigin]): Compilation target string, selects the CPU or GPU path.
  • ​window (Int): Sliding window length in tokens; a query at pos sees positions max(0, pos - window + 1) .. pos.
  • ​split_k (Bool): Whether the SM100 route may split a row's keys across CTAs. Split partial outputs are rounded to the output dtype before the combine; False runs one CTA per row and rounds the output once, as the portable kernel does.
  • ​portable_p_bf16 (Bool): Diagnostic. Runs the portable GPU kernel even on SM100, with each attention weight rounded to bf16 before it multiplies its value row, the one rounding the SM100 decode makes and the portable kernel does not.

Args:

Was this page helpful?