For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo trait
MLAGraphKernels
A GPU vendor's kernels for the MLA graph branches.
Each branch takes one as kernels, so a graph op registered for one vendor
reaches only that vendor's kernels; see MLAKernels.
Implemented traits
comptime members
Kernels
comptime Kernels
The vendor's MLA decode and prefill dispatch.
Required methods
sparse_prefill
static def sparse_prefill[collection_t: KVCollectionT, AttnSinkPtrType: OptionalPointer, //, num_heads: Int, k_cache_dim: Int, kv_latent_dim: Int, indices_stride: Int, nvfp4_convert_ratio: Int = Int(0), nvfp4_convert_max_rows: Int = Int(0)](raw_output: TileTensor[Engine=raw_output.Engine, linear_idx_type=raw_output.linear_idx_type], q: TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], d_indices: MutPointer[Int32, MutAnyOrigin], topk_lengths: MutPointer[Int32, MutAnyOrigin], attn_sink_ptr: AttnSinkPtrType, attn_sink_opt: Optional[Pointer[Float32, ImmutAnyOrigin]], scale: Float32, ctx: DeviceContext)
Runs sparse MLA prefill attention over the paged latent cache.
Parameters:
- collection_t (
KVCollectionT): KV collection type (inferred). - AttnSinkPtrType (
OptionalPointer):OptionalPointertype ofattn_sink_ptr(inferred). - num_heads (
Int): Number of query heads. - k_cache_dim (
Int): Width of a latent cache row. - kv_latent_dim (
Int): Width of the latent (value) part of a row. - indices_stride (
Int): Top-k slots per query token. - nvfp4_convert_ratio (
Int): Seenvfp4_prefill_converts_once; 0 always runs the NVFP4 kernel. - nvfp4_convert_max_rows (
Int): Seenvfp4_prefill_converts_once.
Args:
- raw_output (
TileTensor[Engine=raw_output.Engine, linear_idx_type=raw_output.linear_idx_type]): Latent attention output,[seq_len, heads, latent]. - q (
TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Absorbed Q,[seq_len, heads, k_cache_dim], staged in FP8 for an FP8 or NVFP4 cache. - kv_collection (
collection_t): Paged latent cache collection. - layer_idx (
UInt32): Layer whose cache is read. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Ragged query row offsets. - d_indices (
MutPointer[Int32, MutAnyOrigin]): Physical cache rows,[seq_len, indices_stride]. - topk_lengths (
MutPointer[Int32, MutAnyOrigin]): Valid slots per query token,[seq_len]. - attn_sink_ptr (
AttnSinkPtrType): Attention sink, typed for the NVFP4 convert path. - attn_sink_opt (
Optional[Pointer[Float32, ImmutAnyOrigin]]): The same sink, nullable at runtime. - scale (
Float32): Softmax scale. - ctx (
DeviceContext): Device context.
latent_sparse_attention
static def latent_sparse_attention[q_type: DType, out_type: DType, //, window: Int, split_k: Bool = True, portable_p_bf16: Bool = False](output: TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], comp_indices: TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type], attn_sink: TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type], swa_cache: PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_], comp_cache: PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_], scale: Float32, ctx: DeviceContext)
Runs latent sparse attention; see latent_sparse_attention_ragged_paged.
Parameters:
- q_type (
DType): Query element type (inferred). - out_type (
DType): Output element type (inferred). - window (
Int): Sliding window length in tokens. - split_k (
Bool): Whether the SM100 route may split a row's keys. - portable_p_bf16 (
Bool): Diagnostic; seelatent_sparse_attention_ragged_paged.
Args:
- output (
TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type]):[num_rows, num_heads, head_dim]. - q (
TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type]):[num_rows, num_heads, head_dim]. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]):[batch + 1]ragged row offsets ofq. - comp_indices (
TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type]):[num_rows, num_comp]compressed entry indices. - attn_sink (
TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type]):[num_heads]per-head sink logits. - swa_cache (
PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_]): This layer's sliding-window leaf. - comp_cache (
PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_]): This layer's compressed leaf. - scale (
Float32): Softmax scale. - ctx (
DeviceContext): Device context.