For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
LegacyNVMLAGraph
struct LegacyNVMLAGraph
Pre-SM100 NVIDIA MLA graph kernels.
Implemented traits
AnyType,
Deinitable,
MLAGraphKernels,
Movable
comptime members
Kernels
comptime Kernels = LegacyNVMLA
Methods
sparse_prefill
static def sparse_prefill[collection_t: KVCollectionT, AttnSinkPtrType: OptionalPointer, //, num_heads: Int, k_cache_dim: Int, kv_latent_dim: Int, indices_stride: Int, nvfp4_convert_ratio: Int = Int(0), nvfp4_convert_max_rows: Int = Int(0)](raw_output: TileTensor[Engine=raw_output.Engine, linear_idx_type=raw_output.linear_idx_type], q: TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], d_indices: MutPointer[Int32, MutAnyOrigin], topk_lengths: MutPointer[Int32, MutAnyOrigin], attn_sink_ptr: AttnSinkPtrType, attn_sink_opt: Optional[Pointer[Float32, ImmutAnyOrigin]], scale: Float32, ctx: DeviceContext)
Sparse MLA prefill has no pre-SM100 NVIDIA kernel.
latent_sparse_attention
static def latent_sparse_attention[q_type: DType, out_type: DType, //, window: Int, split_k: Bool = True, portable_p_bf16: Bool = False](output: TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], comp_indices: TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type], attn_sink: TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type], swa_cache: PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_], comp_cache: PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_], scale: Float32, ctx: DeviceContext)
Runs the portable GPU kernel.