IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

LegacyNVMLAGraph

struct LegacyNVMLAGraph

Pre-SM100 NVIDIA MLA graph kernels.

Implemented traits​

AnyType, Deinitable, MLAGraphKernels, Movable

comptime members​

Kernels​

comptime Kernels = LegacyNVMLA

Methods​

sparse_prefill​

static def sparse_prefill[collection_t: KVCollectionT, AttnSinkPtrType: OptionalPointer, //, num_heads: Int, k_cache_dim: Int, kv_latent_dim: Int, indices_stride: Int, nvfp4_convert_ratio: Int = Int(0), nvfp4_convert_max_rows: Int = Int(0)](raw_output: TileTensor[Engine=raw_output.Engine, linear_idx_type=raw_output.linear_idx_type], q: TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], d_indices: MutPointer[Int32, MutAnyOrigin], topk_lengths: MutPointer[Int32, MutAnyOrigin], attn_sink_ptr: AttnSinkPtrType, attn_sink_opt: Optional[Pointer[Float32, ImmutAnyOrigin]], scale: Float32, ctx: DeviceContext)

Sparse MLA prefill has no pre-SM100 NVIDIA kernel.

latent_sparse_attention​

static def latent_sparse_attention[q_type: DType, out_type: DType, //, window: Int, split_k: Bool = True, portable_p_bf16: Bool = False](output: TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], comp_indices: TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type], attn_sink: TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type], swa_cache: PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_], comp_cache: PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_], scale: Float32, ctx: DeviceContext)

Runs the portable GPU kernel.

Was this page helpful?