IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo trait

MLAGraphKernels

A GPU vendor's kernels for the MLA graph branches.

Each branch takes one as kernels, so a graph op registered for one vendor reaches only that vendor's kernels; see MLAKernels.

Implemented traits​

AnyType

comptime members​

Kernels​

comptime Kernels

The vendor's MLA decode and prefill dispatch.

Required methods​

sparse_prefill​

static def sparse_prefill[collection_t: KVCollectionT, AttnSinkPtrType: OptionalPointer, //, num_heads: Int, k_cache_dim: Int, kv_latent_dim: Int, indices_stride: Int, nvfp4_convert_ratio: Int = Int(0), nvfp4_convert_max_rows: Int = Int(0)](raw_output: TileTensor[Engine=raw_output.Engine, linear_idx_type=raw_output.linear_idx_type], q: TileTensor[Engine=q.Engine, linear_idx_type=q.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], d_indices: MutPointer[Int32, MutAnyOrigin], topk_lengths: MutPointer[Int32, MutAnyOrigin], attn_sink_ptr: AttnSinkPtrType, attn_sink_opt: Optional[Pointer[Float32, ImmutAnyOrigin]], scale: Float32, ctx: DeviceContext)

Runs sparse MLA prefill attention over the paged latent cache.

Parameters:

  • ​collection_t (KVCollectionT): KV collection type (inferred).
  • ​AttnSinkPtrType (OptionalPointer): OptionalPointer type of attn_sink_ptr (inferred).
  • ​num_heads (Int): Number of query heads.
  • ​k_cache_dim (Int): Width of a latent cache row.
  • ​kv_latent_dim (Int): Width of the latent (value) part of a row.
  • ​indices_stride (Int): Top-k slots per query token.
  • ​nvfp4_convert_ratio (Int): See nvfp4_prefill_converts_once; 0 always runs the NVFP4 kernel.
  • ​nvfp4_convert_max_rows (Int): See nvfp4_prefill_converts_once.

Args:

latent_sparse_attention​

static def latent_sparse_attention[q_type: DType, out_type: DType, //, window: Int, split_k: Bool = True, portable_p_bf16: Bool = False](output: TileTensor[out_type, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[q_type, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], comp_indices: TileTensor[.int32, Engine=comp_indices.Engine, linear_idx_type=comp_indices.linear_idx_type], attn_sink: TileTensor[.float32, Engine=attn_sink.Engine, linear_idx_type=attn_sink.linear_idx_type], swa_cache: PagedKVCache[scale_dtype_=swa_cache.scale_dtype_, quantization_granularity_=swa_cache.quantization_granularity_], comp_cache: PagedKVCache[scale_dtype_=comp_cache.scale_dtype_, quantization_granularity_=comp_cache.quantization_granularity_], scale: Float32, ctx: DeviceContext)

Runs latent sparse attention; see latent_sparse_attention_ragged_paged.

Parameters:

  • ​q_type (DType): Query element type (inferred).
  • ​out_type (DType): Output element type (inferred).
  • ​window (Int): Sliding window length in tokens.
  • ​split_k (Bool): Whether the SM100 route may split a row's keys.
  • ​portable_p_bf16 (Bool): Diagnostic; see latent_sparse_attention_ragged_paged.

Args:

Was this page helpful?