For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mla
GPU kernels for Multi-head Latent Attention (MLA) decoding and prefill.
Provides the flare_mla_decoding and flare_mla_prefill entrypoints plus
their platform-specific dispatch and kernel implementations targeting NVIDIA
(SM80/SM100) and AMD (gfx950) GPUs, including split-K reduction, multi-token
prediction (MTP) query folding, per-token scale, and sparse-attention support.
comptime values
AMD_MLA_DECODE_FOLD_M_MAX
comptime AMD_MLA_DECODE_FOLD_M_MAX = 2048
AMD_MLA_PREFILL_COMBINE_WARPS
comptime AMD_MLA_PREFILL_COMBINE_WARPS = 4
AMD_MLA_PREFILL_FULL_GRID_SPLIT_MIN_KEYS
comptime AMD_MLA_PREFILL_FULL_GRID_SPLIT_MIN_KEYS = 16384
AMD_MLA_PREFILL_MAX_SPLITS
comptime AMD_MLA_PREFILL_MAX_SPLITS = 32
AMD_MLA_PREFILL_RESIDENT_CTAS
comptime AMD_MLA_PREFILL_RESIDENT_CTAS = 512
AMD_MLA_PREFILL_SPLIT_MIN_KEYS
comptime AMD_MLA_PREFILL_SPLIT_MIN_KEYS = 1024
AMD_MLA_PREFILL_SPLIT_TARGET_CTAS
comptime AMD_MLA_PREFILL_SPLIT_TARGET_CTAS = 1024
MLA_DECODE_FOLD_MAX_SEQ_LEN
comptime MLA_DECODE_FOLD_MAX_SEQ_LEN = AMD_MLA_DECODE_FOLD_MAX_SEQ_LEN
MLA_DECODE_MAX_SEQ_LEN
comptime MLA_DECODE_MAX_SEQ_LEN = AMD_MLA_DECODE_MTP_SEQ_LEN
Structs
-
AMDMLA: AMD MLA kernels. -
LegacyNVMLA: Pre-Blackwell NVIDIA MLA kernels. -
SM100MLA: SM100 (Blackwell datacenter) MLA kernels.
Traits
-
MLAKernels: A GPU vendor's MLA decode and prefill dispatch.
Functions
-
amd_mla_prefill_num_splits: Returns the KV split count for the AMD MLA prefill grid. -
flare_mla_decoding: MLA decoding kernel that would only be called in the optimized compute graph. -
flare_mla_decoding_rows: Runs MLA decode over a K operand withnum_keyskeys per entry. -
flare_mla_prefill: MLA prefill kernel that would only be called in the optimized compute graph. Only supports ragged Q/K/V inputs. -
mla_decode_max_seq_len: Max query tokens (S) the MLA decode branch can fold for this config. -
mla_decoding_amd: MLA decoding kernel that computes single-batch attention per CTA. -
mla_decoding_legacy: MLA decoding kernel that computes single-batch attention per CTA. -
mla_decoding_single_batch: Flash attention v2 algorithm. -
mla_mask_places_query_span: Returns whether the named mask derives its key spans from the query width. -
mla_prefill_amd: -
mla_prefill_legacy: -
mla_prefill_plan: This calls a GPU kernel that plans how to process a batch of sequences with varying lengths using a fixed-size buffer. -
mla_prefill_plan_kernel: Plans how to process a batch of varying-length sequences through a fixed-size buffer. -
mla_prefill_single_batch: MLA for encoding where seqlen > 1. -
mla_prefill_split_kv_combine: Merges the KV splits of an AMD MLA prefill into the output. -
mla_splitk_reduce: -
set_buffer_lengths_to_zero: Zeroes out every element of a 1D buffer-lengths tensor.