For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
flare_mla_decoding_rows
def flare_mla_decoding_rows[k_t: MHAOperand, mask_t: MHAMask, dtype: DType, //, config: MHAConfig[dtype]](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: k_t, num_keys: Int, mask_functor: mask_t, scale: Float32, ctx: DeviceContext, scalar_args_buf: NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type], num_partitions: Optional[Int] = None)
Runs MLA decode over a K operand with num_keys keys per entry.
The operand form of the TileTensor overload: k decides where each
key row lives, so a SparseRowsMHAOperand attends a top-k selection
straight from the paged cache, with no gathered copy. Each entry has one
absorbed-latent KV head.
Parameters:
- k_t (
MHAOperand): The K operand type (inferred). - mask_t (
MHAMask): The mask type (inferred). - dtype (
DType): Q's element type (inferred). - config (
MHAConfig[dtype]): The attention configuration.
Args:
- output (
TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]):[batch, 1, num_heads, v_depth]output. - q (
TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]):[batch, 1, num_heads, depth]queries. - k (
k_t): The K (and aliased V) operand. - num_keys (
Int): Keys per batch entry;mask_functorhides the invalid ones. - mask_functor (
mask_t): The attention mask. - scale (
Float32): The softmax scale. - ctx (
DeviceContext): The device context. - scalar_args_buf (
NullableTileTensor[.int64, Engine=scalar_args_buf.Engine, linear_idx_type=scalar_args_buf.linear_idx_type]): Precomputed dispatch scalars, or null. - num_partitions (
Optional[Int]): Split-K partition count;Nonepicks heuristically.