For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_prefill_branch_bf16
def mla_prefill_branch_bf16[collection_t: KVCollectionT, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, mask_str: StaticString, OutputGatePtrType: OptionalPointer = NullPointer[.bfloat16], *, kernels: MLAGraphKernels](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, buffer_row_offsets: TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type], cache_offsets: TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type], buffer_length: Int, w_k: TileTensor[.bfloat16, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], kv_input_fn: KVInputFnType, ctx: DeviceContext, output_gate: OutputGatePtrType = null_pointer[OutputGatePtrType](), output_gate_row_stride: Int = Int(0))
BF16 MLA prefill path.
Applies RoPE and RMSNorm, up-projects latent KV to full K and V, then runs prefill attention.
Parameters:
- collection_t (
KVCollectionT): Type of the KV collection (inferred). - KVInputFnType (
def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred). - mask_str (
StaticString): Mask variant. - OutputGatePtrType (
OptionalPointer):OptionalPointertype ofoutput_gate(inferred, defaults toNullPointer); a null type applies no gate. - kernels (
MLAGraphKernels): The vendor's MLA graph kernels; seeMLAGraphKernels.
Args:
- output (
TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor of shape [tot_seq_len, num_heads, v_head_dim]. - q (
TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Combined query tensor containing both nope and rope parts. Shape: [tot_seq_len, num_heads, qk_nope_head_dim + qk_rope_head_dim]. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Indicates where each request starts and ends inq. Shape: [num_batches + 1]. - freqs_cis (
TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type]): Precomputed RoPE frequency values for rotary position embeddings. Shape: [max_seq_len, qk_rope_head_dim]. - kv_norm_gamma (
TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type]): RMSNorm gamma weights for normalizing the KV cache. Shape: [kv_lora_rank]. - kv_collection (
collection_t): Paged KV Cache object. - layer_idx (
UInt32): Layer index. - scale (
Float32): Scale for the attention calculation. - epsilon (
Float32): Small constant for numerical stability in RMSNorm. - buffer_row_offsets (
TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type]): Indicates where each request's KV latent values should be stored in the contiguous K buffer. This is a 1D tensor of shape [num_batches + 1]. - cache_offsets (
TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type]): Indicates the starting token position in the KV cache from which to copy KV latent values for each request. This is a 1D tensor of shape [num_batches + 1]. - buffer_length (
Int): The total number of tokens in the KV cache. Scalar. - w_k (
TileTensor[.bfloat16, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type]): Weight matrix for up-projecting the latent cache to full K. Shape: [num_heads * qk_nope_head_dim, kv_latent_dim]. - w_uv (
TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type]): Weight tensor for projecting latent values to V. Shape: [num_heads, v_head_dim, kv_latent_dim]. - kv_input_fn (
KVInputFnType): Closure that loads the KV latent values. Shape: [tot_seq_len, cache_head_dim]. Where cache_head_dim = kv_lora_rank- qk_rope_head_dim. It runs on the device, so it must copy what it captures.
- ctx (
DeviceContext): Device context. - output_gate (
OutputGatePtrType): Optional bfloat16[seq_len, num_heads * v_head_dim]per-token output gate applied in one pass after attention, which is the unfused fallback for a caller that could not get the gate folded into its own kernel. Row-strided byoutput_gate_row_stride. - output_gate_row_stride (
Int): Row stride ofoutput_gate, in elements.