For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_fused_rope_rmsnorm_quantization
def mla_fused_rope_rmsnorm_quantization[dtype: DType, freq_dtype: DType, gamma_dtype: DType, collection_t: KVCollectionT, out_rope_dtype: DType, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, store_only: Bool = False, dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](q_rope_output: TileTensor[out_rope_dtype, Engine=q_rope_output.Engine, address_space=q_rope_output.address_space, linear_idx_type=q_rope_output.linear_idx_type], q_rope: TileTensor[dtype, Engine=q_rope.Engine, address_space=q_rope.address_space, linear_idx_type=q_rope.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[freq_dtype, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type], gamma: TileTensor[gamma_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, epsilon: Float32, kv_input_fn: KVInputFnType, ctx: DeviceContext, dcp_rank: Int = Int(0))
Launches the fused RoPE and RMSNorm kernel for MLA attention.
This function fuses three operations:
- RoPE applied to query and key cache rope parts.
- RMSNorm applied to the non-rope portion of the key cache.
- Quantization of the final results before writing to the KVCache object.
Parameters:
- dtype (
DType): Data type of query tensors. - freq_dtype (
DType): Data type of frequency cosine/sine values. - gamma_dtype (
DType): Data type of RMSNorm gamma weights. - collection_t (
KVCollectionT): Type of the KV cache collection. - out_rope_dtype (
DType): Data type of the RoPE output values. - KVInputFnType (
def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred). - store_only (
Bool): Skip the query projection when storing cache rows. - dcp_size (
Int): Ranks of the decode context-parallel group that shares each request's KV positions; 1 stores every row. - dcp_interleave (
Int): Consecutive positions a rank owns before the next rank's turn (seeContextParallelLayout).
Args:
- q_rope_output (
TileTensor[out_rope_dtype, Engine=q_rope_output.Engine, address_space=q_rope_output.address_space, linear_idx_type=q_rope_output.linear_idx_type]): Output tensor for RoPE-applied query projections. Shape: [tot_seq_len, num_heads, rope_dim]. - q_rope (
TileTensor[dtype, Engine=q_rope.Engine, address_space=q_rope.address_space, linear_idx_type=q_rope.linear_idx_type]): Input query rope projections. Shape: [tot_seq_len, num_heads, rope_dim]. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type]): Row offsets indicating request boundaries. Shape: [num_batches + 1]. - freqs_cis (
TileTensor[freq_dtype, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type]): Precomputed RoPE frequency values. Shape: [max_seq_len, rope_dim]. - gamma (
TileTensor[gamma_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): RMSNorm gamma weights. Shape: [kv_norm_dim]. - kv_collection (
collection_t): Paged KV cache collection. - layer_idx (
UInt32): Index of the current transformer layer. - epsilon (
Float32): Small constant for numerical stability in RMSNorm. - kv_input_fn (
KVInputFnType): Closure that loads the KV latent values. Shape: [tot_seq_len, cache_head_dim]. Where cache_head_dim = kv_lora_rank- qk_rope_head_dim. It runs on the device, so it must copy what it captures.
- ctx (
DeviceContext): Device context for kernel execution. - dcp_rank (
Int): This rank's index in the decode context-parallel group. RoPE uses every token's global position; only the rank that owns a position stores its row, at its local slot.