For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo function

fused_qk_rope_ragged

def fused_qk_rope_ragged[dtype: DType, freq_dtype: DType, collection_t: KVCollectionT, //, cache_t: KVCacheT, *, interleaved: Bool, target: StringSlice[ImmStaticOrigin], mrope_types: TypeList[mrope_types.values] = *?, mrope_section: Optional[Coord[mrope_types]] = None, PositionIdsLayoutType: TensorLayout = Layout[*?, *?]](q_proj: TileTensor[dtype, Storage=q_proj.Storage, address_space=q_proj.address_space, linear_idx_type=q_proj.linear_idx_type], input_row_offsets: TileTensor[DType.uint32, Storage=input_row_offsets.Storage, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type], kv_collection: collection_t, freqs_cis: TileTensor[freq_dtype, Storage=freqs_cis.Storage, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type], position_ids: OptionalReg[TileTensor[DType.uint32, PositionIdsLayoutType, ImmutAnyOrigin]], layer_idx: UInt32, output: TileTensor[dtype, Storage=output.Storage, address_space=output.address_space, linear_idx_type=output.linear_idx_type], context: DeviceContext)

Applies RoPE (Rotary Position Embedding) to query and key tensors.

This function can applies RoPE only to the last rope_dim elements of each head, leaving the first unroped_dim elements unchanged. This is required for DeepSeek models where only part of each head undergoes rotary transformation.

Parameters:

dtype (DType): The element type of q_proj and output (inferred).
freq_dtype (DType): The element type of freqs_cis (inferred).
collection_t (KVCollectionT): The KV cache collection type holding the key cache (inferred).
cache_t (KVCacheT): The KV cache type used to store and load key cache entries.
interleaved (Bool): Whether the RoPE weights use interleaved real and imaginary layout.
target (StringSlice[ImmStaticOrigin]): The compilation target string for the kernel.
mrope_types (TypeList[mrope_types.values]): The coordinate element types for multimodal RoPE sections (defaults to a single CoordLike type).
mrope_section (Optional[Coord[mrope_types]]): Optional section boundaries splitting the head dimension into position-id groups for multimodal RoPE (defaults to None).
PositionIdsLayoutType (TensorLayout): The tensor layout of the position_ids tensor (defaults to RowMajorLayout).

Args:

q_proj (TileTensor[dtype, Storage=q_proj.Storage, address_space=q_proj.address_space, linear_idx_type=q_proj.linear_idx_type]): Query projection tensor of shape [total_tokens, n_heads, head_dim].
input_row_offsets (TileTensor[DType.uint32, Storage=input_row_offsets.Storage, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor of shape [batch + 1] marking where each batch's tokens start and end.
kv_collection (collection_t): The KV cache collection containing the key cache.
freqs_cis (TileTensor[freq_dtype, Storage=freqs_cis.Storage, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type]): Frequency tensor for RoPE of shape [max_seq_len, rope_dim].
position_ids (OptionalReg[TileTensor[DType.uint32, PositionIdsLayoutType, ImmutAnyOrigin]]): Optional position ids overriding cache-derived positions, of shape [n_sections, total_tokens].
layer_idx (UInt32): The layer index for accessing the correct cache.
output (TileTensor[dtype, Storage=output.Storage, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Output tensor for Q with RoPE applied, same shape as q_proj.
context (DeviceContext): Device context for GPU execution.