For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

fp8_index

def fp8_index[dtype: DType, //, num_heads: Int, depth: Int](output: TileTensor[.float32, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], q_s: TileTensor[.float32, Engine=q_s.Engine, address_space=q_s.address_space, linear_idx_type=q_s.linear_idx_type], k: TileTensor[dtype, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type], k_s: TileTensor[.float32, Engine=k_s.Engine, address_space=k_s.address_space, linear_idx_type=k_s.linear_idx_type], valid_length: TileTensor[.uint32, Engine=valid_length.Engine, address_space=valid_length.address_space, linear_idx_type=valid_length.linear_idx_type], cache_row_offsets: TileTensor[.uint32, Engine=cache_row_offsets.Engine, address_space=cache_row_offsets.address_space, linear_idx_type=cache_row_offsets.linear_idx_type], batch_size: Int, max_seq_len: Int, max_num_keys: Int, ctx: DeviceContext)

Dispatches the FP8 index/gather scorer on the given device context.

Selects the Blackwell tcgen05/TMA tensor-core scorer when the device and operand layout support it, otherwise falls back to the scalar fp8_index_kernel path.

Parameters:

dtype (DType): Data type of the query and key tensors.
num_heads (Int): Number of attention heads.
depth (Int): Per-head feature depth.

Args:

output (TileTensor[.float32, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Output score tensor of shape [total_seq_len, max_num_keys].
q (TileTensor[dtype, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type]): Query tensor of shape [total_seq_len, num_heads, depth].
q_s (TileTensor[.float32, Engine=q_s.Engine, address_space=q_s.address_space, linear_idx_type=q_s.linear_idx_type]): Per-query scale tensor of shape [total_seq_len, num_heads].
k (TileTensor[dtype, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type]): Key tensor of shape [total_keys, 1, depth].
k_s (TileTensor[.float32, Engine=k_s.Engine, address_space=k_s.address_space, linear_idx_type=k_s.linear_idx_type]): Per-key scale tensor of shape [total_keys].
valid_length (TileTensor[.uint32, Engine=valid_length.Engine, address_space=valid_length.address_space, linear_idx_type=valid_length.linear_idx_type]): Cumulative sequence offsets of shape [batch_size + 1].
cache_row_offsets (TileTensor[.uint32, Engine=cache_row_offsets.Engine, address_space=cache_row_offsets.address_space, linear_idx_type=cache_row_offsets.linear_idx_type]): Per-batch row offsets into the paged key cache.
batch_size (Int): Number of sequences in the batch.
max_seq_len (Int): Maximum sequence length across the batch.
max_num_keys (Int): Maximum key count across the batch.
ctx (DeviceContext): Device context used to enqueue the selected kernel.

Raises:

When the underlying kernel enqueue reports a device-side error.