For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
fp8_index_score_amd
def fp8_index_score_amd[dtype: DType, KOperand: MHAOperand, KSOperand: MHAOperand, num_heads: Int, depth: Int, _is_cache_length_accurate: Bool, causal: Bool = False, kpool: Int = Int(1), pool_row_tokens: Int = Int(1)](output: TileTensor[.float32, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], q_s: TileTensor[.float32, Engine=q_s.Engine, address_space=q_s.address_space, linear_idx_type=q_s.linear_idx_type], k_operand: KOperand, ks_operand: KSOperand, valid_length: TileTensor[.uint32, Engine=valid_length.Engine, address_space=valid_length.address_space, linear_idx_type=valid_length.linear_idx_type], batch_size: Int, max_seq_len: Int, max_num_keys: Int, ctx: DeviceContext, chunk_begin: Int = Int(0))
Enqueues the gfx950 MFMA FP8 indexer scorer (_fp8_index_score_amd_kernel).
AMD counterpart of fp8_index_score_sm100: writes the same
[chunk_rows, max_num_keys] score buffer as the scalar
fp8_index_kernel, with OOB keys left untouched (the caller's -inf fill
or bounded top-k covers them).
Parameters:
- dtype (
DType): FP8 element type of Q and K (float8_e4m3fn). - KOperand (
MHAOperand):MHAOperandtype for the K values. - KSOperand (
MHAOperand):MHAOperandtype for the per-token K scales. - num_heads (
Int): Query index heads (multiple of 32). - depth (
Int): Head dimension (multiple of 64). - _is_cache_length_accurate (
Bool): When False,num_keys = cache_length + seq_len; when True,cache_lengthalready includes the new tokens. - causal (
Bool): Fuse the causal bound into the store guard (token t sees keys up to cache_len + t + 1); forbidden slots are never written. - kpool (
Int): Tokens per scored key; see_fp8_index_score_amd_kernel. - pool_row_tokens (
Int): Cache rows a pooled key spans; see_fp8_index_score_amd_kernel.
Args:
- output (
TileTensor[.float32, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Score buffer[chunk_rows, max_num_keys], f32. - q (
TileTensor[dtype, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type]): Query tensor[total_seq, num_heads, depth], fp8. - q_s (
TileTensor[.float32, Engine=q_s.Engine, address_space=q_s.address_space, linear_idx_type=q_s.linear_idx_type]): Query scales[total_seq, num_heads], f32. - k_operand (
KOperand): K values as anMHAOperand. - ks_operand (
KSOperand): K scales as anMHAOperand. - valid_length (
TileTensor[.uint32, Engine=valid_length.Engine, address_space=valid_length.address_space, linear_idx_type=valid_length.linear_idx_type]): Ragged query-token offsets[batch + 1]. - batch_size (
Int): Number of sequences in the batch. - max_seq_len (
Int): Upper bound on any batch entry's query-token count. - max_num_keys (
Int): Row stride ofoutput(>= every per-batch key count). - ctx (
DeviceContext): Device context used to enqueue the kernel. - chunk_begin (
Int): Global row held in output row 0.