For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
indexer_q_ssmax_rope_quantize
def indexer_q_ssmax_rope_quantize[rope_dim: Int](q_out: TileTensor[.float8_e4m3fn, Engine=q_out.Engine, address_space=q_out.address_space, linear_idx_type=q_out.linear_idx_type], scale_out: TileTensor[.float8_e8m0fnu, Engine=scale_out.Engine, address_space=scale_out.address_space, linear_idx_type=scale_out.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], positions: TileTensor[.uint32, Engine=positions.Engine, address_space=positions.address_space, linear_idx_type=positions.linear_idx_type], ssmax_s: TileTensor[.bfloat16, Engine=ssmax_s.Engine, address_space=ssmax_s.address_space, linear_idx_type=ssmax_s.linear_idx_type], ssmax_b: TileTensor[.bfloat16, Engine=ssmax_b.Engine, address_space=ssmax_b.address_space, linear_idx_type=ssmax_b.linear_idx_type], freqs_cis: TileTensor[.bfloat16, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type], start_pos: TileTensor[.uint32, Engine=start_pos.Engine, address_space=start_pos.address_space, linear_idx_type=start_pos.linear_idx_type], ctx: DeviceContext)
Scales, rotates and FP8-quantizes an indexer query in one launch.
Computes, per row t at absolute position positions[t] and head h,
q[t, h] * (s[h] * log(positions[t] + 1) + b[h]), rotates its leading
rope_dim columns by the interleaved RoPE at the row's cache position,
and quantizes the head to FP8 with one E8M0 scale. A row past the last
ragged offset belongs to no request and is skipped, as the key op's rows
are.
start_pos has to hold an entry per request, or the rope's batch lookup
would read past it, so a shorter buffer is refused as the key op refuses
one. q_out and scale_out are addressed flat, so a non-contiguous
layout is refused too.
Parameters:
- rope_dim (
Int): Leading columns of each head the RoPE rotates; 0 rotates none.
Args:
- q_out (
TileTensor[.float8_e4m3fn, Engine=q_out.Engine, address_space=q_out.address_space, linear_idx_type=q_out.linear_idx_type]):[rows, heads, 128]FP8 query. - scale_out (
TileTensor[.float8_e8m0fnu, Engine=scale_out.Engine, address_space=scale_out.address_space, linear_idx_type=scale_out.linear_idx_type]):[rows, heads]E8M0 scale of each head. - q (
TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type]):[rows, heads, 128]BF16 query projection, rows contiguous. - positions (
TileTensor[.uint32, Engine=positions.Engine, address_space=positions.address_space, linear_idx_type=positions.linear_idx_type]):[rows]absolute 0-based positions, for SSMax. - ssmax_s (
TileTensor[.bfloat16, Engine=ssmax_s.Engine, address_space=ssmax_s.address_space, linear_idx_type=ssmax_s.linear_idx_type]):[heads]SSMax scale. - ssmax_b (
TileTensor[.bfloat16, Engine=ssmax_b.Engine, address_space=ssmax_b.address_space, linear_idx_type=ssmax_b.linear_idx_type]):[heads]SSMax bias. - freqs_cis (
TileTensor[.bfloat16, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type]):[positions, rope_dim]interleaved RoPE table, rows contiguous. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type]):[batch + 1]ragged row offsets. - start_pos (
TileTensor[.uint32, Engine=start_pos.Engine, address_space=start_pos.address_space, linear_idx_type=start_pos.linear_idx_type]):[batch]cached tokens per request, the RoPE start. - ctx (
DeviceContext): The device context.