IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

indexer_q_ssmax_rope_quantize

def indexer_q_ssmax_rope_quantize[rope_dim: Int](q_out: TileTensor[.float8_e4m3fn, Engine=q_out.Engine, address_space=q_out.address_space, linear_idx_type=q_out.linear_idx_type], scale_out: TileTensor[.float8_e8m0fnu, Engine=scale_out.Engine, address_space=scale_out.address_space, linear_idx_type=scale_out.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], positions: TileTensor[.uint32, Engine=positions.Engine, address_space=positions.address_space, linear_idx_type=positions.linear_idx_type], ssmax_s: TileTensor[.bfloat16, Engine=ssmax_s.Engine, address_space=ssmax_s.address_space, linear_idx_type=ssmax_s.linear_idx_type], ssmax_b: TileTensor[.bfloat16, Engine=ssmax_b.Engine, address_space=ssmax_b.address_space, linear_idx_type=ssmax_b.linear_idx_type], freqs_cis: TileTensor[.bfloat16, Engine=freqs_cis.Engine, address_space=freqs_cis.address_space, linear_idx_type=freqs_cis.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, address_space=input_row_offsets.address_space, linear_idx_type=input_row_offsets.linear_idx_type], start_pos: TileTensor[.uint32, Engine=start_pos.Engine, address_space=start_pos.address_space, linear_idx_type=start_pos.linear_idx_type], ctx: DeviceContext)

Scales, rotates and FP8-quantizes an indexer query in one launch.

Computes, per row t at absolute position positions[t] and head h, q[t, h] * (s[h] * log(positions[t] + 1) + b[h]), rotates its leading rope_dim columns by the interleaved RoPE at the row's cache position, and quantizes the head to FP8 with one E8M0 scale. A row past the last ragged offset belongs to no request and is skipped, as the key op's rows are.

start_pos has to hold an entry per request, or the rope's batch lookup would read past it, so a shorter buffer is refused as the key op refuses one. q_out and scale_out are addressed flat, so a non-contiguous layout is refused too.

Parameters:

  • ​rope_dim (Int): Leading columns of each head the RoPE rotates; 0 rotates none.

Args:

Was this page helpful?