For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
gated_delta_recurrence_verify_ring_gpu
def gated_delta_recurrence_verify_ring_gpu[work_dtype: DType, state_dtype: DType, ring_dtype: DType, KEY_HEAD_DIM: Int, VALUE_HEAD_DIM: Int, RING_LEN: Int, recurrence_output_LT: TensorLayout, qkv_conv_output_LT: TensorLayout, decay_per_token_LT: TensorLayout, beta_per_token_LT: TensorLayout, recurrent_state_LT: TensorLayout, slot_idx_LT: TensorLayout, input_row_offsets_LT: TensorLayout, ring_LT: TensorLayout, ring_slot_idx_LT: TensorLayout, Engine: TensorEngine, RECORDS_ONLY: Bool = False, SKIP_LONG_ROWS: Bool = False](batch_size: Int32, num_value_heads: Int32, num_key_heads: Int32, key_dim: Int32, num_draft_tokens: Int32, recurrence_output: TileTensor[work_dtype, recurrence_output_LT, MutUntrackedOrigin, Engine=Engine], recurrent_state: TileTensor[state_dtype, recurrent_state_LT, MutUntrackedOrigin, Engine=Engine], slot_idx: TileTensor[.uint32, slot_idx_LT, ImmUntrackedOrigin, Engine=Engine], qkv_conv_output: TileTensor[work_dtype, qkv_conv_output_LT, ImmUntrackedOrigin, Engine=Engine], decay_per_token: TileTensor[work_dtype, decay_per_token_LT, ImmUntrackedOrigin, Engine=Engine], beta_per_token: TileTensor[work_dtype, beta_per_token_LT, ImmUntrackedOrigin, Engine=Engine], input_row_offsets: TileTensor[.uint32, input_row_offsets_LT, ImmUntrackedOrigin, Engine=Engine], ring: TileTensor[ring_dtype, ring_LT, MutUntrackedOrigin, Engine=Engine], ring_slot_idx: TileTensor[.uint32, ring_slot_idx_LT, ImmUntrackedOrigin, Engine=Engine])
GPU kernel: runs the gated delta recurrence over a verify window, recording a ring for the rows the window holds.
Produces the same recurrence_output as gated_delta_recurrence_fwd_gpu
and leaves recurrent_state unchanged. Each token of a recorded row
writes the record gated_delta_state_fold_gpu needs to advance the state
over it.
A row longer than one token and its drafts records nothing and commits
itself: the walk stores the state it holds at the row's last real token,
the one before the num_draft_tokens drafts every verify window ends
with, so the fold the caller plans for such a row is zero. The op
rejects a draft width of RING_LEN or more, so a long row always has
that token. Committing inside the walk leaves the state column live no
further than the loop carries it; an exit writeback held it to the exit
and cost the recording path about 30% of its time in spills
(MXSERV-555).
With RECORDS_ONLY the commit is compiled out and a longer row trips
the kernel's safe assert instead. The op launches it when every row is
one token and its drafts, which keeps the commit's conditional store off
the decode verify's hot loop.
With SKIP_LONG_ROWS a longer row writes zeros over its outputs and
nothing else, and the other rows run the RECORDS_ONLY code. A caller
that runs the long rows through a chunked forward launches it, so no
state is written.
Parameters:
- work_dtype (
DType):DTypeof the per-token input and output tensors. - state_dtype (
DType):DTypeof therecurrent_statepool. - ring_dtype (
DType):DTypeof the ring pool. The fold is bit-exact against a forward over the accepted prefix only forfloat32. - KEY_HEAD_DIM (
Int): Compile-time key head dimension. - VALUE_HEAD_DIM (
Int): Compile-time value head dimension, equal toKEY_HEAD_DIM. - RING_LEN (
Int): Compile-time record capacity of one ring row. - recurrence_output_LT (
TensorLayout):TensorLayoutforrecurrence_output. - qkv_conv_output_LT (
TensorLayout):TensorLayoutforqkv_conv_output. - decay_per_token_LT (
TensorLayout):TensorLayoutfordecay_per_token. - beta_per_token_LT (
TensorLayout):TensorLayoutforbeta_per_token. - recurrent_state_LT (
TensorLayout):TensorLayoutforrecurrent_state. - slot_idx_LT (
TensorLayout):TensorLayoutforslot_idx. - input_row_offsets_LT (
TensorLayout):TensorLayoutforinput_row_offsets. - ring_LT (
TensorLayout):TensorLayoutforring. - ring_slot_idx_LT (
TensorLayout):TensorLayoutforring_slot_idx. - Engine (
TensorEngine): Engine shared by all tile operands. - RECORDS_ONLY (
Bool): Compile-time contract that every row is one token and its drafts; the long-row commit is compiled out. - SKIP_LONG_ROWS (
Bool): Compile-time: a longer row is the caller's; its outputs are zeroed and its state is left alone.
Args:
- batch_size (
Int32): Number of sequences in the ragged batch. - num_value_heads (
Int32): Number of value heads (nv). - num_key_heads (
Int32): Number of key heads (nk). - key_dim (
Int32):num_key_heads * key_head_dim. - num_draft_tokens (
Int32): Draft widthKthe verify rides every row with; only a longer row reads it. - recurrence_output (
TileTensor[work_dtype, recurrence_output_LT, MutUntrackedOrigin, Engine=Engine]):[total_seq_len, value_dim]readout, written for every token of the window. - recurrent_state (
TileTensor[state_dtype, recurrent_state_LT, MutUntrackedOrigin, Engine=Engine]):[max_slots, nv, KEY_HEAD_DIM, VALUE_HEAD_DIM]dense live pool, written once for a longer row, at that row's last real token. - slot_idx (
TileTensor[.uint32, slot_idx_LT, ImmUntrackedOrigin, Engine=Engine]):[batch_size]live pool row per batch item. - qkv_conv_output (
TileTensor[work_dtype, qkv_conv_output_LT, ImmUntrackedOrigin, Engine=Engine]):[total_seq_len, conv_dim]conv output. - decay_per_token (
TileTensor[work_dtype, decay_per_token_LT, ImmUntrackedOrigin, Engine=Engine]):[total_seq_len, nv]per-token decay. - beta_per_token (
TileTensor[work_dtype, beta_per_token_LT, ImmUntrackedOrigin, Engine=Engine]):[total_seq_len, nv]per-token beta gate. - input_row_offsets (
TileTensor[.uint32, input_row_offsets_LT, ImmUntrackedOrigin, Engine=Engine]):[batch_size + 1]ragged offsets. - ring (
TileTensor[ring_dtype, ring_LT, MutUntrackedOrigin, Engine=Engine]):[ring_rows, nk, RING_LEN, record_stride]ring pool, whererecord_strideis at leastgated_delta_ring_record_elements(nv // nk). - ring_slot_idx (
TileTensor[.uint32, ring_slot_idx_LT, ImmUntrackedOrigin, Engine=Engine]):[batch_size]ring row per batch item.