For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mamba2_replay_fold_gpu
def mamba2_replay_fold_gpu[dtype: DType, conv_dtype: DType, state_dtype: DType, DSTATE: Int, DSTATE_SPLIT: Int, CH_PER_THREAD: Int, CONV_WIDTH: Int, NUM_LAYERS: Int, conv_pool_LT: TensorLayout, ssm_pool_LT: TensorLayout, conv_rows_LT: TensorLayout, ssm_rows_LT: TensorLayout, bounds_LT: TensorLayout, offsets_LT: TensorLayout, width_LT: TensorLayout, Engine: TensorEngine, dt_softplus: Bool = True](projected: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], convolved: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], A: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], dt_bias: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], projected_stride: Int32, convolved_stride: Int32, ngroups: Int32, conv_pool: TileTensor[conv_dtype, conv_pool_LT, MutUntrackedOrigin, Engine=Engine], ssm_pool: TileTensor[state_dtype, ssm_pool_LT, MutUntrackedOrigin, Engine=Engine], conv_rows: TileTensor[.uint32, conv_rows_LT, ImmUntrackedOrigin, Engine=Engine], ssm_rows: TileTensor[.uint32, ssm_rows_LT, ImmUntrackedOrigin, Engine=Engine], replay_bounds: TileTensor[.int32, bounds_LT, ImmUntrackedOrigin, Engine=Engine], merged_offsets: TileTensor[.uint32, offsets_LT, ImmUntrackedOrigin, Engine=Engine], verify_width: TileTensor[.int64, width_LT, ImmUntrackedOrigin, Engine=Engine])
GPU kernel: folds each request's accepted rows into every layer's state.
The grid is one linear x axis. It decomposes with the channel chunk
fastest, then the column, then the layer and request, where
lb = x // (head_chunks * columns) is request lb % batch of layer
lb // batch. Grid y and z stop at 65535 blocks, which
batch * NUM_LAYERS can pass. This order keeps the launch order of a
three-axis (head_chunks, columns, batch * NUM_LAYERS) grid, whose
locality the fold is tuned for. Columns below nheads fold that head of
the SSM state, laid out as the token-serial scan's: DSTATE_SPLIT
threads share a channel, each owning DSTATE // DSTATE_SPLIT contiguous
lanes and CH_PER_THREAD channels. The columns above them each move 128
channels of the conv window.
Grid: (batch * NUM_LAYERS * columns * head_chunks, 1, 1) with
head_chunks = ceildiv(head_dim, CH), CH = (128 // DSTATE_SPLIT) * CH_PER_THREAD and columns = nheads + ceildiv(ceildiv(conv_dim, 128), head_chunks), block: (DSTATE_SPLIT, 128 // DSTATE_SPLIT, 1).
Each request continues its live state, so no initial-state flag is read.
A row longer than K + 1 tokens is a prefill row and folds nothing:
the verify committed its state itself.
Parameters:
- dtype (
DType): Element type of the activations,Aanddt_bias. - conv_dtype (
DType): Storage dtype ofconv_pool. - state_dtype (
DType): Storage dtype ofssm_pool; the fold accumulates in fp32. - DSTATE (
Int): State dimension per head. - DSTATE_SPLIT (
Int): Threads cooperating on one channel's state. - CH_PER_THREAD (
Int): Channels each thread carries. - CONV_WIDTH (
Int): Convolution width; the window holdsCONV_WIDTH - 1rows. - NUM_LAYERS (
Int): Number of Mamba layers. - conv_pool_LT (
TensorLayout): Tensor layout ofconv_pool. - ssm_pool_LT (
TensorLayout): Tensor layout ofssm_pool. - conv_rows_LT (
TensorLayout): Tensor layout ofconv_rows. - ssm_rows_LT (
TensorLayout): Tensor layout ofssm_rows. - bounds_LT (
TensorLayout): Tensor layout ofreplay_bounds. - offsets_LT (
TensorLayout): Tensor layout ofmerged_offsets. - width_LT (
TensorLayout): Tensor layout ofverify_width. - Engine (
TensorEngine): Engine shared by all tile operands. - dt_softplus (
Bool): Whether to apply softplus todt + dt_bias.
Args:
- projected (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Each layer'sin_projoutput rows,(rows, intermediate + conv_dim + nheads)row-major: the gate, the conv input anddt. - convolved (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Each layer's conv output rows,(rows, conv_dim)row-major: the scan'sx,BandC. Each row'sBis read asDSTATE // DSTATE_SPLIT-wide vectors, which the launcher checks are aligned. - A (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Each layer's per-head decay,(nheads,), already-exp(A_log). - dt_bias (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Each layer's per-headdtbias,(nheads,). - projected_stride (
Int32): Row stride of everyprojectedtensor. - convolved_stride (
Int32): Row stride of everyconvolvedtensor. - ngroups (
Int32): Number ofBgroups. - conv_pool (
TileTensor[conv_dtype, conv_pool_LT, MutUntrackedOrigin, Engine=Engine]): Conv windows,(max_slots, conv_dim, CONV_WIDTH - 1). - ssm_pool (
TileTensor[state_dtype, ssm_pool_LT, MutUntrackedOrigin, Engine=Engine]): SSM states,(max_slots, nheads, head_dim, DSTATE). - conv_rows (
TileTensor[.uint32, conv_rows_LT, ImmUntrackedOrigin, Engine=Engine]): Each layer's conv-window row per request,(NUM_LAYERS, batch). - ssm_rows (
TileTensor[.uint32, ssm_rows_LT, ImmUntrackedOrigin, Engine=Engine]): Each layer's SSM-state row per request,(NUM_LAYERS, batch). - replay_bounds (
TileTensor[.int32, bounds_LT, ImmUntrackedOrigin, Engine=Engine]): Each request's accepted(start, end)rows,(2 * batch,). - merged_offsets (
TileTensor[.uint32, offsets_LT, ImmUntrackedOrigin, Engine=Engine]): The batch's cumulative row offsets,(batch + 1,). - verify_width (
TileTensor[.int64, width_LT, ImmUntrackedOrigin, Engine=Engine]): The verify widthK,(K,); read for its length only.