IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mamba2_replay_fold_gpu

def mamba2_replay_fold_gpu[dtype: DType, conv_dtype: DType, state_dtype: DType, DSTATE: Int, DSTATE_SPLIT: Int, CH_PER_THREAD: Int, CONV_WIDTH: Int, NUM_LAYERS: Int, conv_pool_LT: TensorLayout, ssm_pool_LT: TensorLayout, conv_rows_LT: TensorLayout, ssm_rows_LT: TensorLayout, bounds_LT: TensorLayout, offsets_LT: TensorLayout, width_LT: TensorLayout, Engine: TensorEngine, dt_softplus: Bool = True](projected: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], convolved: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], A: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], dt_bias: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], projected_stride: Int32, convolved_stride: Int32, ngroups: Int32, conv_pool: TileTensor[conv_dtype, conv_pool_LT, MutUntrackedOrigin, Engine=Engine], ssm_pool: TileTensor[state_dtype, ssm_pool_LT, MutUntrackedOrigin, Engine=Engine], conv_rows: TileTensor[.uint32, conv_rows_LT, ImmUntrackedOrigin, Engine=Engine], ssm_rows: TileTensor[.uint32, ssm_rows_LT, ImmUntrackedOrigin, Engine=Engine], replay_bounds: TileTensor[.int32, bounds_LT, ImmUntrackedOrigin, Engine=Engine], merged_offsets: TileTensor[.uint32, offsets_LT, ImmUntrackedOrigin, Engine=Engine], verify_width: TileTensor[.int64, width_LT, ImmUntrackedOrigin, Engine=Engine])

GPU kernel: folds each request's accepted rows into every layer's state.

The grid is one linear x axis. It decomposes with the channel chunk fastest, then the column, then the layer and request, where lb = x // (head_chunks * columns) is request lb % batch of layer lb // batch. Grid y and z stop at 65535 blocks, which batch * NUM_LAYERS can pass. This order keeps the launch order of a three-axis (head_chunks, columns, batch * NUM_LAYERS) grid, whose locality the fold is tuned for. Columns below nheads fold that head of the SSM state, laid out as the token-serial scan's: DSTATE_SPLIT threads share a channel, each owning DSTATE // DSTATE_SPLIT contiguous lanes and CH_PER_THREAD channels. The columns above them each move 128 channels of the conv window.

Grid: (batch * NUM_LAYERS * columns * head_chunks, 1, 1) with head_chunks = ceildiv(head_dim, CH), CH = (128 // DSTATE_SPLIT) * CH_PER_THREAD and columns = nheads + ceildiv(ceildiv(conv_dim, 128), head_chunks), block: (DSTATE_SPLIT, 128 // DSTATE_SPLIT, 1).

Each request continues its live state, so no initial-state flag is read.

A row longer than K + 1 tokens is a prefill row and folds nothing: the verify committed its state itself.

Parameters:

  • ​dtype (DType): Element type of the activations, A and dt_bias.
  • ​conv_dtype (DType): Storage dtype of conv_pool.
  • ​state_dtype (DType): Storage dtype of ssm_pool; the fold accumulates in fp32.
  • ​DSTATE (Int): State dimension per head.
  • ​DSTATE_SPLIT (Int): Threads cooperating on one channel's state.
  • ​CH_PER_THREAD (Int): Channels each thread carries.
  • ​CONV_WIDTH (Int): Convolution width; the window holds CONV_WIDTH - 1 rows.
  • ​NUM_LAYERS (Int): Number of Mamba layers.
  • ​conv_pool_LT (TensorLayout): Tensor layout of conv_pool.
  • ​ssm_pool_LT (TensorLayout): Tensor layout of ssm_pool.
  • ​conv_rows_LT (TensorLayout): Tensor layout of conv_rows.
  • ​ssm_rows_LT (TensorLayout): Tensor layout of ssm_rows.
  • ​bounds_LT (TensorLayout): Tensor layout of replay_bounds.
  • ​offsets_LT (TensorLayout): Tensor layout of merged_offsets.
  • ​width_LT (TensorLayout): Tensor layout of verify_width.
  • ​Engine (TensorEngine): Engine shared by all tile operands.
  • ​dt_softplus (Bool): Whether to apply softplus to dt + dt_bias.

Args:

Was this page helpful?