For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mamba2_replay_fold
Commits a speculative verify's accepted rows to every Mamba-2 layer at once.
A Nemotron-H verify runs each Mamba layer on its live state without writing it back. Once acceptance is known, each request's state has to advance by its accepted prefix of the verify's rows. That is two updates per layer:
- the conv window becomes the last
width - 1accepted conv inputs, the stored window filling in when fewer were accepted; - the SSM state folds in each accepted row,
state = state * exp(A * dt) + dt * B * x, withdt = softplus(dt + dt_bias).
Neither needs the other: the scan's x, B and C are the verify's conv
outputs, which were computed causally, so their accepted rows already saw the
accepted conv history. And neither produces an output. One launch does both
for every layer and request, reading each layer's rows where the verify left
them, so a batch-1 replay fills the GPU rather than running one small launch
per layer. The SSM update is the token-serial scan's arithmetic, expression for
expression, so the state it lands is bit-identical to replaying with
mamba2_ssd_chunk_scan_varlen_fwd_inplace.
comptime values
PAD_SLOT_ID
comptime PAD_SLOT_ID = Int32(-1)
A conv-state row id that marks a padded request, which is skipped.
Functions
-
mamba2_replay_fold_gpu: GPU kernel: folds each request's accepted rows into every layer's state. -
mamba2_replay_fold_launch: Launchesmamba2_replay_fold_gpufor every layer and request.