IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mamba2_replay_fold

Commits a speculative verify's accepted rows to every Mamba-2 layer at once.

A Nemotron-H verify runs each Mamba layer on its live state without writing it back. Once acceptance is known, each request's state has to advance by its accepted prefix of the verify's rows. That is two updates per layer:

  • the conv window becomes the last width - 1 accepted conv inputs, the stored window filling in when fewer were accepted;
  • the SSM state folds in each accepted row, state = state * exp(A * dt) + dt * B * x, with dt = softplus(dt + dt_bias).

Neither needs the other: the scan's x, B and C are the verify's conv outputs, which were computed causally, so their accepted rows already saw the accepted conv history. And neither produces an output. One launch does both for every layer and request, reading each layer's rows where the verify left them, so a batch-1 replay fills the GPU rather than running one small launch per layer. The SSM update is the token-serial scan's arithmetic, expression for expression, so the state it lands is bit-identical to replaying with mamba2_ssd_chunk_scan_varlen_fwd_inplace.

comptime values​

PAD_SLOT_ID​

comptime PAD_SLOT_ID = Int32(-1)

A conv-state row id that marks a padded request, which is skipped.

Functions​

Was this page helpful?