For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Mamba2SSDChunkScanVarlenFwdInplace
struct Mamba2SSDChunkScanVarlenFwdInplace[dt_softplus: Bool = True]
Varlen Mamba-2 SSD chunked scan with an in-place SSM state pool.
Matches mamba_chunk_scan_combined semantics for the Nemotron-H
NemotronHMamba2Mixer: per-head scalar A, grouped B/C, per-head
dt + dt_bias softplus, and a state reset at each query_start_loc
boundary. Gating z and MambaRMSNormGated are applied outside this op.
Each sequence starts from ssm_pool[cache_indices[b]] when
has_initial_state[b] is set, and its final state is written back to the
same slot, so the graph never round-trips the pool. The kernel math lives
in state_space.mamba2_ssd_scan (token-serial scan, used for decode,
verify and short launches) and state_space.mamba2_ssd_chunked (chunked
state-space-duality scan on tensor cores). On NVIDIA GPUs with bf16 and
dstate 128, a launch with at least 512 tokens and an average sequence of at
least 128 tokens takes the chunked path.
Tensor shapes:
- y: (total_len, nheads, head_dim), output (dtype) [OUT]
- x: (total_len, nheads, head_dim), input (dtype)
- dt: (total_len, nheads)
- A: (nheads,)
- B: (total_len, ngroups, dstate)
- C: (total_len, ngroups, dstate)
- D: (nheads,) optional/empty
- dt_bias: (nheads,) optional/empty
- ssm_pool: (max_slots, nheads, head_dim, dstate) fp32, or bf16 on
Apple GPUs (storage dtype only; the scan accumulates in fp32) [MUT]
- query_start_loc: (batch + 1,) int32
- has_initial_state: (batch,) bool optional/empty
- cache_indices: (batch,) or (1, batch) uint32, slot indices into
ssm_pool
- update_state: (batch,) bool optional/empty. A clear flag leaves
that sequence's ssm_pool slot untouched (a read-only scan);
empty writes every sequence's final state.
- xbc: (total_len, nheads * head_dim + 2 * ngroups * dstate) or
empty (of any dtype): the conv output that x, B and C are
column slices of, in that order. With it, a long prefill on B200 reads their
rows by TMA (state_space.mamba2_ssd_chunked).
Parameters
- dt_softplus (
Bool): If True (default), apply softplus todt + dt_bias.
Implemented traits
Methods
execute
static def execute(y: T, x: T, dt: T, A: T, B: T, C: T, D: T, dt_bias: T, ssm_pool: T, query_start_loc: T, has_initial_state: T, cache_indices: T, update_state: T, xbc: T, ctx: DeviceContext)