For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
Mamba2SSDChunkScanVarlenFwdInplace
struct Mamba2SSDChunkScanVarlenFwdInplace[dt_softplus: Bool = True]
Varlen Mamba-2 SSD chunked scan with an in-place SSM state pool.
Matches mamba_chunk_scan_combined semantics for the Nemotron-H
NemotronHMamba2Mixer: per-head scalar A, grouped B/C, per-head
dt + dt_bias softplus, and a state reset at each query_start_loc
boundary. Gating z and MambaRMSNormGated are applied outside this op.
Each sequence starts from ssm_pool[cache_indices[b]] when
has_initial_state[b] is set, and its final state is written back to the
same slot, so the graph never round-trips the pool. The kernel math lives
in state_space.mamba2_ssd_scan.
Tensor shapes:
- y: (total_len, nheads, head_dim), output (dtype) [OUT]
- x: (total_len, nheads, head_dim), input (dtype)
- dt: (total_len, nheads)
- A: (nheads,)
- B: (total_len, ngroups, dstate)
- C: (total_len, ngroups, dstate)
- D: (nheads,) optional/empty
- dt_bias: (nheads,) optional/empty
- ssm_pool: (max_slots, nheads, head_dim, dstate) fp32, or bf16 on
Apple GPUs (storage dtype only; the scan accumulates in fp32) [MUT]
- query_start_loc: (batch + 1,) int32
- has_initial_state: (batch,) bool optional/empty
- cache_indices: (batch,) or (1, batch) uint32, slot indices into
ssm_pool
- update_state: (batch,) bool optional/empty. A clear flag leaves
that sequence's ssm_pool slot untouched (a read-only scan);
empty writes every sequence's final state.
Parameters
- dt_softplus (
Bool): If True (default), apply softplus todt + dt_bias.
Implemented traits
Methods
execute
static def execute[dtype: DType, state_dtype: DType, target: StringSpan[ImmStaticOrigin]](y: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=y.static_spec], x: ManagedTensorSlice[IOSpec[_, _].FusedInput, static_spec=x.static_spec], dt: ManagedTensorSlice[IOSpec[_, _].FusedInput, static_spec=dt.static_spec], A: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=A.static_spec], B: ManagedTensorSlice[IOSpec[_, _].FusedInput, static_spec=B.static_spec], C: ManagedTensorSlice[IOSpec[_, _].FusedInput, static_spec=C.static_spec], D: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=D.static_spec], dt_bias: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=dt_bias.static_spec], ssm_pool: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=ssm_pool.static_spec], query_start_loc: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=query_start_loc.static_spec], has_initial_state: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=has_initial_state.static_spec], cache_indices: ManagedTensorSlice[IOSpec[_, _].FusedInput, static_spec=cache_indices.static_spec], update_state: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=update_state.static_spec], ctx: DeviceContextBase)