For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mamba2_ssd_chunked_fwd_inplace_gpu
def mamba2_ssd_chunked_fwd_inplace_gpu[kernel_dtype: DType, state_dtype: DType, XFn: def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, DtFn: def[width: Int, alignment: Int](Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, BFn: def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, CFn: def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, SlotFn: def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width] & RegisterPassable & ImplicitlyCopyable, //, DSTATE: Int, dt_softplus: Bool, grouped_cb_mode: Int = Int(0), ROW_RULE: Bool = False](ctx: DeviceContext, total_len: Int, batch: Int, ngroups: Int, A: TileTensor[kernel_dtype, Engine=A.Engine, address_space=A.address_space, linear_idx_type=A.linear_idx_type], D: TileTensor[kernel_dtype, Engine=D.Engine, address_space=D.address_space, linear_idx_type=D.linear_idx_type], dt_bias: TileTensor[kernel_dtype, Engine=dt_bias.Engine, address_space=dt_bias.address_space, linear_idx_type=dt_bias.linear_idx_type], y: TileTensor[kernel_dtype, Engine=y.Engine, address_space=y.address_space, linear_idx_type=y.linear_idx_type], ssm_pool: TileTensor[state_dtype, Engine=ssm_pool.Engine, address_space=ssm_pool.address_space, linear_idx_type=ssm_pool.linear_idx_type], query_start_loc: TileTensor[.int32, Engine=query_start_loc.Engine, address_space=query_start_loc.address_space, linear_idx_type=query_start_loc.linear_idx_type], has_initial_state: TileTensor[.bool, Engine=has_initial_state.Engine, address_space=has_initial_state.address_space, linear_idx_type=has_initial_state.linear_idx_type], update_state: TileTensor[.bool, Engine=update_state.Engine, address_space=update_state.address_space, linear_idx_type=update_state.linear_idx_type], x_fn: XFn, dt_fn: DtFn, b_fn: BFn, c_fn: CFn, slot_fn: SlotFn, x_addr: Int = Int(0), x_ld: Int = Int(0), b_addr: Int = Int(0), c_addr: Int = Int(0), bc_ld: Int = Int(0), num_draft_tokens: Int = Int(0))
Runs the chunked SSD scan over a varlen batch.
Same operands and semantics as the serial scan kernel, including
has_initial_state and update_state. head_dim must be a multiple of
TILE and DSTATE a multiple of 64.
Parameters:
- kernel_dtype (
DType): Element type ofx,dt,A,B,C,D,dt_biasandy. - state_dtype (
DType): Storage dtype ofssm_pool. - XFn (
def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width]&RegisterPassable&ImplicitlyCopyable): Readsx. - DtFn (
def[width: Int, alignment: Int](Int, Int) -> SIMD[kernel_dtype, width]&RegisterPassable&ImplicitlyCopyable): Readsdt. - BFn (
def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width]&RegisterPassable&ImplicitlyCopyable): ReadsB. - CFn (
def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width]&RegisterPassable&ImplicitlyCopyable): ReadsC. - SlotFn (
def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width]&RegisterPassable&ImplicitlyCopyable): Reads the pool slot of each sequence. - DSTATE (
Int): State dimension per head. - dt_softplus (
Bool): Whether to apply softplus todt + dt_bias. - grouped_cb_mode (
Int):GROUPED_CB_OFF,GROUPED_CB_ON, orGROUPED_CB_AUTO. AUTO selects only the validated B200 bf16 kernel, fp32 state, dimension-128 amortization regime. - ROW_RULE (
Bool): Whether a prefill row (longer than one real token plus its drafts) maps only its real tokens and commits its state after them, whatever itsupdate_stateflag.
Args:
- ctx (
DeviceContext): Device context. - total_len (
Int): Total tokens of the batch. - batch (
Int): Number of sequences. - ngroups (
Int): Number ofB/Cgroups. - A (
TileTensor[kernel_dtype, Engine=A.Engine, address_space=A.address_space, linear_idx_type=A.linear_idx_type]): Per-head scalar decay. - D (
TileTensor[kernel_dtype, Engine=D.Engine, address_space=D.address_space, linear_idx_type=D.linear_idx_type]): Per-head skip connection, or empty. - dt_bias (
TileTensor[kernel_dtype, Engine=dt_bias.Engine, address_space=dt_bias.address_space, linear_idx_type=dt_bias.linear_idx_type]): Per-head bias added todt, or empty. - y (
TileTensor[kernel_dtype, Engine=y.Engine, address_space=y.address_space, linear_idx_type=y.linear_idx_type]): Output of shape(total_len, nheads, head_dim). - ssm_pool (
TileTensor[state_dtype, Engine=ssm_pool.Engine, address_space=ssm_pool.address_space, linear_idx_type=ssm_pool.linear_idx_type]): State pool, read and written in place. - query_start_loc (
TileTensor[.int32, Engine=query_start_loc.Engine, address_space=query_start_loc.address_space, linear_idx_type=query_start_loc.linear_idx_type]): Cumulative sequence offsets of shape(batch + 1,). - has_initial_state (
TileTensor[.bool, Engine=has_initial_state.Engine, address_space=has_initial_state.address_space, linear_idx_type=has_initial_state.linear_idx_type]): Per-sequence flag, or empty. - update_state (
TileTensor[.bool, Engine=update_state.Engine, address_space=update_state.address_space, linear_idx_type=update_state.linear_idx_type]): Per-sequence write-back flag, or empty. UnderROW_RULEa prefill row writes its state at its last real token whatever this flag. - x_fn (
XFn): Readsx. - dt_fn (
DtFn): Readsdt. - b_fn (
BFn): ReadsB. - c_fn (
CFn): ReadsC. - slot_fn (
SlotFn): Reads the pool slot of each sequence. - x_addr (
Int): The first element of x when it is a plain strided tensor (rowsx_ldapart, unit stride inside a row), else 0. Withb_addr,c_addrandbc_ldlikewise, the scan takes the sm100 chunk state and output kernels (mamba2_ssd_state_sm100,mamba2_ssd_out_sm100) where they apply. On AMD (gfx950) the MI355 kernels (mamba2_ssd_state_mi355,mamba2_ssd_out_mi355) are the only chunk state and output, so there the caller checksuse_mi355_ssdfirst. - x_ld (
Int): Row stride of x in elements. - b_addr (
Int): B's first element, or 0. - c_addr (
Int): C's first element, or 0. - bc_ld (
Int): Row stride of B and C in elements. - num_draft_tokens (
Int): Draft tokens per row; a row longer than this plus one is a prefill row underROW_RULE.