IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mamba2_ssd_chunked_fwd_inplace_gpu

def mamba2_ssd_chunked_fwd_inplace_gpu[kernel_dtype: DType, state_dtype: DType, XFn: def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, DtFn: def[width: Int, alignment: Int](Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, BFn: def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, CFn: def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable, SlotFn: def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width] & RegisterPassable & ImplicitlyCopyable, //, DSTATE: Int, dt_softplus: Bool, grouped_cb_mode: Int = Int(0), ROW_RULE: Bool = False](ctx: DeviceContext, total_len: Int, batch: Int, ngroups: Int, A: TileTensor[kernel_dtype, Engine=A.Engine, address_space=A.address_space, linear_idx_type=A.linear_idx_type], D: TileTensor[kernel_dtype, Engine=D.Engine, address_space=D.address_space, linear_idx_type=D.linear_idx_type], dt_bias: TileTensor[kernel_dtype, Engine=dt_bias.Engine, address_space=dt_bias.address_space, linear_idx_type=dt_bias.linear_idx_type], y: TileTensor[kernel_dtype, Engine=y.Engine, address_space=y.address_space, linear_idx_type=y.linear_idx_type], ssm_pool: TileTensor[state_dtype, Engine=ssm_pool.Engine, address_space=ssm_pool.address_space, linear_idx_type=ssm_pool.linear_idx_type], query_start_loc: TileTensor[.int32, Engine=query_start_loc.Engine, address_space=query_start_loc.address_space, linear_idx_type=query_start_loc.linear_idx_type], has_initial_state: TileTensor[.bool, Engine=has_initial_state.Engine, address_space=has_initial_state.address_space, linear_idx_type=has_initial_state.linear_idx_type], update_state: TileTensor[.bool, Engine=update_state.Engine, address_space=update_state.address_space, linear_idx_type=update_state.linear_idx_type], x_fn: XFn, dt_fn: DtFn, b_fn: BFn, c_fn: CFn, slot_fn: SlotFn, x_addr: Int = Int(0), x_ld: Int = Int(0), b_addr: Int = Int(0), c_addr: Int = Int(0), bc_ld: Int = Int(0), num_draft_tokens: Int = Int(0))

Runs the chunked SSD scan over a varlen batch.

Same operands and semantics as the serial scan kernel, including has_initial_state and update_state. head_dim must be a multiple of TILE and DSTATE a multiple of 64.

Parameters:

  • ​kernel_dtype (DType): Element type of x, dt, A, B, C, D, dt_bias and y.
  • ​state_dtype (DType): Storage dtype of ssm_pool.
  • ​XFn (def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable): Reads x.
  • ​DtFn (def[width: Int, alignment: Int](Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable): Reads dt.
  • ​BFn (def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable): Reads B.
  • ​CFn (def[width: Int, alignment: Int](Int, Int, Int) -> SIMD[kernel_dtype, width] & RegisterPassable & ImplicitlyCopyable): Reads C.
  • ​SlotFn (def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width] & RegisterPassable & ImplicitlyCopyable): Reads the pool slot of each sequence.
  • ​DSTATE (Int): State dimension per head.
  • ​dt_softplus (Bool): Whether to apply softplus to dt + dt_bias.
  • ​grouped_cb_mode (Int): GROUPED_CB_OFF, GROUPED_CB_ON, or GROUPED_CB_AUTO. AUTO selects only the validated B200 bf16 kernel, fp32 state, dimension-128 amortization regime.
  • ​ROW_RULE (Bool): Whether a prefill row (longer than one real token plus its drafts) maps only its real tokens and commits its state after them, whatever its update_state flag.

Args:

Was this page helpful?