IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mamba2_replay_fold_launch

def mamba2_replay_fold_launch[dtype: DType, conv_dtype: DType, state_dtype: DType, NUM_LAYERS: Int, conv_pool_LT: TensorLayout, ssm_pool_LT: TensorLayout, conv_rows_LT: TensorLayout, ssm_rows_LT: TensorLayout, bounds_LT: TensorLayout, offsets_LT: TensorLayout, width_LT: TensorLayout, Engine: TensorEngine, //, dt_softplus: Bool = True](projected: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], convolved: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], A: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], dt_bias: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], projected_stride: Int, convolved_stride: Int, ngroups: Int, conv_pool: TileTensor[conv_dtype, conv_pool_LT, Engine=Engine], ssm_pool: TileTensor[state_dtype, ssm_pool_LT, Engine=ssm_pool.Engine], conv_rows: TileTensor[.uint32, conv_rows_LT, Engine=conv_rows.Engine], ssm_rows: TileTensor[.uint32, ssm_rows_LT, Engine=ssm_rows.Engine], replay_bounds: TileTensor[.int32, bounds_LT, Engine=replay_bounds.Engine], merged_offsets: TileTensor[.uint32, offsets_LT, Engine=merged_offsets.Engine], verify_width: TileTensor[.int64, width_LT, Engine=verify_width.Engine], ctx: DeviceContext)

Launches mamba2_replay_fold_gpu for every layer and request.

Parameters:

  • ​dtype (DType): Element type of the activations, A and dt_bias.
  • ​conv_dtype (DType): Storage dtype of conv_pool.
  • ​state_dtype (DType): Storage dtype of ssm_pool.
  • ​NUM_LAYERS (Int): Number of Mamba layers.
  • ​conv_pool_LT (TensorLayout): Tensor layout of conv_pool.
  • ​ssm_pool_LT (TensorLayout): Tensor layout of ssm_pool.
  • ​conv_rows_LT (TensorLayout): Tensor layout of conv_rows.
  • ​ssm_rows_LT (TensorLayout): Tensor layout of ssm_rows.
  • ​bounds_LT (TensorLayout): Tensor layout of replay_bounds.
  • ​offsets_LT (TensorLayout): Tensor layout of merged_offsets.
  • ​width_LT (TensorLayout): Tensor layout of verify_width.
  • ​Engine (TensorEngine): Engine of conv_pool. The launched kernel takes one engine for every tile operand, so the others must match it.
  • ​dt_softplus (Bool): Whether to apply softplus to dt + dt_bias.

Args:

Was this page helpful?