For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mamba2_replay_fold_launch
def mamba2_replay_fold_launch[dtype: DType, conv_dtype: DType, state_dtype: DType, NUM_LAYERS: Int, conv_pool_LT: TensorLayout, ssm_pool_LT: TensorLayout, conv_rows_LT: TensorLayout, ssm_rows_LT: TensorLayout, bounds_LT: TensorLayout, offsets_LT: TensorLayout, width_LT: TensorLayout, Engine: TensorEngine, //, dt_softplus: Bool = True](projected: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], convolved: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], A: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], dt_bias: StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS], projected_stride: Int, convolved_stride: Int, ngroups: Int, conv_pool: TileTensor[conv_dtype, conv_pool_LT, Engine=Engine], ssm_pool: TileTensor[state_dtype, ssm_pool_LT, Engine=ssm_pool.Engine], conv_rows: TileTensor[.uint32, conv_rows_LT, Engine=conv_rows.Engine], ssm_rows: TileTensor[.uint32, ssm_rows_LT, Engine=ssm_rows.Engine], replay_bounds: TileTensor[.int32, bounds_LT, Engine=replay_bounds.Engine], merged_offsets: TileTensor[.uint32, offsets_LT, Engine=merged_offsets.Engine], verify_width: TileTensor[.int64, width_LT, Engine=verify_width.Engine], ctx: DeviceContext)
Launches mamba2_replay_fold_gpu for every layer and request.
Parameters:
- dtype (
DType): Element type of the activations,Aanddt_bias. - conv_dtype (
DType): Storage dtype ofconv_pool. - state_dtype (
DType): Storage dtype ofssm_pool. - NUM_LAYERS (
Int): Number of Mamba layers. - conv_pool_LT (
TensorLayout): Tensor layout ofconv_pool. - ssm_pool_LT (
TensorLayout): Tensor layout ofssm_pool. - conv_rows_LT (
TensorLayout): Tensor layout ofconv_rows. - ssm_rows_LT (
TensorLayout): Tensor layout ofssm_rows. - bounds_LT (
TensorLayout): Tensor layout ofreplay_bounds. - offsets_LT (
TensorLayout): Tensor layout ofmerged_offsets. - width_LT (
TensorLayout): Tensor layout ofverify_width. - Engine (
TensorEngine): Engine ofconv_pool. The launched kernel takes one engine for every tile operand, so the others must match it. - dt_softplus (
Bool): Whether to apply softplus todt + dt_bias.
Args:
- projected (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Seemamba2_replay_fold_gpu. - convolved (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Seemamba2_replay_fold_gpu. - A (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Seemamba2_replay_fold_gpu. - dt_bias (
StaticTuple[Pointer[Scalar[dtype], ImmutAnyOrigin], NUM_LAYERS]): Seemamba2_replay_fold_gpu. - projected_stride (
Int): Seemamba2_replay_fold_gpu. - convolved_stride (
Int): Seemamba2_replay_fold_gpu. - ngroups (
Int): Seemamba2_replay_fold_gpu. - conv_pool (
TileTensor[conv_dtype, conv_pool_LT, Engine=Engine]): Seemamba2_replay_fold_gpu. - ssm_pool (
TileTensor[state_dtype, ssm_pool_LT, Engine=ssm_pool.Engine]): Seemamba2_replay_fold_gpu. - conv_rows (
TileTensor[.uint32, conv_rows_LT, Engine=conv_rows.Engine]): Seemamba2_replay_fold_gpu. - ssm_rows (
TileTensor[.uint32, ssm_rows_LT, Engine=ssm_rows.Engine]): Seemamba2_replay_fold_gpu. - replay_bounds (
TileTensor[.int32, bounds_LT, Engine=replay_bounds.Engine]): Seemamba2_replay_fold_gpu. - merged_offsets (
TileTensor[.uint32, offsets_LT, Engine=merged_offsets.Engine]): Seemamba2_replay_fold_gpu. - verify_width (
TileTensor[.int64, width_LT, Engine=verify_width.Engine]): Seemamba2_replay_fold_gpu. - ctx (
DeviceContext): The device context to launch on.