IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

causal_conv1d_varlen_fwd_gpu

def causal_conv1d_varlen_fwd_gpu[dtype: DType, conv_states_dtype: DType, weight_LT: TensorLayout, bias_LT: TensorLayout, query_start_loc_LT: TensorLayout, has_initial_state_LT: TensorLayout, update_state_LT: TensorLayout, conv_states_LT: TensorLayout, output_LT: TensorLayout, Engine: TensorEngine, XFn: def[width: Int, alignment: Int](Int, Int) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, SlotFn: def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width] & RegisterPassable & ImplicitlyCopyable, //, WIDTH: Int, silu_activation: Bool = False, use_residual: Bool = False, channels_last: Bool = False](weight: TileTensor[dtype, weight_LT, Engine=Engine], bias: TileTensor[dtype, bias_LT, Engine=Engine], query_start_loc: TileTensor[.int32, query_start_loc_LT, Engine=Engine], has_initial_state: TileTensor[.bool, has_initial_state_LT, Engine=Engine], update_state: TileTensor[.bool, update_state_LT, Engine=Engine], conv_states: TileTensor[conv_states_dtype, conv_states_LT, Engine=Engine], output: TileTensor[dtype, output_LT, Engine=Engine], x_addr: Int, x_row_stride: Int, x_fn: XFn, slot_fn: SlotFn, ctx: DeviceContextBase, x_vector_loads: Bool = True, x_col_stride: Int = Int(1))

Launches the causal conv1d forward kernel with variable length sequences.

Prefill moves 8 bytes of channels per thread when channels_last and the operands are aligned for it, and one channel per thread otherwise. Pure decode, where no sequence has more than a token on average, gives each thread one channel and each grid row a whole sequence.

x_addr, x_row_stride and x_col_stride describe the view x_fn reads, copied out before the call so the tensor and the closure are not both live arguments. They are the address and strides of that view's first element. A fused input does not have them: the closure applies the slice offset inside the load, and the slice pointer is not that address. Pass x_vector_loads=False unless those three values are the ones the load uses, in which case x_fn may only issue a channel vector when they are 8-byte aligned and the channel stride is 1.

Parameters:

  • ​dtype (DType): Element type of x, weight, bias and output.
  • ​conv_states_dtype (DType): Element type of conv_states.
  • ​weight_LT (TensorLayout): Layout type of weight.
  • ​bias_LT (TensorLayout): Layout type of bias.
  • ​query_start_loc_LT (TensorLayout): Layout type of query_start_loc.
  • ​has_initial_state_LT (TensorLayout): Layout type of has_initial_state.
  • ​update_state_LT (TensorLayout): Layout type of update_state.
  • ​conv_states_LT (TensorLayout): Layout type of conv_states.
  • ​output_LT (TensorLayout): Layout type of output.
  • ​Engine (TensorEngine): Engine shared by all tile operands.
  • ​XFn (def[width: Int, alignment: Int](Int, Int) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable): Reads x at its physical (row, column) index, claiming alignment elements of alignment.
  • ​SlotFn (def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width] & RegisterPassable & ImplicitlyCopyable): Reads the conv-state slot of each sequence.
  • ​WIDTH (Int): Convolution width (number of taps).
  • ​silu_activation (Bool): Whether to apply SiLU to the output.
  • ​use_residual (Bool): If True, adds x[d, s] to the convolution sum before the activation.
  • ​channels_last (Bool): If True, x and output are (total_seqlen, dim) rather than (dim, total_seqlen).

Args:

Was this page helpful?