For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
causal_conv1d_varlen_fwd_gpu
def causal_conv1d_varlen_fwd_gpu[dtype: DType, conv_states_dtype: DType, weight_LT: TensorLayout, bias_LT: TensorLayout, query_start_loc_LT: TensorLayout, has_initial_state_LT: TensorLayout, update_state_LT: TensorLayout, conv_states_LT: TensorLayout, output_LT: TensorLayout, Engine: TensorEngine, XFn: def[width: Int, alignment: Int](Int, Int) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, SlotFn: def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width] & RegisterPassable & ImplicitlyCopyable, //, WIDTH: Int, silu_activation: Bool = False, use_residual: Bool = False, channels_last: Bool = False](weight: TileTensor[dtype, weight_LT, Engine=Engine], bias: TileTensor[dtype, bias_LT, Engine=Engine], query_start_loc: TileTensor[.int32, query_start_loc_LT, Engine=Engine], has_initial_state: TileTensor[.bool, has_initial_state_LT, Engine=Engine], update_state: TileTensor[.bool, update_state_LT, Engine=Engine], conv_states: TileTensor[conv_states_dtype, conv_states_LT, Engine=Engine], output: TileTensor[dtype, output_LT, Engine=Engine], x_addr: Int, x_row_stride: Int, x_fn: XFn, slot_fn: SlotFn, ctx: DeviceContextBase, x_vector_loads: Bool = True, x_col_stride: Int = Int(1))
Launches the causal conv1d forward kernel with variable length sequences.
Prefill moves 8 bytes of channels per thread when channels_last and the
operands are aligned for it, and one channel per thread otherwise. Pure
decode, where no sequence has more than a token on average, gives each
thread one channel and each grid row a whole sequence.
x_addr, x_row_stride and x_col_stride describe the view x_fn reads,
copied out before the call so the tensor and the closure are not both live
arguments. They are the address and strides of that view's first element.
A fused input does not have them: the closure applies the slice offset
inside the load, and the slice pointer is not that address. Pass
x_vector_loads=False unless those three values are the ones the load
uses, in which case x_fn may only issue a channel vector when they are
8-byte aligned and the channel stride is 1.
Parameters:
- dtype (
DType): Element type ofx,weight,biasandoutput. - conv_states_dtype (
DType): Element type ofconv_states. - weight_LT (
TensorLayout): Layout type ofweight. - bias_LT (
TensorLayout): Layout type ofbias. - query_start_loc_LT (
TensorLayout): Layout type ofquery_start_loc. - has_initial_state_LT (
TensorLayout): Layout type ofhas_initial_state. - update_state_LT (
TensorLayout): Layout type ofupdate_state. - conv_states_LT (
TensorLayout): Layout type ofconv_states. - output_LT (
TensorLayout): Layout type ofoutput. - Engine (
TensorEngine): Engine shared by all tile operands. - XFn (
def[width: Int, alignment: Int](Int, Int) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): Readsxat its physical(row, column)index, claimingalignmentelements of alignment. - SlotFn (
def[width: Int, alignment: Int](Int) -> SIMD[.uint32, width]&RegisterPassable&ImplicitlyCopyable): Reads the conv-state slot of each sequence. - WIDTH (
Int): Convolution width (number of taps). - silu_activation (
Bool): Whether to apply SiLU to the output. - use_residual (
Bool): If True, addsx[d, s]to the convolution sum before the activation. - channels_last (
Bool): If True,xandoutputare(total_seqlen, dim)rather than(dim, total_seqlen).
Args:
- weight (
TileTensor[dtype, weight_LT, Engine=Engine]): Weight tensor of shape(dim, width). - bias (
TileTensor[dtype, bias_LT, Engine=Engine]): Bias tensor of shape(dim,), or empty. - query_start_loc (
TileTensor[.int32, query_start_loc_LT, Engine=Engine]): Cumulative sequence lengths of shape(batch + 1,). - has_initial_state (
TileTensor[.bool, has_initial_state_LT, Engine=Engine]): Per-sequence flags of shape(batch,), or empty. - update_state (
TileTensor[.bool, update_state_LT, Engine=Engine]): Per-sequence flags of shape(batch,)selecting whether the sequence writes its new window toconv_states, or empty to write every sequence's. A clear flag leaves the slot untouched. - conv_states (
TileTensor[conv_states_dtype, conv_states_LT, Engine=Engine]): Convolution states of shape(max_slots, dim, width - 1), updated in place, or empty. - output (
TileTensor[dtype, output_LT, Engine=Engine]): Output tensor, same shape asx. - x_addr (
Int): Address of the first elementx_fnloads, whenx_vector_loadsis True. - x_row_stride (
Int): Stride, in elements, of that view's first axis. - x_fn (
XFn): Readsx. - slot_fn (
SlotFn): Reads each sequence's slot inconv_states.PAD_SLOT_IDskips the sequence. - ctx (
DeviceContextBase): The device context to launch on. - x_vector_loads (
Bool): Whetherx_fnissues channel vectors. False when the address of those loads is not the one inx_addr. - x_col_stride (
Int): Stride, in elements, of that view's channel axis. A channel vector needs this to be 1.