For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
dispatch_im2col_matmul_conv3d
def dispatch_im2col_matmul_conv3d[input_type: DType, filter_type: DType, output_type: DType, EpilogueFnType: ConvSimdEpilogueFn = inflated$def[comptime[dtype: DType, width: SIMDLength, alignment: Int = Int(1)] #kgen.func.symbol<@nn::@conv::@conv_utils::@"_no_conv_simd_epilogue_body[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], ComputeFnType: ConvSimdComputeFn = inflated$def[comptime[dtype: DType, width: SIMDLength, alignment: Int = Int(1)] #kgen.func.symbol<@nn::@conv::@conv_utils::@"_identity_conv_simd_compute_body[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], //, filter_is_fcrs: Bool = False, has_epilogue_fn: Bool = False, m_tile_byte_budget: Int = Int(268435456), has_compute_fn: Bool = False](input: TileTensor[input_type, Engine=input.Engine, address_space=input.address_space, linear_idx_type=input.linear_idx_type], filter: TileTensor[filter_type, Engine=filter.Engine, address_space=filter.address_space, linear_idx_type=filter.linear_idx_type], output: TileTensor[output_type, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], stride: Coord[Int, Int, Int], dilation: Coord[Int, Int, Int], symmetric_padding: Coord[Int, Int, Int], num_groups: Int, ctx: DeviceContext, epilogue_fn: EpilogueFnType = no_conv_simd_epilogue_fn, compute_fn: ComputeFnType = no_conv_simd_compute_fn) -> Bool
Try to dispatch a 3-D conv as explicit im2col + generic matmul.
Returns True if the conv was handled; False if the caller should fall back to another implementation (naive Mojo kernel, cuDNN, etc.).
Skips on: non-bf16 dtype, grouped conv, dilation != 1, kernel size 1x1x1 (the vectorized naive kernel wins on tiny shapes), and K too small for the matmul fast path.
Parameters:
- input_type (
DType): Element type of the input tensor; must beDType.bfloat16. - filter_type (
DType): Element type of the filter tensor. - output_type (
DType): Element type of the output tensor. - EpilogueFnType (
ConvSimdEpilogueFn): Type ofepilogue_fn(inferred). - ComputeFnType (
ConvSimdComputeFn): Type ofcompute_fn(inferred). - filter_is_fcrs (
Bool): True if the filter is inFCQRSlayout, False if inQRSCFlayout (inferred, defaults to False). - has_epilogue_fn (
Bool): True whenepilogue_fnstores the output instead of the matmul (defaults to False). - m_tile_byte_budget (
Int): Upper bound on bytes allocated for one M-tile of the im2col matrix, controlling scratch memory usage (inferred, defaults to 256 MiB). - has_compute_fn (
Bool): True when the matmul storescompute_fnof each output value (defaults to False).
Args:
- input (
TileTensor[input_type, Engine=input.Engine, address_space=input.address_space, linear_idx_type=input.linear_idx_type]): Input tensor inNDHWClayout with shape[batch, D, H, W, C]. - filter (
TileTensor[filter_type, Engine=filter.Engine, address_space=filter.address_space, linear_idx_type=filter.linear_idx_type]): Filter tensor with static shape, eitherFCQRSorQRSCFdepending onfilter_is_fcrs. - output (
TileTensor[output_type, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Output tensor inNDHWClayout with shape[batch, D_out, H_out, W_out, F]. - stride (
Coord[Int, Int, Int]): Spatial strides for the depth, height, and width axes. - dilation (
Coord[Int, Int, Int]): Spatial dilation for the depth, height, and width axes; must be 1 for each. - symmetric_padding (
Coord[Int, Int, Int]): Symmetric padding for the depth, height, and width axes. - num_groups (
Int): Number of convolution groups; must be 1. - ctx (
DeviceContext): Device context for enqueueing GPU kernels and allocating scratch buffers. - epilogue_fn (
EpilogueFnType): Elementwise epilogue applied to each output element with 5-D coordinates; used only whenhas_epilogue_fnis true. - compute_fn (
ComputeFnType): Elementwise transform of each output element with 5-D coordinates; used only whenhas_compute_fnis true.
Returns:
Bool: True if the conv was handled, False if the caller should fall
back to another implementation.