For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Build custom ops for GPUs
When the built-in ops in MAX don't cover your computation (a specialized
activation, a fused operation, or a domain-specific kernel), you can write your
own in Mojo and call it from Python. In this tutorial, you'll learn to write
custom operations (ops) in Mojo that run on GPUs and CPUs, then call them from
eager tensors with F.custom() and from a model layer written with
Module.
We'll start with a custom op that adds 1 to every element in the graph's
input tensor, using an API that handles CPU and GPU device management. Then
you'll write specialized functions for CPUs and GPUs. GPU functions that run in
parallel are also known as kernels.
This page assumes a basic understanding of MAX graphs, which are computation graphs written in Python. These graphs are the foundation for high-performance AI models that run on MAX. To learn more, see build a model graph with Module. It also helps to know the Mojo language basics.
Requirementsβ
Mac
Linux
WSL
GPU
Although these examples work on both CPUs and GPUs, in order for the GPU code paths to run, your system must meet the GPU requirements.
Get the examplesβ
This tutorial is a walkthrough of a couple code examples from our GitHub repo. Start by cloning and running one of the examples:
-
Clone the repo:
git clone https://github.com/modular/modular.git -
To set up a compatible developer environment, use
pixito create a virtual environment and manage the package dependencies. If you don't have it, install it:curl -fsSL https://pixi.sh/install.sh | shThen restart your terminal for the changes to take effect.
-
Make sure everything works by running the first example:
cd modular/max/examples/custom_ops pixi run python addition.pyThe exact output will vary based on random initialization of the input tensor, but the "Graph result" and "Expected result" must match:
Graph result: [[1.7736697 1.4688652 1.7971799 1.4553597 1.8967733 1.3691401 1.1297637 1.7047229 1.1314526 1.3924606] # ... shorten for brevity Expected result: [[1.7736697 1.4688652 1.7971799 1.4553597 1.8967733 1.3691401 1.1297637 1.7047229 1.1314526 1.3924606] # ... shorten for brevity
Example 1: An elementwise add opβ
To learn how to create a custom op, let's look at a simple "hello world" example
that adds 1 to each element of a tensor, in parallel on a GPU when one is
available.
Define the custom op in Mojoβ
Take a look at the custom op defined in
custom_ops/kernels/add_one.mojo.
You'll see a Mojo struct called
AddOne with an execute() function. A custom op defines either an execute()
function or an elementwise() method.
Write a fusible elementwise op covers the
second.
Depending on the purpose of your custom op, the execute() function accepts
zero or more inputs and produces one or more outputs, as specified by the
function arguments.
Let's inspect the struct and function signatures:
@extensibility.register("add_one")
struct AddOne:
@staticmethod
def execute(
output: Some[Output & Tensor],
x: Some[Input & Tensor],
ctx: DeviceContext,
) raises:
# See below for the restThe struct must include the
@extensibility.register()
decorator, which registers the custom op with MAX. The add_one name set here
is the name we'll use to call the op from Python.
The rest of the execute() signature describes the custom op's graph node to
the graph compiler: the parameters, inputs, and outputs. The execute()
function must take output as the first runtime argument; the graph compiler
uses this order to identify the op's output slots:
-
This op has no compile-time parameters. An op declares them when it needs values fixed at compile time, as later examples do. The kind of hardware the function is compiled for comes from the
ctxargument instead (we'll use it in the next example). -
The runtime arguments are the op's inputs and outputs. Each argument's type is a generic parameter bounded by tensor traits:
InputorOutputmarks its role, andTensorexposes its dtype, rank, and shape.Some[Output & Tensor]is shorthand for declaring that parameter inline, the same as writing[O: Output & Tensor](output: O). The graph compiler supplies the concrete tensor type for each argument, whose memory is managed outside of the operation.
Now let's look at the body of the execute() function. It adds 1 to each
element inside the elementwise_add_one() closure.
@extensibility.register("add_one")
struct AddOne:
@staticmethod
def execute(
output: Some[Output & Tensor],
x: Some[Input & Tensor],
ctx: DeviceContext,
) raises:
comptime assert conforms_to(type_of(output), TileTensorable)
comptime assert conforms_to(type_of(x), TileTensorable)
comptime dtype = type_of(output).dtype
comptime output_alignment = type_of(output).alignment
comptime assert type_of(x).rank == type_of(output).rank
var output_tile = output.to_tile_tensor[dtype]()
var x_tile = x.to_tile_tensor[dtype]().as_imm()
@inline(.always)
def elementwise_add_one[
width: Int, alignment: Int = 1
](idx: Coord) {var}:
var result = (
x_tile.load[width=width, alignment=align_of[dtype]()](idx) + 1
)
output_tile.store[
width=width,
alignment=min(output_alignment, alignment * align_of[dtype]()),
](idx, result)
elementwise[simd_width=get_kernel_simd_width[dtype, ctx.target]()](
elementwise_add_one, output.shape(), ctx
)The body first checks that both arguments conform to
TileTensorable,
then projects each one to a
TileTensor view over its memory
with to_tile_tensor(). The input's view is immutable (as_imm()).
We call elementwise_add_one() using
elementwise(),
which distributes an elementwise computation in parallel across all elements of
the output's shape. Each call loads and stores a SIMD vector of width
elements, and alignment tells the closure how far its index is aligned, in
elements. At compile time, elementwise() specializes the computation for the
device behind ctx, so the same code runs on CPU or GPU with no changes.
get_kernel_simd_width() picks the SIMD width for the device that
ctx.target names. The
next example branches on
ctx.target explicitly.
Declare the output shapeβ
If you don't annotate output shapes manually in the Python graph, you must tell
the compiler how to infer them by registering a shape function with
@extensibility.register_shape_function().
A shape function that reads only its inputs' shapes takes them as
Some[Tensor], which exposes a tensor's shape, rank, and dtype but not its
elements:
@extensibility.register_shape_function("add_one")
def add_one_shape(x: Some[Tensor]) -> IndexList[type_of(x).rank]:
return rebind[IndexList[type_of(x).rank]](
coord_to_index_list(x.shape().tuple())
)Shape functions must use tensor traits for tensor arguments. If a shape function
reads element values, use Some[TileTensorable] and access them through
to_tile_tensor(). For variadic inputs, use a TensorTuple whose elements bind
the traits they need. Legacy aliases such as InputTensor and
InputVariadicTensors are not accepted on shape functions; execute() can
still use them.
For an op on a GPU, each tensor input not bound to Some[Tensor] is copied to
the host before the shape function runs, and the host waits for the copy. Use
Some[Tensor] for inputs whose elements the shape function doesn't read.
Call the custom op from Pythonβ
An op reaches Python through custom_extensions, which points at the Mojo
package directory that holds the kernels. The directory needs an __init__.mojo
file (which can be empty) so the compiler can find the ops.
On eager tensors, call the op directly with
F.custom(), as the
custom_ops/eager_vector_addition.py
example does:
mojo_kernels = Path(__file__).parent / "kernels"
vector_width = 10
dtype = DType.float32
# Place the op on a GPU, if available. Fall back to CPU if not.
device = CPU() if accelerator_count() == 0 else Accelerator()
lhs = random.uniform([vector_width], dtype=dtype, device=device)
rhs = random.uniform([vector_width], dtype=dtype, device=device)
result = F.custom(
name="vector_addition",
device=device,
values=[lhs, rhs],
out_types=[lhs.type],
custom_extensions=mojo_kernels,
)[0]Inside a model, call the op from a layer's forward(). The module then compiles
the kernel as part of its graph along with everything else. The custom layer
guide shows that wrapper.
Add the custom op to a graph directlyβ
When you build a graph with max.graph, add the op with
ops.custom(). The
custom_ops/addition.py
example does that. The path passed to custom_extensions takes one of two
forms:
- A Mojo package directory, which needs an
__init__.mojofile (which can be empty) so the compiler can find the ops. MAX precompiles the directory each time it builds a graph or session, so it picks up your edits automatically. Use this form during development. - A
.mojocfile built withmojo precompile. Use this form to ship ops, and build it with the same Mojo version that the consumer runs.
A single .mojo file or a .mojopkg file is rejected. This example uses a
package directory:
mojo_kernels = Path(__file__).parent / "kernels"
rows = 5
columns = 10
dtype = DType.float32
device = CPU() if accelerator_count() == 0 else Accelerator()
graph = Graph(
"addition",
forward=lambda x: (
ops.custom(
name="add_one",
device=DeviceRef.from_device(device),
values=[x],
out_types=[
TensorType(
dtype=x.dtype,
shape=x.tensor.shape,
device=DeviceRef.from_device(device),
)
],
)[0].tensor
),
input_types=[
TensorType(
dtype,
shape=[rows, columns],
device=DeviceRef.from_device(device),
),
],
custom_extensions=[mojo_kernels],
)The Graph() takes an input tensor
with five rows and ten columns, runs the custom add_one operation on it, and
returns the result.
That's the whole custom op: the graph now contains it and is ready to compile.
Now we can run an inference with
InferenceSession. First
we compile the graph for the selected device, either a CPU or GPU
"accelerator", then initialize it for execution:
session = InferenceSession(
devices=[device],
)
compiled = session.compile(graph)
model = session.init(compiled)Finally, we generate some random data and pass it as input:
# Fill an input matrix with random values.
x_values = np.random.uniform(size=(rows, columns)).astype(np.float32)
# Create a buffer and move it to the device (CPU or GPU).
x = Buffer.from_numpy(x_values).to(device)
# Run inference with the input tensor.
result = model.execute(x)[0]
# Copy values back to the CPU to be read.
assert isinstance(result, Buffer)
result = result.to(CPU())The Buffer is
initially resident on the host (CPU), so we move it to the accelerator before
use with the graph on that device. If the device is the host CPU and not an
accelerator, to() is a no-op. After the run, we move the result back to the
CPU to read it.
You can run this example as follows:
pixi run python addition.pyWrite a fusible elementwise opβ
The graph compiler can fuse an elementwise custom op with the ops around it. It combines a chain of elementwise ops into one kernel, which skips writing each intermediate tensor to memory and reading it back. It can also apply the chain as the epilogue of a matmul that produces the chain's input.
To make an op fusible, define it with a static elementwise() method. The
method receives SIMD vectors of input values and returns a SIMD vector of
output values. The compiler handles loading, storing, and device dispatch.
The following op adds 1 to every element, like add_one:
@extensibility.register("add_one_elementwise")
struct AddOneElementwise(ElementwiseUnaryOp):
@staticmethod
def elementwise[
dtype: DType,
width: SIMDLength,
](x: SIMD[dtype, width]) -> SIMD[dtype, width]:
return x + 1Declaring conformance to
ElementwiseUnaryOp
makes the compiler check the method signature. The
operation traits module defines the
signatures for unary, binary, and comparison ops.
Add the op to a graph with ops.custom(), the same way as add_one. In
custom_ops/elementwise_addition.py,
the op and a multiply follow a matmul, so the compiler can apply both as the
matmul's epilogue:
def add_one(x: TensorValue, device_ref: DeviceRef) -> TensorValue:
return ops.custom(
name="add_one_elementwise",
device=device_ref,
values=[x],
out_types=[TensorType(dtype=x.dtype, shape=x.shape, device=device_ref)],
)[0].tensor
def main() -> None:
# ...
graph = Graph(
"elementwise_addition",
forward=lambda a, b: add_one(a @ b, device_ref) * 2,
input_types=[
TensorType(dtype, shape=[rows, inner], device=device_ref),
TensorType(dtype, shape=[inner, columns], device=device_ref),
],
custom_extensions=[mojo_kernels],
)Run it the same way as the first example:
pixi run python elementwise_addition.pyAn op that defines execute() with Input and Output tensor arguments runs
as its own kernel, including when it uses elementwise(). Use execute() when
an output element depends on input elements at other positions, or when the op
needs control over device dispatch.
Example 2: Write device-specific kernelsβ
elementwise()
is a device-independent abstraction for elementwise computation: it handles
hardware dispatch and covers a range of CPUs and GPUs. The add_one custom op
uses it to run its computation. When you need direct control over thread
dispatch, block sizes, or GPU-specific memory patterns, write separate code
paths for CPU and GPU with Mojo's comptime if and the GPU thread model.
So let's now look at the
vector_addition.mojo
custom op, which adds two vectors in parallel on a GPU.
This example uses a programming model that may be familiar if you've used general purpose GPU programming in CUDA or similar frameworks: we're going to write separate functions for GPUs and CPUs, and the GPU function (kernel) is written specifically for parallelism across GPU threads.
The vector addition op in kernels/vector_addition.mojo looks like this:
@extensibility.register("vector_addition")
struct VectorAddition:
@staticmethod
def execute(
output: Some[Output & Tensor],
lhs: Some[Input & Tensor],
rhs: Some[Input & Tensor],
ctx: DeviceContext,
) raises:
comptime assert conforms_to(type_of(output), TileTensorable)
comptime assert conforms_to(type_of(lhs), TileTensorable)
comptime assert conforms_to(type_of(rhs), TileTensorable)
comptime dtype = type_of(output).dtype
comptime assert type_of(output).rank == 1
comptime assert type_of(lhs).rank == 1
comptime assert type_of(rhs).rank == 1
var output_tile = output.to_tile_tensor[dtype]()
var lhs_tile = lhs.to_tile_tensor[dtype]().as_imm()
var rhs_tile = rhs.to_tile_tensor[dtype]().as_imm()
comptime if ctx.target.is_cpu():
_vector_addition_cpu(output_tile, lhs_tile, rhs_tile, ctx)
elif ctx.target.is_gpu():
_vector_addition_gpu(output_tile, lhs_tile, rhs_tile, ctx)
else:
raise Error("No known target:", ctx.target.device_kind_label())Using the
comptime if
statement, Mojo checks at compile time if the target device has a GPU or not. If
it does, it uses the _vector_addition_gpu() function; otherwise, it uses
_vector_addition_cpu(). comptime if resolves that branch at compile time, so
the CPU build contains no GPU code.
The _vector_addition_gpu() function looks like this:
def _vector_addition_gpu(
output: TileTensor[mut=True, ...],
lhs: TileTensor[output.dtype, ...],
rhs: TileTensor[output.dtype, ...],
ctx: DeviceContext,
) raises:
comptime BLOCK_SIZE = 16
var vector_length = Int(output.dim[0]())
var length_dev = Int32(vector_length)
def vector_addition_gpu_kernel() {var output, var lhs, var rhs, var length_dev}:
var length = Int(length_dev)
var tid = block_dim.x * block_idx.x + thread_idx.x
if tid < length:
var result = lhs.load[width=1](Coord(tid)) + rhs.load[width=1](
Coord(tid)
)
output.store[width=1](Coord(tid), result)
var num_blocks = ceildiv(vector_length, BLOCK_SIZE)
ctx.enqueue_function(
vector_addition_gpu_kernel, grid_dim=num_blocks, block_dim=BLOCK_SIZE
)The vector_addition_gpu_kernel() closure runs once per thread on the GPU,
adding an element from the lhs vector to the matching element in the rhs
vector and then saving the result at the correct position in the output
vector. The closure copies the tensors and the length into its captures
({var ...}), so enqueue_function() can launch the closure itself as the
kernel across a grid of BLOCK_SIZE blocks of threads.
BLOCK_SIZE is untuned for this device. elementwise() picks the launch
parameters for the target and is the better choice for a simple elementwise op.
This kernel shows the thread-level mapping that elementwise() hides.
You can execute this one in the same manner as the previous example, by running the Python-based graph that uses the custom op:
pixi run python vector_addition.pyChoosing a dispatch styleβ
You wrote custom ops and integrated them into MAX graphs:
AddOne, which useselementwise()for automatic device dispatchAddOneElementwise, which the graph compiler can fuse with neighboring opsVectorAddition, which usescomptime ifto dispatch separate CPU and GPU implementations
Define elementwise() for elementwise ops you want the compiler to fuse. Use
elementwise() inside execute() for other ops where device-specific tuning
isn't needed. Write device-specific kernels when you need direct control over
thread layout and memory access patterns.
Next stepsβ
- Build a model graph with Module: Learn how to compose modules, write custom modules with explicit weights, load checkpoint data, and construct a model graph.
- Optimize custom ops for GPUs with Mojo: Learn to use Mojo's GPU programming abstractions to progressively optimize a matrix multiplication.