For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Build custom ops for GPUs
When the built-in ops in a MAX graph don't cover your computation (a specialized activation, a fused operation, or a domain-specific kernel), you can write your own in Mojo and load them at graph construction time. In this tutorial, you'll learn to write custom graph operations (ops) in Mojo that run on GPUs and CPUs, and then load them into a MAX graph written in Python.
We'll start with a simple custom op that just adds 1 to every element in the
graph's input tensor, using an API that abstracts-away the CPU and GPU device
management. Then you'll learn to write specialized functions for CPUs and GPUs.
(GPU functions that run in parallel are also known as kernels.)
Before you begin, you should have a basic understanding of MAX graphs, which are computation graphs written in Python. These graphs are the foundation for high-performance AI models that run on MAX. To learn more, see build a model graph with Module. It also helps to know the Mojo language basics.
Requirements
Mac
Linux
WSL
GPU
Although these examples work on both CPUs and GPUs, in order for the GPU code paths to run, your system must meet the GPU requirements.
Get the examples
This tutorial is a walkthrough of a couple code examples from our GitHub repo. Start by cloning and running one of the examples:
-
Clone the repo:
git clone https://github.com/modular/modular.git -
To ensure you have a compatible developer environment, we recommend using
pixito create a virtual environment and manage the package dependencies. If you don't have it, install it:curl -fsSL https://pixi.sh/install.sh | shThen restart your terminal for the changes to take effect.
-
Make sure everything works by running the first example:
cd modular/max/examples/custom_ops pixi run python addition.pyThe exact output will vary based on random initialization of the input tensor, but the "Graph result" and "Expected result" should match:
Graph result: [[1.7736697 1.4688652 1.7971799 1.4553597 1.8967733 1.3691401 1.1297637 1.7047229 1.1314526 1.3924606] # ... shorten for brevity Expected result: [[1.7736697 1.4688652 1.7971799 1.4553597 1.8967733 1.3691401 1.1297637 1.7047229 1.1314526 1.3924606] # ... shorten for brevity
Now let's dive into the implementation details to understand how this custom operation works under the hood.
Example 1: Learn the custom op basics
To learn how to create a custom op, let's look as simple "hello world" example
that adds 1 to each element of a tensor—doing so in parallel on a GPU, if
available.
Define the custom op in Mojo
Take a look at the custom op defined in
custom_ops/kernels/add_one.mojo.
You'll see a Mojo struct called
AddOne with an execute() function. A custom op defines either an execute()
function, described below, or an elementwise() method, described in
Write a fusible elementwise op.
Depending on the purpose of your custom op, the execute() function accepts
zero or more inputs and produces one or more outputs, as specified by the
function arguments.
Let's inspect the struct and function signatures:
@extensibility.register("add_one")
struct AddOne:
@staticmethod
def execute[
target: StaticString,
](
output: OutputTensor,
x: InputTensor[dtype=output.dtype, rank=output.rank, ...],
ctx: DeviceContext,
) raises:
# See below for the restThe struct must include the
@extensibility.register()
decorator, which registers the custom op with MAX. The add_one name we set
here is the name we'll use to add the op to our Python graph in the next
section.
The rest of the execute() signature describes the custom op's graph node to
the graph compiler: the parameters, inputs, and outputs. The execute()
function must include target as the first compile-time parameter and output
as the first runtime argument; the graph compiler uses this order to identify
the op's hardware target and output slots:
-
There's one compile-time parameter,
target, which tells the function what kind of hardware it's being compiled for (either"cpu"or"gpu"; we'll use this in the next example). -
The runtime arguments include the op's inputs and outputs, which take the form of
InputTensorandOutputTensor, respectively. These are specialized versions of theManagedTensorSlicetype, which represents a tensor of a specific rank and datatype whose memory is managed outside of the operation.
Now let's look at the body of the execute() function, which is highlighted in
the following code. The op's core computation that adds 1 to each element in
the tensor happens in the elementwise_add_one() closure function.
@extensibility.register("add_one")
struct AddOne:
@staticmethod
def execute[
target: StaticString,
](
output: OutputTensor,
x: InputTensor[dtype=output.dtype, rank=output.rank, ...],
ctx: DeviceContext,
) raises:
@inline(.always)
def elementwise_add_one[
width: Int
](idx: Coord) {var} -> SIMD[x.dtype, width]:
return x.load[width](idx) + 1
foreach[target=target](elementwise_add_one, output, ctx)We call elementwise_add_one() using
foreach(),
which distributes an elementwise computation in parallel across all elements in
the output tensor. At compile time, foreach() specializes the computation for
the target hardware, so the same code runs on CPU or GPU with no changes.
Also notice that we pass the target parameter to the foreach() function,
which allows it to optimize for the hardware device. (We'll do this ourselves in
the next example.)
Declare the output shape
If you don't annotate output shapes manually in the Python graph, you must tell
the compiler how to infer them by registering a shape function with
@extensibility.register_shape_function().
A shape function that reads only its inputs' shapes takes them as
Some[Tensor], which exposes a tensor's shape, rank, and dtype but not its
elements:
@extensibility.register_shape_function("add_one")
def add_one_shape(x: Some[Tensor]) -> IndexList[type_of(x).rank]:
return rebind[IndexList[type_of(x).rank]](
coord_to_index_list(x.shape().tuple())
)Shape functions must use tensor traits for tensor arguments. If a shape function
reads element values, use Some[TileTensorable] and access them through
to_tile_tensor(). For variadic inputs, use a TensorTuple whose elements bind
the traits they need. Legacy aliases such as InputTensor and
InputVariadicTensors are not accepted on shape functions; execute() can
still use them.
Add the custom op to a Python graph
Let's now look at the corresponding
custom_ops/addition.py
file where we load the add_one custom op into a graph using
ops.custom().
Here's the code that specifies the path to the Mojo custom op and adds it to the
graph. The path passed to custom_extensions takes one of two forms:
- A Mojo package directory, which needs an
__init__.mojofile (which can be empty) so the compiler can find the ops. MAX precompiles the directory each time it builds a graph or session, so it picks up your edits automatically. Use this form during development. - A
.mojocfile built withmojo precompile. Use this form to ship ops, and build it with the same Mojo version that the consumer runs.
A single .mojo file or a .mojopkg file is rejected. This example uses a
package directory:
mojo_kernels = Path(__file__).parent / "kernels"
rows = 5
columns = 10
dtype = DType.float32
device = CPU() if accelerator_count() == 0 else Accelerator()
graph = Graph(
"addition",
forward=lambda x: (
ops.custom(
name="add_one",
device=DeviceRef.from_device(device),
values=[x],
out_types=[
TensorType(
dtype=x.dtype,
shape=x.tensor.shape,
device=DeviceRef.from_device(device),
)
],
)[0].tensor
),
input_types=[
TensorType(
dtype,
shape=[rows, columns],
device=DeviceRef.from_device(device),
),
],
custom_extensions=[mojo_kernels],
)The Graph() takes an input tensor
with five rows and ten columns, runs the custom add_one operation on it, and
returns the result.
That's basically it. We created a custom op and added it to a graph!
Now we can run an inference, using
InferenceSession. We
start by compiling the graph for the selected device (see above; either a CPU
or GPU "accelerator"), then initializing it for execution:
session = InferenceSession(
devices=[device],
)
compiled = session.compile(graph)
model = session.init(compiled)Finally, we generate some random data and pass it as input:
# Fill an input matrix with random values.
x_values = np.random.uniform(size=(rows, columns)).astype(np.float32)
# Create a buffer and move it to the device (CPU or GPU).
x = Buffer.from_numpy(x_values).to(device)
# Run inference with the input tensor.
result = model.execute(x)[0]
# Copy values back to the CPU to be read.
assert isinstance(result, Buffer)
result = result.to(CPU())Notice that the Buffer is
initially resident on the host (CPU), so we move it to the accelerator to be
ready for use with the graph on that device (if the device is the host CPU and
not an accelerator, the to() function is a no-op). Likewise, after we get
results, we move the result back to the CPU to read it.
As shown above, you can run this example as follows:
pixi run python addition.pyWrite a fusible elementwise op
The graph compiler can fuse an elementwise custom op with the ops around it. It combines a chain of elementwise ops into one kernel, which skips writing each intermediate tensor to memory and reading it back. It can also apply the chain as the epilogue of a matmul that produces the chain's input.
To make an op fusible, define it with a static elementwise() method. The
method receives SIMD vectors of input values and returns a SIMD vector of
output values. The compiler handles loading, storing, and device dispatch.
The following op adds 1 to every element, like add_one:
@extensibility.register("add_one_elementwise")
struct AddOneElementwise(ElementwiseUnaryOp):
@staticmethod
def elementwise[
dtype: DType,
width: SIMDLength,
](x: SIMD[dtype, width]) -> SIMD[dtype, width]:
return x + 1Declaring conformance to
ElementwiseUnaryOp
makes the compiler check the method signature. The
operation traits module defines the
signatures for unary, binary, and comparison ops.
Add the op to a graph with ops.custom(), the same way as add_one. In
custom_ops/elementwise_addition.py,
the op and a multiply follow a matmul, so the compiler can apply both as the
matmul's epilogue:
def add_one(x: TensorValue, device_ref: DeviceRef) -> TensorValue:
return ops.custom(
name="add_one_elementwise",
device=device_ref,
values=[x],
out_types=[TensorType(dtype=x.dtype, shape=x.shape, device=device_ref)],
)[0].tensor
def main() -> None:
# ...
graph = Graph(
"elementwise_addition",
forward=lambda a, b: add_one(a @ b, device_ref) * 2,
input_types=[
TensorType(dtype, shape=[rows, inner], device=device_ref),
TensorType(dtype, shape=[inner, columns], device=device_ref),
],
custom_extensions=[mojo_kernels],
)Run it the same way as the first example:
pixi run python elementwise_addition.pyAn op that defines execute() with InputTensor and OutputTensor arguments
runs as its own kernel, including when it uses foreach(). Use execute()
when an output element depends on input elements at other positions, or when
the op needs control over device dispatch.
Example 2: Write device-specific kernels
foreach()
is a device-independent abstraction designed for elementwise computation: it
handles hardware dispatch and achieves excellent performance for that class of
operation. When you need direct control over thread dispatch, block sizes, or
GPU-specific memory patterns, you write separate code paths for CPU and GPU
using Mojo's comptime if and the GPU thread model.
The add_one custom op above uses
foreach()
to run our custom op computation. This function is a device-independent
abstraction to perform calculations on each element of a tensor, and it offers
great out-of-the-box performance for a range of CPUs and GPUs. However, there
might also be situations in which you want to write your own hardware-specific
algorithms.
So let's now look at the
vector_addition.mojo
custom op, which adds two vectors together in parallel on a GPU.
This example uses a programming model that may be familiar if you've used general purpose GPU programming in CUDA or similar frameworks: We're going to write separate functions for GPUs and CPUs, and the GPU function (kernel) is written specifically for parallelism across GPU threads.
The vector addition op in kernels/vector_addition.mojo looks like this:
@extensibility.register("vector_addition")
struct VectorAddition:
@staticmethod
def execute[
target: StaticString,
](
output: OutputTensor[rank=1, ...],
lhs: InputTensor[dtype=output.dtype, rank=output.rank, ...],
rhs: InputTensor[dtype=output.dtype, rank=output.rank, ...],
ctx: DeviceContext,
) raises:
comptime if target == "cpu":
_vector_addition_cpu(output, lhs, rhs, ctx)
elif target == "gpu":
_vector_addition_gpu(output, lhs, rhs, ctx)
else:
raise Error("No known target:", target)Using the
comptime if
statement, Mojo checks at compile time if the target device has a GPU or not. If
it does, it uses the _vector_addition_gpu() function; otherwise, it uses
_vector_addition_cpu(). Compile-time specialization like this is a unique and
powerful feature of Mojo that makes it easy to optimize code for specific
hardware.
The _vector_addition_gpu() function looks like this:
def _vector_addition_gpu(
output: ManagedTensorSlice[mut=True, ...],
lhs: ManagedTensorSlice[dtype=output.dtype, rank=output.rank, ...],
rhs: ManagedTensorSlice[dtype=output.dtype, rank=output.rank, ...],
ctx: DeviceContext,
) raises:
comptime BLOCK_SIZE = 16
var vector_length = output.dim_size(0)
var length_dev = Int32(vector_length)
def vector_addition_gpu_kernel() {var output, var lhs, var rhs, var length_dev}:
var length = Int(length_dev)
var tid = block_dim.x * block_idx.x + thread_idx.x
if tid < length:
var idx = IndexList[output.rank](tid)
var result = lhs.load[1](idx) + rhs.load[1](idx)
output.store[1](idx, result)
var num_blocks = ceildiv(vector_length, BLOCK_SIZE)
ctx.enqueue_function(
vector_addition_gpu_kernel, grid_dim=num_blocks, block_dim=BLOCK_SIZE
)The vector_addition_gpu_kernel() closure runs once per thread on the GPU,
adding an element from the lhs vector to the matching element in the rhs
vector and then saving the result at the correct position in the output
vector. The closure copies the tensors and the length into its captures
({var ...}), so enqueue_function() can launch the closure itself as the
kernel across a grid of BLOCK_SIZE blocks of threads.
The block size is arbitrary here, and is not tuned for the specific GPU hardware
this will be run on. The previously-used foreach() abstraction will do
hardware-specific tuning for this style of dispatch, and is what we recommend
for simple elementwise calculations like this. However, this example shows how
you might mentally map CUDA C-style code to thread-level GPU operations in MAX.
You can execute this one in the same manner as the previous example, by running the Python-based graph that uses the custom op:
pixi run python vector_addition.pyConclusion
In this tutorial, you wrote custom ops and integrated them into MAX graphs:
AddOne, which usesforeach()for automatic device dispatchAddOneElementwise, which the graph compiler can fuse with neighboring opsVectorAddition, which usescomptime ifto dispatch separate CPU and GPU implementations
Define elementwise() for elementwise ops you want the compiler to fuse.
Use foreach() inside execute() for other ops where device-specific tuning
isn't needed. Write device-specific kernels when you need direct control over
thread layout and memory access patterns.
Next steps
- Build a model graph with Module: Learn how to compose modules, write custom modules with explicit weights, load checkpoint data, and construct a model graph.
- Optimize custom ops for GPUs with Mojo: Learn to use Mojo's GPU programming abstractions to progressively optimize a matrix multiplication.