For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

LoRAManagerV3

LoRAManagerV3

class max.pipelines.lora.LoRAManagerV3(config, base_model_path, base_dtype, n_heads, n_kv_heads, head_dim, max_lora_seq_len, targets=())

source

Bases: object

Loads and routes ModuleV3 LoRA adapters (adapters-as-inputs).

It owns an LRU slot cache and loads adapters unfused (per-projection q/k/v), then supplies the batch routing and slot->adapter map that the serving path feeds as graph inputs. Adapters ride in as inputs, so there is no alias-buffer hot-swap.

Parameters:

• config (LoRAConfig)
• base_model_path (str)
• base_dtype (DType)
• n_heads (int)
• n_kv_heads (int)
• head_dim (int)
• max_lora_seq_len (int)
• targets (Sequence[LoRATargetModule])

activate_adapter()

activate_adapter(name)

source

Marks an adapter active, assigning it an LRU slot.

Parameters:

name (str)

Return type:

None

active_adapters_by_slot()

active_adapters_by_slot(context_batch)

source

Maps each batch adapter to its active LRU slot id.

Base requests and inactive adapters are skipped.

Parameters:

context_batch (Sequence[Any])

Return type:

dict[int, LoRAModel]

bind_inputs()

bind_inputs(model, lora_inputs)

source

Distributes the LoRA graph inputs into the model’s LoRA layers.

Called inside the model’s forward during tracing: the batch-wide routing goes to every layer and each adapter stack to its slot, in the order symbolic_inputs() produced.

Raises:

ValueError – If the input count does not match the routing triple plus the discovered adapter slots.

Parameters:

• model (Module[..., Any])
• lora_inputs (Sequence[Tensor])

Return type:

None

get_lora_graph_inputs()

get_lora_graph_inputs(context_batch, input_row_offsets, device)

source

The batch routing triple (lora_ids, grouped_offsets, end).

Groups consecutive same-adapter requests and truncates at the first base-model group. end carries the LoRA-routed token count that the boundary-aware expand kernel reads (see LoRA.set_lora_batch_info); lora_ids/grouped_offsets live on device and end on CPU, matching symbolic_inputs().

Parameters:

• context_batch (Sequence[Any])
• input_row_offsets (ndarray[tuple[Any, ...], dtype[integer[Any]]])
• device (Device)

Return type:

tuple[Buffer, Buffer, Buffer]

input_buffers()

input_buffers(context_batch, input_row_offsets, device)

source

Builds the per-call LoRA input tuple.

The batch routing triple then the per-slot adapter buffers, in symbolic_inputs() order.

Parameters:

• context_batch (Sequence[Any])
• input_row_offsets (ndarray[tuple[Any, ...], dtype[Any]])
• device (Device)

Return type:

tuple[Buffer, …]

is_active_lora()

is_active_lora(name)

source

Whether name is an active (LRU-slotted) adapter.

The serving scheduler calls this to decide whether a request can reuse an already-active slot or needs to claim a new one.

Parameters:

name (str)

Return type:

bool

is_lora()

is_lora(name)

source

Whether name is a loaded adapter.

Parameters:

name (str)

Return type:

bool

load_adapter()

load_adapter(path)

source

Loads one adapter (name=path or path), keeping it unfused.

A path that is not a local directory is treated as a Hugging Face repo id and downloaded to a local snapshot before loading.

Parameters:

path (str)

Return type:

LoRAStatus

lora_weight_adapter()

lora_weight_adapter(base_adapter, *, wrapper_prefix='model')

source

Wraps a format weight adapter to emit the wrapped LoRA layout.

Runs base_adapter (the format’s normal conversion), rewrites this manager’s target projections via _fuse_projections_for_lora() (concatenating base q/k/v only for stacked targets; name-transparent targets keep their native per-projection keys), then prefixes each key with wrapper_prefix – the attribute under which the fanout wrapper nests the inner model ("model" by convention). This lets the base _load_state_dict produce the LoRA-ready state dict, so an arch needs no _load_state_dict override.

Parameters:

• base_adapter (Callable[[...], dict[str, WeightData]])
• wrapper_prefix (str)

Return type:

Callable[[…], dict[str, WeightData]]

loras

property loras: list[str]

source

The loaded adapter names.

sort_lora_batch()

sort_lora_batch(context_batch)

source

Sorts a batch so same-adapter requests are adjacent, base last.

get_lora_graph_inputs() groups consecutive same-adapter requests and truncates at the first base group, so the batch must be ordered by adapter id (base id -1 sorts last) for the ragged routing to cover every LoRA request.

Parameters:

context_batch (list[_CtxT])

Return type:

list[_CtxT]

symbolic_inputs()

symbolic_inputs(device_ref)

source

Returns the extra compile-input types for a ModuleV3 LoRA model.

The routing triple first, then one adapter-stack type per slot in lora_parameters() order – the same order fed at runtime and consumed by bind_inputs() during tracing.

Parameters:

device_ref (DeviceRef)

Return type:

list[TensorType]

unload_adapter()

unload_adapter(name)

source

Unloads name from the registry and frees its LRU slot.

The serving request processor calls this to release a loaded adapter.

Parameters:

name (str)

Return type:

LoRAStatus

wrap()

wrap(model)

source

Makes model ModuleV3 LoRA ready and returns it.

Wraps each of this manager’s target projections in place with LoRA, then returns (and remembers) a top-level module that fans the per-call LoRA inputs into those layers, so the inner model’s forward stays LoRA-oblivious. The returned wrapper holds model at attribute model (adding a model. prefix that lora_weight_adapter() compensates for).

Parameters:

model (Module[..., Any])

Return type:

Module[…, Any]