For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DeviceGraphCache
struct DeviceGraphCache
Holds the device graphs a model has already built, keyed for reuse.
A graph is expensive to record and instantiate, so
DeviceGraph.create()
consults a cache before doing either. The key is derived from the graph's
inputs, so a later call with equivalent inputs replays the graph the first
call built.
Each individual operation is safe to call concurrently. A miss is not
exclusive, though: callers that miss together will each build a graph and the
last one to add() wins, so concurrent first use of one key costs duplicated
recording rather than a single shared graph.
Implemented traits
Methods
__init__
def __init__(out self)
Creates an empty cache.
make_key
static def make_key[*Ts: DeviceGraphInput](build: T, *inputs: *Ts.values) -> String
Derives the cache key for a graph built from the given inputs.
Two calls agree on a key exactly when they pass the same work function and their inputs write the same contributions in the same order.
Parameters:
- *Ts (
DeviceGraphInput): Types of the device graph inputs.
Args:
- build (
T): The work function that records the graph. Its type is what identifies the graph independently of its inputs; the function is never called here. - *inputs (
*Ts.values): The inputs whose contributions distinguish this graph.
Returns:
String: The cache key.
lookup
def lookup(mut self, key: String) -> Optional[DeviceGraph]
Returns the graph stored under a key, if there is one.
Args:
- key (
String): The cache key to look up.
Returns:
Optional[DeviceGraph]: The cached graph, or None on a miss.
cache
def cache(mut self, var key: String, var graph: DeviceGraph) -> DeviceGraph
Interns the device graph identified by the given key.
If there is already a graph identified by the given key, that graph is returned. Otherwise, the supplied graph is inserted into the cache and returned.
Args:
- key (
String): The cache key to store the graph under. - graph (
DeviceGraph): The graph to store.
Returns:
DeviceGraph: The canonicalized graph in the cache.