For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
use_chunked_ssd
def use_chunked_ssd(total_len: Int, batch: Int, head_dim: Int, min_tokens: Int) -> Bool
Returns whether a launch should take the chunked scan over the serial one.
The chunk setup costs a fixed ~0.2 ms, so the launch needs at least
min_tokens tokens in total and an average sequence of at least MIN_AVG_SEQ_LEN tokens:
a batch of short verify or decode rows stays on the serial kernel.
head_dim must be a multiple of TILE.
Returns: