For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
attention
Registers attention graph ops (MHA, MLA, fused QKV) and dispatches them to the nn.attention kernels.
comptime values
logger
comptime logger = Logger(stdout, prefix=String(""), source_location=False)
SM100
comptime SM100 = ((ACCELERATOR & CUDA) & arch("sm_100a"))
Holds for CUDA devices with the sm_100a architecture.
SM103
comptime SM103 = ((ACCELERATOR & CUDA) & arch("sm_103a"))
Holds for CUDA devices with the sm_103a architecture.
Structs
-
FlashAttentionCPU: Registers the CPU kernel of theattention.mha.no_cachegraph op with the graph compiler. -
FlashAttentionGPU: Registers theattention.mha.no_cachegraph op with the graph compiler. -
MLACPMergeTopk: Registers theattention.mla.cp_merge_topkgraph op. -
MLAIndexerRaggedFloat8Paged: Registers theattention.mla.indexer.ragged.float8.pagedgraph op with the graph compiler. -
MLAIndexerRaggedFloat8PagedRemap: Registers theattention.mla.indexer.ragged.float8.paged.remapgraph op. -
MLAIndexerRaggedFloat8PagedRemapCP: Registers theattention.mla.indexer.ragged.float8.paged.remap.cpgraph op. -
MLAIndexerRaggedFloat8PagedRemapCPNVFP4: Registers theattention.mla.indexer.ragged.float8.paged.remap.cp.nvfp4graph op. -
MLAIndexerRaggedFloat8PagedRemapFiltered: Registers theattention.mla.indexer.ragged.float8.paged.remap.filteredgraph op. -
MLAIndexerRaggedFloat8PagedRemapNVFP4: Registers theattention.mla.indexer.ragged.float8.paged.remap.nvfp4graph op. -
MLAKPoolAppendPaged: Registers theattention.mla.kpool.append.pagedgraph op with the graph compiler. -
MLAKPoolCompress: Registers theattention.mla.kpool.compressgraph op with the graph compiler. -
MLAKPoolExpandTopK: Registers theattention.mla.kpool.expand_topkgraph op with the graph compiler. -
MLAKPoolRingClose: Registers theattention.mla.kpool.ring_closegraph op with the graph compiler. -
MLAKPoolSeedTail: Registers theattention.mla.kpool.seed_tailgraph op with the graph compiler. -
PaddedFlashAttentionGPU: Registers theattention.mha.padded.no_cachegraph op with the graph compiler. -
RaggedFlashAttentionGPU: Registers theattention.mha.ragged.no_cachegraph op with the graph compiler. -
Struct_compress_windows_ragged_paged: Registers theattention.compress_windows.ragged.pagedgraph op with the graph compiler. -
Struct_cross_attention_ragged_paged: Registers theattention.cross.ragged.pagedgraph op with the graph compiler. -
Struct_fused_qk_rope_ragged_paged: Registers thekv_cache.fused_qk_rope.ragged.pagedgraph op with the graph compiler. -
Struct_fused_qk_rope_ragged_paged_with_position_id: Registers thekv_cache.fused_qk_rope.ragged.paged.with_position_idgraph op with the graph compiler. -
Struct_fused_qkv_index_matmul_padded_ragged: -
Struct_fused_qkv_index_matmul_padded_ragged_scale_mxfp6_amd: Registers thekv_cache.fused_qkv_index_matmul.ragged.paged.scale.mxfp6.amdgraph op with the graph compiler. -
Struct_fused_qkv_index_matmul_padded_ragged_scale_mxfp8: Registers thekv_cache.fused_qkv_index_matmul.ragged.paged.scale.mxfp8graph op with the graph compiler. -
Struct_fused_qkv_index_matmul_padded_ragged_scale_mxfp8_amd: Registers thekv_cache.fused_qkv_index_matmul.ragged.paged.scale.mxfp8.amdgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_paged: Registers thekv_cache.fused_qkv_matmul.padded.pagedgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged: Registers thekv_cache.fused_qkv_matmul.ragged.pagedgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_bias: Registers thekv_cache.fused_qkv_matmul.ragged.paged.biasgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_bias_quantized: Registers thekv_cache.fused_qkv_matmul.ragged.paged.bias.quantizedgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_quantized: Registers thekv_cache.fused_qkv_matmul.ragged.paged.quantizedgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_scale: Registers thekv_cache.fused_qkv_matmul.ragged.paged.scalegraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_scale_bias: Registers thekv_cache.fused_qkv_matmul.ragged.paged.scale.biasgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_scale_float4: Registers thekv_cache.fused_qkv_matmul.ragged.paged.scale.float4graph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_scale_mxfp6_amd: Registers thekv_cache.fused_qkv_matmul.ragged.paged.scale.mxfp6.amdgraph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_scale_mxfp8: Registers thekv_cache.fused_qkv_matmul.ragged.paged.scale.mxfp8graph op with the graph compiler. -
Struct_fused_qkv_matmul_padded_ragged_scale_mxfp8_amd: Registers thekv_cache.fused_qkv_matmul.ragged.paged.scale.mxfp8.amdgraph op with the graph compiler. -
Struct_indexer_score_ragged_paged: Registers theattention.indexer_score.ragged.pagedgraph op with the graph compiler. -
Struct_latent_sparse_attention_ragged_paged: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_latent_sparse_attention_ragged_paged_amd: Registers the op for AMD GPUs. -
Struct_latent_sparse_attention_ragged_paged_sm100: Registers the op for B100 and B200. -
Struct_latent_sparse_attention_ragged_paged_sm103: Registers the op for B300. -
Struct_mha_padded_paged: Registers theattention.mha.padded.pagedgraph op with the graph compiler. -
Struct_mha_ragged_paged_scalar_args: Registers theattention.mha.ragged.pagedgraph op with the graph compiler. -
Struct_mha_ragged_paged_sink_weights_scalar_args: Registers theattention.mha.ragged.paged.sink_weightsgraph op with the graph compiler. -
Struct_mla_decode_graph_bf16_paged: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_decode_graph_bf16_paged_amd: Registers the op for AMD GPUs. -
Struct_mla_decode_graph_bf16_paged_sm100: Registers the op for B100 and B200. -
Struct_mla_decode_graph_bf16_paged_sm103: Registers the op for B300. -
Struct_mla_decode_graph_bf16_paged_sparse: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_decode_graph_bf16_paged_sparse_amd: Registers the op for AMD GPUs. -
Struct_mla_decode_graph_bf16_paged_sparse_cp: Registers theattention.mla.graph.decode.paged.sparse.cpgraph op. -
Struct_mla_decode_graph_bf16_paged_sparse_nvfp4: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_decode_graph_bf16_paged_sparse_nvfp4_amd: Registers the op for AMD GPUs. -
Struct_mla_decode_graph_bf16_paged_sparse_nvfp4_cp: Registers theattention.mla.graph.decode.paged.sparse.nvfp4.cpgraph op. -
Struct_mla_decode_graph_bf16_paged_sparse_nvfp4_sm100: Registers the op for B100 and B200. -
Struct_mla_decode_graph_bf16_paged_sparse_nvfp4_sm103: Registers the op for B300. -
Struct_mla_decode_graph_bf16_paged_sparse_sm100: Registers the op for B100 and B200. -
Struct_mla_decode_graph_bf16_paged_sparse_sm103: Registers the op for B300. -
Struct_mla_decode_graph_paged_fp8: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_decode_graph_paged_fp8_amd: Registers the op for AMD GPUs. -
Struct_mla_decode_graph_paged_fp8_sm100: Registers the op for B100 and B200. -
Struct_mla_decode_graph_paged_fp8_sm103: Registers the op for B300. -
Struct_mla_decode_graph_paged_fp8_sparse: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_decode_graph_paged_fp8_sparse_amd: Registers the op for AMD GPUs. -
Struct_mla_decode_graph_paged_fp8_sparse_cp: Registers theattention.mla.graph.decode.paged.fp8.sparse.cpgraph op. -
Struct_mla_decode_graph_paged_fp8_sparse_sm100: Registers the op for B100 and B200. -
Struct_mla_decode_graph_paged_fp8_sparse_sm103: Registers the op for B300. -
Struct_mla_decode_ragged_paged: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_decode_ragged_paged_amd: Registers the op for AMD GPUs. -
Struct_mla_decode_ragged_paged_scaled: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_decode_ragged_paged_scaled_amd: Registers the op for AMD GPUs. -
Struct_mla_decode_ragged_paged_scaled_sm100: Registers the op for B100 and B200. -
Struct_mla_decode_ragged_paged_scaled_sm103: Registers the op for B300. -
Struct_mla_decode_ragged_paged_sm100: Registers the op for B100 and B200. -
Struct_mla_decode_ragged_paged_sm103: Registers the op for B300. -
Struct_mla_decompress_k_cache_ragged_paged: Registers theattention.mla.decompress.k.cache.ragged.pagedgraph op with the graph compiler. -
Struct_mla_prefill_graph_bf16_paged: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_bf16_paged_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_bf16_paged_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_bf16_paged_sm103: Registers the op for B300. -
Struct_mla_prefill_graph_decode_bf16_paged: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_decode_bf16_paged_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_decode_bf16_paged_quantized: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_decode_bf16_paged_quantized_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_decode_bf16_paged_quantized_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_decode_bf16_paged_quantized_sm103: Registers the op for B300. -
Struct_mla_prefill_graph_decode_bf16_paged_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_decode_bf16_paged_sm103: Registers the op for B300. -
Struct_mla_prefill_graph_decode_paged_fp8: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_decode_paged_fp8_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_decode_paged_fp8_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_decode_paged_fp8_sm103: Registers the op for B300. -
Struct_mla_prefill_graph_decode_paged_fp8_sparse: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_decode_paged_fp8_sparse_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_decode_paged_fp8_sparse_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_decode_paged_fp8_sparse_sm103: Registers the op for B300. -
Struct_mla_prefill_graph_decode_paged_sparse: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_decode_paged_sparse_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_decode_paged_sparse_nvfp4: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_decode_paged_sparse_nvfp4_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_decode_paged_sparse_nvfp4_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_decode_paged_sparse_nvfp4_sm103: Registers the op for B300. -
Struct_mla_prefill_graph_decode_paged_sparse_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_decode_paged_sparse_sm103: Registers the op for B300. -
Struct_mla_prefill_graph_paged: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_graph_paged_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_graph_paged_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_graph_paged_sm103: Registers the op for B300. -
Struct_mla_prefill_ragged_paged: Registers the op for pre-SM100 NVIDIA GPUs and any other device. -
Struct_mla_prefill_ragged_paged_amd: Registers the op for AMD GPUs. -
Struct_mla_prefill_ragged_paged_sm100: Registers the op for B100 and B200. -
Struct_mla_prefill_ragged_paged_sm103: Registers the op for B300. -
Struct_mla_prefill_ragged_plan: Registers theattention.mla.prefill.ragged.plangraph op with the graph compiler. -
Struct_mla_store_ragged_paged: Registers theattention.mla.store.ragged.pagedgraph op with the graph compiler. -
Struct_mla_store_ragged_paged_nvfp4: Registers theattention.mla.store.ragged.paged.nvfp4graph op with the graph compiler.