IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_mla_decode_graph_paged_fp8_sparse_cp

struct Struct_mla_decode_graph_paged_fp8_sparse_cp

Registers the attention.mla.graph.decode.paged.fp8.sparse.cp graph op.

attention.mla.graph.decode.paged.fp8.sparse for decode context parallelism. Each query token t attends over the first sparse_lengths[t] entries of its sparse_indices row (0 is allowed), and the op also writes lse, the attention log-sum-exp per (token, head) in base 2: log2(sum_k exp2(q.k * scale * log2(e))), before W_UV. A token with no keys gets lse = -inf and output = 0. Ranks merge as w_r = exp2(lse_r - max_r lse_r), O = sum_r w_r O_r / sum_r w_r. split_len_bound (>0) bounds every length and sizes split-K in place of indices_stride; it only affects performance. As in the base op with dcp_size > 1, the fused latent store writes a new token only on the rank that owns its position (ContextParallelLayout[dcp_size, dcp_interleave]), at that rank's local slot; dcp_rank is a host scalar after split_len_bound. SM100 only.

Implemented traitsโ€‹

AnyType, Deinitable, Movable

Methodsโ€‹

executeโ€‹

static def execute[m_scale_granularity: Int, n_scale_granularity: Int, k_scale_granularity: Int, mask_str: StaticString, dcp_size: Int, dcp_interleave: Int, indices_stride: Int, physical_indices: Bool](output: T, lse: T, q: T, kv: T, input_row_offsets: T, freqs_cis: T, kv_norm_gamma: T, w_uk: T, w_uv: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, scale: Float32, epsilon: Float32, w_uk_scale: T, w_uv_scale: T, scalar_args: T, sparse_indices: T, sparse_lengths: T, split_len_bound: Int32, dcp_rank: Int32, context: DeviceContext)

Was this page helpful?