IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_prefill_branch_bf16

def mla_prefill_branch_bf16[collection_t: KVCollectionT, KVInputFnType: def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable, //, mask_str: StaticString, OutputGatePtrType: OptionalPointer = NullPointer[.bfloat16], *, kernels: MLAGraphKernels](output: TileTensor[.bfloat16, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[.bfloat16, Engine=q.Engine, linear_idx_type=q.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], freqs_cis: TileTensor[Engine=freqs_cis.Engine, linear_idx_type=freqs_cis.linear_idx_type], kv_norm_gamma: TileTensor[Engine=kv_norm_gamma.Engine, linear_idx_type=kv_norm_gamma.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, epsilon: Float32, buffer_row_offsets: TileTensor[.uint32, Engine=buffer_row_offsets.Engine, linear_idx_type=buffer_row_offsets.linear_idx_type], cache_offsets: TileTensor[.uint32, Engine=cache_offsets.Engine, linear_idx_type=cache_offsets.linear_idx_type], buffer_length: Int, w_k: TileTensor[.bfloat16, Engine=w_k.Engine, linear_idx_type=w_k.linear_idx_type], w_uv: TileTensor[.bfloat16, Engine=w_uv.Engine, linear_idx_type=w_uv.linear_idx_type], kv_input_fn: KVInputFnType, ctx: DeviceContext, output_gate: OutputGatePtrType = null_pointer[OutputGatePtrType](), output_gate_row_stride: Int = Int(0))

BF16 MLA prefill path.

Applies RoPE and RMSNorm, up-projects latent KV to full K and V, then runs prefill attention.

Parameters:

  • ​collection_t (KVCollectionT): Type of the KV collection (inferred).
  • ​KVInputFnType (def[width: Int](IndexList[Int(2)]) -> SIMD[.bfloat16, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the KV latent values (inferred).
  • ​mask_str (StaticString): Mask variant.
  • ​OutputGatePtrType (OptionalPointer): OptionalPointer type of output_gate (inferred, defaults to NullPointer); a null type applies no gate.
  • ​kernels (MLAGraphKernels): The vendor's MLA graph kernels; see MLAGraphKernels.

Args:

Was this page helpful?