For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
k_matmul_ragged_paged_scale
def k_matmul_ragged_paged_scale[dtype: DType, weight_dtype: DType, scale_dtype: DType, target: StringSpan[ImmStaticOrigin], scales_granularity_mnk: IndexList[Int(3)]](hidden_state: TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], weight: TileTensor[weight_dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type], input_scale: TileTensor[scale_dtype, Engine=input_scale.Engine, linear_idx_type=input_scale.linear_idx_type], weight_scale: TileTensor[scale_dtype, Engine=weight_scale.Engine, linear_idx_type=weight_scale.linear_idx_type], kv_collection: PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_], layer_idx: UInt32, ctx: DeviceContextBase)
Performs a matmul, writing the output into a mutable PagedKVCacheCollection object.
Parameters:
- dtype (
DType): Element type of thehidden_stateinput tensor and of the key entries written to the KV cache. - weight_dtype (
DType): Element type of theweighttensor; must matchdtype. - scale_dtype (
DType): Element type of theinput_scaleandweight_scaletensors. - target (
StringSpan[ImmStaticOrigin]): Compilation target string; must be a GPU target. - scales_granularity_mnk (
IndexList[Int(3)]): Block sizes along the M, N, and K matmul dimensions used to tile the scale application;-1selects per-tensor scaling,1selects per-channel scaling, and any other value selects blockwise scaling.
Args:
- hidden_state (
TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type]): Ragged input tensor with shape (sum(seq_lens), hidden_size). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the seq_len dimension. - weight (
TileTensor[weight_dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type]): Key projection weights with shape (num_kv_heads * head_size, hidden_size). - input_scale (
TileTensor[scale_dtype, Engine=input_scale.Engine, linear_idx_type=input_scale.linear_idx_type]): Scale to be multiplied to the input Tensor. - weight_scale (
TileTensor[scale_dtype, Engine=weight_scale.Engine, linear_idx_type=weight_scale.linear_idx_type]): Scale to be multiplied to the weight Tensor. - kv_collection (
PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_]): The historical KVCache for keys and values. The KVCache for this layer is retrieved via layer_idx. - layer_idx (
UInt32): The index of the layer being executed. Used to retrieve the KVCache for the given layer from kv_collection. - ctx (
DeviceContextBase): The call context pointer, passed by the graph compiler.