For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
QkArgs
struct QkArgs[cache_dtype: DType, freq_dtype: DType = .float32]
qk_norm_rope_store's operands, one kernel parameter.
Fields
- qkv (
In[.bfloat16]): The projection,qkv_strideelements a token. - q_out (
Out[.bfloat16]): The roped Q[tokens, n_q * HEAD_DIM]. - freqs (
In[freq_dtype]): Interleaved (cos, sin) pairs[positions, HEAD_DIM]. - offsets (
In[.uint32]): Ragged row offsets[batch + 1]. - cache_lengths (
In[.uint32]): - lut (
In[.uint32]): The page table,lut_colspages a sequence. - k_rows (
Out[cache_dtype]): This layer's key plane: rows ofn_kv * HEAD_DIM,page_rowsrows from one page to the next. - v_rows (
Out[cache_dtype]): - qkv_stride (
Int64): - tokens (
Int64): - batch (
Int64): - lut_cols (
Int64): - page_rows (
Int64): - eps (
Float32): - q_scale (
Float32): The Q factor, already rounded to bf16.
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
Movable
comptime members
device_type
comptime device_type = QkArgs[cache_dtype, freq_dtype]