mirror of
https://github.com/microsoft/BitNet.git
synced 2026-08-23 17:36:31 +00:00
Add bitnet-embeddings-270m model adaptation with F16 and I2_S GGUF conversion
- Add LLM_ARCH_GEMMA3 in llama.cpp for gemma3_text model type (embedding scaling, GELU, post-attn/post-FFN norms, GQA) - Add GGUF conversion support for Gemma3-based 270m models (SPM tokenizer, RMSNorm w+1 offset, arch-specific tensor mapping) - Add tokenizer hash for multilingual-e5-0.6b-260311 - Add conversion documentation
This commit is contained in:
isHuangXin
@@ -0,0 +1,410 @@
|
||||
# BitNet Embeddings GGUF Conversion Implementation
|
||||
|
||||
## 1. Background
|
||||
|
||||
BitNet embedding models apply per-projection RMSNorm (`BitLinear`) before each linear projection (q/k/v/o/gate/up/down). Each projection has a `.norm.weight` that applies RMSNorm to the input **before** the matmul:
|
||||
|
||||
```
|
||||
x → RMSNorm(x, norm.weight) → activation_quant(8bit) → matmul(weight_quant(ternary))
|
||||
```
|
||||
|
||||
This pattern does **not** exist in any standard llama.cpp architecture:
|
||||
- Standard Qwen3/Gemma3: no per-projection norms
|
||||
- Standard BitNet: has `attn_sub_norm`/`ffn_sub_norm` at different positions (after attention/gate*up, not before each projection)
|
||||
|
||||
Currently two base architectures are supported:
|
||||
|
||||
| | bitnet-embeddings-0.6b (Qwen3) | bitnet-embeddings-270m (Gemma3) |
|
||||
|---|---|---|
|
||||
| Architecture | `Qwen3Model` | `Gemma3TextModel` |
|
||||
| hidden_size | 1024 | 640 |
|
||||
| num_attention_heads | 16 | 4 |
|
||||
| num_key_value_heads | 8 | 1 |
|
||||
| head_dim | 128 (note: != hidden_size/num_heads = 64) | 256 (note: != hidden_size/num_heads = 160) |
|
||||
| intermediate_size | 3072 | 2048 |
|
||||
| num_hidden_layers | 28 | 18 |
|
||||
| hidden_activation | SiLU | gelu_pytorch_tanh |
|
||||
| vocab_size | 151936 | 262144 |
|
||||
| rope_theta | 1000000 | 10000.0 |
|
||||
| rms_norm_eps | 1e-06 | 1e-06 |
|
||||
| query_pre_attn_scalar | N/A | 256 |
|
||||
| tie_word_embeddings | true | true |
|
||||
|
||||
### Gemma3 vs Qwen3 Key Differences
|
||||
|
||||
| Feature | Qwen3 | Gemma3 |
|
||||
|---------|-------|--------|
|
||||
| Post-attn norm | No | Yes (`post_attention_norm`) |
|
||||
| Post-FFW norm | No | Yes (`post_ffw_norm`) |
|
||||
| Pre-FFW norm naming | `post_attention_layernorm` → `ffn_norm` | `pre_feedforward_layernorm` → `ffn_norm` |
|
||||
| QK head norms | Yes | Yes |
|
||||
| Activation | SiLU | GELU |
|
||||
| Embedding scaling | No | sqrt(n_embd) |
|
||||
| EOS token override | Yes (`<\|endoftext\|>` 151643) | No (auto from tokenizer) |
|
||||
|
||||
### Per-Layer Tensors (7 extra norm tensors per layer)
|
||||
|
||||
| Tensor | Qwen3 Shape | Gemma3 Shape |
|
||||
|--------|-------------|--------------|
|
||||
| `self_attn.q_proj.norm.weight` | [1024] | [640] |
|
||||
| `self_attn.k_proj.norm.weight` | [1024] | [640] |
|
||||
| `self_attn.v_proj.norm.weight` | [1024] | [640] |
|
||||
| `self_attn.o_proj.norm.weight` | [2048] | [1024] |
|
||||
| `mlp.gate_proj.norm.weight` | [1024] | [640] |
|
||||
| `mlp.up_proj.norm.weight` | [1024] | [640] |
|
||||
| `mlp.down_proj.norm.weight` | [3072] | [2048] |
|
||||
|
||||
---
|
||||
|
||||
## 2. GGUF Tensor Name Mapping
|
||||
|
||||
### Common Tensors (both architectures)
|
||||
|
||||
| HF Name | GGUF Name | Notes |
|
||||
|----------|-----------|-------|
|
||||
| `embed_tokens.weight` | `token_embd.weight` | |
|
||||
| `norm.weight` | `output_norm.weight` | |
|
||||
| `layers.{i}.input_layernorm.weight` | `blk.{i}.attn_norm.weight` | |
|
||||
| `layers.{i}.self_attn.q_proj.weight` | `blk.{i}.attn_q.weight` | |
|
||||
| `layers.{i}.self_attn.k_proj.weight` | `blk.{i}.attn_k.weight` | |
|
||||
| `layers.{i}.self_attn.v_proj.weight` | `blk.{i}.attn_v.weight` | |
|
||||
| `layers.{i}.self_attn.o_proj.weight` | `blk.{i}.attn_output.weight` | |
|
||||
| `layers.{i}.self_attn.q_norm.weight` | `blk.{i}.attn_q_norm.weight` | QK head norm |
|
||||
| `layers.{i}.self_attn.k_norm.weight` | `blk.{i}.attn_k_norm.weight` | QK head norm |
|
||||
| `layers.{i}.self_attn.q_proj.norm.weight` | `blk.{i}.attn_q_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.self_attn.k_proj.norm.weight` | `blk.{i}.attn_k_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.self_attn.v_proj.norm.weight` | `blk.{i}.attn_v_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.self_attn.o_proj.norm.weight` | `blk.{i}.attn_output_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.mlp.gate_proj.weight` | `blk.{i}.ffn_gate.weight` | |
|
||||
| `layers.{i}.mlp.up_proj.weight` | `blk.{i}.ffn_up.weight` | |
|
||||
| `layers.{i}.mlp.down_proj.weight` | `blk.{i}.ffn_down.weight` | |
|
||||
| `layers.{i}.mlp.gate_proj.norm.weight` | `blk.{i}.ffn_gate_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.mlp.up_proj.norm.weight` | `blk.{i}.ffn_up_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.mlp.down_proj.norm.weight` | `blk.{i}.ffn_down_norm_in.weight` | BitNet per-projection |
|
||||
|
||||
### Architecture-Specific Tensors
|
||||
|
||||
**Qwen3:**
|
||||
|
||||
| HF Name | GGUF Name |
|
||||
|----------|-----------|
|
||||
| `layers.{i}.post_attention_layernorm.weight` | `blk.{i}.ffn_norm.weight` |
|
||||
|
||||
**Gemma3 (additional):**
|
||||
|
||||
| HF Name | GGUF Name |
|
||||
|----------|-----------|
|
||||
| `layers.{i}.post_attention_layernorm.weight` | `blk.{i}.post_attention_norm.weight` |
|
||||
| `layers.{i}.pre_feedforward_layernorm.weight` | `blk.{i}.ffn_norm.weight` |
|
||||
| `layers.{i}.post_feedforward_layernorm.weight` | `blk.{i}.post_ffw_norm.weight` |
|
||||
|
||||
---
|
||||
|
||||
## 3. Conversion Script
|
||||
|
||||
### `utils/convert-bitnet-embedding-to-gguf.py`
|
||||
|
||||
Unified standalone conversion script (safetensors → GGUF) that **auto-detects** the model architecture from `config.json`'s `model_type` field (`qwen3` or `gemma3_text`). Key features:
|
||||
|
||||
- Hardcoded HF→GGUF tensor name mapping (no dependency on llama.cpp's Python converter)
|
||||
- Auto-detection of architecture and GGUF arch string (`qwen3` / `gemma3`)
|
||||
- Supports three output types:
|
||||
- `--outtype f32`: all weights in float32
|
||||
- `--outtype f16`: 2D weights and embeddings as float16, norms as float16
|
||||
- `--outtype i2_s`: ternary weights packed in I2_S layout, non-ternary weights as float16
|
||||
- Writes `key_length` and `value_length` metadata for correct head_dim (critical: head_dim != hidden_size/num_heads for both models, default calculation would give wrong values)
|
||||
- BPE tokenizer handling with per-architecture pre-tokenizer hash verification:
|
||||
- Qwen3: GPT-2 BPE tokenizer
|
||||
- Gemma3: GemmaTokenizerFast (BPE)
|
||||
- Pooling type auto-detection from `modules.json` / `1_Pooling/config.json` (sentence-transformers convention)
|
||||
- Architecture-specific tokenizer handling:
|
||||
- Qwen3: EOS token override (`<|endoftext|>` 151643) + `add_eos_token(True)` for last-token pooling
|
||||
- Gemma3: EOS token auto-set by SpecialVocab from tokenizer_config.json (eos_token_id=1)
|
||||
- Gemma3: writes `query_pre_attn_scalar = 256` for correct attention scaling
|
||||
|
||||
### I2_S Ternary Packing
|
||||
|
||||
The I2_S format packs ternary weights {-1, 0, +1} into 2-bit representation:
|
||||
|
||||
- Quantization: `scale = 1/mean(|w|)`, `q = round(w * scale).clamp(-1, 1)`
|
||||
- Encoding: `-1 → 0`, `0 → 1`, `+1 → 2`
|
||||
- Every 128 values form a block, packed into 32 bytes
|
||||
- Each byte stores 4 values: `byte = (c0 << 6) | (c1 << 4) | (c2 << 2) | c3`
|
||||
- Scale (float32) is appended at the end of the packed data buffer
|
||||
|
||||
### Tensor Type Assignment
|
||||
|
||||
| Tensor Type | f16 mode | i2_s mode |
|
||||
|-------------|----------|-----------|
|
||||
| 2D linear weights | float16 | I2_S ternary packed |
|
||||
| Embedding weights | float16 | float16 |
|
||||
| Norm weights (1D) | float16 | float16 |
|
||||
|
||||
Note: `output.weight` (lm_head) is skipped for embedding models — it is not needed (no token generation).
|
||||
|
||||
---
|
||||
|
||||
## 4. C++ Modifications (`3rdparty/llama.cpp/src/llama.cpp`)
|
||||
|
||||
### 4.1 New Architecture: `LLM_ARCH_GEMMA3`
|
||||
|
||||
Added after `LLM_ARCH_GEMMA2` in the `llm_arch` enum with name mapping `"gemma3"`. Qwen3 (`LLM_ARCH_QWEN3`) was added by the 0.6b adaptation.
|
||||
|
||||
### 4.2 New Tensor Enums (shared across architectures)
|
||||
|
||||
Added 7 new entries after `LLM_TENSOR_FFN_SUB_NORM`:
|
||||
|
||||
```cpp
|
||||
LLM_TENSOR_ATTN_Q_NORM_IN,
|
||||
LLM_TENSOR_ATTN_K_NORM_IN,
|
||||
LLM_TENSOR_ATTN_V_NORM_IN,
|
||||
LLM_TENSOR_ATTN_OUT_NORM_IN,
|
||||
LLM_TENSOR_FFN_GATE_NORM_IN,
|
||||
LLM_TENSOR_FFN_UP_NORM_IN,
|
||||
LLM_TENSOR_FFN_DOWN_NORM_IN,
|
||||
```
|
||||
|
||||
### 4.3 Layer Struct Fields
|
||||
|
||||
Added to `struct llama_layer`:
|
||||
|
||||
```cpp
|
||||
struct ggml_tensor * attn_q_norm_in;
|
||||
struct ggml_tensor * attn_k_norm_in;
|
||||
struct ggml_tensor * attn_v_norm_in;
|
||||
struct ggml_tensor * attn_out_norm_in;
|
||||
struct ggml_tensor * ffn_gate_norm_in;
|
||||
struct ggml_tensor * ffn_up_norm_in;
|
||||
struct ggml_tensor * ffn_down_norm_in;
|
||||
```
|
||||
|
||||
### 4.4 Tensor Name Mappings
|
||||
|
||||
Both `LLM_ARCH_QWEN3` and `LLM_ARCH_GEMMA3` include the 7 per-projection norm tensor mappings plus standard tensors (see Section 2 for full mapping). Key differences:
|
||||
|
||||
- Qwen3 includes `LLM_TENSOR_OUTPUT` (`"output"`); Gemma3 does not (uses tied embeddings directly)
|
||||
- Gemma3 additionally includes `LLM_TENSOR_ATTN_POST_NORM` (`"blk.%d.post_attention_norm"`) and `LLM_TENSOR_FFN_POST_NORM` (`"blk.%d.post_ffw_norm"`)
|
||||
|
||||
### 4.5 load_tensors
|
||||
|
||||
Both architectures load the 7 per-projection norm tensors as optional (`TENSOR_NOT_REQUIRED`):
|
||||
|
||||
```cpp
|
||||
layer.attn_q_norm_in = create_tensor(tn(...), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.attn_k_norm_in = create_tensor(tn(...), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.attn_v_norm_in = create_tensor(tn(...), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.attn_out_norm_in = create_tensor(tn(...), {n_embd_head_k * n_head}, TENSOR_NOT_REQUIRED);
|
||||
layer.ffn_gate_norm_in = create_tensor(tn(...), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.ffn_up_norm_in = create_tensor(tn(...), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.ffn_down_norm_in = create_tensor(tn(...), {n_ff}, TENSOR_NOT_REQUIRED);
|
||||
```
|
||||
|
||||
Note: `o_proj.norm` input dimension is `n_embd_head_k * n_head` (Qwen3: 2048, Gemma3: 1024), `down_proj.norm` input dimension is `n_ff` (Qwen3: 3072, Gemma3: 2048).
|
||||
|
||||
Both graph functions use the same per-projection norm pattern. The logic is fully backward compatible — when no `*_norm_in` tensors exist, behavior is identical to the original.
|
||||
|
||||
**Attention per-projection norms:**
|
||||
```
|
||||
// Before Q/K/V matmul:
|
||||
if (layer.attn_q_norm_in) {
|
||||
cur_q = ggml_rms_norm(ctx, cur, hparams.f_norm_rms_eps);
|
||||
cur_q = ggml_mul(ctx, cur_q, layer.attn_q_norm_in);
|
||||
} else {
|
||||
cur_q = cur;
|
||||
}
|
||||
Qcur = ggml_mul_mat(ctx, layer.wq, cur_q);
|
||||
// QK head norms applied after projection
|
||||
Qcur = ggml_rms_norm(ctx, Qcur, hparams.f_norm_rms_eps);
|
||||
Qcur = ggml_mul(ctx, Qcur, layer.attn_q_norm);
|
||||
```
|
||||
|
||||
**O_proj norm** requires special handling because `llm_build_kv()` normally applies `wo` internally. Solution: pass `wo=NULL` to `llm_build_kv()`, then apply norm + wo manually:
|
||||
|
||||
```
|
||||
cur = llm_build_kv(..., wo=NULL, ...); // returns attention output without o_proj
|
||||
if (layer.attn_out_norm_in) {
|
||||
cur = ggml_rms_norm(ctx, cur, hparams.f_norm_rms_eps);
|
||||
cur = ggml_mul(ctx, cur, layer.attn_out_norm_in);
|
||||
}
|
||||
cur = ggml_mul_mat(ctx, layer.wo, cur);
|
||||
```
|
||||
|
||||
**FFN per-projection norms:**
|
||||
```
|
||||
// Instead of llm_build_ffn(), manually:
|
||||
if (layer.ffn_gate_norm_in) {
|
||||
tmp_gate = rms_norm(cur) * gate_norm_in;
|
||||
} else {
|
||||
tmp_gate = cur;
|
||||
}
|
||||
tmp_gate = matmul(gate_proj, tmp_gate);
|
||||
tmp_gate = activation(tmp_gate); // SiLU for Qwen3, GELU for Gemma3
|
||||
// Similarly for up_proj
|
||||
tmp = tmp_gate * tmp_up;
|
||||
|
||||
if (layer.ffn_down_norm_in) {
|
||||
tmp = rms_norm(tmp) * down_norm_in;
|
||||
}
|
||||
cur = matmul(down_proj, tmp);
|
||||
```
|
||||
|
||||
**Gemma3-specific differences:**
|
||||
- Embedding scaling by `sqrt(n_embd)` (Gemma convention)
|
||||
- GELU activation instead of SiLU
|
||||
- Post-attention and post-FFN layer norms
|
||||
- `query_pre_attn_scalar` for attention scaling
|
||||
|
||||
---
|
||||
|
||||
## 5. GGUF Conversion Process
|
||||
|
||||
Each model variant requires two GGUF files from **two different source models**:
|
||||
|
||||
### 5.1 Qwen3 (0.6b)
|
||||
|
||||
| GGUF Output | Source Model | Description |
|
||||
|-------------|-------------|-------------|
|
||||
| `embeddings-0.6b-f16.gguf` | `multilingual-e5-0.6b` (standard Qwen3) | F16 baseline |
|
||||
| `bitnet-embeddings-0.6b-f16-i2_s.gguf` | `bitnet-embeddings-0.6b` (BitNet ternary) | I2_S ternary packed |
|
||||
|
||||
**F16 (from standard Qwen3 model):**
|
||||
```bash
|
||||
python3 utils/convert-bitnet-embedding-to-gguf.py \
|
||||
/path/to/multilingual-e5-0.6b \
|
||||
--outtype f16 \
|
||||
--outfile embeddings-0.6b-f16.gguf
|
||||
```
|
||||
|
||||
What happens:
|
||||
1. Load `model.safetensors` (standard Qwen3 weights, bfloat16)
|
||||
2. Convert all 2D weights (projections, embeddings) to float16
|
||||
3. Convert norm weights to float16
|
||||
4. Write GGUF with `qwen3` architecture metadata and tokenizer
|
||||
|
||||
**Output:** ~1.11 GiB (595.78M params)
|
||||
|
||||
**I2_S (from BitNet model):**
|
||||
```bash
|
||||
python3 utils/convert-bitnet-embedding-to-gguf.py \
|
||||
/path/to/bitnet-embeddings-0.6b \
|
||||
--outfile bitnet-embeddings-0.6b-f16-i2_s.gguf --outtype i2_s
|
||||
```
|
||||
|
||||
What happens:
|
||||
1. Load `model.safetensors` (BitNet ternary weights, bfloat16)
|
||||
2. Map HF tensor names to GGUF names, including 7 extra `*_norm_in` tensors per layer
|
||||
3. For each 2D linear weight: quantize to I2_S ternary packed format
|
||||
4. Keep embeddings (`token_embd.weight`) in float16
|
||||
5. Keep all norm weights in float16
|
||||
6. Skip `output.weight` (lm_head, not needed for embedding models)
|
||||
7. Write GGUF with `I2_S` type tag for quantized tensors
|
||||
|
||||
**Output:** ~699 MiB (~50% of F16 size)
|
||||
|
||||
### 5.2 Gemma3 (270m)
|
||||
|
||||
| GGUF Output | Source Model | Description |
|
||||
|-------------|-------------|-------------|
|
||||
| `multilingual-e5-270m-f16.gguf` | `multilingual-e5-270m-260311` (standard Gemma3) | F16 baseline |
|
||||
| `bitnet-embeddings-270m-i2_s.gguf` | `bitnet-embeddings-270m` (BitNet ternary) | I2_S ternary packed |
|
||||
|
||||
**F16 (from standard Gemma3 model):**
|
||||
```bash
|
||||
python3 utils/convert-bitnet-embedding-to-gguf.py \
|
||||
/path/to/multilingual-e5-270m-260311 \
|
||||
--outtype f16
|
||||
```
|
||||
|
||||
What happens:
|
||||
1. Load `model.safetensors` (standard Gemma3 weights, bfloat16)
|
||||
2. Convert all 2D weights (projections, embeddings) to float16
|
||||
3. Convert norm weights to float16
|
||||
4. Write GGUF with `gemma3` architecture metadata and tokenizer
|
||||
|
||||
**I2_S (from BitNet model):**
|
||||
```bash
|
||||
python3 utils/convert-bitnet-embedding-to-gguf.py \
|
||||
/path/to/bitnet-embeddings-270m \
|
||||
--outtype i2_s
|
||||
```
|
||||
|
||||
What happens:
|
||||
1. Load `model.safetensors` (BitNet ternary weights, bfloat16)
|
||||
2. Map HF tensor names to GGUF names, including 7 extra `*_norm_in` tensors per layer
|
||||
3. For each 2D linear weight: quantize to I2_S ternary packed format
|
||||
4. Keep embeddings (`token_embd.weight`) in float16
|
||||
5. Keep all norm weights in float16
|
||||
6. Skip `output.weight` (lm_head, not needed for embedding models)
|
||||
7. Write GGUF with `I2_S` type tag for quantized tensors
|
||||
|
||||
### 5.3 Why Two Different Source Models?
|
||||
|
||||
- `multilingual-e5-*` is the **teacher/baseline model** with standard float weights, used as the F16 performance reference
|
||||
- `bitnet-embeddings-*` is the **1-bit quantized student model** with ternary weights and per-projection BitLinear norms, converted to I2_S for efficient CPU inference
|
||||
- Benchmarking compares both to measure the throughput gain and quality trade-off of ternary quantization
|
||||
|
||||
### 5.4 Tensor Type Summary
|
||||
|
||||
| Tensor | F16 (baseline) | I2_S (BitNet) |
|
||||
|--------|----------------|---------------|
|
||||
| Linear projections (q/k/v/o/gate/up/down) | float16 | I2_S (2-bit packed + float32 scale) |
|
||||
| Embedding (`token_embd.weight`) | float16 | float16 |
|
||||
| Per-projection norms (`*_norm_in`) | N/A (not present) | float16 |
|
||||
| Layer norms (attn_norm, ffn_norm, etc.) | float16 | float16 |
|
||||
| QK head norms (`attn_q_norm`, `attn_k_norm`) | float16 | float16 |
|
||||
| `output.weight` (lm_head) | skipped | skipped |
|
||||
|
||||
---
|
||||
|
||||
## 6. Additional Changes
|
||||
|
||||
### 6.1 ggml.c: F16 Norm Weight Support
|
||||
|
||||
Added `ggml_compute_forward_mul_f32_f16()` function to support element-wise multiplication where norm weights are stored in float16. Modified `ggml_compute_forward_mul()` to dispatch based on `src1->type`.
|
||||
|
||||
### 6.2 gguf-py: I2_S Type
|
||||
|
||||
Added `I2_S = 36` to `GGMLQuantizationType` enum and `(4, 1)` quant size in `constants.py`.
|
||||
|
||||
### 6.3 CMakeLists.txt: BitNet LUT Kernels Guard
|
||||
|
||||
Guarded `bitnet-lut-kernels.h` include with `if (GGML_BITNET_ARM_TL1 OR GGML_BITNET_X86_TL2)` to prevent build errors when LUT kernels are not available.
|
||||
|
||||
### 6.4 ggml-bitnet-mad.cpp: AVX512 SIMD
|
||||
|
||||
Added AVX512BW SIMD paths for I2_S dot product functions:
|
||||
- `ggml_vec_dot_i2_i8_s_1x1`
|
||||
- `ggml_vec_dot_i2_i8_s_1xN`
|
||||
- `ggml_vec_dot_i2_i8_s_Nx1`
|
||||
|
||||
---
|
||||
|
||||
## 7. Build and Run
|
||||
|
||||
```bash
|
||||
# Build with BitNet repo (includes I2_S support)
|
||||
cmake -S /path/to/BitNet -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --target llama-embedding llama-bench -j$(nproc)
|
||||
|
||||
# Run embedding inference (Qwen3 example)
|
||||
build/bin/llama-embedding -m bitnet-embeddings-0.6b-f16-i2_s.gguf \
|
||||
-p "hello world" --embd-normalize 2 --embd-output-format array
|
||||
|
||||
# Run embedding inference (Gemma3 example)
|
||||
build/bin/llama-embedding -m bitnet-embeddings-270m-i2_s.gguf \
|
||||
-p "hello world" --embd-normalize 2 --embd-output-format array
|
||||
|
||||
# Benchmark: F16 vs I2_S (Qwen3)
|
||||
build/bin/llama-bench -m embeddings-0.6b-f16.gguf \
|
||||
-t 8 -p 128,256,512,1024,2048 -n 32,64 -r 3 -ngl 0
|
||||
|
||||
build/bin/llama-bench -m bitnet-embeddings-0.6b-f16-i2_s.gguf \
|
||||
-t 8 -p 128,256,512,1024,2048 -n 32,64 -r 3 -ngl 0
|
||||
|
||||
# Benchmark: F16 vs I2_S (Gemma3)
|
||||
build/bin/llama-bench -m multilingual-e5-270m-f16.gguf \
|
||||
-t 8 -p 128,256,512,1024,2048 -n 32,64 -r 3 -ngl 0
|
||||
|
||||
build/bin/llama-bench -m bitnet-embeddings-270m-i2_s.gguf \
|
||||
-t 8 -p 128,256,512,1024,2048 -n 32,64 -r 3 -ngl 0
|
||||
```
|
||||
@@ -1,302 +0,0 @@
|
||||
# BitNet Embeddings (Qwen3) GGUF Conversion Implementation
|
||||
|
||||
## 1. Background
|
||||
|
||||
`bitnet-embeddings-0.6b` is a Qwen3-based embedding model with BitNet per-projection RMSNorm (`BitLinear`). Each linear projection (q/k/v/o/gate/up/down) has a `.norm.weight` that applies RMSNorm to the input **before** the matmul:
|
||||
|
||||
```
|
||||
x → RMSNorm(x, norm.weight) → activation_quant(8bit) → matmul(weight_quant(ternary))
|
||||
```
|
||||
|
||||
This pattern does **not** exist in any standard llama.cpp architecture:
|
||||
- Standard Qwen3: no per-projection norms
|
||||
- Standard BitNet: has `attn_sub_norm`/`ffn_sub_norm` at different positions (after attention/gate*up, not before each projection)
|
||||
|
||||
### Model Config
|
||||
|
||||
- Architecture: `Qwen3Model`
|
||||
- hidden_size: 1024, num_attention_heads: 16, num_key_value_heads: 8
|
||||
- head_dim: 128 (note: != hidden_size/num_heads = 64)
|
||||
- intermediate_size: 3072, num_hidden_layers: 28
|
||||
- tie_word_embeddings: true
|
||||
- rope_theta: 1000000, rms_norm_eps: 1e-06
|
||||
|
||||
### Per-Layer Tensors (7 extra norm tensors per layer)
|
||||
|
||||
| Tensor | Shape |
|
||||
|--------|-------|
|
||||
| `self_attn.q_proj.norm.weight` | [1024] |
|
||||
| `self_attn.k_proj.norm.weight` | [1024] |
|
||||
| `self_attn.v_proj.norm.weight` | [1024] |
|
||||
| `self_attn.o_proj.norm.weight` | [2048] |
|
||||
| `mlp.gate_proj.norm.weight` | [1024] |
|
||||
| `mlp.up_proj.norm.weight` | [1024] |
|
||||
| `mlp.down_proj.norm.weight` | [3072] |
|
||||
|
||||
---
|
||||
|
||||
## 2. GGUF Tensor Name Mapping
|
||||
|
||||
| HF Name | GGUF Name | Notes |
|
||||
|----------|-----------|-------|
|
||||
| `embed_tokens.weight` | `token_embd.weight` | |
|
||||
| `norm.weight` | `output_norm.weight` | |
|
||||
| `layers.{i}.input_layernorm.weight` | `blk.{i}.attn_norm.weight` | |
|
||||
| `layers.{i}.post_attention_layernorm.weight` | `blk.{i}.ffn_norm.weight` | |
|
||||
| `layers.{i}.self_attn.q_proj.weight` | `blk.{i}.attn_q.weight` | |
|
||||
| `layers.{i}.self_attn.k_proj.weight` | `blk.{i}.attn_k.weight` | |
|
||||
| `layers.{i}.self_attn.v_proj.weight` | `blk.{i}.attn_v.weight` | |
|
||||
| `layers.{i}.self_attn.o_proj.weight` | `blk.{i}.attn_output.weight` | |
|
||||
| `layers.{i}.self_attn.q_norm.weight` | `blk.{i}.attn_q_norm.weight` | QK head norm |
|
||||
| `layers.{i}.self_attn.k_norm.weight` | `blk.{i}.attn_k_norm.weight` | QK head norm |
|
||||
| `layers.{i}.self_attn.q_proj.norm.weight` | `blk.{i}.attn_q_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.self_attn.k_proj.norm.weight` | `blk.{i}.attn_k_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.self_attn.v_proj.norm.weight` | `blk.{i}.attn_v_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.self_attn.o_proj.norm.weight` | `blk.{i}.attn_output_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.mlp.gate_proj.weight` | `blk.{i}.ffn_gate.weight` | |
|
||||
| `layers.{i}.mlp.up_proj.weight` | `blk.{i}.ffn_up.weight` | |
|
||||
| `layers.{i}.mlp.down_proj.weight` | `blk.{i}.ffn_down.weight` | |
|
||||
| `layers.{i}.mlp.gate_proj.norm.weight` | `blk.{i}.ffn_gate_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.mlp.up_proj.norm.weight` | `blk.{i}.ffn_up_norm_in.weight` | BitNet per-projection |
|
||||
| `layers.{i}.mlp.down_proj.norm.weight` | `blk.{i}.ffn_down_norm_in.weight` | BitNet per-projection |
|
||||
|
||||
---
|
||||
|
||||
## 3. Conversion Script
|
||||
|
||||
### `utils/convert-bitnet-embedding-to-gguf.py`
|
||||
|
||||
Standalone conversion script (safetensors → GGUF). Key features:
|
||||
|
||||
- Hardcoded HF→GGUF tensor name mapping (no dependency on llama.cpp's Python converter)
|
||||
- Supports three output types:
|
||||
- `--outtype f32`: all weights in float32
|
||||
- `--outtype f16`: 2D weights and embeddings as float16, norms as float16
|
||||
- `--outtype i2_s`: ternary weights packed in I2_S layout, non-ternary weights as float16
|
||||
- Writes `key_length` and `value_length` metadata for head_dim=128 (critical: default calculation would give wrong value 64)
|
||||
- GPT-2 BPE tokenizer handling with pre-tokenizer hash verification
|
||||
- Pooling type auto-detection from `modules.json` / `1_Pooling/config.json` (sentence-transformers convention)
|
||||
- EOS token override: uses `<|endoftext|>` (151643) for correct last-token pooling
|
||||
- Architecture string: `"qwen3"`
|
||||
|
||||
### I2_S Ternary Packing
|
||||
|
||||
The I2_S format packs ternary weights {-1, 0, +1} into 2-bit representation:
|
||||
|
||||
- Quantization: `scale = 1/mean(|w|)`, `q = round(w * scale).clamp(-1, 1)`
|
||||
- Encoding: `-1 → 0`, `0 → 1`, `+1 → 2`
|
||||
- Every 128 values form a block, packed into 32 bytes
|
||||
- Each byte stores 4 values: `byte = (c0 << 6) | (c1 << 4) | (c2 << 2) | c3`
|
||||
- Scale (float32) is appended at the end of the packed data buffer
|
||||
|
||||
### Tensor Type Assignment
|
||||
|
||||
| Tensor Type | f16 mode | i2_s mode |
|
||||
|-------------|----------|-----------|
|
||||
| 2D linear weights | float16 | I2_S ternary packed |
|
||||
| Embedding weights | float16 | float16 |
|
||||
| Norm weights (1D) | float16 | float16 |
|
||||
|
||||
Note: `output.weight` (lm_head) is skipped for embedding models — it is not needed (no token generation).
|
||||
|
||||
---
|
||||
|
||||
## 4. C++ Modifications (`3rdparty/llama.cpp/src/llama.cpp`)
|
||||
|
||||
### 4.1 New Tensor Enums
|
||||
|
||||
Added 7 new entries after `LLM_TENSOR_FFN_SUB_NORM`:
|
||||
|
||||
```cpp
|
||||
LLM_TENSOR_ATTN_Q_NORM_IN,
|
||||
LLM_TENSOR_ATTN_K_NORM_IN,
|
||||
LLM_TENSOR_ATTN_V_NORM_IN,
|
||||
LLM_TENSOR_ATTN_OUT_NORM_IN,
|
||||
LLM_TENSOR_FFN_GATE_NORM_IN,
|
||||
LLM_TENSOR_FFN_UP_NORM_IN,
|
||||
LLM_TENSOR_FFN_DOWN_NORM_IN,
|
||||
```
|
||||
|
||||
### 4.2 Tensor Name Mappings
|
||||
|
||||
Added to `LLM_ARCH_QWEN3` tensor name map:
|
||||
|
||||
```cpp
|
||||
{ LLM_TENSOR_ATTN_Q_NORM_IN, "blk.%d.attn_q_norm_in" },
|
||||
{ LLM_TENSOR_ATTN_K_NORM_IN, "blk.%d.attn_k_norm_in" },
|
||||
{ LLM_TENSOR_ATTN_V_NORM_IN, "blk.%d.attn_v_norm_in" },
|
||||
{ LLM_TENSOR_ATTN_OUT_NORM_IN, "blk.%d.attn_output_norm_in" },
|
||||
{ LLM_TENSOR_FFN_GATE_NORM_IN, "blk.%d.ffn_gate_norm_in" },
|
||||
{ LLM_TENSOR_FFN_UP_NORM_IN, "blk.%d.ffn_up_norm_in" },
|
||||
{ LLM_TENSOR_FFN_DOWN_NORM_IN, "blk.%d.ffn_down_norm_in" },
|
||||
```
|
||||
|
||||
### 4.3 Layer Struct Fields
|
||||
|
||||
Added to `struct llama_layer`:
|
||||
|
||||
```cpp
|
||||
struct ggml_tensor * attn_q_norm_in;
|
||||
struct ggml_tensor * attn_k_norm_in;
|
||||
struct ggml_tensor * attn_v_norm_in;
|
||||
struct ggml_tensor * attn_out_norm_in;
|
||||
struct ggml_tensor * ffn_gate_norm_in;
|
||||
struct ggml_tensor * ffn_up_norm_in;
|
||||
struct ggml_tensor * ffn_down_norm_in;
|
||||
```
|
||||
|
||||
### 4.4 load_tensors (LLM_ARCH_QWEN3)
|
||||
|
||||
Added optional loading with `TENSOR_NOT_REQUIRED`:
|
||||
|
||||
```cpp
|
||||
layer.attn_q_norm_in = create_tensor(tn(LLM_TENSOR_ATTN_Q_NORM_IN, "weight", i), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.attn_k_norm_in = create_tensor(tn(LLM_TENSOR_ATTN_K_NORM_IN, "weight", i), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.attn_v_norm_in = create_tensor(tn(LLM_TENSOR_ATTN_V_NORM_IN, "weight", i), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.attn_out_norm_in = create_tensor(tn(LLM_TENSOR_ATTN_OUT_NORM_IN, "weight", i), {n_embd_head_k * n_head}, TENSOR_NOT_REQUIRED);
|
||||
layer.ffn_gate_norm_in = create_tensor(tn(LLM_TENSOR_FFN_GATE_NORM_IN, "weight", i), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.ffn_up_norm_in = create_tensor(tn(LLM_TENSOR_FFN_UP_NORM_IN, "weight", i), {n_embd}, TENSOR_NOT_REQUIRED);
|
||||
layer.ffn_down_norm_in = create_tensor(tn(LLM_TENSOR_FFN_DOWN_NORM_IN, "weight", i), {n_ff}, TENSOR_NOT_REQUIRED);
|
||||
```
|
||||
|
||||
Note: `o_proj.norm` input dimension is `n_embd_head_k * n_head` (=2048), `down_proj.norm` input dimension is `n_ff` (=3072).
|
||||
|
||||
### 4.5 build_qwen3() Graph Modifications
|
||||
|
||||
The `build_qwen3()` function was modified to conditionally apply per-projection RMSNorm. The logic is fully backward compatible — when no `*_norm_in` tensors exist, behavior is identical to original.
|
||||
|
||||
**Attention per-projection norms:**
|
||||
```
|
||||
// Before Q/K/V matmul:
|
||||
if (layer.attn_q_norm_in) {
|
||||
cur_q = ggml_rms_norm(ctx, cur, hparams.f_norm_rms_eps);
|
||||
cur_q = ggml_mul(ctx, cur_q, layer.attn_q_norm_in);
|
||||
} else {
|
||||
cur_q = cur;
|
||||
}
|
||||
Qcur = ggml_mul_mat(ctx, layer.wq, cur_q);
|
||||
// Similarly for K, V
|
||||
```
|
||||
|
||||
**O_proj norm** requires special handling because `llm_build_kv()` normally applies `wo` internally. Solution: pass `wo=NULL` to `llm_build_kv()`, then apply norm + wo manually:
|
||||
|
||||
```
|
||||
cur = llm_build_kv(..., wo=NULL, ...); // returns attention output without o_proj
|
||||
if (layer.attn_out_norm_in) {
|
||||
cur = ggml_rms_norm(ctx, cur, hparams.f_norm_rms_eps);
|
||||
cur = ggml_mul(ctx, cur, layer.attn_out_norm_in);
|
||||
}
|
||||
cur = ggml_mul_mat(ctx, layer.wo, cur);
|
||||
```
|
||||
|
||||
**FFN per-projection norms:**
|
||||
```
|
||||
// Instead of llm_build_ffn(), manually:
|
||||
if (layer.ffn_gate_norm_in) {
|
||||
tmp_gate = rms_norm(cur) * gate_norm_in;
|
||||
} else {
|
||||
tmp_gate = cur;
|
||||
}
|
||||
tmp_gate = matmul(gate_proj, tmp_gate);
|
||||
// Similarly for up_proj
|
||||
tmp = silu(tmp_gate) * tmp_up;
|
||||
|
||||
if (layer.ffn_down_norm_in) {
|
||||
tmp = rms_norm(tmp) * down_norm_in;
|
||||
}
|
||||
cur = matmul(down_proj, tmp);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. GGUF Conversion Process
|
||||
|
||||
There are two GGUF files to produce, from **two different source models**:
|
||||
|
||||
| GGUF Output | Source Model | Description |
|
||||
|-------------|-------------|-------------|
|
||||
| `embeddings-0.6b-f16.gguf` | `multilingual-e5-0.6b` (standard Qwen3) | F16 baseline, standard float16 weights |
|
||||
| `bitnet-embeddings-0.6b-f16-i2_s.gguf` | `bitnet-embeddings-0.6b` (BitNet ternary) | I2_S ternary packed weights |
|
||||
|
||||
### 5.1 F16 GGUF: from multilingual-e5-0.6b
|
||||
|
||||
The F16 GGUF is converted from the **standard (non-BitNet) model** `multilingual-e5-0.6b`, which has normal float weights and no per-projection RMSNorm. This uses llama.cpp's standard converter since it is a vanilla Qwen3 model:
|
||||
|
||||
```bash
|
||||
python3 /path/to/llama.cpp/convert_hf_to_gguf.py \
|
||||
/path/to/multilingual-e5-0.6b \
|
||||
--outtype f16 \
|
||||
--outfile embeddings-0.6b-f16.gguf
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Load `model.safetensors` (standard Qwen3 weights, bfloat16)
|
||||
2. Convert all 2D weights (projections, embeddings) to float16
|
||||
3. Convert norm weights to float32
|
||||
4. Write GGUF with `qwen3` architecture metadata and tokenizer
|
||||
|
||||
**Output:** ~1.11 GiB (595.78M params)
|
||||
|
||||
### 5.2 I2_S GGUF: from bitnet-embeddings-0.6b
|
||||
|
||||
The I2_S GGUF is converted from the **BitNet ternary model** `bitnet-embeddings-0.6b`, which has ternary weights {-1, 0, +1} and 7 extra per-projection RMSNorm tensors per layer. This uses the custom converter because the standard llama.cpp converter does not handle per-projection norms or I2_S quantization:
|
||||
|
||||
```bash
|
||||
python3 utils/convert-bitnet-embedding-to-gguf.py \
|
||||
/path/to/bitnet-embeddings-0.6b \
|
||||
--outfile bitnet-embeddings-0.6b-f16-i2_s.gguf --outtype i2_s
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Load `model.safetensors` (BitNet ternary weights, bfloat16)
|
||||
2. Map HF tensor names to GGUF names, including 7 extra `*_norm_in` tensors per layer (see Section 2)
|
||||
3. For each 2D linear weight (q/k/v/o/gate/up/down projections):
|
||||
- Compute scale: `scale = 1 / mean(|w|)`
|
||||
- Quantize: `q = round(w * scale).clamp(-1, 1)`
|
||||
- Encode: `-1 -> 0`, `0 -> 1`, `+1 -> 2`
|
||||
- Pack every 128 values into 32 bytes (4 values per byte, 2 bits each)
|
||||
- Append per-row float32 scale
|
||||
4. Keep embeddings (`token_embd.weight`) in float16 (not ternary)
|
||||
5. Keep all norm weights in float16
|
||||
6. Skip `output.weight` (lm_head, not needed for embedding models)
|
||||
7. Write GGUF with `I2_S` type tag for quantized tensors
|
||||
|
||||
**Output:** ~699 MiB (~50% of F16 size)
|
||||
|
||||
### 5.3 Why Two Different Source Models?
|
||||
|
||||
- `multilingual-e5-0.6b` is the **teacher/baseline model** with standard float weights, used as the F16 performance reference
|
||||
- `bitnet-embeddings-0.6b` is the **1-bit quantized student model** with ternary weights and per-projection BitLinear norms, converted to I2_S for efficient CPU inference
|
||||
- Benchmarking compares both to measure the throughput gain and quality trade-off of ternary quantization
|
||||
|
||||
### 5.4 Tensor Type Summary
|
||||
|
||||
| Tensor | F16 (from e5-0.6b) | I2_S (from bitnet-0.6b) |
|
||||
|--------|---------------------|-------------------------|
|
||||
| Linear projections (q/k/v/o/gate/up/down) | float16 | I2_S (2-bit packed + float32 scale) |
|
||||
| Embedding (`token_embd.weight`) | float16 | float16 |
|
||||
| Per-projection norms (`*_norm_in`) | N/A (not present) | float16 |
|
||||
| Layer norms (`attn_norm`, `ffn_norm`) | float32 | float16 |
|
||||
| QK head norms (`attn_q_norm`, `attn_k_norm`) | float32 | float32 |
|
||||
| `output.weight` (lm_head) | present | skipped |
|
||||
|
||||
---
|
||||
|
||||
## 6. Build and Run
|
||||
|
||||
```bash
|
||||
# Build with BitNet repo (includes I2_S support)
|
||||
cmake -S /path/to/BitNet -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --target llama-embedding llama-bench -j$(nproc)
|
||||
|
||||
# Run embedding inference
|
||||
build/bin/llama-embedding -m bitnet-embeddings-0.6b-f16-i2_s.gguf \
|
||||
-p "hello world" --embd-normalize 2 --embd-output-format array
|
||||
|
||||
# Benchmark: F16 vs I2_S
|
||||
build/bin/llama-bench -m embeddings-0.6b-f16.gguf \
|
||||
-t 8 -p 128,256,512,1024,2048 -n 32,64 -r 3 -ngl 0
|
||||
|
||||
build/bin/llama-bench -m bitnet-embeddings-0.6b-f16-i2_s.gguf \
|
||||
-t 8 -p 128,256,512,1024,2048 -n 32,64 -r 3 -ngl 0
|
||||
```
|
||||
@@ -23,11 +23,17 @@ import gguf
|
||||
|
||||
logger = logging.getLogger("convert-bitnet-embedding")
|
||||
|
||||
# Supported architectures: model_type -> gguf arch name
|
||||
SUPPORTED_ARCHS = {
|
||||
"qwen3": "qwen3",
|
||||
"gemma3_text": "gemma3",
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tensor name mapping: HuggingFace -> GGUF
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def build_tensor_name_map(n_layers: int) -> dict[str, str]:
|
||||
def build_tensor_name_map(n_layers: int, arch: str) -> dict[str, str]:
|
||||
"""Build HF tensor name -> GGUF tensor name mapping."""
|
||||
mapping: dict[str, str] = {
|
||||
"embed_tokens.weight": "token_embd.weight",
|
||||
@@ -41,7 +47,6 @@ def build_tensor_name_map(n_layers: int) -> dict[str, str]:
|
||||
mapping.update({
|
||||
# Layer norms
|
||||
f"{pfx}.input_layernorm.weight": f"{blk}.attn_norm.weight",
|
||||
f"{pfx}.post_attention_layernorm.weight": f"{blk}.ffn_norm.weight",
|
||||
|
||||
# Self-attention projections
|
||||
f"{pfx}.self_attn.q_proj.weight": f"{blk}.attn_q.weight",
|
||||
@@ -49,7 +54,7 @@ def build_tensor_name_map(n_layers: int) -> dict[str, str]:
|
||||
f"{pfx}.self_attn.v_proj.weight": f"{blk}.attn_v.weight",
|
||||
f"{pfx}.self_attn.o_proj.weight": f"{blk}.attn_output.weight",
|
||||
|
||||
# QK head norms (standard Qwen3)
|
||||
# QK head norms
|
||||
f"{pfx}.self_attn.q_norm.weight": f"{blk}.attn_q_norm.weight",
|
||||
f"{pfx}.self_attn.k_norm.weight": f"{blk}.attn_k_norm.weight",
|
||||
|
||||
@@ -70,20 +75,29 @@ def build_tensor_name_map(n_layers: int) -> dict[str, str]:
|
||||
f"{pfx}.mlp.down_proj.norm.weight": f"{blk}.ffn_down_norm_in.weight",
|
||||
})
|
||||
|
||||
if arch == "qwen3":
|
||||
mapping[f"{pfx}.post_attention_layernorm.weight"] = f"{blk}.ffn_norm.weight"
|
||||
elif arch == "gemma3_text":
|
||||
mapping.update({
|
||||
f"{pfx}.post_attention_layernorm.weight": f"{blk}.post_attention_norm.weight",
|
||||
f"{pfx}.pre_feedforward_layernorm.weight": f"{blk}.ffn_norm.weight",
|
||||
f"{pfx}.post_feedforward_layernorm.weight": f"{blk}.post_ffw_norm.weight",
|
||||
})
|
||||
|
||||
return mapping
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tokenizer handling (GPT-2 / BPE for Qwen3)
|
||||
# Tokenizer handling
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def get_vocab_base_pre(tokenizer) -> str:
|
||||
def get_vocab_base_pre(tokenizer, arch: str) -> str:
|
||||
# encoding this string and hashing the resulting tokens would (hopefully) give us a unique identifier that
|
||||
# is specific for the BPE pre-tokenizer used by the model
|
||||
# we will use this unique identifier to write a "tokenizer.ggml.pre" entry in the GGUF file which we can
|
||||
# use in llama.cpp to implement the same pre-tokenizer
|
||||
|
||||
chktxt = '\n \n\n \n\n\n \t \t\t \t\n \n \n \n \n\U0001f680 (normal) \U0001f636\U0001f32b️ (multiple emojis concatenated) ✅ \U0001f999\U0001f999 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច\U0001f601 ?我想在apple工作1314151天～ ------======= нещо на Български \'\'\'\'\'\'```````""""......!!!!!!?????? I\'ve been \'told he\'s there, \'RE you sure? \'M not sure I\'ll make it, \'D you like some tea? We\'Ve a\'lL'
|
||||
chktxt = '\n \n\n \n\n\n \t \t\t \t\n \n \n \n \n\U0001f680 (normal) \U0001f636\U0001f32b️ (multiple emojis concatenated) ✅ \U0001f999\U0001f999 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច\U0001f601 ?我想在apple工作1314151天～ ------======= нещо на Български \'\'\'\'\'\'```````""""""......!!!!!!?????? I\'ve been \'told he\'s there, \'RE you sure? \'M not sure I\'ll make it, \'D you like some tea? We\'Ve a\'lL'
|
||||
|
||||
chktok = tokenizer.encode(chktxt)
|
||||
chkhsh = sha256(str(chktok).encode()).hexdigest()
|
||||
@@ -93,27 +107,38 @@ def get_vocab_base_pre(tokenizer) -> str:
|
||||
|
||||
res = None
|
||||
|
||||
# NOTE: if you get an error here, you need to update the convert_hf_to_gguf_update.py script
|
||||
# or pull the latest version of the model from Huggingface
|
||||
# don't edit the hashes manually!
|
||||
if chkhsh == "0ef9807a4087ebef797fc749390439009c3b9eda9ad1a097abbe738f486c01e5":
|
||||
# ref: https://huggingface.co/meta-llama/Meta-Llama-3-8B
|
||||
res = "llama-bpe"
|
||||
if chkhsh == "049ecf7629871e3041641907f3de7c733e4dbfdc736f57d882ba0b0845599754":
|
||||
# ref: https://huggingface.co/deepseek-ai/deepseek-llm-7b-base
|
||||
res = "deepseek-llm"
|
||||
if chkhsh == "347715f544604f9118bb75ed199f68779f423cabb20db6de6f31b908d04d7821":
|
||||
# ref: https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base
|
||||
res = "deepseek-coder"
|
||||
if chkhsh == "8aeee3860c56296a157a1fe2fad249ec40aa59b1bb5709f4ade11c4e6fe652ed":
|
||||
# ref: https://huggingface.co/tiiuae/falcon-7b
|
||||
res = "falcon"
|
||||
if chkhsh == "3ce83efda5659b07b1ad37ca97ca5797ea4285d9b9ab0dc679e4a720c9da7454":
|
||||
# ref: https://huggingface.co/openai-community/gpt2
|
||||
res = "gpt-2"
|
||||
if chkhsh == "d4540891389ea895b53b399da6ac824becc30f2fba0e9ddbb98f92e55ca0e97c":
|
||||
# ref: https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
|
||||
res = "qwen2"
|
||||
if arch == "qwen3":
|
||||
# NOTE: if you get an error here, you need to update the convert_hf_to_gguf_update.py script
|
||||
# or pull the latest version of the model from Huggingface
|
||||
# don't edit the hashes manually!
|
||||
if chkhsh == "0ef9807a4087ebef797fc749390439009c3b9eda9ad1a097abbe738f486c01e5":
|
||||
# ref: https://huggingface.co/meta-llama/Meta-Llama-3-8B
|
||||
res = "llama-bpe"
|
||||
if chkhsh == "049ecf7629871e3041641907f3de7c733e4dbfdc736f57d882ba0b0845599754":
|
||||
# ref: https://huggingface.co/deepseek-ai/deepseek-llm-7b-base
|
||||
res = "deepseek-llm"
|
||||
if chkhsh == "347715f544604f9118bb75ed199f68779f423cabb20db6de6f31b908d04d7821":
|
||||
# ref: https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base
|
||||
res = "deepseek-coder"
|
||||
if chkhsh == "8aeee3860c56296a157a1fe2fad249ec40aa59b1bb5709f4ade11c4e6fe652ed":
|
||||
# ref: https://huggingface.co/tiiuae/falcon-7b
|
||||
res = "falcon"
|
||||
if chkhsh == "3ce83efda5659b07b1ad37ca97ca5797ea4285d9b9ab0dc679e4a720c9da7454":
|
||||
# ref: https://huggingface.co/openai-community/gpt2
|
||||
res = "gpt-2"
|
||||
if chkhsh == "d4540891389ea895b53b399da6ac824becc30f2fba0e9ddbb98f92e55ca0e97c":
|
||||
# ref: https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
|
||||
res = "qwen2"
|
||||
if chkhsh == "855d9fb74bb0b28ce2305e9cd037ff6d8c798f18d19381ddfc14bea3dc9c002f":
|
||||
# ref: multilingual-e5-0.6b-260311 (Qwen3 tokenizer variant)
|
||||
res = "qwen2"
|
||||
elif arch == "gemma3_text":
|
||||
if chkhsh == "fcb6bf9f20f6c40fa4aa4f7f99607bd6c106ca2348efdacacdca8152e59dcfe9":
|
||||
# ref: multilingual-e5-270m-260311 (Gemma3 tokenizer)
|
||||
res = "default"
|
||||
if chkhsh == "a8594e3edff7c29c003940395316294b2c623571571fc8d3d2d6571f5571cbe6":
|
||||
# ref: google/gemma-2-9b
|
||||
res = "default"
|
||||
|
||||
if res is None:
|
||||
logger.warning("\n")
|
||||
@@ -146,13 +171,29 @@ def _does_token_look_special(token: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def set_vocab(gguf_writer: gguf.GGUFWriter, dir_model: Path, hparams: dict):
|
||||
"""Set GPT-2 BPE vocab for Qwen3."""
|
||||
def set_vocab(gguf_writer: gguf.GGUFWriter, dir_model: Path, hparams: dict, arch: str):
|
||||
"""Set tokenizer vocab.
|
||||
|
||||
- Qwen3: BPE tokenizer (tokenizer.ggml.model = "gpt2")
|
||||
- Gemma3: SPM-compatible tokenizer from tokenizer.json (tokenizer.ggml.model = "llama")
|
||||
Gemma uses SentencePiece-style tokenization with ▁ space prefix and byte fallback.
|
||||
Using "llama" model type ensures llama.cpp uses the correct SPM pre-tokenizer
|
||||
instead of the BPE regex-based pre-tokenizer which breaks CJK tokenization.
|
||||
"""
|
||||
from transformers import AutoTokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained(dir_model)
|
||||
vocab_size = hparams.get("vocab_size", len(tokenizer.vocab))
|
||||
|
||||
tokpre = get_vocab_base_pre(tokenizer)
|
||||
if arch == "gemma3_text":
|
||||
_set_vocab_gemma3(gguf_writer, dir_model, tokenizer, vocab_size)
|
||||
else:
|
||||
_set_vocab_bpe(gguf_writer, dir_model, tokenizer, vocab_size, arch)
|
||||
|
||||
|
||||
def _set_vocab_bpe(gguf_writer: gguf.GGUFWriter, dir_model: Path,
|
||||
tokenizer, vocab_size: int, arch: str):
|
||||
"""Set BPE vocab (for Qwen3)."""
|
||||
tokpre = get_vocab_base_pre(tokenizer, arch)
|
||||
|
||||
tokens: list[str] = []
|
||||
toktypes: list[int] = []
|
||||
@@ -176,7 +217,6 @@ def set_vocab(gguf_writer: gguf.GGUFWriter, dir_model: Path, hparams: dict):
|
||||
if added_tokens_decoder[i].special or _does_token_look_special(token):
|
||||
toktypes.append(gguf.TokenType.CONTROL)
|
||||
else:
|
||||
# Pre-normalize user-defined spaces (for Gemma-style tokenizers)
|
||||
token = token.replace(b"\xe2\x96\x81".decode("utf-8"), " ")
|
||||
toktypes.append(gguf.TokenType.USER_DEFINED)
|
||||
|
||||
@@ -191,14 +231,105 @@ def set_vocab(gguf_writer: gguf.GGUFWriter, dir_model: Path, hparams: dict):
|
||||
gguf_writer.add_token_types(toktypes)
|
||||
|
||||
special_vocab = gguf.SpecialVocab(dir_model, load_merges=True)
|
||||
# Override EOS token: PyTorch tokenizer appends <|endoftext|> (151643) as the
|
||||
# sentence-end marker, not <|im_end|> (151645). For last-token pooling to work
|
||||
# correctly, llama.cpp must append the same token.
|
||||
special_vocab.special_token_ids["eos"] = 151643
|
||||
|
||||
if arch == "qwen3":
|
||||
# Override EOS token: PyTorch tokenizer appends <|endoftext|> (151643) as the
|
||||
# sentence-end marker, not <|im_end|> (151645). For last-token pooling to work
|
||||
# correctly, llama.cpp must append the same token.
|
||||
special_vocab.special_token_ids["eos"] = 151643
|
||||
|
||||
special_vocab.add_to_gguf(gguf_writer)
|
||||
|
||||
# Embedding models need EOS token appended for last-token pooling
|
||||
gguf_writer.add_add_eos_token(True)
|
||||
if arch == "qwen3":
|
||||
# Embedding models need EOS token appended for last-token pooling
|
||||
gguf_writer.add_add_eos_token(True)
|
||||
|
||||
|
||||
def _set_vocab_gemma3(gguf_writer: gguf.GGUFWriter, dir_model: Path,
|
||||
tokenizer, vocab_size: int):
|
||||
"""Set SPM-compatible vocab for Gemma3.
|
||||
|
||||
Gemma's tokenizer is SentencePiece-based (BPE variant with ▁ space prefix
|
||||
and byte fallback). We read tokenizer.json to extract vocab and compute
|
||||
BPE merge scores, then write as tokenizer.ggml.model = "llama" so llama.cpp
|
||||
uses the SPM code path (correct pre-tokenizer behavior for CJK etc.).
|
||||
|
||||
Score assignment:
|
||||
- BPE merge results get scores derived from merge rank (lower rank = higher score)
|
||||
- Single-char / byte tokens get score 0
|
||||
- Special / added tokens get score -1000
|
||||
"""
|
||||
tokenizer_json_file = dir_model / "tokenizer.json"
|
||||
if not tokenizer_json_file.exists():
|
||||
raise FileNotFoundError(f"tokenizer.json not found in {dir_model}")
|
||||
|
||||
with open(tokenizer_json_file, encoding="utf-8") as f:
|
||||
tokenizer_json = json.load(f)
|
||||
|
||||
bpe_vocab = tokenizer_json["model"]["vocab"] # token_str -> token_id
|
||||
bpe_merges = tokenizer_json["model"].get("merges", [])
|
||||
|
||||
# Build merge result -> rank mapping for score computation
|
||||
# merge_scores[result_token] = -rank (lower rank = earlier merge = higher priority)
|
||||
merge_scores: dict[str, float] = {}
|
||||
for rank, merge in enumerate(bpe_merges):
|
||||
if isinstance(merge, list):
|
||||
result = "".join(merge)
|
||||
else:
|
||||
parts = merge.split(" ", 1)
|
||||
result = "".join(parts)
|
||||
if result not in merge_scores:
|
||||
merge_scores[result] = -float(rank)
|
||||
|
||||
# Build token arrays
|
||||
reverse_vocab = {v: k for k, v in bpe_vocab.items()}
|
||||
added_tokens_decoder = tokenizer.added_tokens_decoder
|
||||
|
||||
tokens: list[bytes] = []
|
||||
scores: list[float] = []
|
||||
toktypes: list[int] = []
|
||||
|
||||
for i in range(vocab_size):
|
||||
if i not in reverse_vocab:
|
||||
tokens.append(f"[PAD{i}]".encode("utf-8"))
|
||||
scores.append(-10000.0)
|
||||
toktypes.append(gguf.TokenType.UNUSED)
|
||||
continue
|
||||
|
||||
token_str = reverse_vocab[i]
|
||||
token_bytes = token_str.encode("utf-8")
|
||||
|
||||
# Determine token type
|
||||
if i in added_tokens_decoder:
|
||||
tok_data = added_tokens_decoder[i]
|
||||
if tok_data.special or _does_token_look_special(token_str):
|
||||
toktypes.append(gguf.TokenType.CONTROL)
|
||||
else:
|
||||
toktypes.append(gguf.TokenType.USER_DEFINED)
|
||||
scores.append(-1000.0)
|
||||
elif token_str.startswith("<0x") and token_str.endswith(">") and len(token_str) == 6:
|
||||
# Byte token: <0xHH>
|
||||
toktypes.append(gguf.TokenType.BYTE)
|
||||
scores.append(0.0)
|
||||
elif token_str == "<unk>":
|
||||
toktypes.append(gguf.TokenType.UNKNOWN)
|
||||
scores.append(0.0)
|
||||
else:
|
||||
toktypes.append(gguf.TokenType.NORMAL)
|
||||
# Score from merge rank, or 0 for single-char tokens
|
||||
scores.append(merge_scores.get(token_str, 0.0))
|
||||
|
||||
tokens.append(token_bytes)
|
||||
|
||||
gguf_writer.add_tokenizer_model("llama")
|
||||
gguf_writer.add_tokenizer_pre("default")
|
||||
gguf_writer.add_token_list(tokens)
|
||||
gguf_writer.add_token_scores(scores)
|
||||
gguf_writer.add_token_types(toktypes)
|
||||
gguf_writer.add_add_space_prefix(False)
|
||||
|
||||
special_vocab = gguf.SpecialVocab(dir_model, load_merges=False)
|
||||
special_vocab.add_to_gguf(gguf_writer)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -260,7 +391,7 @@ def set_gguf_parameters(gguf_writer: gguf.GGUFWriter, hparams: dict, dir_model:
|
||||
pooling_type = gguf.PoolingType.MEAN
|
||||
gguf_writer.add_pooling_type(pooling_type)
|
||||
|
||||
logger.info(f" n_layers={n_layers}, n_embd={n_embd}, n_head={n_head}, n_head_kv={n_head_kv}, n_ff={n_ff}")
|
||||
logger.info(f" n_layers={n_layers}, n_embd={n_embd}, n_head={n_head}, n_head_kv={n_head_kv}, n_ff={n_ff}, head_dim={head_dim}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -366,7 +497,7 @@ def quantize_to_i2_s(w: np.ndarray) -> np.ndarray:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Convert bitnet-embeddings to GGUF")
|
||||
parser = argparse.ArgumentParser(description="Convert bitnet-embeddings (Qwen3/Gemma3) to GGUF")
|
||||
parser.add_argument("model", type=Path, help="Model directory")
|
||||
parser.add_argument("--outfile", type=Path, default=None, help="Output GGUF file")
|
||||
parser.add_argument("--outtype", choices=["f32", "f16", "i2_s"], default="f16",
|
||||
@@ -390,9 +521,12 @@ def main():
|
||||
with open(dir_model / "config.json") as f:
|
||||
hparams = json.load(f)
|
||||
|
||||
arch = hparams.get("model_type", "qwen3")
|
||||
assert arch == "qwen3", f"Expected qwen3 architecture, got {arch}"
|
||||
arch = hparams.get("model_type", "")
|
||||
if arch not in SUPPORTED_ARCHS:
|
||||
logger.error(f"Unsupported model_type '{arch}'. Supported: {list(SUPPORTED_ARCHS.keys())}")
|
||||
sys.exit(1)
|
||||
|
||||
gguf_arch = SUPPORTED_ARCHS[arch]
|
||||
n_layers = hparams["num_hidden_layers"]
|
||||
|
||||
# Determine ftype
|
||||
@@ -403,20 +537,20 @@ def main():
|
||||
else: # i2_s
|
||||
ftype = 40 # LLAMA_FTYPE_MOSTLY_I2_S
|
||||
|
||||
logger.info(f"Converting {dir_model.name} to GGUF ({args.outtype})")
|
||||
logger.info(f"Converting {dir_model.name} (arch={arch}) to GGUF ({args.outtype})")
|
||||
|
||||
# Create GGUF writer
|
||||
gguf_writer = gguf.GGUFWriter(str(args.outfile), "qwen3")
|
||||
gguf_writer = gguf.GGUFWriter(str(args.outfile), gguf_arch)
|
||||
|
||||
# Set parameters
|
||||
set_gguf_parameters(gguf_writer, hparams, dir_model, ftype)
|
||||
|
||||
# Set vocab
|
||||
logger.info("Setting tokenizer/vocab...")
|
||||
set_vocab(gguf_writer, dir_model, hparams)
|
||||
set_vocab(gguf_writer, dir_model, hparams, arch)
|
||||
|
||||
# Build tensor name map
|
||||
tensor_map = build_tensor_name_map(n_layers)
|
||||
tensor_map = build_tensor_name_map(n_layers, arch)
|
||||
|
||||
# Process tensors
|
||||
logger.info("Processing tensors...")
|
||||
@@ -473,6 +607,16 @@ def main():
|
||||
|
||||
else:
|
||||
# norms, 1D tensors
|
||||
# Gemma3 RMSNorm uses (1+w)*x instead of w*x; preprocess w -> w+1
|
||||
# so llama.cpp's standard RMSNorm produces correct results.
|
||||
# NOTE: *_norm_in weights are BitLinear standard RMSNorm (initialized ~1.0),
|
||||
# NOT Gemma3RMSNorm (initialized ~0.0), so they must NOT get +1.
|
||||
is_gemma3_native_norm = (arch == "gemma3_text" and is_norm
|
||||
and not gguf_name.endswith("_norm_in.weight"))
|
||||
if is_gemma3_native_norm:
|
||||
data = data.astype(np.float32) + 1.0
|
||||
logger.info(f" [Gemma3 norm offset] {gguf_name}: applied w = w + 1")
|
||||
|
||||
if args.outtype in ("f16", "i2_s"):
|
||||
data = data.astype(np.float16)
|
||||
logger.info(f" {gguf_name}: {list(data_torch.shape)} {old_dtype} -> float16")
|
||||
|
||||
Reference in New Issue
Block a user