- Add docs/bitnet-embeddings-i2s-guide.md with model overview, I2_S GGUF
conversion details, accuracy verification, inference performance benchmarks,
and quick start examples for both 0.6B (Qwen3) and 270M (Gemma3) models
- Add embedding quantization chart (fig1_quant_per_task.png)
- Remove old docs/bitnet-embeddings-gguf-conversion.md (replaced by new guide)
- Add bitnet-embedding-0.6b and bitnet-embedding-270m to supported HF models
in setup_env.py
- Update README.md What's New section with link to the new guide
- Add quantize_to_i2_s() for direct ternary-to-I2_S packing in conversion script
- Support offline-quantized models (uint8 packed weights + weight_scale)
- Fix weight_quant double-quantization bug for offline-quantized models
- Fix I2_S scale computation to use first nonzero absolute value
- Add I2_S ftype mapping and BitNetForCausalLM registration
- Refactor ggml-bitnet-lut T-MAC wrapper with proper mul_mat implementation
- Update llama.cpp submodule with I2_S ftype and 2B model type support
Mitigate unsafe deserialization vulnerability (CWE-502) in the GPU
inference pipeline. torch.load without weights_only=True allows
arbitrary code execution via malicious pickle payloads in checkpoint
files.
Affected locations:
- gpu/convert_checkpoint.py:37 (checkpoint conversion utility)
- gpu/generate.py:67,69 (fp16 and int2 checkpoint loading)
The utils/ scripts already applied this parameter correctly; this
commit brings the GPU pipeline to the same safety standard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GCC does not recognize Clang-specific warning flags like
-Wunreachable-code-break and -Wunreachable-code-return, which are passed
by upstream submodules (e.g., ggml). This patch forces CMake to use Clang
via command-line arguments, avoiding the need to patch nested submodules.
This resolves compiler errors without modifying submodule source code.
* Update CMakeLists.txt
I added a CMake option to compile the Llama.cpp server. This update allows us to easily build and deploy the server using BitNet
* Create run_inference_server.py
same as run_inference, but for use with llama.cpp's built in server, for some extra comfort
In particular:
- The build directory is determined based on whether the system is running on Windows or not.
- A list of arguments (`--model`, `-m` etc.) is created.
- The main argument list is parsed and passed to the `subprocess.run()` method to execute the system command.
Xin Huang