The Transformers project has added support for loading and running GGUF checkpoints—the quantized format popularized by llama.cpp—inside the transformers Python API. The work reuses ggml kernels to reduce overhead and targets local inference on Apple Silicon, with Qwen3.5 as the initial architecture.
GGUF files and quantization options
GGUF packages model weights and metadata (tokenizer info and optional chat templates) in a single file and supports multiple quantization types. The team published comparative sizes for Unsloth’s Qwen3.5-4B variants: BF16 8.42 GB (unquantized), Q6_K 3.53 GB, Q5_K_M 3.14 GB, and Q4_K_M 2.74 GB. The developers suggest starting with Q4_K_M for local use and trying Q5_K_M or Q6_K if more memory is available. They emphasize evaluating quantization on the actual tasks users need.
How to run GGUF models in transformers and performance notes
Loading a GGUF file requires passing the Hub model_id and the filename as gguf_file to from_pretrained (the same call that produces AutoModelForCausalLM and AutoTokenizer objects). When compatible ggml/Metal kernels are available, transformers will load them automatically; otherwise the loader falls back to a dequantized path or to an “sdpa” attention implementation with a warning. The team also showed how the same checkpoint can be served with transformers serve, which exposes an OpenAI-compatible API and accepts model arguments in the form repository:filename.gguf.
To approach llama.cpp’s efficiency, transformers integrates several ggml-derived Metal kernels distributed via the kernels library: ggml-quantization, ggml-norm, ggml-attn, ggml-gated-delta-net, and a topk implementation for MoE routing. The project reports that, on a MacBook Pro M2 Max with 32 GB unified memory (macOS 26.6, PyTorch 2.12.1, kernels 0.17.0), generate throughput is close to llama.cpp across three GGUF checkpoints. The team notes that llama.cpp remains the recommended engine when efficiency is the primary goal.
Two changes to the generation loop also help performance: removing an unnecessary all-ones attention mask when inputs have no padding, and deferring the stopping check so the CPU can keep scheduling GPU work. These adjustments reduce synchronization and improve token throughput.
Current limitations include an MPS-only packed inference path for now, incomplete padding and batching optimizations, and initial architecture coverage limited to Qwen3.5 dense and MoE checkpoints (including compatible Qwen3.8). The team asks users with GGUF models to open an issue citing the checkpoint and their use case to help prioritize further support.
Original source: Hugging Face Blog