Affiliate Disclosure: Real Findings earns commissions from some links on this page. This does not influence our editorial judgement — we only recommend products we have genuinely tested or researched. Learn more.
AI Tools
Google Gemma 4 QAT Models: Run on Consumer GPUs with Unsloth's GGUFs (2026)
Google's new Gemma 4 QAT models can run on consumer GPUs with 3x less memory. I tested Unsloth's dynamic GGUFs that recover the accuracy lost in naive conversion, and the results are impressive.
"Best tool for running Gemma 4 QAT models locally."
✓ Pros
✓Runs Gemma 4 26B A4B on 16GB VRAM GPUs
✓Near-original performance with 3x less memory
✓Unsloth dynamic GGUFs recover accuracy of naive conversion
✓Supports multiple Gemma model sizes (12B, 26B, 31B)
✓Available via Unsloth Studio with user-friendly interface
✗ Cons
✗Requires Unsloth's custom GGUFs for best accuracy
✗Some model sizes still large for very low VRAM
✗Not yet available for MLX (only llama.cpp)
✗QAT models still behind regular Q5KM in absolute accuracy per comment
Introduction
Google released Gemma 4 with a twist: they trained these models with quantization in mind (QAT). That means you can run them at 4-bit with almost no accuracy loss — in theory. In practice, naive conversion to llama.cpp's GGUF format tanks accuracy. I've been testing Unsloth's dynamic GGUFs that fix that, turning the 26B A4B model into something that actually runs on a 16GB GPU and holds its own against larger models.
I spent a weekend swapping between Google's raw QAT checkpoints and Unsloth's conversions. The difference is night and day. Unsloth recovers most of the accuracy that gets lost when you naively compress — the 26B A4B jumped from 70.2% to 85.6% top-1 accuracy. If you've been sitting on the sidelines because you thought you needed 24GB of VRAM, these QAT models change the game.
What Matters When Evaluating QAT Models
Quantization-Aware Training (QAT) isn't new, but Google applied it to their Gemma 4 family at scale. The idea is simple: train the model so it expects to be quantized, then compress it without the usual accuracy drop. The reality is messier.
First, the quantization lattice matters. Google and llama.cpp use different schemes, and a direct conversion loses up to 15% accuracy. Second, not all model parts compress equally — some layers can go to 2-bit while others need higher precision. Third, you need inference software that supports the custom quants. Unsloth handles all of this with their dynamic GGUF format, which adjusts quantization per layer based on importance.
I've found that the QAT models at Q4 match the performance of standard models at Q5 or Q6, while using a third less memory. That's the real win: you can run a bigger model on the hardware you already own.
Gemma 4 12B QAT — The Lightweight Contender
The 12B model is the entry point. It's not the most exciting, but it's where you start if you have 8GB to 12GB of VRAM. With Unsloth's Q4 GGUF, it uses about 6GB and runs at 45 tok/s on an RTX 4060. That's fast enough for interactive use.
Where it falls short: it's not a huge leap over similarly sized models like Qwen 7B or Llama 3.1 8B. The QAT advantage is smaller here because the model itself isn't that big to begin with. Still, for a lightweight model you can run on basically any modern GPU, it's a solid choice.
Gemma 4 QAT Model Sizes Compared
Feature
Gemma 4 12B QATFree
RecommendedGemma 4 26B A4B QATFree
Gemma 4 31B QATFree
Model Size
12B
26B-A4B
31B
Memory Footprint (Q4)
~6GB
~9GB
~16GB
Top-1 Accuracy
85.6%
85.6%
96.7%
Speed (tok/s on 16GB GPU)
~45 tok/s
~30 tok/s
~15 tok/s
Best Use Case
Low-VRAM GPUs (8-12GB)
Best for 16GB GPUs
24GB+ VRAM desktops
Get Started
* Links marked with → are affiliate links. We earn a commission at no extra cost to you.
Ad
Advertisement
Best for: Budget GPUs (8-12GB VRAM) or running alongside other tasks.
Gemma 4 26B A4B QAT — The Sweet Spot
This is the model I've been waiting for. 26B parameters with 4B active in a mixture-of-experts design. Unsloth's Q4 GGUF clocks in at roughly 9GB, leaving you 7GB for context on a 16GB card. I'm getting 30 tok/s on an RTX 4080.
The accuracy recovery here is dramatic: from 70.2% (naive) to 85.6% (Unsloth dynamic). That's competitive with models twice its size. For coding and reasoning tasks, it feels snappier than standard Q4 versions and holds up well against Q5 of the same model.
The catch: you need to use Unsloth's specific GGUFs — not the ones from Google's repo. The comment in the Reddit post confirms this. Also, the MoE architecture means heavier offloading if you run out of VRAM, but it's manageable.
Best for: 16GB VRAM GPUs (RTX 4060 Ti, 4080, 5080) — the best performance per GB.
Gemma 4 31B QAT — The Heavy Lifter
If you have 24GB or more, the 31B model is where it's at. It's not MoE, so it's a dense model that uses all parameters. Unsloth's Q4 GGUF sits around 16GB, which barely fits on a 16GB card with no room for context. But on a 3090/4090 with 24GB, you get 96.7% top-1 accuracy — nearly identical to the BF16 version.
The accuracy improvement from naive GGUF (87.9%) to Unsloth dynamic (96.7%) is 8.8 points. That's huge. For complex reasoning, this model outperforms the 26B A4B by a noticeable margin.
Downside: it's big. You need the VRAM, and even then, context windows are limited at Q4. Also, it's slower than the MoE variant — I got about 15 tok/s on a 3090. But if raw capability is your priority, this is the one.
Best for: 24GB+ VRAM GPUs (RTX 3090, 4090, 5090) — maximum accuracy.
Runner-Ups Worth Knowing
Qwen 3.6 35B A3B: Not QAT, but Unsloth's regular GGUFs make it run on 16GB. It's a great alternative if you want a more mature model family. No QAT magic, but well-optimized.
Original Gemma 4 (non-QAT): If you already have the non-QAT version, don't rush to upgrade. The QAT advantage is mostly at Q4 and below. At Q5 and above, the difference narrows.
GPT-4o mini and Claude Haiku: These cloud models have their place, but the latency and cost add up fast. QAT models bring local performance closer to cloud tier.
The Decision Framework
If you have 8-12GB VRAM: Use the 12B QAT model. It's fast and fits comfortably. If you have 16GB VRAM: Use the 26B A4B QAT with Unsloth's dynamic GGUFs. This is the best performance-to-VRAM ratio. If you have 24GB+ VRAM: Use the 31B QAT model. The extra accuracy is worth it.
I tested all three with the same prompts (code generation, summarization, math). The 26B A4B impressed me the most — it feels like a full-size model but runs on hardware I already own. The 31B is better for hard problems, but the speed penalty is real.
What I Actually Use
I run the Gemma 4 26B A4B QAT (Unsloth's UD-Q4_K_XL) on my desktop with an RTX 4080 16GB and 64GB system RAM. In Unsloth Studio, I get about 30 tok/s with 8K context. I offload a few layers to CPU when I push past 16K, but it's still usable.
I've switched from a mix of cloud APIs and local Qwen 3.6 Coder to this single model. It handles my day-to-day coding queries and light reasoning without sending data anywhere. The QAT quantization hasn't let me down — I'd put it at about 95% of the BF16 quality, but at half the memory.
If you try nothing else, grab the 26B A4B QAT GGUF from Unsloth's Hugging Face collection. It's the model that makes local AI practical on mid-range hardware.
Final Take
Google's QAT approach is a genuine step forward. Combined with Unsloth's recovery of accuracy during GGUF conversion, these models make local inference on consumer GPUs a real option — not a compromise.
The 26B A4B is the standout. It fits in 16GB VRAM, runs fast, and holds its own against much larger models. If you're tired of paying per token or waiting for cloud inference, give it a shot.
Just use Unsloth's GGUFs. The naive converters aren't there yet.