Dustin Morgan

What I Got Wrong About SQ4

· 2 min read

In May I published SQ4, a 4-bit weight format with per-block percentile bands, along with a whitepaper and a llama.cpp PR. I called its quality “indistinguishable from FP16.”

I hadn’t measured that. I’d eyeballed model outputs and liked what I saw. When I finally ran proper comparisons, SQ4 didn’t hold up, so I’ve retired it.

What the measurements showed

In simulated post-training quantization against the usual 4-bit formats, with everything else matched, SQ4 came out 0.3–0.7 nats of KL divergence worse than the whole field. Not some of them. All of them.

So “put the codes where the weights are” sounds right, and it’s wrong. Plain uniform codes with a per-block scale and offset are cheaper to decode and more accurate at the same size.

What I use now

Boring formats: 8-bit with 64-weight blocks, and 5-bit with 32-weight blocks for the bigger models. Nothing novel about them, and that’s fine. They’re well understood and they measure better.

SQ4 did have one interesting trick: training stayed stable at very high learning rates. That turned out less useful than I hoped, but the idea lives on in later work.

What I’m changing

  • The original post now starts with a correction that links here, and the Projects page lists SQ4 as retired.
  • No more quality claims without a matched baseline. I set that rule for benchmarks back in I owe PyTorch a fair fight, and then didn’t follow it for SQ4.

Update, October 2026

Two later measurements made it worse, not better:

  • It’s worse at equal size. In a test of compressing activations between machines, SQ4 had 2.6–3.5× the error of a plain uniform 4-bit code that used the same number of bytes.
  • My one good number was a cherry-pick. On GPT-2, whole-model KL against FP32 was 0.238 for SQ4 and 0.638 for MXFP4, and I’d have happily led with that. On the layer weights alone, MXFP4 was slightly better (0.158 vs 0.179). The whole-model number was mostly about how each format handled the embedding table.

I also closed the llama.cpp PR with a note explaining why. Nobody should merge a format its own author has retired.