From Fine-Tune to NVFP4 on a DGX Spark

I put my first quantized model on Hugging Face: simonlehmann/clef-NVFP4, Cloudflare’s 27B decision model, 55 GB down to 23. The quantizing itself wasn’t new. Since July every fine-tune of Emil has gone through the same pipeline and ended up in NVFP4 on the DGX Spark next to my desk. Clef is just the first build that left the box.

Why NVFP4

The Spark has 273 GB/s of memory bandwidth. Generating one token means reading every weight once, so for one user at a time the speed limit is bandwidth divided by model size. A 9B model in BF16 is 18 GB, which caps it at 15 tok/s, and it measured 12.9. The same model in NVFP4 measured about 35.

Every 4-bit format stores small values plus a scale. NVFP4 shares one FP8 scale per 16 weights. MXFP4 shares a power-of-two scale per 32 weights, so it fits each block more loosely. Blackwell supports FP4 in hardware.

Most of Emil’s builds are weight-only (NVFP4A16): weights in 4 bits, activations in BF16. When the bottleneck is reading weights, that is where the gain is, and it needs no calibration data.

What stays BF16

Not everything gets quantized. The vision encoder, lm_head, MoE routers and the multi-token-prediction draft head stay BF16, and for Nemotron I follow NVIDIA’s choice of attention layers plus the Mamba layer feeding each one. It matters: a uniform NVFP4 build of Qwen3.6 ran at 44.1 tok/s, a mixed one at 71.6, and 110.9 with its draft head doing speculative decoding.

The pipeline

1. Fine-tune a LoRA adapter on Emil’s own play, on the BF16 base.
2. Merge the adapter into the BF16 weights, so the quantizer sees the weights that get served and no adapter runs per token. The merge counts tensors against the base, because transformers drops a module it fails to build with only a warning.
3. Quantize with llm-compressor, skipping the keep-list. When activations are quantized too, calibration uses Emil’s training data rendered through the same chat template and tool definitions the server sends. A row the template rejects fails the run. The model loads to CPU and moves to the GPU one layer at a time, because loading it straight to the GPU briefly holds two copies in the shared memory.
4. Verify from the saved safetensors headers that every keep-listed tensor is in the output.
5. Serve in vLLM.
6. Bench against the previous checkpoint. It only goes live if the score moved.

A lineage.json travels with every build, so a bench number traces back to its dataset, training run and adapter.

Clef

Clef is Cloudflare’s 27B decision model. It takes a state and a set of questions and returns a probability for every allowed answer in one forward pass. I want it as a fast decision layer next to Emil’s main model, which needed it smaller.

It’s built on Qwen3.8-27B, but existing Qwen quants don’t fit: the upper layers were retrained, and Clef reads its answer options from the rows of lm_head. I used the community Qwen3.8-27B recipe and changed one thing, keeping lm_head in BF16:

part	format
MLP, layers 0–55	NVFP4, weights and activations
MLP, layers 56–63, attention	FP8
lm_head, vision encoder, norms, embeddings, answer head	BF16

Calibration used 374 Clef-format records, none of them reused for eval. The model’s output is the probabilities, so I compared it against the BF16 release question by question on vLLM’s real FP4 kernels:

eval set	questions	same top answer as BF16	accuracy BF16 → NVFP4
BANKING77, 77 options	400	98.5%	94.25% → 94.50%
Flickr30k photos	256	96.9%	no labels
game-agent states	150	96.7%	68.7% → 67.3%*

* synthetic labels that BF16 itself only matches 69% of the time.

On the Spark one decision takes 177 ms for a short text state, 257 ms with an image and 705 ms for all 77 banking options. The recipe and the full eval are on the model card.