Fitting in VRAM Is the Easy Part

I spent a day trying to replace the model on my two-RTX-3090 inference lane. Four candidates got a serious look. None earned the job.

That sounds like a failed project. It wasn’t. The current model kept its place because every alternative crossed the same predeclared gates and failed for a specific reason. There was no “this one feels faster” promotion at the end of the night.

The current baseline is a 27B Qwen-family model using AutoRound W4A16, tensor parallelism across both cards, prefix caching, FP8 KV cache, and speculative decoding. It is not the fastest option in every isolated benchmark. It is still the best complete system I tested.

The candidate table

Each candidate was restored out of production before the next one started. Direct and routed tool calls had to pass after every rollback.

W4A8 made prefill much faster

The W4A8 experiment was the closest thing to a tempting result.

AutoRound checkpoints did not engage Marlin INT8 activations correctly with the environment toggle alone. Two targeted fixes were needed: one to propagate the requested input dtype through vLLM’s INC path, and another to handle negative group scales without corrupting output. These are tracked in vLLM #48904 and vLLM #48905.

Once both fixes were active, the gains were real:

The model stayed coherent and passed all 30 structural and native-tool checks. If I had measured only time to first token, I would have promoted it.

Decode told the other half of the story. Mean decode throughput landed at 0.892x the W4A16 control, just beyond the registered ten-percent regression limit. Speculative-token acceptance dropped from 65.55% to 61.57%. The lane handles long agent sessions, so trading decode and speculation for prefill was not an automatic win.

The result was useful, but it was still a rejection.

Tess worked, then made every tool loop expensive

The tested Tess recipe was LeaderboardModel1/Tess-4-27B-AutoRound-W4A16-Tuning with MTP n=5 and prefix caching disabled as shipped.

Basic operation was fine. The model loaded, called tools, and reached correct final states. An external pairwise judge also liked its general answers. The operational comparison was much less flattering:

Tess also lost the registered safety/minimality comparison. The exact recipe was technically valid, but I could not justify roughly half-speed narrative decode and a repeated-turn wall time more than five times worse than the current model.

This matters because single-response benchmarks hide re-prefill costs. Agent workloads do not. A model that looks acceptable for one answer can feel broken by the fifth tool turn.

Nemotron fit, but the first tool call ended the test

The 75B-A9B Nemotron Puzzle W4A16 checkpoint did what its author hoped in one important respect: it fit on two real RTX 3090s without CPU offload. It used about 21.32 GiB per card and left roughly 247 MiB per card in the tested 49K-context profile.

Startup was clean. A plain exact-text request passed. The next request asked for exactly one weather tool call.

The response contained eleven calls. Ten were identical duplicates. The last had the truncated function name get_ and empty arguments. Assistant content was empty.

That is not a minor benchmark loss. An agent could execute the same side effect ten times and then trip over a malformed call. The canary stopped immediately, as it was designed to do. The exact result has been shared with the model community as a narrow reproducer, not a claim that every runtime configuration fails.

The safety prompt made the agent safe by making it useless

A prior AWQ candidate had better aggregate quality than the production AutoRound model, but it also showed repeatable safety and edit-discipline regressions. In three-for-three reproductions it emitted a blanket chmod -R 777, broadened permissions when an exact repair was available, and rewrote a file that only needed a patch.

I tested one isolated system-policy mitigation against a focused safety suite and the existing quality packs. It ran three times and produced the same scores each time:

• Bug finding: 13/15
• Agent behavior: 0/20
• CLI work: 24/40
• Total: 37/75

Across 60 agent scenarios, the model emitted zero native tool events. The policy reduced some bad behavior by suppressing ordinary tool use.

That is a useful failure. Broad prompt rules are a poor substitute for narrow enforcement. If I revisit this path, destructive-command approval and exact permission checks belong at the tool boundary, where they can block a known hazard without rewriting the model’s personality.

The harness mattered as much as the models

The model results were not the only output from the day.

The benchmark sandbox could not reach a proxy bound to host loopback, so the proxy moved into a disposable container on the model network. The inference lane also never became idle by itself; new requests arrived every few seconds. The runner now drains new admissions, waits for in-flight work to finish, and requires two consecutive idle polls before mutation.

One early rollback trap recreated the healthy production model after a preflight timeout, even though no mutation had happened. The trap now arms only after the first live change. That small state bit prevents a safety mechanism from causing its own outage.

Rollback verification also got stricter. A running container is not enough. The restored lane must pass direct readiness, routed readiness, a direct native-tool call, a routed native-tool call, and checks for OOMs and unexpected restarts.

What I kept

The W4A8 patches are real and worth upstreaming. The Nemotron fit data and tool-call failure are worth reporting. Tess’s repeated-turn result is a useful warning for anyone testing an APC-off agent recipe.

The production model did not change.

That is the boring answer, and I trust it more than a benchmark winner selected after the fact. A replacement needs to beat the whole lane: startup, memory, prefill, decode, speculative acceptance, tool integrity, safety, repeated turns, and rollback. Fitting in VRAM only gets it into the room.