Every Improvement I Made to My 1.7B Agent Made It Worse

I fine-tuned a 1.7B model on a narrow tool-calling task and it beat its own base model on every metric I track. Then I set about making the training data better. Two of the three "better" datasets produced measurably worse agents, and one of them landed level with the untrained baseline — a full training cycle that bought nothing.

Final ranking, all four models served through the same inference path, scored by the same reward: v7 0.825 ≈ v8.2 0.803 > v8 0.792 >> v8.1 0.700 ≈ baseline 0.697.

This post is the postmortem. Two of the lessons are about measurement, not modelling, and each nearly cost me a wrong conclusion.

What Crab-1 Does

Crab-1 is a small tool-calling agent: Qwen3-1.7B, QLoRA fine-tuned. It gets a French company name and nothing else, and has to return that company's website, its department/city, and its sector. It has four tools:

• registry_lookup — company registry by name, returns legal identity and the head office department
• web_search — web results for a query
• web_extract — fetch and clean a page
• submit_answer — the terminal action, with the structured profile as arguments

The goal is not general capability. The goal is a specialist that beats much larger general models on this one task at near-zero inference cost, on a single 8 GB consumer GPU. Reward is computed per episode over the fields that have ground truth, with a dynamic denominator: if a row has no sector label, sector isn't scored.

The First Fine-Tune Won Cleanly

Generate teacher trajectories, keep the ones that score ≥ 0.7, convert to native tool-calling episodes, QLoRA train, evaluate. The first complete cycle — call it v7 — came out like this against the untrained Qwen3-1.7B:

Every metric moved the right way. Submit rate going to 1.0 matters more than it looks: an episode that never calls submit_answer scores zero, no matter how good its intermediate reasoning was.

That's the good news, and it's the last of it.

My Eval Harness Was Measuring My Eval Harness

Before those numbers existed, I had different ones. An earlier in-process evaluation put the trained model at 0.415 and the baseline at 0.305 — a win on the average, but with a website regression. I stared at that regression for a long time. I wrote a paragraph about how supervised fine-tuning on teacher trajectories teaches label memorisation and therefore hallucinated URLs. It was a good paragraph. It was about a bug.

That harness loaded the model in-process and ran a hand-rolled loop: call generate, regex the output for a <tool_call> block, parse the JSON, execute the tool, append the observation, generate again. It was not how the model would ever be served. It depressed both models — 0.415 and 0.305 are far below what either does in production — and it hit the trained model's website field hardest, because that field needs a long, exactly-formatted argument string, which gave my parser the most to mangle.

Serving the model properly and evaluating through the real inference path — native tool calls, tools executed by the harness, same code path as production — gave 0.839 vs 0.691. The regression was gone. It had never been in the model.

The Reward Was Marking Correct Answers Wrong

There was an earlier bug, and it was worse, because it was silent. City accuracy sat near zero and I assumed the model was bad at geography.

It wasn't. The evaluation ground truth labelled locations as communes — PARIS, LYON, VILLEJUIF. The training data lake labelled them as a mix of departments (Val-de-Marne) and whole regions (Auvergne-Rhône-Alpes, Occitanie). The reward function compared those strings directly. A model that answered "Val-de-Marne" for a company in Villejuif was completely right and scored zero.

The fix was in the scoring, not the model. I built a small geography module with a department code↔name table (including 2A/2B and the overseas departments) and a department→region table, then made resolution strict: an exact department code or an exact department name resolves, and a region name must not resolve to some representative department inside it. Then city_score branches on granularity — department level when the row carries a department code, region level when it only carries a region. The registry gives the department code directly, so there is no postal-code arithmetic and nothing is fabricated.

One detail worth stealing: name-based registry lookups are unreliable for this. Matching by name produced 19 collisions in my lake — a small Paris company matching an unrelated firm elsewhere that shared its name. So the lake stores only the coarse region label it can justify, and the by-name match is kept for filtering, not scoring.

Improvement #1: Ground Everything (v8)

With honest scoring and an honest harness, I went after the data. v7's teacher trajectories copied labels: the answer came from the dataset row, not from the observations in the episode. That felt like cheating, and it should generalise badly.

So v8 was "grounded and verified". Every website in the training set was actually fetched and verified before being written into an episode. Where verification failed, the teacher was allowed to abstain — emit a null website rather than a guess.

Result: 0.792, against v7's 0.825 under the same strict reward. Worse.

A note on why v7's number moves between tables: the 0.839 / 0.691 pair above predates a tightening of the website score, which became TLD-strict before the four-way comparison. Every number from here down — 0.825, 0.803, 0.792, 0.700, 0.697 — is under that stricter reward, on the same eval set, through the same serving path. Compare within a table, not across.

And it wasn't uniformly worse. v8 hit perfect department accuracy — 1.0. Grounding genuinely wins on verifiable fields: when the answer is sitting in a registry observation two turns back, a model trained to read it off the observation does exactly that, every time.

Website accuracy is where it lost: 0.617 against v7's 0.717. Two causes, both mine:

• The verified dataset was smaller. 82 usable examples versus v7's 112. Verification drops and name collisions ate thirty examples. For a 1.7B, a 27% cut in training episodes is not a rounding error.
• Abstention was over-taught. Roughly 20% of v8 episodes ended with a null website. The model learned the lesson I encoded — "when unsure, don't answer" — and on a find-the-website task an abstention scores the same as a wrong answer. Zero.

There was also a noise tax: for common-word company names, "verified" websites were sometimes confidently wrong — a real, reachable site belonging to somebody else. Verification narrows the error distribution; it does not eliminate it.

Improvement #2: Search Harder (v8.1)

The obvious response to "verification loses too many websites" is to find more websites. v8.1 used multi-query search plus anchor verification: several search formulations per company, then verify by looking for identifying anchors on the candidate page. It worked, as a data pipeline. 103 of 111 companies got a verified website. Abstention fell to about 7%.

Result: 0.700. The worst trained model of the four, essentially equal to the 0.697 untrained baseline. A complete data-generation, training and evaluation cycle that produced a model no better than the one I started with.

The cause is the part I did not see coming, and it has nothing to do with the websites. Multi-query search means two search calls per training episode, and that taught the small model to over-search and then loop — search, search again, search a third time, reconsider. Two of thirty eval episodes hit the turn limit without ever calling submit_answer. Zero reward each.

The second-order damage was worse. Longer episodes pushed the registry_lookup observation further from the final answer. Department accuracy, which v8 had at a perfect 1.0, dropped to 0.867. Nothing about the department pipeline changed. The fact just got further away from the moment the model had to use it, and a 1.7B could not hold onto it.

Improvement #3: Delete the Sophistication (v8.2)

v8.2 kept the grounding and threw away everything clever. Single search. Always commit — no abstention, ever, submit the best candidate you have. Drop unverified examples rather than teaching the model to hedge. Keep v7's short, uniform four-turn episode shape; only make the website verified-correct instead of label-copied.

Result: 0.803. Most of the loss recovered. v8.2 is v8's grounded single-search recipe with the abstention taken out — still a verified website behind every episode, but in v7's short, uniform shape. v8.1 and v8.2 were both built on verified websites, and v8.1 verified more of them. What separates 0.700 from 0.803 is episode shape, not verification coverage.

The Full Ranking

Same strict reward, same eval set, all served through the real inference path.

The Design Rule

What I take away, and the reason this post exists:

Every improvement I added traded a small gain for a bigger loss, and the pattern is identical each time. Grounding bought department accuracy (1.0, perfect) and cost website coverage through abstention. Search diversity bought website recall (103/111 verified) and cost submit rate through looping, plus department accuracy through episode length. Both times the thing I was optimising got better and the model got worse.

The underlying constraint: capability you add to the data has to be capability the model has the capacity to use. "Abstain when unsure" requires a calibrated sense of uncertainty. "Search again with a different phrasing" requires knowing when the first search was insufficient and when to stop. Those are real capabilities and a 1.7B does not have the budget for them — it spends its entire budget on the shape of the episode. A large model can afford to be taught nuance. A small one can only be taught a habit, so the habit had better be a good one.

The Honest Caveat About That Ranking

v7 tops the table and I am not shipping v7.

v7 partly won by copying labels — the training answers came from the dataset row, and the eval draws from a related distribution. Its website edge over v8.2 is eval-distribution overfit, and it will not survive a company that isn't shaped like my eval set. v8.2 learns find-and-verify from the observations in the episode; on a new company name it does the work rather than recalling the answer.

And 0.825 versus 0.803 is a two-point gap on a thirty-episode eval; I have not measured the run-to-run variance, so I would not defend that ordering. Scores rank models on the eval you happen to have. They do not rank generalisation, and I have no metric that does.

Four Mechanics That Cost Real Time

None of this is conceptual. All of it cost real time.

The trainer. I had to use the plain transformers.Trainer with manual tokenisation instead of the convenience SFT wrapper, because of an end-of-sequence token bug with this model family. Symptom: models that train without complaint and then never stop generating. Manual tokenisation means owning the label masking — tedious, and probably worth owning anyway.

Episode format. Training episodes must be native tool-calling format — system message carrying the tool schemas, assistant emits a structured tool call, the tool result comes back as an observation, the next action follows. Not a flattened chat transcript with the tool calls pasted in as text. Flatten it and the result is a chatbot that talks about calling tools:

system    tools: registry_lookup, web_search, web_extract, submit_answer
user      Profile this company: <name>
assistant tool_call registry_lookup {"name": "<name>"}
tool      {"siege": {"departement": "94", ...}}
assistant tool_call web_search {"query": "<name> site officiel"}
tool      [{"title": "...", "url": "..."}, ...]
assistant tool_call submit_answer {"website": "...", "city": "...", "sector": "..."}

Serving. The runtime I evaluate through (Ollama 0.31.2) cannot convert this architecture's safetensors internally, but it runs the equivalent GGUF without complaint. The path that works: merge the adapter to 16-bit on CPU — the GPU already has another workload resident and merging there causes contention — then convert, then create the served model.

# merge adapter -> 16-bit, CPU only, from the local cache
HF_HUB_OFFLINE=1 HF_HUB_DISABLE_XET=1 python merge_adapter.py --device cpu

# convert to GGUF
python llama.cpp/convert_hf_to_gguf.py ./merged --outtype f16 --outfile crab1_v82.gguf

ollama create crab1_v82 -f Modelfile

Disk. The GPU box's root filesystem was around 96% full. ollama create copies the GGUF into its own blob store, so the merged weights, the 3.4 GB converted file and a second 3.4 GB copy all have to fit at once. There wasn't room, and the existing models on that box were not mine to delete.

The unblock was a tmpfs RAM-disk for the intermediate file — there was 14 GB of free RAM and no free disk:

mount -t tmpfs -o size=5G tmpfs /mnt/ramgguf
mv crab1_v82.gguf /mnt/ramgguf/
ollama create crab1_v82 -f Modelfile   # blob copy lands on disk, source never does

Delete the standalone GGUF only after the model list confirms creation. This is now my default serve pattern.

What I'd Tell Myself At The Start

Fix the reward before reading a single number. Run the eval through the production path or don't run it at all. And on a 1.7B, "smarter training data" and "a harder job for the model" are frequently the same sentence.

Simple, committed, verified. In that order.