Last month I hit a wall: generic LLMs were useless at a task I actually cared about — turning a company name into a structured OSINT profile. So I stopped prompting big models and trained a small one instead. The result surprised me: a distilled 1.7B model beats Llama 3.1 8B by every metric on the same task, same tools, verifiable ground truth. Here is the receipt.

The Task

Given a company name, build a profile: website, sector, headcount, location, key people, and recent signals (jobs, funding, news). Everything is checked against a ground-truth dataset — no vibes, no self-reported accuracy. A profile counts as passed only when the extracted facts match reality.

  • Verifiable reward — facts compared to ground truth, per-field accuracy
  • Agentic — the model calls real tools (search, scrape, geo lookup) in a harness
  • Narrow by design — one domain, deep, not a jack-of-all-trades

The Setup

I distilled trajectories from a strong teacher (Claude) doing the task well, then fine-tuned Qwen3-1.7B with QLoRA SFT. No chain-of-thought tricks, no huge context — just the model, a tool harness, and a reward signal.

  • Teacher: Claude-generated trajectories on real companies
  • Student: Qwen3-1.7B, QLoRA SFT (runs on a laptop)
  • Harness: search + scrape + geo tools, max ~6 turns per profile
  • Eval: 30 real companies with ground truth

Results

Same task, same tools. The only variable is what is doing the thinking:

  • Llama 3.1 8B, classic prompting: 0% pass rate (avg reward 0.23)
  • Llama 3.1 8B + scripted pipeline: 66.7% (avg reward 0.60)
  • Qwen3-1.7B, prompted, no training: 56.7% on 30 companies (0.69)
  • Crab-1 v7 (SFT): 76.7% on 30 companies (avg reward 0.84)
  • Crab-1 final (SFT): 100% on holdout batch, avg reward 0.90

The final model hits 100% submit rate, 100% valid tool calls, and finishes a profile in a median of 5.4 turns — under 20 seconds per company on commodity hardware.

What Actually Mattered

  • Training data beats parameters. 30 well-curated trajectories did more than 8B of extra weights.
  • Narrow agents win. A small model that knows your task beats a generalist that sort of knows everything.
  • Cost: 1.7B runs locally. This pipeline costs dollars per month, not thousands.

Honest Caveats

The holdout batch is small (n=5). The honest number at n=30 is 76.7% — still a 40-point jump over Llama 8B prompted, and 10 points over the scripted 8B pipeline. The task is deliberately narrow. This is not a general reasoning benchmark; it is a blueprint for your narrow task.

The Takeaway

Stop renting 400B models for tasks you actually do every day. Distill, fine-tune a 1.7B, and let it run for pennies. That is the whole trick. 🦀