Training the Harness Instead of the Model

When I released Agent Neo in May, it leaned on whatever model you gave it. Since then it’s changed a lot, mostly in how little it trusts the model.

Less trust in the model

Most signals never reach the model. Deterministic observers, dedup, and correlation handle the bulk of it. One LLM pass diagnoses what’s left, and code checks that answer before anything acts on it.

It can only run compiled skills. Every skill is a YAML template that gets compiled into Go. The model gets exactly one tool, whose operations come from that compiled registry, with typed arguments. It can’t make up a command or pass it arbitrary flags. If you want it to do something new, you write a template and it goes through code review like everything else.

Config can only make it more careful. Safety settings combine with the model’s own checks, so configuration can take autonomy away but never add it.

It runs on a local model. Day to day it runs on a self-hosted open-weight model in a 64K context, not a paid API.

Training the harness instead of the model

Over the summer I fine-tuned dozens of 3B–8B open-weight models for this job, with LoRA SFT and then GRPO, scored on a sandboxed fault-injection eval with control arms. None of them beat the base model with a better harness. That was humbling, but it was also the answer, so I shipped the harness and wrote down the negative result.

So I started training the harness instead. I combined Google’s Dream-RSI and Microsoft’s SkillOpt into one loop that learns how Neo investigates (which operations to try, how wide to go, when to stop) without ever touching the compiled skills. Every candidate strategy is gated on held-out incidents before it’s kept. Two 30-hour cycles took it from 5 to 10 out of 12 hard correlation investigations. Moving the loop onto RLM and ADK made each cycle 1.5–2.5× faster.

What’s next

I’m cleaning this version up for a new open-source release. The public repo is still the May snapshot.