Research · Field dossier · Coding agents

The exit-code contract

A coding agent succeeds when its process exits zero with tests green, and tells the truth when it fails. A field dossier on Kestrel — the harness-over-model argument, the competitive frame, and the build plan.

This is a field dossier: our internal working notes on Kestrel, a one-shot CLI coding agent, edited lightly for public reading. It is a build document — nothing here is shipped fact, and the risks section names what would falsify the whole position.

Four claims

  1. K1 — Value = P(first-shot correctness) × time saved. Everything else is UX polish on top of those two factors.
  2. K2 — Base models commoditize; durable value is the harness. Context assembly, safe file operations, a verification loop. The model is rented; the harness compounds.
  3. K3 — “One-shot” means something operational or it means nothing. Defined here: the process exits 0 with tests or build green, or it prints an honest failure report. Never a plausible half-change.
  4. K4 — Setup budget: one command plus one environment variable. A GEMINI_API_KEY and nothing else. Setup that costs more than this loses the audience before the first edit.

Harness over model

Our own working tree supplied the evidence that the harness thesis holds. A pack-based design rulebook — flat color, accessibility floors, a five-step verification protocol, a stack mandate — turned out to be effectively Kestrel's first pack: domain rulebooks injected into context, with precedence rules already written down. An execution-trace format existed in our session logs. A tested TypeScript library with a vitest configuration gave us a real project to validate jobs against. None of these assets came from a model; they came from discipline encoded outside the model.

Which sharpens the differentiator: Kestrel's value is not another agent loop. Competitors ship chat loops; Kestrel ships rulebooks and exit codes. Packs are the extensibility model — markdown rulebooks loaded by domain, so a design job gets the design pack and a migration job gets whatever pack applies.

Competitive frame

  • Claude Code / Codex CLI / Gemini CLI: general interactive agents, session-based, strong but chatty. Success is conversational, not gated on tests.
  • Aider: git-native diffs, genuinely good — but verification remains opt-in rather than contractual.
  • Kestrel's claim: the only CLI whose contract is the exit code. Batch-friendly — CI, cron, scripts — precisely because it is non-interactive by design.
Claude Code is a pair programmer. Kestrel is a merge gate you can run from cron.

Build plan

MVP, weeks 1–2:

  1. CLI shell: parse the task, load repo context (tree, git status, relevant files via ripgrep heuristics), call Gemini with a pack-augmented system prompt.
  2. Edit executor: unified-diff apply with dry-run preview and atomic per-file rollback on failure.
  3. Verification gate: detect the package manager, run build/test commands from config or convention, and block success on red.
  4. Report: a green summary, or a failure report carrying hypothesis, diff stat and the tail of the failing command's output.

V2, in order:

  1. Packs: markdown rulebooks loaded by domain — the design pack exists — with pack precedence rules already defined.
  2. Headless mode for CI: a --check flag that fails the pipeline if the fix does not verify.
  3. Opt-in telemetry: first-shot green rate, edits per task, token burn. These numbers are the marketing; publish them the way heyIAS publishes calibration.

One harness discipline, three products

Kestrel is also internal infrastructure. Numera's reconciler and filing validators, and heyIAS's evaluation pipeline, should run as kestrel-style verified jobs: deterministic inputs, coded rules, exit codes, audit logs. One harness discipline across accounting, exam prep and coding is itself the velofy engineering thesis made real — trust scales with verification, not with intelligence.

Risks, stated honestly

  • R1 — Latency and rate limits on large repos. Mitigate with aggressive context pruning and cached file summaries.
  • R2 — Overpromising “one-shot.” Define the contract narrowly: the task must include or imply a verification command; otherwise refuse politely and suggest one. Honesty is the brand — one inflated success claim destroys the entire position.
  • R3 — Model drift. Pin an evaluation suite of 50 tasks with known-green outcomes and run it against every model version before promoting defaults.

The short version

Rent the model, own the harness. Ship a CLI whose entire promise fits in one byte: exit zero means tests were green, anything else is an honest failure report. Make it runnable from cron because it never needed a conversation. Publish the green rate. If we cannot keep that contract narrow and true, the product does not exist.

This dossier extends the coding case derived in three markets, one thesis, and implements the confirmation step of the verified agent loop: for Kestrel, the test gate decides.