All work

AI agent engineering

AI agents that have to prove their work

My AI coding agents now run the real tests before they can call a job done, and I measure it before making it mandatory.

Claude CodeCodexPythonSQLiteNotionTest gates
System
Richie OS, my private operating platform
Agents
Claude Code and Codex, one shared task list
My role
Designed, directed, verified and operated. The agents write much of the code; I review and approve it.
Shown here
Real measurements from my own system

01The situation

I use AI coding agents every day on my own platform, on automation work and on a Telegram community bot. They are fast, but an agent saying “done” is not proof.

In one week the bot needed 31 bug-fix commits after agents had reported their work finished. The old completion check never ran a single test.

Before“done” meant the agent said so
  1. Task
  2. Agent edits code
  3. Agent says done
  4. Person finds bug later

02What I found

  • I rejected trusting the agent’s summary. It describes intent, not results.
  • I rejected blocking on every check from day one. The old gate let agents give up 289 times against 240 clean passes, nearly all false alarms, and a noisy gate gets ignored.
  • I chose to run the real tests at the moment an agent tries to finish, watch-only for a week, and make it mandatory only if false alarms stay low.

03What I built

  1. One task list that both agents and I work from. Each task has one accountable lead, and the other agent reviews consequential work.
  2. A completion check: when Claude changes the bot’s code, the full suite of 586 tests runs first, in about 7 seconds, on a throwaway database with internet calls blocked.
  3. Bounded retries: two attempts per check and at most six blocks a session, then it stops and asks me.
  4. Every pilot task writes one run record, so a weekly report shows pass rate, retries, tokens per verified task and how often I had to step in.
  5. Scheduled monitors report into Telegram. I removed four standing false warnings that had fired on 34 of 34 daily runs.
Aftertests decide, not the agent
  1. Taskone accountable lead
  2. Agent works
  3. Tests pass?586 in about 7 s
  4. Done, or ask meafter two retries

04How it fails safely

  • Tests can never touch the live database.
  • Under my operating rules, sending, publishing and spending need my approval.
  • A second agent reviews production and data changes before I approve them.

05Result

Measured so far

  • Before: 31 bug-fix commits to the bot in one week, and 0 tests run at “done”.
  • Now: 586 tests run on every bot change before “done”, currently in watch-only mode.
  • Separately, my wider rule that work is verified before it is called done held 77.8% of the time over 14 days, against a 95% target. I publish the gap, not just the goal.

06Lessons

Make the agent prove it, and make the proof cheap. Seven seconds of tests is a check an agent can run every time.
A check that cries wolf gets switched off. Measure false alarms before you enforce.
For engineers: how it is put together
  • A stop hook maps changed files to test suites and runs them in test mode with a disposable database and outbound HTTP pointed at a dead port.
  • Each run writes a workflow row that feeds the existing watchdog and a weekly report.
Next case studyOne customer, one record

Have a system that keeps drifting?

Tell me where it breaks. I’ll tell you what I’d check first.

Get in touchWork with FlowThinkers