Multi-agent system with independent audit

Problem

One AI agent doing everything is hard to trust. It plans, builds and then grades its own work.

I wanted the jobs split. I also wanted the checking done by someone other than the builder.

What I built

I set up a small team of agents across two runtimes, OpenClaw and Hermes.

• An orchestrator routes work, accepts results and is the only one that talks to me.
• A planner writes plans and acceptance criteria. It never builds.
• A coder only touches program code.
• An executor handles config, commands, files and tests.
• An auditor and a security reviewer are called in for risky or hard-to-check work.

Routine, low-risk changes get one build and one check. Not a full review chain.

Every agent hands back a short, fixed report: outcome, evidence, risks, verdict, next step.

How it’s checked

• A drift-check script compares the live agent setup with a pinned manifest. It fails on any mismatch.
• Audits are blind where possible. A second auditor on a different model family sees only the work and the task. It does not see the builder’s story.
• A clean verdict has to list what was checked.

One audit caught a real gap. An auditor was described as “read-only, network off”, but its sandbox was not on. The audit flagged it and the setting was corrected that day (September 2026). A later config change turned it off again, so it is listed as open below.

Results

• Report leak closed. Agent reports were reaching my chat raw, instead of only the orchestrator’s summary. Three fixes were tried and disproved first. The real fix removed one dispatch path. Proof: 5 out of 5 live marker tests sent nothing to the chat (one test run, September 2026). The auditor confirmed this from the logs on its own.
• Drift check passed once. 83 of 83 checks passed after an audited update (7 September 2026). A later full run failed; see below.

What’s still open

• Delegation. After a model change, the orchestrator often did work itself instead of routing it. I changed the instructions. I have not yet seen the behaviour change over enough turns to call it fixed.
• Long replies. The orchestrator’s replies got much longer after the same change. Not addressed yet.
• Config drift. In late September the full drift check failed on eight model settings that no longer matched the manifest. The policy checks passed. The mismatches were still open at the last status update.
• Soft boundaries. The reviewer agents, including the auditors, are read-only by rule, not by a hard sandbox.
• Leak residual. The leak path is closed for normal dispatch. One rarer path is documented but not closed.