Separate Your Build Model From Your Review Model: Codex as Adversarial Reviewer
Build with one model, review with a different one at high effort. One adversarial Codex pass found several false-green paths in a production gate that showed all green.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
It Looked Exactly Like a Bot Swarm. It Was SQL Server Parameter Sniffing.
Traffic-correlated CPU spikes can mean a scraper swarm or a cached bad execution plan. Here is how to tell them apart and what to do when the answer is the plan.
-
The stale credential that failed two different code paths at once
A sandbox checkout died with "merchant login ID or password is invalid." The exact same credentials had authenticated cleanly in a standalone Python proof minutes earlier, returning I00001.
-
The Ambient-Global Bug: When "Current Account" Is a Mutable Global Your Whole App Reads
A dropdown that let org admins drill into child events broke saves across a 12,000-line handler. Two globals that had always pointed at the same object suddenly didn't. Here's what the fix looked like and why the real hazard is the mutation, not the call sites.
-
rtk Wasn't Broken, It Was Unreachable: How I Found 366 Silent Failures in My AI Agent Sessions
A silent PATH misconfiguration meant my token-optimization tool had actually run 10 times across 4,446 AI agent sessions. Here's how grepping my own transcripts like a flaky test suite found it, and two other live bugs.
-
A warning is not a guard: make conflicting work stop before it overwrites
A system noticed two workers wanted the same shared resource, wrote a warning, and proceeded anyway. The durable lesson is that conflict detection matters only when it prevents unsafe mutation and preserves the current holder's state.
-
Before adding a new option, ask whether the existing one is almost right
A feature request can look like a request for a new branch of the product. Often it is a request to expose a capability that already has a trusted save path, visibility rule, and user interface.
-
The \"queue worker drains it\" story was vapor: the create command said NOT YET IMPLEMENTED
The admin console had a clean button. Click it, and each domain got queued for Cloudflare custom-hostname registration.
-
The runbook said the domain was in account A. It was in account B. Verify ownership before touching live DNS.
My own build runbook, written by me, confidently told me to flip a DNS record in a specific GoDaddy reseller account. The record wasn't there.
-
The customer's own scraper IS the bot swarm you've been fighting
A customer opened a ticket asking us to allow-list their app's scraper so it could pull their own hosted site data.
Follow the Build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.