We Assumed We Were Letting AI Crawlers In. We'd Been Blocking Them for Months.
The alert fired on a rule we'd never named: 200 events in 24 hours through the verified-bot skip lane, and not one of them was the thing we'd come to document.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
I Closed an Unauthenticated Bypass, Then Locked Out My Own Staff
The alert I was chasing had nothing to do with what I found. I was in the codebase for an unrelated ticket, tracing how staff sessions get verified on an internal admin tool, when I hit a line that...
-
The Dependency You Shipped But Can't See: Reverse-DNS and Email Delivery
Between 10 and 20 percent of our signup codes and password resets to Yahoo, AOL and Gmail had been hard-rejected for about ten days, and nothing in our code had changed in that time.
-
The Exemption That Trusted the Wrong Input
A staff login check had one carve-out for the login page itself. The carve-out read a value the caller controlled, not one the server had already resolved.
-
JavaScript in Your Stylesheet: Why Your Mobile UI Silently Broke
One filter button sat in the wrong place on mobile and in the right place on desktop, and the stylesheet that positioned it looked fine in the editor.
-
The Handoff Gauntlet: How Parallel Reviewers Caught a Cache Bug
Four reviewers ran in parallel over a finished diff, and one of them flagged a preview render that would be written into the production page cache and served to real visitors.
-
51 of 55 Verdicts Were Wrong: Why Parallel Reviewers Need a Skeptic
51 of 55. That is how many of the "already cleared" verdicts an adversarial audit refuted after I fanned a backlog review out to sub-sessions and got back 45 tidy files, each one confident.
-
Two Tiers for the Same Content
One of my AI agents called me escapist. I fired back a task asking it to prove it with my own data. It retracted the claim within the hour.
-
The Runbook Said Three Minutes
A maintenance window ran exactly once, against a real production box, and the written plan turned out wrong in four separate places.
-
I Stopped Guessing Categories and Read Them Off My Own Product
The alert wasn't an alert. It was a number: 82 tickets, 382 comments, and nobody had read all of them. I opened the epic planning to work through it top to bottom and closed it inside a minute.
Follow the Build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.