The Quota Error on the Wrong Stream
A Claude seat ran out of weekly quota, and our ticket bot kept calling it for hours. Nothing in its logs read as a quota failure, because as far as the bot could tell, no failure had happened.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
The 697-Second Queue That Was Hiding in Shared Infrastructure
The p90 queue delay on our password reset email was 697 seconds during a campaign burst and 3 seconds outside one. Users had been telling us that resets and signup confirmations felt slow.
-
Expensive Models at the Hub: Why Constraining Costs at the Root Beats Guarding Every Edge
One orchestration session that week ran 644 prompts and pulled 190,414,059 tokens off cache before it closed. Output alone was 800,620 tokens. It wasn't an outlier.
-
The Guard That Works But Can't Undo
One delete against a self-managed customer domain came back blocked, and that was my proof the guard worked.
-
When Parallel Sessions Overwrite Each Other's Work
At 10:29 one morning I wrote the status line for a ticket, and it had to say two things that didn't fit together. One half of the work was finished, tested and pushed.
-
The Small Annoyance That Exposed Our Capacity Ceiling
A powershell.exe window opened on top of my editor every time any agent session finished a turn, and on the morning I chased it I had six sessions open at once.
-
When Product Analytics Finds What Logs Miss
The session-replay connection took one late session, 11:27 PM to about 1:27 AM, and by the end of it a popup crash fix was live in production. Nothing in our error logs had pointed at that crash.
-
I Killed My Code Fork and Built a Daily Agent Pipeline Instead
The alert I actually wanted was a green checkmark on a cron job. What I got instead was a Cloudflare 530 at 10:59 PM, because a cloudflared tunnel had fired at boot before the network was up, exite...
-
Walking the Funnel: Manual Testing Beat a Week of Waiting
New signups had gone bad, and the first thing I typed into a fresh agent session was a request to check whether some unknown bug was killing conversions. That session opened at 5:37 PM.
-
When Enforcement Scatters, Data Silently Leaks
Twenty-two customer tickets closed without a sprint, and only one of them could still be repaired. The other 21 lost that field for good.
Follow the build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.