The Urgent Bug Report That Was Two Years Old
A dashboard number read as 'happening right now.' It was a lifetime total going back more than two years.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
Stop Rediscovering the Same Blocked Work: A Detector That Surfaces Every Session
The queue said ten items. Four of them were real.
-
The Error Log That Lied: When Recent and Lifetime Counts Blur Together
The Error Log table had 492 rows tagged Script timed out over the trailing 120 days, and the ticket that came in described it like a site-wide admin outage.
-
Narrow First, Fix at the Choke Point: The Script-Escape Audit
267 places in the codebase write a string into a <script> block or an inline handler, and I had to decide which of them were dangerous.
-
956 Certificates Could Only Renew If a Customer Did Something. The Watchdog Then Paged 55 Times and Buried Eleven Real Outages.
A bulk migration had put 956 domains on the one renewal method that needs a DNS record from the customer every 90 days. Moving them was easy. The watchdog I built for the next one spent its first real morning crying wolf over the wrong 55.
-
I Asked for JSON and Got My Own Footer Back
Headless runs of my AI CLI loaded my interactive hooks, and a plain JSON request came back as pure footer text. The lint I wrote to protect the fix passed on a broken file.
-
The Batch That Was Aging Out With No Alarm Attached
Nearly a thousand certificates sat on a validation method that needed refreshing every 90 days, and nothing would have noticed a specific one about to fail before it actually did.
-
The Three-Day Blind Spot: Why Agent Health Checks Must Be Automatic
A teammate's agent sat broken for two days, and the way it finally got noticed was someone squinting at Discord reactions and thinking they looked wrong. Then it spammed the channel.
-
Blog Voice Measured: When Artifacts Trump Abstract Rules
Two rewrites in, the post still had a forty-five line code block in it, and it still read like documentation.
-
The Test That Left Real Accounts Behind
An automated task needed a throwaway test site, used the public signup form to get one, and reported that nothing had been created. Two live accounts said otherwise.
Follow the Build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.