The fastest fix was blocking the bot by name
AwarioBot slipped past an existing IP blacklist because the per-request filter had auto-cleared its range. Thirty seconds in the session handler fixed it.
The build log
Notes from running agents in production.
Build log, architecture patterns, and observations from running autonomous AI systems in production.
-
-
The Auto-Approval Feature That Rewrote Curated Data
A payment-confirmation automation overwrote a status that some administrators managed by hand. The remediation separated payment evidence from approval, added explicit tenant controls, and used a narrowly scoped, audited recovery process.
-
Our backups had been dead for three weeks. We found out by accident.
We split a sprawling AI ops monorepo into five sibling repos. The rename exposed runtime automation that had been dead for weeks.
-
"Bamboo is broken" was wrong: a deploy that races the filesystem under CPU pressure
A stale note said the CI pipeline was broken. A bot swarm had starved the box, and the deploy's extract-then-verify step lost a filesystem race.
-
UTC logs, a local clock, and the canary request: timezone discipline in an incident
We spent rounds of an active production incident convinced a firewall rule wasn't working. The log timestamps looked like they fell after the rule went in, and the bad traffic was still showing up.
-
Hotfix branches need different rules than develop
Running /sync from a hotfix branch tried to merge develop in. On a legacy codebase with an active release in flight, that would have dragged unfinished work toward master. Tooling gap caught during a hotfix session, April 2026.
-
The fix that ran on every request
We closed a redirect-loop bug in two hours. Then the fix itself saturated the IIS worker pool on every authenticated admin page load. Two tickets, one afternoon.
-
The bug was wearing a trench coat
Two hotfix branches merged the same day. Three tickets looked like one customer complaint and turned into three different bugs: bad payment data, wrong scope resolution, and a deploy that would have re-broken the fix by Tuesday.
-
We had to stop letting the package tier pretend it was the product
A ticket started as a Packages UI redesign. By the time we closed it, we had touched the organization taxonomy, the credit constants, and the account type routing logic in three separate files. The UI was the last thing we changed.
-
The blog merge that turned into a platform decision
Folding dxdev-blog into dxdev-landing was supposed to be content migration. It became schema design, renderer archaeology, and a question about what the public surface of a product should reveal.
Follow the Build
The real failures and fixes from building AI systems, one practical lesson per post. Get the next one in your inbox.