Converting the first real account found a gate that never existed

The conversion tool had already been built and tested. Moving an account from a generic sport configuration onto that sport’s own dedicated code, the same move every account on that sport would eventually need, was supposed to be routine. Running it against the first real customer account instead of sample data found two things testing had never touched.

The gate that was never built

The sport in question has a stats feature, entry of detailed play-by-play data, that two comparable sports both restrict to a paid tier. This sport didn’t restrict it at all. Every account on it, regardless of tier, had full access to a feature that was supposed to require an upgrade, and the account I was converting was on the entry tier, so the gap was immediately visible: after the fix, that account lost access to a feature it had never actually been entitled to and never should have had.

There was no record of that being an intentional exception. It read as a gap in the sport’s original build, one that had simply never been closed, sitting quietly in code that two other sports got right from the start. It only surfaced now because this was the first time a real account on that tier ran the play-by-play feature through this particular code path with anyone actually checking whether it was gated.

The second gap: photos left behind

The same conversion also had to move photo assets to the new storage location, and that step was a separate option rather than something the main move enforced. Skip it, forget it, and photos are left behind with no error raised, silently, no failed job, no flag, nothing that would tell a future run or a support ticket that anything was incomplete. That gap only shows up when someone runs the conversion without deliberately remembering the extra step, which is exactly what sample data never requires anyone to do.

What verification actually looked like

Both fixes were checked against the real account on both sides of the change: before, confirming the tier gate was genuinely open and the feature genuinely worked without a paid tier; after, confirming the gate now applied and a comparable account on the paid tier kept full access exactly as before. The photo-migration fix was checked with a live dry-run against the real account, confirming the tool now moves photos by default, since that day’s real conversion had already exercised the underlying move code the fix now defaults onto.

The gate fix collided with a mailer

A conversion mailer sent earlier to a large batch of customers had told everyone the play-by-play stats feature came with their account regardless of tier, which was true by accident, because the gate had never existed. The tier-gate fix made that promise false for anyone below the paid tier. Rather than resolve that collision inline, it went to a separate ticket, since it’s a customer-communication decision, not a code fix, and the two shouldn’t be rushed together.

Why “first real run” earns its own review

Nothing here would have shown up in a test suite built against synthetic accounts, because synthetic accounts don’t carry the accumulated shape of years of real usage: a tier that was never checked because nothing before now had reason to check it, an optional step that nothing before now had ever forgotten to take. The first time a new or newly-touched code path runs against a real account, it is doing something a clean test environment structurally cannot: exposing every assumption that was only ever true because nothing real had tested it yet.

dxdev LESSON · paste into your AI coding agent
LESSON: Running the First Real Account Through a New Path Is Its Own Test Suite
SOURCE: dxdev.com/blog/2026-08-19_first-real-run-found-the-gap
WHAT HAPPENED: A tool converted an account from a generic sport configuration onto that sport's dedicated code path. The first time it ran against a real customer account rather than test data, it surfaced two problems test data had never triggered: a paid-tier gate that two comparable sports both enforced had simply never been built for this one, letting every account on it use a paid feature for free since the feature was added years earlier; and moving the account's photo assets to the new storage location was a separate option rather than something the main move enforced, so skipping or forgetting it silently left photos behind with no error to signal the gap. Both were fixed and verified against the live account before and after the change, then shipped as hotfixes. A separate mailer had already told a large batch of customers the paid feature was included at every tier, which the tier-gate fix now contradicted, so that mismatch was handed to a follow-up ticket rather than resolved inline.
THE RULE: A feature can pass every test built against synthetic or sample data and still carry a real gap, because the gap only shows up against the shape of an actual account: real data volume, a real subscription tier, a real history of partial edits. The first time any migration or conversion tool touches a real account, treat that run as a verification pass in its own right, not a routine execution of already-tested code. Check the account against its peers on adjacent, comparable paths (a similar feature on a similar sport, a similar tier on a similar account) rather than only checking it against its own prior state, because a gap that has existed since the original build will look completely normal in isolation.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any feature-gating or tier-restriction logic that is duplicated per category (sport, plan, product line) rather than shared, where one category could silently lack a gate the others enforce.
  2. Any migration, import, or conversion script where moving secondary data (assets, related rows) is a separate opt-in step rather than something the main move enforces, so skipping or forgetting it leaves orphaned or incomplete data with no error surfaced.
  3. Any marketing or customer communication (a mailer, a changelog, an announcement) asserting a feature is available at every tier, cross-checked against the actual gating code for that feature.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.