The cleanup run that retired 29 dead domains took three minutes. What made it safe to run was a tool that refuses to trust our own inventory.

The version that trusted our records

The job was ordinary housekeeping. Over the years we had accumulated SSL certificates and domain entries for sites that no longer existed. The old version of the tool took its list of what was dead from our own records, the inventory we keep of domains and the sites they belong to, and deleted the certificate for each one.

That list was accurate when someone wrote it. Nothing forced it to stay accurate. A domain marked retired in our records could have been pointed at a working site since then, and the tool never asked the database. It read the inventory, saw “dead,” and deleted.

That is the risk, and it is why the tool changed before this run. There is no bug in the deletion code to fix. A tool that does exactly what a stale inventory says will delete something live sooner or later, and a chore that looks safe turns into an incident.

Checking at the moment of the delete

The fix was to stop treating the inventory as evidence. The rebuilt tool has two refusals, and both run at delete time:

  1. Stale record. If the record for a domain no longer matches what the current database says, the tool stops and reports the mismatch. It does not guess which side is right.
  2. Live site. If the domain is attached to a site that is currently live, the tool refuses the delete outright. The record can say anything it wants.

The inventory now only nominates candidates. The current database is what authorizes a deletion, and it gets asked immediately before each one, not at the start of the run. A batch that begins with a correct list can still hit a domain that changed halfway through. Per-item checks cover that case, and one upfront validation does not.

With those refusals in place, the run went through. 29 dead domains were removed, and each one was commented in place, so the reason for the removal stays next to where the entry used to be.

What it declined to do

One domain is still held. A separate cleanup ticket was not approved, and the tool does not route around that. A refusal that gets overridden by whoever is in a hurry is just a delay. So the held domain stays held until someone with the authority says otherwise.

The run also surfaced a security problem next to it. Some of the scripts on the box that do this work carry the SQL login in the script itself. That is a bigger exposure than a stale certificate, and it got its own ticket to lock those scripts down. We have not fixed it yet.

A record is a hint, not a permit

Any tool that destroys something should re-verify its preconditions against the live source, immediately before acting. A record of the world is a hint about the world. It tells you where to look. It cannot tell you that a delete is safe.

Our records were not sloppy. They were just old the moment we stopped touching them, and the tool treated them as current. The three-minute run finished cleanly because each delete had to pass a check it could fail.