When Your Observability Tool Is Wrong, Your Diagnosis Is Wrong

The PowerShell window popped up again at 1:48 PM. Third time that week. I went looking for what spawned it and found something worse: nothing on the machine could tell me what was actually running.

The instinct was to chase the window. Find the process, find the task that launched it, kill the task, done. I did that for the first two instances, and each time a new one showed up somewhere else. A scheduled task, a stray script, a leftover from a Discord bot restart. Three separate root causes that all rendered the same symptom. I was patching sightings, not fixing a system, and I only noticed because the fourth window looked exactly like the first three in the screenshot.

The actual tool I reached for first was the README. We keep one that lists what’s supposed to be running on this box: which agents, which scheduled tasks, which bots. I opened it to cross-check against the process list and it was wrong. Not slightly stale, wrong in ways that mattered: it listed a task by a name that had been renamed two refactors ago, it didn’t mention the platform Discord agent at all, and it claimed a background watcher was disabled when Task Scheduler showed it running hourly. I’d been treating a doc as ground truth for the state of a live machine. The doc doesn’t know when someone renamed a task. The machine does.

So I stopped chasing windows and built ops-cli background, a command that derives its answer from the machine instead of from documentation. It walks the actual process list, cross-references Task Scheduler entries, and inspects what each one can do rather than reading a claim about what it’s supposed to do. No README to fall out of sync, because there’s no README in the loop.

The mechanism is simple: enumerate the scheduled tasks, resolve each one to its actual command line, check whether the referenced script or executable still exists at that path, and flag anything that starts an interactive shell without -WindowStyle Hidden. That last check is what found it. Two of the four window-spawning tasks were calling PowerShell with no window flag at all, launched months apart by different fixes to different problems, both missing the same one-line switch.

Once the tool told the truth about what was running, the actual fix took minutes. We cleaned up the tasks causing the windows, got the platform Discord agent running hidden, and verified with the same tool that it stayed up instead of trusting a log line that says “started” and then goes silent. Verification against live state, not against a status message the process emitted on its way up.

The eighteen-hour session that produced this also touched the thing we were actually supposed to be doing, which was giving each developer their own agent. That work stalled twice for the same reason at a different layer: an agent that had been pushed looked done because the deploy log said so, but the process on the machine was still running the old code. Same failure shape as the README, just one layer down. The fix there was a check that reads what the live process actually started from, not what the last deploy claimed to ship. It cost about three hours across three machines to notice, because “pushed” and “running” look identical in every log that only records the push.

The pattern underneath both is the same: any status that comes from a system telling you about itself is a claim, not an observation. A README is a claim. A deploy log line is a claim. A task name is a claim. The only thing that isn’t a claim is asking the machine directly, right now, what’s actually loaded and what it’s actually doing. Once ops-cli background existed, the recurring popup wasn’t a mystery to solve case by case anymore. It was one line in a table that had been lying to me since before I started looking.

dxdev LESSON · paste into your AI coding agent
LESSON: Derive Operational Status From Live State, Not Claims
SOURCE: dxdev.com/blog/2026-09-07_stale-observability-breaks-diagnosis
WHAT HAPPENED: The author initially chased individual PowerShell windows, but each apparent fix was followed by another popup from a different source. Cross-checking the process list against the README showed that task names, agents, and watcher status in the document no longer matched the machine. They built ops-cli background to enumerate scheduled tasks, resolve their real command lines, verify referenced paths, and flag interactive PowerShell launches missing -WindowStyle Hidden. The tool found two window-spawning tasks with the same missing switch, and live verification confirmed the hidden Discord agent remained up. A related deployment problem also turned out to be old code still running despite logs claiming the agent had been pushed.
THE RULE: Treat documentation, task names, and deployment logs as claims rather than proof of operational state. Diagnose and verify running systems by reading live processes, task definitions, loaded code, and actual command lines.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Scheduled-task audits resolve every task to its current command line and verify that each referenced script or executable exists.
      2. Tasks that launch PowerShell or another interactive shell use the required hidden-window option when they run unattended.
      3. Deployment verification reads the live process's actual loaded code or startup source instead of relying only on a successful push or startup log.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.