A replaced system was made read-only for 30 days. No complaints came in and it was switched off, with its data exported to an archive. Three weeks later the annual regulatory extract fails, because the job read directly from the retired database. Which design decision made this possible?
Show the full answer Hide the answer
The trigger
An annual job. It ran in month thirteen of a system whose entire evidence base was thirty days long. No amount of care applied within a 30-day window can observe a consumer that appears once a year, and the window was chosen because thirty days feels thorough, not because anyone compared it to the system's cycle.
Why it propagated
Traffic-based evidence has a sampling problem that nobody states out loud: it can only prove the absence of consumers that would have run during the observation period. The distribution of batch consumers in an enterprise is not uniform. There is a dense band of daily and weekly jobs, then a long tail at monthly, quarterly, annual and "when the auditor asks" frequencies, and the tail contains the consumers with the worst consequences precisely because they are regulatory and financial.
The read-only stage made it worse in one specific way. A read-only system satisfies a reading consumer silently. Every daily and weekly reader kept working through the whole thirty days and generated no signal at all, so the stage that was meant to smoke out consumers proved only that nothing was writing.
Why detection lagged
Nothing alerted, because nothing failed until the job ran. The gap between shutdown and discovery was three weeks, and had the extract been due in month eleven instead, it would have been ten months. Decommissioning failures have detection latency measured in the consumer's period, not in minutes.
The structural fix versus the tempting local fix
The tempting fix is "keep the archive restorable". Useful, and it does not address the cause: a restored database is worthless to a job whose connection string points at a hostname that no longer resolves, and the team will not know to restore it until something breaks.
The structural fix is to set the observation window from the system's longest business cycle, then stage the shutdown inside it:
- Inventory by evidence, not memory — connection logs, scheduler definitions, firewall rules and the batch calendar, because the calendar lists jobs that have not run yet.
- Observe across at least one full cycle, which for anything touching finance or regulation means thirteen months and not thirty days.
- Stage the shutdown so each stage produces a signal: read-only, then reject all connections with a logged error, then stop, then delete. Rejecting is what converts a silent reader into a page.
- Keep the hostname and the credentials alive after the data moves, answering with a loud error that names the archive, for at least one more cycle.
Why the other options fail
- Archiving instead of keeping a restorable database. Archiving is correct: the obligation to keep data is not an obligation to keep software running. A live standby would have made recovery faster and would not have revealed the consumer any sooner.
- Read-only instead of blocking traffic. This is a real contributor and not the design decision. Blocking earlier would have surfaced the daily readers sooner; the annual job would still have been invisible, because it was not going to run either way inside thirty days.
- No rehearsed restart step. A genuine gap that affects recovery time after the failure. It has no bearing on whether the failure could happen, and fixing it alone leaves the same outage with a shorter duration.
When this is the wrong answer
For a system with only interactive users and no batch, no reporting and no regulatory extracts, thirty days is defensible and thirteen months is theatre that costs a year of licences and hosting. The rule is to set the window from the longest cycle you can evidence, and if the batch calendar is empty, say so in writing and switch it off.