You must remove a column from a published data product. The warehouse query log shows 180 distinct queries from 40 teams touching it in the last 90 days - 22 of them scheduled jobs and the rest ad hoc. Roughly how long does the retirement take end to end, and which assumption dominates the estimate?
Show the full answer Hide the answer
The assumptions, stated
- 22 scheduled jobs have an owning team and a known change process.
- 158 ad-hoc queries come from perhaps 60 to 90 individuals. In practice a third of them no longer need the query at all, a third will change it within days of being asked, and a third will not reply - people change teams, and a query in a notebook has no owner.
- Consuming teams plan on a two-week to one-quarter cadence, and a change that is not an incident enters that cadence rather than jumping it.
The arithmetic
Notice period is set by the slowest consumer you are willing to break, not by the work. One sprint of notice reaches the teams who read announcements; the long tail is only found by making the column visibly degrade. So:
- Week 0: announce, with the query log attached so each team sees its own queries.
- Weeks 1 to 4: the 22 scheduled jobs move. This is the easy, bounded part.
- Weeks 4 to 10: two brownouts - the column returns null for a fixed window, announced in advance, long enough to surface in a daily report. One brownout finds the dashboards; the second finds the monthly and quarterly jobs that the first one missed entirely.
- Weeks 10 to 14: remove. Keep the column physically present but null for one more cycle so a failure is a wrong number you can explain rather than a query that will not parse.
One quarter when the organisation already has a deprecation habit, two when this is the first time. The range is dominated by one thing.
Which assumption dominates
Not the query count. Whether a quarterly job exists is worth more than whether there are 180 queries or 400: a monthly reporting cycle means a single brownout cannot prove safety, and the calendar sets the floor. The second-biggest factor is whether you are permitted to brownout at all. If you cannot degrade the column deliberately, your discovery mechanism is email, and email does not find the long tail - you will remove it, break something unknown, and roll back.
Why the other options fail
- "Two to three weeks after emailing the authors." This is the standard plan and it fails on the third of authors who never reply and the jobs that run monthly. Email proves notice was given; it does not prove nothing depends on you.
- "Two to three quarters with hands-on help per team." Over-serving. Most consumers need the log line showing their query and a replacement expression, not a meeting. Offering 40 meetings is how a deprecation stalls for a year.
- "As long as retention requires." Confuses keeping data from exposing it. Retention is satisfied by the column existing in storage; the interface promise is what you are retiring.
When this is the wrong answer
If the column is an exposure - it carries personal data that should never have been published, or a regulator has asked - the timeline inverts. Remove it today, break the consumers, and send the query log afterwards. The whole courtesy structure above exists to protect consumers from avoidable breakage, and it is the wrong structure when the data itself is the problem.
Choose the courteous sequence unless the column is the risk. The brownouts cost the producing team about six extra weeks of calendar time and buy the long tail a chance to notice - a good trade for a column that is merely unwanted, and a bad one for a column that should never have been in production.