advanced 3 min answer

Customer balances are served from a change-data-capture copy of a mainframe rather than from the mainframe itself. What happens, hour by hour, on the night the month-end close posts a large batch and the capture stream falls forty minutes behind?

mainframecdcoffloadstalenessmonth-end
Show the full answer Hide the answer

Second by second, what happens

22:00. The close begins. Interest, fees and accruals post as a dense batch, and the rate of changes on the source jumps by an order of magnitude over a normal evening.

22:10. The capture stream is throughput-bound on the apply side, not the read side, because the batch arrives faster than the target can write it. Lag starts to climb linearly, which is the signature of a sustained overload rather than a blip.

22:40. Lag reaches forty minutes. Every balance served from the offload is now a pre-close balance. Nothing has errored. The offload is healthy by every dashboard it has: it is up, it is fast, and it is wrong.

22:45. The first divergence a human sees. Statements generated on the mainframe show post-close figures; the app and the call-centre screen show pre-close figures. Agents start telling customers the app is broken.

23:00. The serious failure. Any process that reads from the offload and then writes to the mainframe — a fee waiver, a limit check, a payment authorisation — is now deciding on stale state. The mainframe will accept the write and the decision will have been made against a balance that no longer exists. This is where a freshness problem becomes a correctness problem.

Where it amplifies

The batch does not slow down because the offload is behind. There is no backpressure from a read replica to its source, which is the whole reason offloading is attractive and also why lag is unbounded. If the capture pipeline retries or restarts, it re-reads from its last committed position and falls further behind, so recovery competes with the batch that caused the problem.

What stops it

Not capacity. Sizing the apply path for the close burst costs peak batch throughput every day of the month to protect a few hours, and that overhead buys nothing on the other 29 days. The mechanism is a published freshness contract that the offload enforces on itself:

  • The offload exposes its own lag as a first-class value and stamps every response with an "as at" time.
  • Reads are classified. Display balances tolerate lag and are labelled. Any read that feeds a write decision is routed to the mainframe or refused, which is the rule that prevents the 23:00 failure.
  • When lag exceeds the contracted threshold, the offload sheds the tolerant traffic to a degraded mode rather than silently serving it, so the failure is visible.
  • The close window is declared to the offload in advance, so elevated lag during it is expected behaviour rather than an alert nobody can action.

What would have to be true for it to self-heal

Apply throughput would have to exceed the close's peak change rate with margin, so lag drains inside the window. Measure both before promising anything: if the close posts on the order of millions of rows in an hour and the target applies tens of thousands per second, it drains; if it applies thousands per second, it does not, and no amount of monitoring changes that.

When this is the wrong answer

If every consumer of the offload is analytical and a day old is acceptable, the correct design is a nightly extract after the close rather than a continuous stream. Change capture buys freshness, and freshness you do not need is a pipeline you have to operate in production every hour of the year. Choose continuous capture only if a named consumer's requirement is measured in seconds; otherwise prefer the extract, which has no lag concept to get wrong because it is either yesterday's file or it is missing.