intermediate 3 min answer

Three services have been extracted from a monolith. All three still read the monolith's database directly, which the team describes as a temporary arrangement. The monolith's next release renames `customers.country` to `customers.country_code` with a compatibility view left behind for itself. What happens, minute by minute, and what mechanism would have stopped it?

shared databasecouplingschema changedistributed monolithcontracts
Show the full answer Hide the answer

Minute by minute

T+0: The migration runs. The column is renamed. The monolith is fine - its own code was changed in the same release and its compatibility view resolves the old name.

T+0 to T+2 min: The three extracted services have open connections and cached prepared statements. Statements referencing country begin failing as their plans are invalidated and re-prepared. Any service using SELECT * with strict row mapping fails differently and later: it keeps working until something reads the missing field.

T+2 to T+10 min: Each service's error behaviour decides what the user sees. The one with a circuit breaker around its data access degrades to a cached country and keeps serving. The one that treats a query error as a 500 starts failing checkout. The one that retries without backoff adds load to a database that is already doing a migration.

T+10 min: Alerts fire on the three services. None of them fires on the monolith, which is healthy, and the monolith's release is therefore not the obvious suspect. The deploy that caused the incident is not in the incident's blast radius.

Where it amplifies

The compatibility view is the trap. It made the change look backward compatible, because the author verified compatibility for the only consumer they could see. A shared schema has no list of consumers, so "compatible" is a claim about the author's knowledge, not about the system.

What the user sees

Depending on which service owns which journey: a blank country field, a failed address validation, or a 500 at checkout. A delivery platform of the kind Zomato operates would see order placement fail for new addresses while everything else looked normal, which is the hardest shape of incident to triage.

What stops it

Not discipline, and not a review step. A mechanism:

  1. No service reads another's tables. The extracted services read a published interface - an API, an event stream, or at minimum a per-consumer database view owned by the monolith's team and covered by a test. A view is the cheap option: it turns the implicit dependency into a named one somebody must break on purpose.
  2. Expand-contract on every schema change, with the contract step gated on evidence that nothing reads the old shape. Add country_code, write both, migrate readers, then drop country once query logs and view access show zero use for a full business cycle.
  3. Grants as the enforcement. Revoke the extracted services' direct table access. If the dependency cannot be expressed through the interface, that is information about the decomposition, discovered in a pull request rather than in production.

What would have to be true for it to self-heal

Nothing in this design heals: the data is gone from the old name. Self-healing requires the old shape to still exist, which is exactly what expand-contract preserves. That is the argument for expand-contract - not elegance, but the existence of a rollback.

When this is acceptable anyway

For about the first 3 to 6 months of an extraction, with 2 or 3 consumers and one team owning all of them, direct reads cost nothing and an interface costs weeks. The decision rule is ownership, not time: direct reads are acceptable while one team can change every reader in a single release, and unacceptable the moment two teams must co-ordinate. The price of the shortcut is paid once, as an incident, and it is charged to the team that did not make the change - which is why the shortcut survives review. A per-consumer view is the cheap middle: a few hours of work that converts an invisible dependency into one somebody has to break deliberately.