advanced 3 min answer

Two teams write to the same set of database tables from two separately deployed services. List the problems this creates and describe how you would evolve out of it.

couplingdata-ownershipprinciplesmigrationamazon
Show the full answer Hide the answer

What is being tested

Whether you recognise a shared writable database as an architectural failure rather than an inconvenience, and whether you can sequence a migration out of it without a big-bang rewrite.

The problems, roughly in order of how much they cost

No owner of the schema. A column's meaning is defined by whichever service happens to write it most, and nobody can change it without an archaeology exercise across two codebases.

Migrations require coordination. Any change that is not purely additive needs both services deployed in a specific order, in a change window, with a rollback plan that must also be coordinated. Deployment independence — usually the reason both services exist — is gone.

Invariants cannot be enforced. If service A believes an order must always have at least one line item and service B does not know that, the invariant is a convention. It will be violated.

Contention and locking cross team boundaries. A long-running report in one service holds locks that time out transactions in the other. The team suffering the outage cannot fix it, because the cause is in code they do not own.

Refactoring becomes impossible. You cannot rename, denormalise, or move a table, because you cannot see all the queries.

This is why Amazon's internal mandate that teams may only access one another's data through service interfaces is a principle rather than a guideline. It is expensive — cross-team joins become network calls, reporting becomes a pipeline, and every interface must be versioned for consumers you do not control — and the reason it survived is that the alternative silently converts many teams into one team.

The migration, in stages

  1. Establish ownership. Decide which service owns which tables. Not which team wrote the code first — which team owns the behaviour. This is a business decision and it is the hard part.
  2. Make the non-owner's access explicit. Move every foreign read and write behind an API on the owning service. Do this incrementally, one query at a time, behind feature flags. At this stage the database is still shared but the access is not.
  3. Break the physical share. Now that all traffic goes through the owner, the tables can be moved to a separate schema or instance. The non-owner never notices.
  4. Replace read-heavy access with events or a read model. If the non-owner was reading for display or reporting, a synchronous API is often the wrong replacement — publish changes and let them keep a local projection.
  5. Handle the reports. The reporting query that joined both sets of tables is the last and most painful item. It usually becomes a data pipeline into a warehouse, and it is worth budgeting for this explicitly because it is where these migrations stall.

What a strong answer adds

Naming what gets worse: latency on paths that used to be a join, new failure modes when the owning service is down, eventual consistency in the projections, and the fact that step 5 has no cheap answer. And noting that if the two services always change together anyway, the correct fix may be to merge them rather than to distribute them further.