advanced 3 min answer Multiple choice

A payments service publishes PaymentCaptured to a topic with four subscribers. A fifth team needs the same event plus the payer's full name and address, which the producer does not emit because those fields are restricted. The fifth team proposes subscribing and then calling the producer per event to fetch them - about 900 extra reads a second. Which design fits?

pub-subtopic designauthorisationpiistream join
Pick one
Show the full answer Hide the answer

The deciding property

A topic is the unit of authorisation in every broker worth using, so the sensitivity of a field decides which topic it belongs in - not which entity it belongs to. That single sentence settles the design.

Four subscribers are already granted read on PaymentCaptured. Any design that adds restricted fields to that topic grants those fields to all four, and the grant is invisible because nothing about their consumption changes. Nobody reviews an entitlement they did not ask for.

So: a second topic keyed by the same payment id, carrying only the restricted fields, with its own access-control list and its own subscriber roster. The fifth team joins the two streams on the key. A keyed join with a bounded wait is ordinary stream work, and the access review becomes a readable list of subscribers per topic, which is the artifact an auditor wants and the thing nobody can produce when fields are mixed.

Splitting also buys independent retention, which is not a side benefit. Restricted data on a 7-day retention and the base event retained forever is only expressible if they are separate topics.

Why the other options fail

  • Add the fields and restrict the topic to vetted subscribers. This requires re-vetting four live subscribers, and the first one that fails vetting must be removed from a topic its service depends on. In practice nobody is removed, the exception is documented, and the restriction becomes prose. It is the correct answer only when every current subscriber already holds that entitlement, in which case there was never a second roster to maintain.
  • Call the producer per event and add a read replica. 900 reads a second is affordable, which is exactly why this gets proposed. It makes the consumer's throughput depend on the producer's availability, which is the coupling pub/sub was adopted to remove, and it puts the restricted field one GRANT away from anyone with replica access. It also breaks on replay: rebuilding from a year of events means a year of lookups against today's producer at whatever rate the replay runs.
  • Read change data capture from the producer's database. This works on the first day and makes the producer's table schema a public interface, so a column rename becomes a consumer outage with no API change to review. It also exposes every column in the table, which is more data than the fifth team asked for and the opposite of the goal. CDC is the right tool when you own both sides and the producer genuinely cannot be changed.

What this costs

Two topics to operate, two schemas to evolve together, and a join that can be one-sided when one stream lags. The fifth team needs an explicit decision about what to do when the restricted record never arrives: hold the event, process without the fields, or dead-letter. That decision is work the single-topic design appeared to avoid by making it somebody else's problem.

When not to split and what else would flip it

If this changes Choose Because
Every subscriber needs the restricted fields One topic with a tighter grant list There is no second roster to maintain and no join to write
One subscriber needs them on a handful of events a day An authorised per-event call A second topic is more machinery than the traffic justifies
The restricted fields change independently of the payment Two topics regardless They are separate facts with separate lifecycles and separate retention