Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
5 to work through
-
intermediate
A team needs to change the schema of an event that many consumers read. What is the safe process?
2 min answer -
advanced
A producer added a required field. New consumers work. A consumer replaying from three months ago fails. Why, and what is the policy fix?
2 min answer -
advanced
An event schema must change while consumers and retained history both depend on the old version. How is this managed?
2 min answer -
advanced
Review this. A payments platform runs 300 topics with the schema registry in FULL compatibility mode for every subject, a rule that each new field must be optional with a default, and a schema-change request that goes to an architecture review board with a three-week turnaround. Engineers have started putting new data into a generic metadata map to avoid the queue. What would you remove, what would you change, and what would you leave alone?
3 min answer -
advanced
Walk me through retiring a field from an event consumed by around 40 teams, where the field is populated from a system being decommissioned in six months.
2 min answer
3 terms in this topic
Schema Compatibility Mode
The rule stating which schema changes a registry will accept, which encodes whether producers or consumers are expected to upgrade first.
conceptSemantic Schema Break
A change that keeps a field's wire format valid while changing what its values mean, so every compatibility check passes and every consumer computes …
practiceStreaming Schema Evolution
Changing the shape of events in a topic that has multiple independently deployed producers and consumers, plus retained history that still has to be …
Neighbouring topics
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.