Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
60 results for “Streaming Schema Evolution”
Schema Compatibility Mode
The rule stating which schema changes a registry will accept, which encodes whether producers or consumers are expected to upgrade first.
Schema Evolution
Changing a table's structure over time while keeping existing data readable and existing consumers working.
Compatibility Mode
The registry setting that defines which schema changes are permitted — backward, forward, full, or none — and therefore what upgrade order is safe.
Event Upcasting
Transforming an old event's stored form into the current shape as it is read, so historical events remain replayable after the schema changes.
Lakehouse Table Format
A metadata layer over object storage that adds atomic commits, snapshots and schema evolution to files that otherwise have none.
Open Table Format
A metadata layer over files in object storage that supplies ACID transactions, schema evolution and time travel — the thing that turns a data lake into a lakehouse.
Blue-Green Database Schema
The constraint that makes fast rollback actually work — both application versions must be able to run against one schema at the same time.
Log Schema Consistency
Enforcing the same field names, types and semantics for structured log records across every service, so cross-service queries are possible.
Star Schema
A dimensional model with one central fact table of measurements surrounded by denormalised dimension tables describing them.
Tool Schema Design
Defining the tools available to a model — names, descriptions, parameters and errors — in a way that makes correct selection likely.
gRPC Streaming
Four call patterns — unary, server streaming, client streaming and bidirectional — built on HTTP/2 streams.
Backward Compatibility
The property that a new version of a producer continues to work with clients written against the old version.
Binary Serialisation
Encoding messages compactly using a schema rather than as self-describing text, trading readability for size and parse speed.
CDC Initial Snapshot
The consistent full copy taken when a CDC pipeline starts, before streaming begins — and the step that determines whether the target is correct.
Change Data Capture
Publishing a stream of a database's row-level changes by reading its replication log, without modifying the application that owns it.
Consumer Group
A set of consumers that cooperatively read one stream, with each partition assigned to exactly one member, so the group collectively processes every message once.
Consumer Lag
How far behind the newest record a consumer is, expressed in time rather than in message count, and its rate of change.
Data Catalog
A searchable inventory of datasets with their schema, owner, meaning, freshness, quality and classification.
Data Contract
An explicit, versioned, enforced agreement between a data producer and its consumers about schema, semantics, quality and change policy.
Data Lakehouse
A pattern that puts warehouse-style transactions, schema and governance on top of cheap open-format object storage.
Data Product Interface
The stable surface a data product exposes — its schema, its access path, its documentation and its guarantees — as distinct from the pipeline behind it.
Event Stream
An append-only, retained log of events that many independent consumers read at their own position, and can re-read.
Expand-Contract Migration
Changing a schema in additive steps that keep old and new code both working, so deployment and migration never have to be simultaneous.
Freshness Requirement
How stale data may be before the decision it supports degrades — the only question that justifies streaming over batch.
GraphQL
A query language and runtime where the client specifies exactly which fields it needs, against a typed schema, usually via a single endpoint.
LinkedIn and the Origin of Kafka
Kafka was built to replace point-to-point data integration between many systems with a single durable log that any system could publish to and any number could read.
Log Compaction
A retention policy that keeps only the most recent value for each key rather than deleting by age, so the log becomes a durable snapshot of current state.
A streaming aggregation reports lower totals than the batch job it replaced. Both read the same source. What is likely happening?
The likely cause: late events dropped past the watermark The batch job reads a completed day and sees everything, including records that arrived hours after the
A team wants to move a workload from PostgreSQL to a document store because "the schema keeps changing". What do you ask?
The questions 1. What is actually changing — the shape, or the schema management process? "The schema keeps changing" usually means migrations are painful, not
A vendor claims their streaming platform provides exactly-once processing. How do you evaluate the claim?
Ask where the guarantee ends Nearly always at the platform's boundary. Within it, state and offsets commit together, so internal state reflects each input once.
Two services share a database because it was faster to build that way. Both teams now block each other on every schema change. What do you do?
Name the actual coupling They are not two services. They are one deployable unit split across two repositories, with the coupling moved from code — where a comp
You are handing a streaming pipeline to an operations team who have never run one. What must exist before they accept it?
The signals they will be paged on Consumer lag in time , not in messages — ten thousand messages means nothing without the rate. And alert on the derivative : g
Your streaming aggregate reports 2% lower daily revenue than the batch reconciliation. Both are "correct". Explain what is happening and how you resolve it.
The likely cause: silently dropped late data The streaming job windows by event time and closes each window when the watermark passes. Records arriving after th
A CDC pipeline feeding your warehouse falls three hours behind during a source system's batch job, and the source's transaction log retention is 24 hours. What is the risk and what do you change?
The immediate risk Lag consumes the retention window. At three hours behind against a 24 hour retention, you have 21 hours of margin. If the consumer stops enti
A business sponsor asks for a real-time data platform because "the competition has one". Reporting is currently a nightly batch that lands at 06:00 and nobody has complained. How do you handle this?
Do not answer the technology question "Real time platform" is a solution, and it has arrived without a problem attached. Answering it directly leads either to a
A release must rename a heavily used database column and ships tonight. The team proposes doing the rename in the deployment. What is wrong, and what do you propose?
What is wrong A rename is not additive. During a rolling deployment both versions of the application run simultaneously: one expects the old name, one the new.
Consumer lag on a Kafka topic grows during peak and does not recover overnight. You add consumers and nothing improves. Why?
The mechanism Each partition is assigned to exactly one consumer within a group. With ten partitions and ten consumers, an eleventh consumer joins the group, is
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Message Formats
JSON, Protobuf, Avro — schema evolution and payload economics.
Event Streaming
Retained ordered logs, consumer offsets, partitions and replay.
Schema Registry
Enforcing compatibility on events the way CI enforces it on code.
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Streaming Data
Windowing, watermarks, late arrivals and exactly-once semantics.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Data Contracts
Producers committing to schema, semantics and freshness, and breaking builds when they do not.
Data Observability
Freshness, volume, schema and distribution monitoring for pipelines that fail silently.
Database Migration Under CD
Expand-contract, backwards-compatible schema change, and migrations that cannot roll back.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Dimensional Modelling
Facts, dimensions, grain, and the star schema's continued relevance.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.