Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
60 results for “Streaming & Real-Time Data”
Unbounded Dataset
Input with no known end, which removes the option of waiting for completeness and forces every aggregate to be provisional.
Clock Drift
Device clocks diverging from real time, which corrupts ordering, expires certificates early or late, and produces data that appears to arrive from the past.
Sub-Second Aggregation Store
A database built to ingest continuously and answer aggregate queries over recent data in milliseconds, occupying the gap between OLTP and the warehouse.
Watermark
A moving assertion that no events older than a given event-time will arrive, which is what allows an event-time window to be closed and emitted.
Windowing
Grouping an unbounded stream into finite chunks so aggregation can produce results, defined over event time rather than arrival time.
Change Data Capture
Publishing a stream of a database's row-level changes by reading its replication log, without modifying the application that owns it.
Data Minimisation
Collecting and retaining only what a stated purpose requires, which reduces both regulatory exposure and breach impact at the same time.
Field Data Versus Lab Data
The difference between what real users on real devices experience and what a synthetic run measures, and why the second is systematically optimistic.
Synthetic Data Fidelity
How closely generated data reproduces the shape, distribution and awkwardness of the real thing, which decides what the data can validly be used for.
Time Travel
Querying a table as it existed at a previous version or timestamp, made possible by keeping the metadata and files of prior commits.
Bandwidth-Delay Product
Bandwidth multiplied by round-trip time — the amount of data that must be in flight to keep a link fully utilised.
CDC Initial Snapshot
The consistent full copy taken when a CDC pipeline starts, before streaming begins — and the step that determines whether the target is correct.
Consumer Lag
How far behind the newest record a consumer is, expressed in time rather than in message count, and its rate of change.
Denormalisation
Deliberately duplicating data across records to make reads cheap, accepting the write-time cost of keeping copies in step.
Discord's Message Store Migrations
Discord moved from MongoDB to Cassandra to ScyllaDB as message volume grew from millions to trillions, each time for a specific and different reason.
Freshness Requirement
How stale data may be before the decision it supports degrades — the only question that justifies streaming over batch.
LinkedIn and the Origin of Kafka
Kafka was built to replace point-to-point data integration between many systems with a single durable log that any system could publish to and any number could read.
Log-Based CDC
Capturing changes by reading the database's own write-ahead log, which sees every change with no load on the source and no application involvement.
Netflix's Recommendation Architecture
Netflix splits personalisation into offline, nearline and online layers so that expensive computation happens ahead of time and the request path stays fast.
Open Table Format
A metadata layer over files in object storage that supplies ACID transactions, schema evolution and time travel — the thing that turns a data lake into a lakehouse.
RTO and RPO
How long recovery may take (RTO) and how much data may be lost (RPO), the two numbers that determine the cost of a resilience design.
Retrieval-Augmented Generation
Retrieving relevant documents at query time and putting them in the model's context, so answers are grounded in your data rather than in training data.
Schema Evolution
Changing a table's structure over time while keeping existing data readable and existing consumers working.
Serving Latency Budget
The end-to-end time from an event occurring to its effect being queryable, allocated across ingest, processing and serving.
Slowly Changing Dimension
A strategy for handling attributes that change over time, deciding whether history is preserved and how facts attach to the correct version.
Steward Accountability
Making stewardship a defined role with time, authority and consequences, rather than a title added to somebody's existing job.
Stream Enrichment
Attaching reference data to a stream by lookup against a materialised table rather than by joining two unbounded streams.
A business sponsor asks for a real-time data platform because "the competition has one". Reporting is currently a nightly batch that lands at 06:00 and nobody has complained. How do you handle this?
Do not answer the technology question "Real time platform" is a solution, and it has arrived without a problem attached. Answering it directly leads either to a
A streaming aggregation reports lower totals than the batch job it replaced. Both read the same source. What is likely happening?
The likely cause: late events dropped past the watermark The batch job reads a completed day and sees everything, including records that arrived hours after the
A core mainframe system with no API supports nightly batch file exchange only. The business needs near-real-time order status. Design the integration.
Establish the real constraint "No API" usually means no API the mainframe team will build on your timeline . Find out what exists: message queue interfaces, dat
Quarterly access reviews take two weeks of manager time and everyone approves everything. How do you make this a real control?
Recognise what it currently is A review where everything is approved is not producing a decision; it is producing a record of a decision shaped activity. Two we
Netflix personalises an entire home page in the time a TV takes to draw a screen. How, given that scoring every title for every member on request is impossible?
What the interviewer is testing Whether you reach for the offline/online split — the single most reusable idea in latency critical personalisation — rather than
Your streaming aggregate reports 2% lower daily revenue than the batch reconciliation. Both are "correct". Explain what is happening and how you resolve it.
The likely cause: silently dropped late data The streaming job windows by event time and closes each window when the watermark passes. Records arriving after th
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
Streaming Data
Windowing, watermarks, late arrivals and exactly-once semantics.
Real User Monitoring
Field data from real devices and networks, against the synthetic run that looked fine.
Synthetic Data
Generating data with the shape and edge cases of the real thing, and where it misleads.
Open Table Formats
Iceberg, Delta and Hudi — transactions, snapshots and time travel over object storage.
Query Optimisation
Reading a plan, fixing statistics, and finding the real bottleneck.