Streaming Data
Windowing, watermarks, late arrivals and exactly-once semantics.
3 to work through
-
advanced
A live-streaming platform must compute real-time viewer counts, chat rate and moderation signals during events where concurrent viewers spike tenfold within minutes. What streaming architecture handles this, and what happens if the stream processor falls behind?
3 min answer -
advanced
A streaming aggregation reports lower totals than the batch job it replaced. Both read the same source. What is likely happening?
2 min answer -
advanced Multiple choice
A streaming aggregation stops emitting results. The job is healthy, consuming messages and using CPU. What is the most likely cause?
2 min answer
2 terms in this topic
Streaming Data Architecture
Processing unbounded data continuously, where the central problems are time semantics, lateness and state rather than throughput.
conceptWindowing
Grouping an unbounded stream into finite chunks so aggregation can produce results, defined over event time rather than arrival time.
Neighbouring topics
Data Architecture
General material on structuring, storing and governing data.
Relational Modelling
Normalisation, keys, constraints and the invariants a schema enforces.
NoSQL Stores
Key-value, document, wide-column and graph — what each buys and forbids.
Indexing
Designing indexes per query shape, and paying for them on every write.
Query Optimisation
Reading a plan, fixing statistics, and finding the real bottleneck.
Transactions & Isolation
ACID, isolation levels, and the anomalies each level permits.
Replication
Primaries, replicas, lag, and synchronous versus asynchronous durability.
Partitioning & Sharding
Splitting data across machines, and the one-way door of a partition key.
Caching Strategies
Cache-aside, read-through, write-through and where each belongs.
Cache Invalidation
Stampedes, penetration, staleness windows and versioned keys.
CQRS
Separating the write model from the read models that serve queries.
Event Sourcing
Storing the change log as the system of record, and what that costs forever.
Change Data Capture
Turning a database's replication log into a stream, and its coupling risk.
Data Warehousing
Dimensional modelling, star schemas and analytical workloads.
Data Lakes & Lakehouses
Open formats on object storage with transactional metadata on top.
ETL & ELT
Where transformation happens, and how much raw history you keep.
Data Governance
Ownership, lineage, quality, catalogues and who may see what.
Data Lifecycle & Retention
How long data is kept, where it ages to, and how it is actually deleted.
Polyglot Persistence
Choosing a store per workload, and the operational cost of variety.