Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
60 results for “Watermarks & Late Data”
Watermark Lag
How far behind the newest event a watermark is held, which trades result latency against how much late data is included.
Clock Drift
Device clocks diverging from real time, which corrupts ordering, expires certificates early or late, and produces data that appears to arrive from the past.
Capital One's Data Centre Exit
A major US bank closed all eight of its data centres and moved fully to public cloud, treating governance automation as the enabling technology rather than a constraint.
Change Data Capture
Publishing a stream of a database's row-level changes by reading its replication log, without modifying the application that owns it.
Control Plane and Data Plane
The separation between the machinery that makes changes to a system and the machinery that serves its traffic.
Cross-Zone Data Transfer
Charges incurred when data moves between availability zones within a region — invisible on architecture diagrams and a recurring surprise on cloud bills.
Data Catalog
A searchable inventory of datasets with their schema, owner, meaning, freshness, quality and classification.
Data Contract
An explicit, versioned, enforced agreement between a data producer and its consumers about schema, semantics, quality and change policy.
Data Discovery
Automatically scanning stores to find where sensitive data actually resides, as distinct from where the documentation says it should.
Data Lakehouse
A pattern that puts warehouse-style transactions, schema and governance on top of cheap open-format object storage.
Data Lineage
A record of where each dataset came from, what transformed it, and what depends on it — traced at table and ideally column level.
Data Lineage View
A view showing how data moves and is transformed through a system, independent of the components that do the moving.
Data Mesh
An organisational approach that gives domain teams ownership of their analytical data as a product, with a self-serve platform and federated governance.
Data Minimisation
Collecting and retaining only what a stated purpose requires, which reduces both regulatory exposure and breach impact at the same time.
Data Plane Proxy
The per-workload proxy that actually carries mesh traffic, applying mTLS, retries, timeouts, routing and telemetry outside the application.
Data Product Interface
The stable surface a data product exposes — its schema, its access path, its documentation and its guarantees — as distinct from the pipeline behind it.
Data Residency
A requirement that specific data be stored and sometimes processed only within a defined geography.
Data Retention Policy
A defined rule for how long each class of data is kept, where, and what happens at the end of it.
A streaming aggregation reports lower totals than the batch job it replaced. Both read the same source. What is likely happening?
The likely cause: late events dropped past the watermark The batch job reads a completed day and sees everything, including records that arrived hours after the
Your streaming aggregate reports 2% lower daily revenue than the batch reconciliation. Both are "correct". Explain what is happening and how you resolve it.
The likely cause: silently dropped late data The streaming job windows by event time and closes each window when the watermark passes. Records arriving after th
A CDO proposes moving to a data mesh because the central data team is a bottleneck with a nine-month backlog. How do you assess the proposal?
Agree with the diagnosis, examine the prescription The bottleneck is real and it is structural rather than a matter of capacity. A central team receives data fr
A GDPR erasure request arrives for a customer. Where does their data actually live, and what makes this expensive to retrofit?
Where the data lives Longer than people expect, and enumerating it is most of the work: Primary database · read replicas · caches · search indexes · analytical
A business sponsor asks for a real-time data platform because "the competition has one". Reporting is currently a nightly batch that lands at 06:00 and nobody has complained. How do you handle this?
Do not answer the technology question "Real time platform" is a solution, and it has arrived without a problem attached. Answering it directly leads either to a
A new platform must serve a public partner API, three internal front-ends with different data needs, and high-volume service-to-service traffic. Choose the API styles and defend the choice.
Resist "pick one" These are three different problems with three different consumers. Standardising on one style optimises for architectural tidiness at the expe
A query that ran in 50ms for two years now takes 90 seconds. Nothing was deployed and the data volume grew normally. What happened?
The most likely cause: a plan flip The optimiser's choice is a function of estimated row counts. As the data grows or its distribution shifts, an estimate cross
A regulated client requires that no traffic between their data centre and your SaaS platform traverses the public internet. Design the connectivity and justify the cost.
Two distinct requirements hiding in one sentence Traffic must not traverse the public internet — a routing requirement. The client must be able to demonstrate i
A regulator asks whether customer data is encrypted. The team says yes, disks are encrypted. Is that a sufficient answer?
What disk encryption actually protects against Someone obtaining the physical medium or a raw storage snapshot. In a cloud context that means a provider employe
Choose storage for four workloads: a Postgres data directory, user-uploaded images, a shared build cache, and seven years of audit records.
Postgres data directory — block storage It needs low latency random reads and writes and a filesystem, and it attaches to one instance. That is precisely block
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Streaming Data
Windowing, watermarks, late arrivals and exactly-once semantics.
Change Data Capture
Turning a database's replication log into a stream, and its coupling risk.
Client Caching & Data Layer
Stale-while-revalidate, invalidation and optimistic updates on the client.
Data Access Models
Role, attribute and purpose-based access over analytical data, and how they compose.
Data Architecture
General material on structuring, storing and governing data.
Data Catalog
Discovery, ownership and technical metadata, and why catalogues go stale.
Data Classification
Knowing which fields are regulated, because every control depends on it.
Data Contracts
Producers committing to schema, semantics and freshness, and breaking builds when they do not.
Data Governance
Ownership, lineage, quality, catalogues and who may see what.
Data Governance & Semantics
General material on ownership, meaning, quality and control of data at enterprise scale.
Data Lakes & Lakehouses
Open formats on object storage with transactional metadata on top.
Data Lifecycle & Retention
How long data is kept, where it ages to, and how it is actually deleted.
Data Mesh
Domain ownership, data as a product, self-serve platform, and federated governance.
Data Migration Strategies
Backfill, dual-write, reconciliation and verification.
Data Observability
Freshness, volume, schema and distribution monitoring for pipelines that fail silently.
Data Platform Architecture
General material on designing the analytical data estate end to end.
Data Platform Tenancy
Multiple domains on shared storage and compute, with separable access and cost.
Data Products
A dataset with an owner, an interface, an SLO, and consumers who can rely on it.
Data Quality Dimensions
Completeness, accuracy, timeliness, consistency, validity and uniqueness as testable claims.
Data Residency
Keeping data within a jurisdiction, including backups, logs and support access.
Data Sharing & Clean Rooms
Collaborating on data neither party may hand over, with computation as the interface.
Data Stewardship
The operating model that makes ownership a role rather than a slide.
Data Architecture
Where state lives, how it is modelled, replicated, partitioned and governed.
Data Governance & Semantics
Who owns data, what it means, whether it can be trusted, and who may see it.
Data Platform Architecture
The analytical estate: storage layout, ingestion, transformation and the compute that reads it.