Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
60 results for “CDC Pipeline Design”
Log-Based Ingestion
Building the pipeline around a database's own change log — an initial snapshot followed by a continuous delta stream, with the two stitched together.
CDC Initial Snapshot
The consistent full copy taken when a CDC pipeline starts, before streaming begins — and the step that determines whether the target is correct.
Deployment Pipeline
The automated path from commit to production, structured so that each stage increases confidence and the whole is fast enough to be run on every change.
Design Review Trigger
The stated conditions under which a change requires security review, so that review capacity goes to what warrants it and everything else proceeds.
Domain-Driven Design
Modelling software around the business domain, with boundaries drawn where the language of the business changes.
Idempotent Pipeline
A pipeline whose task can be re-run for the same input window any number of times and produce the same result.
Log-Based CDC
Capturing changes by reading the database's own write-ahead log, which sees every change with no load on the source and no application involvement.
Pipeline Anomaly Detection
Monitoring row counts, distributions, freshness and schema for unexplained change, because data pipelines fail silently far more often than they error.
Pipeline Fan-Out
Splitting independent verification work to run in parallel, bounded by what genuinely has no ordering dependency on anything else.
Pipeline Orchestration
Coordinating the execution of data tasks by dependency rather than by clock, with retries, backfill and observability built in.
Pipeline Stage Contract
What each pipeline stage promises the next — an artifact of a stated shape plus a specific claim about it that has been verified.
Privacy by Design
Building privacy protections into a system's structure from the start, rather than adding controls to a design that already collects and keeps everything.
Query-Based CDC
Detecting changes by repeatedly querying for rows modified since the last run — simple, universally available, and lossy in specific ways.
Replay Pipeline
A single processing path that produces both live and historical results by re-running the same code over retained input, replacing the two-path Lambda arrangement.
Retrieval Pipeline Stages
The stages that turn a user question into grounded context — query processing, retrieval, reranking and assembly — each independently tunable.
A CDC pipeline feeding your warehouse falls three hours behind during a source system's batch job, and the source's transaction log retention is 24 hours. What is the risk and what do you change?
The immediate risk Lag consumes the retention window. At three hours behind against a 24 hour retention, you have 21 hours of margin. If the consumer stops enti
A core mainframe system with no API supports nightly batch file exchange only. The business needs near-real-time order status. Design the integration.
Establish the real constraint "No API" usually means no API the mainframe team will build on your timeline . Find out what exists: message queue interfaces, dat
A design review presents a new event-driven platform. What cost questions do you ask before approving it?
What the interviewer is testing Whether cost is part of your architecture review or an afterthought handled by finance later, and whether you know the specific
A downstream team needs to react to order changes. The order service can publish events, or they can consume CDC from its database. Which, and why?
The recommendation: published events, with CDC as the mechanism if needed The distinction that matters is what the consumer becomes coupled to . CDC consumed di
A nightly pipeline failed halfway and the retry produced duplicate rows. Walk me through fixing this properly.
The immediate problem The task appends rather than replacing its window , so a partial run followed by a retry writes some rows twice. The data is now wrong in
A payments API must guarantee that a network retry never charges a customer twice. Design the mechanism end to end.
Why this is unavoidable A client that times out on a POST cannot know whether the request succeeded. Not retrying risks a lost payment; retrying risks a duplica
A platform of 40 services has logs only, and incidents take hours to diagnose. Design the observability strategy and its rollout order.
Why logs alone fail at this size Logs answer "what happened in this service". They cannot answer "where did this request spend its time across twelve services",
A product catalogue page does 40,000 reads per second against a database that can serve 5,000. Walk me through the caching design, including what happens at 3 AM when the cache is empty.
What the interviewer is testing Whether you can design a cache including its failure modes, rather than saying "put Redis in front of it". The base design Cache
A regulated client requires that no traffic between their data centre and your SaaS platform traverses the public internet. Design the connectivity and justify the cost.
Two distinct requirements hiding in one sentence Traffic must not traverse the public internet — a routing requirement. The client must be able to demonstrate i
A team proposes exposing their service's database change stream via CDC so other teams can consume it, avoiding the work of building an event API. What is your assessment?
Name what is actually being proposed The proposal is to publish the service's internal schema as its integration contract. CDC does not emit domain events; it e
A team shows you a design with eight services. Without knowing the domain, what questions tell you whether the boundaries are right?
What the interviewer is testing Whether you can evaluate a structure from its properties rather than needing to be a domain expert in every system you review. T
A team's pipeline takes 80 minutes from commit to a deployable artifact and engineers have stopped watching it. How do you approach fixing it?
Measure per stage before changing anything Get the duration of every stage over the last few hundred runs, with the distribution rather than the mean. The shape
A twenty-year-old core system supports most of the business. It is on unsupported technology, three people understand it, and the last replacement attempt was abandoned after two years. Design the programme.
Learn from the abandoned attempt first The most important input is why the last one failed, and the reasons are usually structural rather than technical: a big
An estate has database passwords in environment variables across 200 services. Design the migration to a secrets manager.
Sequence it by risk, not by convenience Phase 0 — stop the bleeding. Secret scanning in CI and on the existing repositories, blocking new commits containing cre
Architecture decisions in your organisation are either ignored or bottlenecked in a review board. Design something better.
Why both failure modes have the same cause A review board is a synchronous, low bandwidth, high latency channel: teams wait, decisions are made by people distan
Classify DynamoDB, Spanner and Cassandra under PACELC, and say which half of the classification you would actually design around.
The classifications Store Partition Else Why DynamoDB (default reads) PA EL Serves from any replica; eventually consistent reads avoid a quorum round trip Cassa
Design a URL shortener handling 100 million new links per month and 10 billion redirects. Where is the real difficulty?
What the interviewer is testing The classic warm up. What is being assessed is not whether you can shorten a URL — it is whether you do capacity arithmetic befo
Design a webhook delivery system for a platform with 10,000 customers. What are the hard parts?
Delivery, and its failure modes Persist the event first, deliver asynchronously. Delivery in the request path couples your latency and availability to every cus
Design an order submission API that is safe when the client cannot tell whether its request succeeded. What exactly do you store, and when?
What the interviewer is testing Whether you know that "make it idempotent" is a design with specific failure modes, not a checkbox. The core design The client g
Design rate limiting for a multi-tenant API where a single customer's traffic spike currently degrades service for everyone.
Name the problem precisely This is the noisy neighbour problem. Rate limiting is one control for it, and on its own it is incomplete — a customer within their l
Design the audit logging for a system handling financial transactions. What is logged, where does it go, and what makes it hold up?
What is logged Significant actions only , defined explicitly rather than logging everything — an audit trail nobody can search is not usable evidence: Authentic
Design the network layout for a three-tier application in one cloud region. What are the decisions you cannot easily change later?
What the interviewer is testing Whether you know which network decisions are cheap and which are effectively permanent. This is a knowledge question with a clea
Design the timeout configuration for a request that passes through gateway, orders, pricing and inventory. What numbers, and what rule generates them?
The rule that generates the numbers One budget at the edge, decreasing inward, with room for a retry at exactly one layer. Start from what the caller will actua
In a design review, a respected senior engineer proposes an approach you believe is wrong. The room defers to them. How do you handle it?
What the interviewer is testing Whether you can be effective without authority, which is most of the architect's job. It is also testing whether you assume you
Incidents at your company are chaotic: unclear ownership, no communication, and postmortems that produce nothing. Design the improvement.
Roles, so that coordination exists The failure is everyone investigating and nobody coordinating. Ten people debugging is slower than three debugging and one di
Leadership asks for "five nines" across the platform. Engineering says it is impossible. Design the response.
Reframe the request "Five nines" is almost never what the business actually wants. 26 seconds of downtime per month is a number chosen for its rhetorical weight
Maersk rebuilt roughly 4,000 servers and 45,000 PCs in about ten days after NotPetya in 2017, and recovered its directory only because one data centre had been offline during the attack. What does this say about DR design?
The case, as publicly reported In June 2017 the NotPetya malware — destructive rather than financially motivated — propagated through Maersk's network, encrypti
Mobile conversion is 40% below desktop. Product blames the design; analytics shows users leaving before the page is usable. How do you investigate?
Get field data before touching anything A laboratory run on a developer machine measures one configuration, usually the fastest one anyone uses. Real traffic in
Placing an order must reserve stock, charge the card and create a shipment across three services. Design it, and justify why not a distributed transaction.
First: question the boundary A transaction spanning three services often means one invariant has been split across three owners. Before designing a protocol, ch
Product wants to add "customers who bought this also bought" using purchase history. What does privacy by design require here?
The first question is lawful basis, not architecture Purchase history was collected to fulfil orders. Using it for recommendations is a new purpose , and purpos
CDC Pipeline Design
Building on a change stream: snapshot plus delta, tombstones, and merge into the target.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Control Design vs Operation
A control that is well designed and never runs fails exactly like one that is absent.
Design Authority
How an ARB should decide, what it should not review, and how it avoids becoming a queue.
Design Patterns
Reusable solutions at code level, and when they become ceremony.
Design Systems
Components as a versioned internal product, with adoption and deprecation like any API.
Domain-Driven Design
Ubiquitous language, bounded contexts and context mapping.
Human-in-the-Loop Design
Meaningful review rather than a rubber stamp, and designing against automation bias.
Performance Test Design
Workload models, warm-up, think time, and the distribution the average hides.
Pipeline Architecture
Stages, fan-out, caching, and the difference between a pipeline and a long script.
Pipeline Secrets
Short-lived credentials, workload identity, and why the CI system is a prime target.
Privacy by Design
Data minimisation, default protection and purpose binding as structural decisions.
REST Design
Resources, uniform methods, status codes and statelessness.
Secure API Design
Object-level authorisation, input validation and safe error responses.