Portfolio ↗

ETL & ELT

Where transformation happens, and how much raw history you keep.

3Questions
9Flashcards
3Terms
Drill this topic All of Data Architecture
Questions

3 to work through

  1. intermediate

    A grocery platform ingests catalogue and inventory feeds from hundreds of retailers, each with different formats, quality and update frequency. Should transformation happen before loading or after, and where should data quality be enforced?

    2 min answer
  2. intermediate

    A nightly pipeline failed halfway and the retry produced duplicate rows. Walk me through fixing this properly.

    2 min answer
  3. advanced

    A daily pipeline has been producing wrong numbers for three weeks due to a logic bug. What must be true of the architecture for the fix to be a one-day job?

    3 min answer
Terminology

3 terms in this topic

practice

ETL and ELT

Whether to transform before loading or after — a choice that follows from where compute is cheap and who owns the transformation.

 practice

Idempotent Pipeline

A pipeline whose task can be re-run for the same input window any number of times and produce the same result.

 practice

Pipeline Orchestration

Coordinating the execution of data tasks by dependency rather than by clock, with retries, backfill and observability built in.
Data Architecture

Neighbouring topics

Data Architecture

General material on structuring, storing and governing data.

10 quiz 31 cards 25 terms

Relational Modelling

Normalisation, keys, constraints and the invariants a schema enforces.

3 quiz 10 cards 4 terms

NoSQL Stores

Key-value, document, wide-column and graph — what each buys and forbids.

5 quiz 13 cards 4 terms

Indexing

Designing indexes per query shape, and paying for them on every write.

5 quiz 12 cards 6 terms

Query Optimisation

Reading a plan, fixing statistics, and finding the real bottleneck.

3 quiz 11 cards 4 terms

Transactions & Isolation

ACID, isolation levels, and the anomalies each level permits.

3 quiz 10 cards 4 terms

Replication

Primaries, replicas, lag, and synchronous versus asynchronous durability.

4 quiz 14 cards 6 terms

Partitioning & Sharding

Splitting data across machines, and the one-way door of a partition key.

6 quiz 12 cards 10 terms

Caching Strategies

Cache-aside, read-through, write-through and where each belongs.

7 quiz 15 cards 6 terms

Cache Invalidation

Stampedes, penetration, staleness windows and versioned keys.

3 quiz 7 cards 2 terms

CQRS

Separating the write model from the read models that serve queries.

4 quiz 10 cards 5 terms

Event Sourcing

Storing the change log as the system of record, and what that costs forever.

5 quiz 13 cards 4 terms

Change Data Capture

Turning a database's replication log into a stream, and its coupling risk.

5 quiz 14 cards 5 terms

Data Warehousing

Dimensional modelling, star schemas and analytical workloads.

4 quiz 13 cards 3 terms

Data Lakes & Lakehouses

Open formats on object storage with transactional metadata on top.

4 quiz 11 cards 4 terms

Streaming Data

Windowing, watermarks, late arrivals and exactly-once semantics.

3 quiz 10 cards 2 terms

Data Governance

Ownership, lineage, quality, catalogues and who may see what.

3 quiz 9 cards 4 terms

Data Lifecycle & Retention

How long data is kept, where it ages to, and how it is actually deleted.

4 quiz 11 cards 5 terms

Polyglot Persistence

Choosing a store per workload, and the operational cost of variety.

6 quiz 12 cards 4 terms