Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
16 results for “File Formats & Compaction”
Small File Problem
The degradation that follows from many tiny files — metadata operations dominating, compression failing, and columnar formats losing their advantage.
File-Based Integration
Exchanging data as files on a schedule via SFTP or shared storage — dated, still ubiquitous, and workable when its failure modes are handled explicitly.
Log Compaction
A retention policy that keeps only the most recent value for each key rather than deleting by age, so the log becomes a durable snapshot of current state.
Binary Serialisation
Encoding messages compactly using a schema rather than as self-describing text, trading readability for size and parse speed.
Dropbox's Move Off S3
Dropbox moved the majority of its file storage off Amazon S3 onto custom infrastructure, reporting savings that its S-1 filing put at roughly $75 million over two years.
Lakehouse Table Format
A metadata layer over object storage that adds atomic commits, snapshots and schema evolution to files that otherwise have none.
Message Translator
A component converting a message between two systems' formats, so neither has to adopt the other's model.
Terraform State
The file mapping declared resources to real infrastructure, without which the tool cannot tell what it already created — and which becomes critical infrastructure in its own right.
A core mainframe system with no API supports nightly batch file exchange only. The business needs near-real-time order status. Design the integration.
Establish the real constraint "No API" usually means no API the mainframe team will build on your timeline . Find out what exists: message queue interfaces, dat
A team is choosing between Iceberg, Delta and Hudi for a new lakehouse. How would you approach the decision?
The formats are more alike than the debate suggests All three provide the same core: atomic commits over object storage, snapshots and time travel, schema evolu
File Formats & Compaction
Columnar formats, the small-file problem, and the maintenance nobody schedules.
Message Formats
JSON, Protobuf, Avro — schema evolution and payload economics.
Open Table Formats
Iceberg, Delta and Hudi — transactions, snapshots and time travel over object storage.
Cloud Storage
Object, block and file storage, and the access patterns each suits.
Data Lakes & Lakehouses
Open formats on object storage with transactional metadata on top.
Regulatory Reporting Pipelines
Submissions with fixed deadlines, fixed formats, and a regulator who audits the lineage.