Data Lakes & Lakehouses: Apache Iceberg, Delta Lake, & Parquet

Historically, organizations managed two separate data architectures:

1. The Data Lake (Hadoop/S3): Stored petabytes of raw files (CSV, JSON, Parquet) at extremely low cloud storage costs, but suffered from severe reliability issues: no ACID transactions, no atomic multi-file writes, inconsistent directory listings, and accidental data corruption during concurrent writes.
2. The Data Warehouse (Snowflake, Teradata): Provided fast, secure, ACID-compliant SQL queries, but locked data into proprietary formats at high compute and storage costs.

The Hive Metastore Problem:

In classic data lakes, table partitions were mapped directly to filesystem directories (s3://bucket/table/year=2026/month=09/). The Hive Metastore tracked directory paths. Because S3 is an object store without atomic rename operations:

• A multi-gigabyte query had to issue expensive LIST API calls across millions of S3 objects, causing multi-minute query latency before scanning any data.
• Concurrent writes had no atomic isolation; failed ETL jobs left orphaned files that corrupted downstream queries.

The Lakehouse Solution:

A Lakehouse provides the best of both worlds: open, cheap object storage (Parquet files on S3) paired with an open transactional table format (Apache Iceberg, Delta Lake, Apache Hudi) that brings full ACID transactions, metadata-level pruning, and time travel directly to the data lake.

Apache Parquet is an open-source, columnar storage format optimized for deep analytics and complex nested data structures:

• File Header & Footer Structure: Every Parquet file begins with a 4-byte magic number PAR1 and terminates with a comprehensive File Footer. The footer contains all schema definitions, column statistics, dictionary locations, and row group offsets. Because the footer is read first, query engines read only the file metadata before streaming columnar chunks.
• Row Groups (Typically 128MB to 512MB): Data is divided into horizontal row groups. Within each row group, data is partitioned by column into Column Chunks.
• Pages (Typically 1MB): Column chunks are divided into indivisible data pages containing dictionary values, run-length-encoded repetition levels (Dremel encoding for nested JSON structures), and compressed payloads (ZSTD, Snappy, LZ4).
• Embedded Statistics for Push-Down Filtering: Every column chunk footer records the exact min_value, max_value, null_count, and distinct count. If a query filters on WHERE user_id = 9521, and the chunk footer declares [min: 100, max: 5000], the engine skips reading the entire 128MB chunk without decompressing a single byte.

Apache Iceberg decouples table definitions from physical directory paths by organizing table state into a hierarchical, immutable metadata tree:

The Iceberg Hierarchy:

1. Iceberg Catalog (AWS Glue, Nessie, Snowflake Catalog, REST): Holds a single atomic pointer to the current Table Metadata JSON file.
2. Table Metadata File (vN.metadata.json): Defines the full schema, partition specifications, snapshot log, and points to the current Snapshot ID.
3. Manifest List (snap-ID.avro): Contains a list of all active Manifest Files that constitute that specific snapshot, alongside partition boundary summaries.
4. Manifest Files (manifest-ID.avro): Tracks individual data files (part-*.parquet), their exact physical S3 URIs, and column-level upper/lower bound metrics.
5. Data Files (*.parquet): The raw columnar data stored immutably on S3.

Atomic Commits via Optimistic Concurrency Control (OCC):

When a writer commits new data:

1. It writes new Parquet data files to S3.
2. It writes new manifest files and a new snapshot manifest list.
3. It writes a new table metadata file (v5.metadata.json).
4. It attempts an atomic swap at the Catalog layer: CompareAndSwap(expected: v4, target: v5).
5. If another writer committed first, the losing transaction detects the conflict, re-validates that its writes do not conflict with the winner, and retries the commit without rewriting data files.

Maintaining high query performance in a production Lakehouse requires background maintenance jobs:

• Small File Compaction (Bin-Packing): High-frequency streaming pipelines (e.g., Flink ingesting Kafka) produce thousands of tiny 5MB Parquet files. Compaction jobs asynchronously read these small files and rewrite them into optimized 256MB–512MB columnar files, atomically committing a new Iceberg snapshot.
• In-Place Schema Evolution: Renaming, adding, or reordering columns is a purely metadata-level operation in Iceberg. Each column is assigned a permanent unique integer ID. Unlike classic Hive tables, renaming a column never requires rewriting underlying petabyte Parquet data.
• Snapshot Expiration & Orphan File Cleanup: Periodic vacuuming jobs prune historical snapshots older than a retention threshold (e.g., 30 days) and delete unreferenced physical Parquet files to reduce cloud storage billing.