Apache Iceberg

Apache Iceberg is an open table format for analytic datasets, designed for large tables. Engines such as Spark, Trino, Flink, Presto, Hive, and Impala can work with the same tables. SQL commands support merging new data, updating rows, and deleting selected data. Columns can be added, renamed, or reordered without rewriting a table, while hidden partitioning helps avoid scanning unneeded partitions and files.

Iceberg records table snapshots that can be queried by version or timestamp, and a table can be rolled back to an earlier state. Its documentation describes atomic updates, serializable isolation, and consistent snapshot reads. Concurrent writers use optimistic concurrency; a failed commit can be retried against the current table state. Data files can be rewritten using strategies such as bin-packing or sorting. Documented integrations include Spark, Flink, Kafka Connect, Hive, and catalogs such as AWS Glue, JDBC, and Nessie. Iceberg is self-hosted, uses storage supplied by the user, and is licensed under Apache License 2.0. Version 1.12.0 removes support for Spark 3.4 and Flink 2.0.

Apache Iceberg provides table-level capabilities for shared analytic datasets, including schema evolution, snapshots, and concurrent writes. Teams using Spark 3.4 or Flink 2.0 should note those versions are no longer supported in release 1.12.0.

Apache Iceberg suits teams managing analytic tables who need multiple query or processing engines to work with shared data. It is also relevant to teams that need table snapshots, schema changes, or concurrent writes in a self-hosted setup.

Apache Iceberg is a self-hosted table format for teams managing large analytic datasets across several processing engines. It is strongest when shared tables need durable history, controlled schema changes, and reliable multi-writer operation; organizations wanting a hosted workspace should look elsewhere.

Overview

Iceberg supplies the table layer while you provide the storage and deployment. Spark, Trino, Flink, Presto, Hive, and Impala can use the same tables, so it fits data platforms where different engines must read and write a common dataset. That breadth is valuable, but the format is infrastructure rather than a finished database or visual analytics application.

Table snapshots make operational recovery practical. A team can query a specific version or timestamp and restore an earlier state. Atomic table updates, serializable isolation, and consistent snapshot reads protect readers during changes, while optimistic concurrency lets multiple writers proceed and retries failed commits against the newest table state.

Key features

• SQL data changes: Merge incoming data, update existing rows, and issue targeted deletes with SQL. This suits incremental maintenance better than repeatedly replacing an entire table.
• Schema evolution: Columns can be added, renamed, or reordered without rewriting the table. Teams can adapt changing data contracts with less migration work.
• Hidden partitioning: Iceberg derives partition values and automatically skips irrelevant partitions and files. Query planning is less dependent on users manually encoding partition logic.
• Snapshots and branching: Table-level history supports point-in-time queries, rollback, and dataset branching, giving pipelines a controlled way to inspect or recover prior states.
• Compaction: Data files can be rewritten with bin-packing or sorting strategies, useful for keeping accumulated files manageable as workloads evolve.
• Integrations: Documented connections include Spark, Flink, Kafka Connect, Hive, and catalogs such as AWS Glue, JDBC, and Nessie. Encryption can use AWS, Azure, or GCP KMS clients, or a custom KMS client; catalogs must preserve the encryption key ID for the table's lifetime.

Pricing

Apache Iceberg is free and open source under the Apache License 2.0. The Apache Iceberg plan costs nothing, with no trial because the software itself is free. There are no hosted seats or service quotas in this plan; you bring your own storage and operate the deployment, so infrastructure and administration remain your responsibility.

Platforms

Iceberg is offered through an API and runs on Linux, macOS, Windows, and self-hosted environments. Downloads include source archives, Spark and Flink runtime JARs, and AWS, GCP, and Azure bundles. Its deployment model is self-hosted, with tables as the data scope and bring-your-own storage as the backend.

Who it's for

Choose Iceberg for an engineering-led lakehouse or analytics platform that must serve the same tables from several engines, preserve table history, and accept concurrent writes. It is a particularly good match for large datasets whose schemas and partitions change over time. Smaller teams seeking managed operations, a hosted interface, or a conventional application database will face more platform work than they need.

Pros and cons

• Pro: Cross-engine tables let Spark, Trino, Flink, Presto, Hive, and Impala share one table representation.
• Pro: Snapshot queries, rollback, atomic updates, and serializable isolation give data operations a clear recovery and consistency model.
• Pro: Schema evolution and hidden partitioning reduce disruptive rewrites and manual partition management.
• Con: Self-hosting and bring-your-own storage shift deployment, catalog, security, and maintenance responsibilities to your team.
• Con: Release 1.12.0 removes support for Spark 3.4 and Flink 2.0, so pinned engine versions can block an upgrade.

Alternatives

Data Version Control Tools is the broader category to consider when you want to compare data-versioning products rather than adopt one table format.

Apache Paimon is a free, open-source data lake platform when connector and storage-plugin choices are more important than Iceberg's table format. Delta Lake is another free, open-source storage framework for compatible engines and platforms. git-annex fits large-file management without storing file contents in Git. Weights & Biases, Roboflow, Databricks Notebooks, Dolt, and Graviti are alternatives when you need their respective hosted, notebook, dataset, or collaboration workflows instead.

Verdict

Apache Iceberg is the right choice for teams building self-hosted analytic infrastructure around shared, evolving tables and needing dependable snapshots and concurrent writes. Its main reason to choose it is the combination of cross-engine interoperability and table-level reliability. Look elsewhere if you want managed service convenience or if your required Spark or Flink versions are no longer supported by release 1.12.0.