- Home
- Data Version Control Tools
- Apache Paimon
Apache Paimon
What it is
Apache Paimon is a free, open-source lake format for building realtime lakehouse architectures with streaming and batch operations through Flink and Spark. Primary-key tables support large-scale updates with configurable merge engines for deduplication, partial updates, aggregation, and first-row updates. Append tables support batch and streaming processing, automatic small-file merging, and compaction with Z-order sorting. Paimon also provides ACID transactions, time travel, schema evolution, and metadata for petabyte-scale datasets and many partitions. Listed integrations include Flink, Spark, Hive, Trino, Presto, StarRocks, and Doris, though supported operations vary by engine and version. The project documents CDC pipelines for MySQL, PostgreSQL, Kafka, MongoDB, and Pulsar. Other capabilities include blob tables, vector storage and search, full-text search, and global indexing. PyPaimon offers catalog, table, Arrow, and pandas APIs plus a command-line tool, with Ray, PyTorch, and Pandas integrations. Core table reads and writes do not require a JVM or a running Flink or Spark cluster.
For and against
For
- Free and open source
- Supports ACID transactions and time travel
- Primary-key tables support configurable merge engines
- Core Python table reads and writes need no JVM
Against
- Supported operations vary by engine and version
- Engine integrations need matching connectors or built-in integration
- Users must choose connectors and storage plugins
The verdict
Apache Paimon offers table management and streaming capabilities across a broad listed ecosystem. Its suitability depends on connector compatibility and the storage and deployment setup required for your environment.
Who it suits
Apache Paimon may suit teams building lakehouse systems that need streaming and batch operations, table time travel, or CDC ingestion. It also offers Python APIs and listed integrations for several query engines.
Plans and prices
Apache Paimon
Open-source data lake platform; choose connectors and storage plugins for your environment
Read 4 Oct 2026 · paimon.apache.org
On the record
- Analytics
- The project describes petabyte-scale tables with time travel, fast scan planning, schema evolution, and incremental clustering.paimon.apache.org · read 4 Oct 2026
- Append processing
- Append tables support large-scale batch and streaming processing, automatic small-file merging, and data compaction with Z-order sorting.paimon.apache.org · read 4 Oct 2026
- CDC
- The documentation lists CDC pipelines for MySQL, PostgreSQL, Kafka, MongoDB, and Pulsar.paimon.apache.org · read 4 Oct 2026
- CDC ingestion
- The project homepage lists CDC pipelines for MySQL, PostgreSQL, Kafka, MongoDB, and Pulsar.paimon.apache.org · read 4 Oct 2026
- Compute integrations
- The ecosystem compatibility page lists integrations for Flink, Spark, Hive, Trino, Presto, StarRocks, and Doris, with supported operations varying by engine.paimon.apache.org · read 4 Oct 2026
- Data management
- Paimon supports ACID transactions, time travel, schema evolution, and metadata for petabyte-scale datasets and many partitions.paimon.apache.org · read 4 Oct 2026
- Data security
- The filesystems guide documents OSS server-side encryption headers and configuration for AES256, KMS, or SM4.paimon.apache.org · read 4 Oct 2026
- Deployment model
- The download guide offers engine JARs, filesystem plugins, and a Java API bundle to run Paimon with query engines or embed it in a Java application.paimon.apache.org · read 4 Oct 2026
- Deployment requirement
- Engine integrations require a matching connector or built-in integration and access to the catalog and warehouse storage; distributed clusters need shared storage available to participating processes.paimon.apache.org · read 4 Oct 2026
- Iceberg access
- Paimon can publish Iceberg metadata so applications can read its existing data files through Iceberg connectors; writers and maintenance remain in Paimon.paimon.apache.org · read 4 Oct 2026
- Integration limits
- The compatibility matrix lists engine version ranges and shows that supported read, write, and table operations vary by engine.paimon.apache.org · read 4 Oct 2026
- Multimodal and Python
- The homepage describes vector search, full-text search, blob tables, and a Python SDK with integrations including Ray, PyTorch, and Pandas.paimon.apache.org · read 4 Oct 2026
- Multimodal data
- The project lists blob storage, vector storage, full-text search, and global indexing capabilities.paimon.apache.org · read 4 Oct 2026
- Purpose
- Apache Paimon is a lake format for unified batch, streaming, and multimodal AI workloads.paimon.apache.org · read 4 Oct 2026
- Python and AI
- PyPaimon is described as a Python SDK with Ray, PyTorch, and Pandas integrations for AI and multimodal workloads.paimon.apache.org · read 4 Oct 2026
- Python client
- PyPaimon provides catalog, table, Arrow, and pandas APIs plus a command-line tool; core table reads and writes do not require a JVM or a running Flink or Spark cluster.paimon.apache.org · read 4 Oct 2026
- Query engines
- The ecosystem documentation lists Flink, Spark, Hive, Trino, Presto, StarRocks, and Doris integrations.paimon.apache.org · read 4 Oct 2026
- Realtime updates
- Primary key tables support large-scale updates and configurable merge engines, including deduplication, partial updates, aggregation, and first-row updates.paimon.apache.org · read 4 Oct 2026
