Apache Paimon brings a streaming-first approach to lakehouse storage, making it ideal for real-time analytics and evolving data workloads.
Modern data platforms increasingly demand real-time analytics, where insights are generated as data arrives rather than hours later. Traditional data lakes work well for batch processing but often struggle with low-latency updates, frequent data changes, and incremental processing.
Streaming systems, on the other hand, are optimised for real-time ingestion but lack robust storage semantics such as schema evolution, time travel, and efficient updates. This gap has led to the rise of lakehouse architectures, combining the reliability of data lakes with the performance of streaming systems.
Apache Paimon addresses this challenge by providing an open source table storage format designed specifically for streaming-first, real-time analytics.
Overview of Apache Paimon
Apache Paimon is an open source lake house storage framework built for continuous data ingestion, upserts, and incremental processing. It uses LSM-tree (Log-Structured Merge Tree) principles, like modern databases, to efficiently merge small updates into larger files.
Unlike traditional append-only table formats, Paimon is designed around primary-key tables, making it suitable for:
- Change Data Capture (CDC)
- Streaming updates
- Near real-time analytics
- Apache Paimon sits between:
- Streaming engines (for ingestion and updates)
- Storage systems (HDFS or object storage)
- Analytics and ML workloads
It is most used with Apache Flink, enabling unified streaming and batch workflows.
Apache Paimon architecture
At a high level, Apache Paimon consists of three core layers:
- Query/compute layer (Flink/batch jobs)
- Apache Paimon tables (metadata and data files)
- Storage layer (HDFS/S3/GCS)
Key components are table metadata, data files, snapshots, and compaction which maintains performance by merging small files. This architecture allows Paimon to support streaming writes and batch reads efficiently.
Setting up Apache Paimon on Linux
Apache Paimon runs smoothly on Linux and integrates easily with existing big data stacks.
For running Paimon, the prerequisites are:
- Linux-based system (Ubuntu/CentOS)
- Java 8 or later
- Apache Flink (standalone or cluster mode)
- Basic setup steps are:
- Download Apache Flink and Apache Paimon binaries.
- Configure Flink to include the Paimon connector JAR.
- Set up a warehouse directory on HDFS or local file system.
Example configuration in flink-conf.yaml:
table.store.path: file:///data/paimon/warehouse
Basic data operations with Apache Paimon
Apache Paimon tables can be created using SQL via Flink:
CREATE TABLE orders ( order_id BIGINT, customer_id BIGINT, amount DOUBLE, order_time TIMESTAMP(3), PRIMARY KEY (order_id) NOT ENFORCED ) WITH ( ‘connector’ = ‘paimon’, ‘path’ = ‘file:///data/paimon/warehouse/orders’
This creates a primary-key table, allowing updates and deletes.
You can insert data as usual:
INSERT INTO orders VALUES (1, 101, 250.50, CURRENT_TIMESTAMP), (2, 102, 175.00, CURRENT_TIMESTAMP); Updating an existing record is handled automatically via the primary key: INSERT INTO orders VALUES (1, 101, 300.00, CURRENT_TIMESTAMP);
Paimon stores this as a change, not a duplicate record.
Common use cases
Real-time analytics:
- Continuous ingestion from Kafka
- Up-to-date dashboards with minimal latency
Streaming pipelines:
- CDC ingestion from transactional databases
- Unified streaming + batch processing
AI and ML-ready data:
- Incremental feature computation
- Consistent, versioned datasets for training and inference
Apache Paimon’s snapshot and incremental-read capabilities make it especially useful for modern AI pipelines. Its open source nature, strong integration with Apache Flink, and support for upserts and incremental processing position it as a powerful choice for modern data platforms. As organisations move towards real-time and AI-driven systems, Apache Paimon provides a solid foundation for building scalable, maintainable, and open data architectures.
