As officially announced by the Apache Software Foundation, we are thrilled to share that Apache Fluss has graduated from the Apache Incubator to become a Top Level Project (TLP).

The project's graduation proposal received unanimous approval from the Apache Incubator Project Management Committee (IPMC) and was subsequently approved by the ASF Board of Directors. This milestone not only marks a new stage in Fluss's journey, but also further advances the convergence of streaming storage, the real-time Lakehouse, and AI data infrastructure, opening a new chapter for real-time data infrastructure.

How We Got Here​

Fluss was initiated by the Flink team at Alibaba Cloud in July 2023 to address long-standing challenges in streaming storage for analytical workloads, including unified streaming and batch storage, Flink state management, and complex data pipelines. Its goal was to build a unified streaming storage system for real-time analytics in the Lakehouse era. The name “Fluss” comes from Flink Unified Streaming Storage and also means “river” in German, reflecting the vision of data flowing continuously like a river before eventually joining the open Lakehouse.

After more than a year of development and large-scale production use within Alibaba, Fluss was officially open-sourced at Flink Forward Asia 2024 in Shanghai in November 2024. In June 2025, Fluss entered the Apache Incubator, evolving from a technology initiative originating at Alibaba into an open source project built by developers around the world.

During incubation, both the contributor community and interest in the project continued to grow. Today, the community includes 157 contributors, and the project has earned 2,000+ stars on GitHub, merged 1700+ pull requests, bringing together developers from different countries and organizations in an active open source community.

At the same time, Fluss has been deployed in production at Alibaba, Xiaohongshu (RedNote), Fresha, JD.com, Ant Group, iQIYI, and other companies. It is used across log collection and analytics, real-time data warehousing, search and recommendation, indexing pipelines, and real-time feature serving. These deployments help organizations reduce cross-system data copies, lower real-time data processing and storage costs, and improve Lakehouse data freshness and analytical efficiency.

"When we started Fluss, I believed strongly in the problem we were solving, but the project has grown faster and reached further than I expected," said Jark Wu, PMC Chair of Apache Fluss. "The energy of the community and the growing number of companies running Fluss in production have been especially encouraging. Graduation is a new beginning, and I am confident Fluss will become a foundational real-time data layer for lakehouse architectures and AI applications that depend on fresh, continuously updated data."

From Lakehouse to Lakestream: Unify Stream and Lakehouse for the Agentic Era​

In the Agentic era, AI is evolving from chatbots that answer questions into agents capable of making autonomous decisions, using tools, and executing tasks. Data is no longer just an input to analytics; it is the real-time context agents rely on to reason and act. That context must span the full historical record while accurately reflecting the latest state of the business. Traditional Lakehouse architectures excel at storing and governing massive volumes of historical data, but struggle to continuously meet data-freshness requirements measured in seconds or even milliseconds.

As an open source pioneer of the Lakestream architecture, Apache Fluss adds a Lakehouse-native, real-time streaming storage layer on top of the data lake. It unifies continuously updated, real-time data with long-term history, giving AI agents fresh and complete context from the past through the present and enabling production-grade, real-time decisions and actions.

Apache Fluss brings the following core capabilities to the traditional Lakehouse, creating a unified Lakestream foundation that makes both historical and real-time data readily available:

• Stream & Lakehouse Unification: Long-term historical data resides in open data lake formats such as Apache Paimon, Apache Iceberg, Apache Hudi, and Lance, while Fluss serves the latest data in real time. With Union Read, agents can access everything from historical data to the latest state through a unified table view, avoiding fragmented context.
• Columnar Streaming Storage: Built on the Apache Arrow columnar format, Fluss supports server-side column pruning, predicate pushdown, and partition pruning, reducing data reads and transfers for efficient streaming and real-time analytics.
• Real-time Updates and Point Queries: Primary-key tables natively support streaming updates, partial updates, changelogs, key-value lookups, and Delta Join, allowing the same real-time data to serve both stream processing and point queries.
• Real-time Context for Agents: Fluss unifies continuously updated real-time state with long-term historical data, giving AI agents fresh, complete, low-latency, and trustworthy context for real-time decisions and actions.
• Open Ecosystem with Multi-Language Access: Fluss works with compute engines such as Apache Flink, Apache Spark, and StarRocks and provides Java, Rust, Python, and C++ clients, making it easy for real-time analytics and AI applications to connect.

Growing Community and Adoption​

In the year since joining the Apache Software Foundation, Fluss has built an active global contributor community and received broad interest and support from leading companies and open source ecosystem partners.

Feng Wang, Head of Open Data Platform at Alibaba Cloud: “Apache Fluss’s graduation reflects a mature community shaped by open, collaborative governance at the ASF. Deployed and proven at scale in Alibaba’s core e-commerce production workloads, Fluss’s Lakestream architecture unifies streams with Lakehouse data, making the Lakehouse real-time while reducing duplication and pipeline complexity. This led us to open-source and donate Fluss to the ASF. I look forward to Fluss becoming the open data foundation for the real-time Lakehouse, advancing analytics and AI across the open data ecosystem.”

Yu Li, ASF Member and Mentor in the Incubator Program: “Congratulations to the Apache Fluss community on reaching this important milestone and becoming an ASF Top-Level Project. Serving as the project’s champion, I have been delighted to see Fluss grow into an open and collaborative community guided by the Apache Way. During incubation, the community delivered substantial releases, advanced streaming storage for real-time analytics, and steadily welcomed new contributors. I look forward to seeing Fluss build on this strong foundation in its next chapter.”

Igor Kersic, CTO at Ververica: “I'm truly excited to see Fluss graduate and become a top-level ASF project. It marks the beginning of next evolution of streaming landscape, but it's real power is in unlocking orders of magnitude more business value in streaming analytics. The design choices are just a perfect mix for one stop native streaming storage combined in just one project. My expectations is for it be de-facto foundation streaming storage go-to standard in real-time AI era. I see quicker than expected enterprise adoption due to it's power to collapse costs of complex lakehouse landscapes, thus enterprise maturity is important.”

Jiangjie Qin, Principal Staff Software Engineer at LinkedIn: “Congratulations to Fluss on its successful graduation! As a unified storage layer that brings together streaming, lakehouse, and key-value store capabilities, Fluss has demonstrated unique and significant value in the data infrastructure ecosystem. Over the past year in the Apache Incubator, the Fluss community has grown rapidly and healthily, and the project has demonstrated remarkable momentum. Its graduation to an Apache Top-Level Project is a well-deserved milestone. Wishing Fluss continued success as it grows into another flagship project in the data infrastructure ecosystem.”

Jingsong Li, Apache Paimon PMC Chair: “Apache Paimon's original vision included a serving and acceleration layer that would keep data moving, but for a long time that part of the vision remained unrealized. With the launch and continued development of Apache Fluss, this acceleration layer now brings sub-second data freshness to the Lakehouse, making it truly real-time and unified. Congratulations to Fluss on its graduation!”

Han Liu, Head of Fluss and Kafka at Xiaohongshu: “As an early adopter and contributor to Fluss, Xiaohongshu is delighted to have witnessed and participated in the community's journey from incubation to graduation. Fluss's columnar reads and writes, hot and cold data tiering, and Stream-Lakehouse unification have delivered significant value in core workloads such as Xiaohongshu's indexing data pipelines, substantially reducing real-time pipeline costs while improving data build and online consumption efficiency. We look forward to the Apache Fluss community continuing to thrive, creating value in more enterprise real-time data scenarios, and establishing Fluss as an important open source project for the next generation of real-time data infrastructure.”

Emiliano Mancuso, VP of Architecture and Data Engineering at Fresha: "At Fresha, Apache Fluss is deployed in production as part of our real-time data platform. Its combination of low-latency streaming access and open Lakehouse integration helps us simplify data movement and build fresher analytical and operational data products, while capabilities such as Delta Join make our Apache Flink pipelines simpler to operate. We are proud to contribute directly to the project and look forward to helping shape its next chapter as an Apache Top-Level Project."

Peibin Wang, Senior Big Data Expert at Taobao Instant Commerce: “Fluss has redefined the paradigm of real-time data infrastructure. Innovations such as Delta Join and Stream-Lakehouse unification mean that streaming storage is no longer simply an alternative to message queues, but an indispensable real-time data layer in the Lakehouse architecture. As an early and deeply involved adopter of Fluss, Taobao Instant Commerce is proud to have used Fluss to systematically address state growth and pipeline complexity at the scale of hundreds of billions of events, while reliably supporting core real-time decision-making pipelines during major campaigns such as the 618 shopping festival. We look forward to Fluss unlocking even greater value in the AI × Data era!”

Fei Han, Head of the Real-Time Data Platform at JD Retail: “By starting with streaming storage, Fluss opens up new possibilities for data infrastructure in the era of real-time analytics and AI. JD.com is combining its broad range of business scenarios with active participation in Fluss's technical development and community building, helping its capabilities take root in enterprise environments. Becoming a Top-Level Project is a new beginning. We look forward to Fluss continuing to embrace open innovation, growing its global developer ecosystem, and helping more organizations unlock the value of their data!”

Xin Wang, ASF Member and Head of Real-Time Intelligence at Ant Group: “Fluss has steadily evolved around streaming storage into an important open source project connecting stream processing and the Lakehouse ecosystem, earning increasingly broad industry adoption. Ant Group is also actively exploring and deploying Fluss. We look forward to Apache Fluss continuing to foster an open, transparent, and collaborative community guided by the Apache Way and becoming a stronger, more trusted foundation for real-time data and AI.”

What’s Next for Fluss?​

As an Apache Top Level Project, Fluss will continue to advance Lakehouse-native streaming storage: providing stream processing with a more efficient and reusable storage foundation, bringing true real-time capabilities to the Lakehouse, and offering an open data foundation for AI applications that depend on real-time features, dynamic state, and continuous context.

Thank you to every contributor who has submitted code, improved documentation, participated in discussions, reported issues, shared their experience, or helped newcomers. Thank you as well to all the users and partners who chose Fluss for their production environments and helped move the project forward by bringing real-world challenges to the community. We would also like to extend our special thanks to the Apache Fluss incubation mentors: Yu Li (Champion), Zili Chen, Jingsong Li, Jiangjie Qin, and Jean-Baptiste Onofré, whose continued guidance and support throughout incubation helped the community better embrace the Apache Way and successfully complete the journey from incubation to graduation.

Join the Apache Fluss Community​

Whether you want to use Fluss for real-time analytics and AI or learn more and contribute to the project, now is a great time to join the community:

• Website: https://fluss.apache.org/
• GitHub: https://github.com/apache/fluss (⭐ stars and contributions are welcome!)
• Contribution guide: https://fluss.apache.org/community/how-to-contribute/overview/

A new journey has begun. Let this open river of data keep flowing. 🌊

Part 1 and Part 2 built up everything you need to know about how tiering behaves: the mental model, the dials, the queue dynamics, the scale-out story. This part is about what to do with all of that. What breaks at runtime, and which of those failures self-heal versus need operator action. The design mistakes that look fine on day one but come back to bite you on day two. And the operator's daily view: which five numbers tell you whether tiering is healthy on a Tuesday afternoon, where each one comes from, and why two of them can only come from your Flink-side dashboards.

Tiering Service Deep Dive, 3-parts:

• Part 1 - The Mental Model: how one tiering round actually works, from timer fire to lake commit.
• Part 2 - Tuning: per-table dials, multi-table dynamics, and scaling out.
• Part 3 - In Production: failure modes, design pitfalls, and the dashboard that tells you everything is fine.

Failure Modes: What Self-Heals And What Doesn't​

The tiering service has a small number of well-defined failure modes, and the design absorbs most of them gracefully. The point of this section isn't to enumerate every edge case. It's to give you a mental model for what self-heals and what doesn't, so when something goes wrong you know whether to wait, restart, or escalate.

The Tiering Job Dies Mid-Round​

This is the most common failure and the easiest to reason about. Job A is halfway through tiering orders: readers have written some Parquet files, but the writer hasn't completed the lake commit yet. The Flink TaskManager crashes, or the JobManager loses leadership.

From Fluss's perspective, heartbeats stop arriving. Roughly two minutes later (the 2-minute liveness threshold from Part 2, plus up to one 15-second checker tick, so detection lands at about 2 to 2.25 minutes), the coordinator declares the job dead, fences its in-flight assignment, and returns orders to the back of the pending queue, where it waits behind whatever else is already queued. The next tiering job to heartbeat picks up the table at the front of the queue, which may or may not be the fenced table, depending on what else was waiting. The interrupted writer's Parquet files in the lake are orphaned; they're committed to neither Fluss nor the lake catalog, so no reader will ever see them. They sit until an external lake-side garbage collector cleans them up.

The new job starts the round from the last committed lake offset, not from where the dead job left off. That's the cost: all the work since the last commit is redone. The benefit: you never get a half-committed table.

Three invariants hold throughout this recovery:

• The lake never sees a partial commit. Readers can't observe a half-written snapshot.
• The redo cannot be double-counted into Fluss. The epoch fences the heartbeat path: the dead attempt's finish, fail, and heartbeat messages carry a stale epoch and are rejected by the coordinator (validateTieringServiceRequest). This fences the bookkeeping, not the lake write itself; the lake commit and the offset-advance RPC are not epoch-gated. Exactly-once into the lake is instead protected by per-snapshot atomicity plus a reconciliation check on the next round (getMissingLakeSnapshot), which detects and repairs the case where the lake advanced but Fluss didn't.
• The orphaned Parquet files are lake-side garbage. They rely on an external lake-side garbage collector to clean them up; as of the Fluss version this series tracks, there is no Fluss-side reaper for orphaned lake files.

These invariants assume the dead job stopped before its lake commit. A job that is fenced but still alive (a JobManager GC pause, or a JobManager-to-coordinator partition that doesn't affect the committer) can still complete its lake commit after the table was reassigned, because that write carries no fencing token derived from the 2-minute lease. In that narrow case, exactly-once depends on the lake committer's own idempotency.

Call to action: If it's Kubernetes or any restart-on-failure scheduler there is no required action. The job comes back, registers, and pulls from the queue normally. The 2-minute fencing window is your only real cost.

A Reader Fails Inside A Healthy Job​

A single TaskManager dies while the JobManager keeps running. This isn't a localized restart: the tiering job runs with a full-restart failover strategy, so one failed task restarts the entire job, readers, enumerator, and committer together. The in-flight round's partial work, which lives only in memory, is thrown away.

After the restart, the enumerator re-registers with the coordinator and asks for fresh work; it does not resume the table it was tiering. That table stays in Tiering on the coordinator until the same ~2-minute liveness timeout from the job-death case fences it (Failed to Pending, epoch bumped) and re-queues it. So a reader failure collapses into the same recovery path as a dead job: full restart, then the interrupted table waits out the ~2-minute fence before it is picked up again. What the surviving JobManager buys you is a faster restart of the job process, not faster recovery of that one table. No progress is lost and nothing double-commits, because the durable record of completed rounds lives in Fluss metadata (committed lake offsets).

Call to action: nothing; it self-heals. Watch the job's restart count and the Fluss-side tierLag and queue depth: a healthy recovery is a brief restart followed by tierLag settling back down.

The Fluss Coordinator Restarts​

The coordinator holds the queue and the per-table assignment state in memory. When it restarts, the queue empties and the assignments are dropped. The durable truth lives in Fluss's persistent metadata (ZooKeeper for table-level configuration, with committed lake offsets recorded through the lake-snapshot commit path). On restart, initWithLakeTables rebuilds the in-memory state from cluster metadata and, importantly, resets every table's epoch counter to 0 as part of registerLakeTable.

Right after restart, any job that was mid-round holds a non-zero epoch, so its next heartbeat is fenced with FencedTieringEpochException (returned as a per-table error in the heartbeat response, which the enumerator treats as "throw away in-flight state and ask for fresh work"). One caveat worth knowing: this epoch lives only in the coordinator's memory and restarts from 0, so it is not monotonic across restarts. Fencing here relies on the stale job being detected and reassigned before the per-table epoch happens to climb back to the same value: in practice fine, but it's a lease-style guarantee, not an absolute one.

Practically, any tiering job that was mid-round when the coordinator restarted loses its assignment, and the next freshness firing re-enqueues the affected tables. That can mean a freshness lag of up to the per-table target on the worst-affected table, but no data loss: the writes that were in flight just hadn't been committed to the lake yet, and they'll be re-tiered on the next round.

Call to action: nothing on the tiering side. Coordinator availability is the broader story; in HA mode another coordinator takes over. The tiering jobs reconnect to the new leader through the same metadata path as everything else.

The Asymmetric Failure: Two Jobs, One Healthy​

You're running two tiering jobs (the scale-out pattern from Part 2). Job A is healthy; Job B is in a crash loop because of some error. What happens?

From the coordinator's perspective, Job B's heartbeats stop arriving. After roughly two minutes (the same 2-minute threshold plus up to one 15-second checker tick), the coordinator fences whatever Job B was holding. Job A picks up the slack: everything Job B would have done now flows through Job A. Effective throughput drops from two-job back to one-job behavior, with the queue-starvation pattern returning if your workload has the mixed-size shape.

This is the silent failure mode worth watching for. If you scaled out specifically to unblock a small table from a large one, and one of your jobs goes dark, the small table's freshness gets worse without anything obviously broken on the Fluss side. The challenge is that Fluss itself doesn't track tiering-job identity (as we saw when scaling out), so the coordinator cannot tell you "I expected 2 jobs and only see 1". The monitoring signal has to come from your Flink-side dashboards (per-JobManager liveness, checkpoint health) combined with the Fluss-side queue and lag metrics from the operations section. If your Flink dashboard shows two healthy JobManagers but pendingTablesCount is climbing and tierLag on small tables is degrading, you have evidence that one of the jobs has effectively gone dark even if its process is alive.

The Pattern: What Self-Heals And What Doesn't​

Anything that's transient and respects the 2-minute liveness window self-heals: job crashes, network blips, reader restarts, coordinator failover. Anything structural (a bad catalog credential, a missing Iceberg table, a misconfigured lake bucket) will crash-loop your tiering job and not affect Fluss's hot tier at all. The hot tier keeps accepting writes regardless of lake health. Your cluster doesn't break when tiering breaks; it just stops aging out. That's the design's most important safety property, and also the reason monitoring the tiering side separately from the Fluss side matters.

Common Mistakes: What To Avoid In Production​

This section is the failure-modes section's sibling. Not "what breaks when the system is healthy" but "what looks fine but is actually about to break". None of these will crash your cluster. All of them produce slow, expensive, or surprising behavior down the road. Worth catching at design time rather than three months in.

Mistake 1: Sizing Buckets For The Wrong Dimension​

The most common one. You pick bucket count based on ingest throughput: "we have 200K writes/sec, give it 32 buckets". That sizes hot-path throughput just fine. What you didn't size for is the tiering round, which is per-bucket parallelism on a single Flink job. With 32 buckets, your tiering job needs 32 reader slots to run in parallel, otherwise the round serializes. And with 32 readers, the Flink-side write to the lake fans out to 32 concurrent writers, which means 32 concurrent S3 PUTs per round.

And bucket count isn't something you can walk back: in current Fluss there's no online ALTER path to change a table's bucket count at all. It's part of the table's distribution, fixed at create time, not an alterable table.* property, so for both log and PK tables you're committing to the count you pick at creation. PK tables make this constraint conceptually sharper still: the hash function that assigns rows to buckets depends on the count, so even if a future release exposed a reshard, changing the count would invalidate every existing key's placement.

The practical takeaway: pick a bucket count at table creation time that reflects your steady-state volume, not your peak burst. Eight buckets handle a lot of throughput when the tiering round is healthy, and you'll thank yourself later when the round time stays predictable. For most tables, fewer buckets is better, until you have measured evidence that ingest is bucket-bound.

Mistake 2: One Tiering Job For Everything​

Covered at length in the multi-table and scale-out sections of Part 2, but worth repeating as a discrete anti-pattern: deploying one tiering job and pointing all 50 of your tables at it. It works, for a while. Then one table grows, its round duration extends, and every other table's effective freshness silently degrades because the queue gets longer. The fix when it bites is to scale out to multiple jobs, but the better play is to think about workload mix at the design stage. If you know you have a heavy table and a hot table, plan for two jobs from day one. The cost is two Flink deployments instead of one, which is marginal on most clusters. The benefit is that one bad week of growth on the heavy table doesn't take out your latency SLO on the hot one.

Mistake 3: Confusing Freshness Target With Freshness Guarantee​

The table.datalake.freshness config is the "the maximum amount of time that the datalake table's content should lag behind updates", but it's also a "target freshness". Those aren't the same thing. The coordinator schedules the next round as a re-enqueue delay (freshness − (now − last_tiered)), so freshness controls how often a table becomes eligible to tier again, not a hard ceiling on how stale the lake can get. Under queue contention, the "maximum" is aspirational, not enforced. If the queue ahead of your table is empty and the round is fast, you'll get something close to 1 minute. If the queue is full of bigger tables, you'll get whatever you get; the multi-table walkthrough in Part 2 goes through this case in detail.

This can look confusing when users, set up dashboards that read directly from the lake and expect "1-minute freshness" to mean "data is never more than 1 minute behind real time". It can mean that under light load.

Under realistic load it means "this table is allowed to be re-tiered as often as every minute, queue permitting". If you genuinely need bounded freshness on a lake-side read, your options are: read from Fluss directly, or scale out tiering jobs to make sure your table never queues.

Mistake 4: Enabling Lake Tiering On A Tiny Table​

This one isn't so much a mistake as a waste. You have a dim_country table with 200 rows, updated once a month. Someone enables table.datalake.enabled = true on it because "we tier everything". Now your tiering job is scheduling rounds on this 200-row table at the configured freshness cadence, each round writing a Parquet file that's mostly metadata overhead, each commit allocating a snapshot in the lake catalog. The lake-side Parquet directory fills up with thousands of tiny files. Compaction will eventually clean them up, but in the meantime your S3 LIST operations are paying for them and your catalog has thousands of unnecessary snapshot entries.

For small, slow-changing reference tables, the correct play is usually to keep them in Fluss only (no lake tiering) and let lake-side queries do a join through the union-read path. Or, if they really need to be in the lake, materialize them once via a batch job and refresh on a much longer cadence. The tiering service is the wrong tool for low-velocity data.

Mistake 5: Ignoring The Cross-Table Effects Of Compaction​

The Fluss-specific hook is simple: because tiering parallelism is per-bucket, each round writes at least one Parquet file per bucket. So a 16-bucket table tiering at 1-minute freshness produces on the order of 16 small files a minute, roughly 960 an hour and 23,040 a day, per table. That count compounds fast.

Everything past that point is standard lakehouse behavior, not a Fluss bug: small files degrade reads and bloat catalog metadata, and both Iceberg and Paimon lean on compaction to claw it back, which isn't free and isn't always automatic. The takeaway for a tiering deployment is just to size and schedule compaction deliberately rather than discover it reactively.

The Pattern: A Production-Readiness Checklist For Tiering​

Before you call a tiering deployment production-ready, walk through:

1. Bucket counts are sized for round duration, not just ingest throughput, and you've explicitly defined that at table creation time.
2. Tables of similar round duration are grouped onto the same tiering job; tables of dissimilar round duration are split across jobs.
3. Freshness targets reflect what you actually need at the lake-tier read path, not "we just tier everything".
4. Compaction is configured on the lake side and you have a dashboard showing file count and snapshot count growth.
5. You have Flink-side liveness monitoring on every tiering job you deployed (see the operations section on why Fluss itself can't tell you a job has gone dark).

What To Monitor: The Operator's View​

The previous sections covered design-time decisions and failure-mode reasoning. This one is about the running system: what to look at on a Tuesday afternoon when you're trying to figure out if tiering is healthy, slow, or quietly broken. The tiering service exposes its state through a combination of Flink metrics, Fluss coordinator metrics, and the lake catalog itself. It's worth knowing which metrics matter.

What The Fluss Coordinator Actually Exposes​

A short list of the metrics actually registered in LakeTableTieringManager, so you know what you can build alarms against without inventing things.

Cluster-level:​

• pendingTablesCount: how many tables are waiting in the pending queue right now.
• runningTablesCount: how many tables are currently in Tiering (that is, assigned to some tiering job).

Per-table:​

• tierLag: milliseconds since the last successful tiering of this table.
• tierDuration: wall-clock duration of the last completed tiering round.
• pendingTime: how long this table has been waiting in the pending queue right now.
• failuresTotal: a counter of total tiering failures observed for this table.
• fileSize / recordCount: cumulative lake-side size and record count after the last round.
• freshness: the per-table configured freshness, in milliseconds.

Round Duration​

If you only watch one number, watch per-table tierDuration against the table's configured freshness. This is the metric that tells you whether your freshness targets are realistic. If orders's configured freshness is 2 minutes and its observed tierDuration is 90 seconds, you have headroom. If tierDuration is 110 seconds and creeping up week over week, you have a problem coming.

Build a per-table dashboard. The pattern you want to see is tierDuration < freshness, with stable variance round over round.

Queue Depth​

pendingTablesCount is the leading indicator of starvation. In a healthy system, the queue is usually short (0 to 2 tables): tables enter, get assigned, get committed, exit. When the queue starts growing (5 tables pending, then 10, then 15), you're past the point where adding tables can keep up with tiering throughput. Either tiering rounds are slowing down, or tables are being added to the cluster faster than they're being drained, or one of your tiering jobs has effectively gone dark.

This is your cue to scale out tiering jobs (the multi-job pattern), or revisit bucket counts, or investigate a sick job. By the time freshness misses are showing up in the lake, queue depth has been climbing for hours. Watch it ahead of the symptom.

Remote-Tier Storage Growth​

It's tempting to assume a broken lake tier shows up as hot-tier (local) disk filling on the tablet servers. It usually doesn't. Local disk is bounded by table.log.tiered.local-segments (default 2): older local segments are continuously moved to the remote log tier no matter what the lakehouse tiering service is doing. The remote-log tier and the lakehouse tier are two independent mechanisms.

What actually grows when lake tiering stalls is the remote log tier. With table.datalake.enabled, Fluss won't delete a TTL-expired remote segment until it has been tiered to the lake: cleanup only frees segments whose log end offset is at or below the lake-synced offset. So a stuck lake tier overrides table.log.ttl, expired segments pile up in remote storage, and your object-store footprint for that table's log climbs even though local disk looks fine.

So the signal to watch is remote-tier storage size, not local disk. Remote log storage that keeps growing and never drops after table.log.ttl should have expired it, combined with a climbing tierLag, is the most direct "the lake side is failing" sign you have. For a table with no lake tiering enabled, table.log.ttl alone bounds remote growth and there's nothing lake-specific to watch here.

The operator's dashboard is five tiles, each answering one question. If all five are green, tiering is healthy. If any is red, the section it points to tells you where to look.

The rule that ties it together: four of these tiles come from Fluss and the lake catalog. Tiles 1 and 2 are Fluss coordinator metrics, tile 3 is the Fluss remote-log tier's storage footprint, and tile 4 comes from the lake catalog. Only tile 5 has to come from your Flink-side dashboard: because Fluss has no notion of tiering-job identity, no Fluss metric can answer "are all my deployed jobs still up?" That's why tile 5 lives outside Fluss, and why monitoring tiering means wiring both sides into one view.

The Pattern: The Five-Number Daily Check​

A complete daily-operations view of the tiering service is five numbers:

1. Per-table tierDuration vs freshness: is anything trending toward the limit?
2. pendingTablesCount: is it growing?
3. Remote-tier storage growth: is TTL cleanup stuck behind a stalled lake tier?
4. Lake-side file count growth: is compaction keeping up?
5. Flink-side JobManager liveness for every tiering job you provisioned: does it match what you deployed?

If all five are green, the tiering service is healthy. If any of them are red, the previous sections tell you where to look next. That's the whole operator's playbook for this subsystem.

Where To Go From Here​

This three-part walkthrough was deliberately scoped to the tiering service, the mechanism that moves data from Fluss's hot tier into the lake. There are three adjacent topics worth exploring next, in roughly increasing order of depth.

Reading from the lake side. The whole point of tiering is to make data queryable from Flink batch, Spark, Trino, or any other engine that speaks Paimon or Iceberg. The union-read path, which seamlessly stitches together lake-side historical data with Fluss-side hot data, is what makes this useful. It's separate machinery from the tiering service, but it depends on it.

Compaction on the lake side. Mentioned several times across this series as a downstream concern. The actual mechanics of Paimon's compaction (full vs minor, the role of the dedicated compaction job) and Iceberg's (rewrite manifests, expire snapshots) are their own topics. If you're going to operate tiering in production, you need to operate compaction alongside it; they're a pair.

Schema evolution. The tiering service handles schema changes by carrying the Fluss-side schema through to the lake on the next round. The lake-side catalog needs to accept the new schema; Paimon and Iceberg both support evolution, but with different rules. The interaction between Fluss schema changes (ALTER TABLE on the streaming side) and lake-side schema (the corresponding Paimon or Iceberg evolution) has its own corner cases and is worth a separate walkthrough.

The tiering service is the kind of subsystem that's invisible when it works and confusing when it doesn't. The goal of this walkthrough was to give you the mental model that lets you reason about it without having to dig back through the source every time something looks off.

Part 1 built the mental model. What tiering is, who does what, how the round runs end-to-end.

This part adds the dials. Buckets and splits determine how a round parallelizes. Log and PK tables behave so differently on round one that the difference deserves its own treatment. The freshness setting, the one knob most users actually touch, does two different jobs that share the same value. Once a single job is handling many tables, queue position starts to dominate effective freshness more than any per-table setting. And once that happens, you have a deployment-shape decision: stay with one job, or scale out. By the end, you'll know which levers matter most and how to use them.

Tiering Service Deep Dive, 3-parts:

• Part 1 - The Mental Model: how one tiering round actually works, from timer fire to lake commit.
• Part 2 - Tuning: per-table dials, multi-table dynamics, and scaling out.
• Part 3 - In Production: failure modes, design pitfalls, and monitoring.

Buckets, Splits, And How The Work Gets Divided​

In Part 1, we had a table with 4 buckets that ended up with 4 splits. That's not accidental: the rule is one split per bucket. Let's slow down and look at the parallelism story end-to-end, because this is where bucket count actually starts to matter.

The mental model is straightforward: buckets are how the data is physically partitioned, splits are the chunks of work the tiering job hands out, and readers are the workers that consume splits. The number of buckets is fixed at table creation time. The number of readers is your Flink job parallelism. The number of splits per round equals the number of buckets.

What happens when the number of readers doesn't match the number of buckets? Two cases:

More readers than buckets. A table with 4 buckets and 16 readers means only 4 readers can work on that table. Bucket count caps how many readers a single table's round can use. The other 12 don't necessarily sit idle, though: if other tables are waiting in the queue, the enumerator hands those spare readers the next table's splits, and they start reading ahead (we'll see exactly how in the multi-table section). They only truly idle when there's no other work queued.

Fewer readers than buckets. A table with 16 buckets and 4 readers means each reader will process 4 splits sequentially. Reader 0 takes one split, finishes it, asks for the next, takes another, and so on. The whole round takes roughly 4× longer than it would with 16 readers. This is the common production case, and it's fine, just slower.

One more thing: the splits are shuffled (randomized) before being assigned. This is a small but important detail. Without it, reader 0 would always get bucket 0, reader 1 would always get bucket 1, and so on. If bucket 0 is consistently the heaviest (because of a bad key distribution), reader 0 would consistently be the bottleneck. Shuffling spreads the bad luck around: over many rounds, every reader gets stuck with the heavy bucket about equally often.

The Slow-Bucket Effect​

In the normal commit path, the commit operator waits for every split of a table before committing. If 3 readers finish quickly and the 4th is still processing a much larger bucket, the table's commit can't happen until that last bucket lands. The 3 fast readers move on to other queued tables (or idle, if there's nothing else queued), but this table's round isn't done until its slowest bucket is. The lake commit itself is always atomic. Readers can't see a half-written snapshot, and most of the time every bucket of a round lands in the same commit.

The end result: the slowest bucket determines how long a round takes. Skewed bucket keys can hurt you here. (There's one exception: when freshness fires and the round force-finishes, whatever each bucket has tiered so far gets committed, and the unread remainder rolls to the next round. We'll cover this in the freshness section.)

Two Kinds of Tables: Log vs. Primary-Key Tables​

Log Tables: The Simple Case​

A log table is exactly what it sounds like: an append-only stream. Records arrive, they get assigned an offset, and they sit there. You can't update or delete a record; you can only append. Examples: a clickstream, an event bus, an audit log.

Tiering a log table is straightforward. Every round, the Flink job reads "everything since the last commit" (just the new offsets) and writes those records as Parquet files in the lake. The lake table grows monotonically. The work per round is proportional to how much you wrote since the last round.

Primary-Key Tables: The Trickier Case​

A primary-key (PK) table is like a row-keyed materialized view. Each record has a primary key and the latest value for that key replaces any previous value. Internally, Fluss stores PK tables in two parts: a changelog (a log of all the upserts and deletes that have happened) and a KV state (the current value for each key, derived from the changelog). It's similar to how a database has a write-ahead log plus the actual table.

The changelog records every event using four operation codes you'll see throughout the Flink and Fluss world: +I for an insert, -U and +U together for an update (the old row going out, the new row coming in), and -D for a delete. Apply all of those to an empty state in order and you get the current row for every key; that's the KV state. Here's a small example to make this concrete:

Now think about what "tier this to the lake" means for a PK table. The lake needs the current state of every row, not just the log of what changed. So the very first round of a PK table has to copy the entire KV state to the lake. That can be huge: think 200 GB if you have a big customer-profile table. After that first round, subsequent rounds only need the changelog records since the last snapshot offset, because the lake's merge engine knows how to apply upserts and deletes to existing rows.

The numbers here are illustrative. A 200 GB PK table might take 10+ minutes to read on its first round (because all 200 GB must be copied). The second round, where only the upserts and deletes from the last few minutes need to be applied, might take 30 seconds. This is an important difference, and the single most important thing to understand about PK tiering: the first round is in a different category from every round that follows.

The Summary Table​

The Freshness Knob​

Every table that's enabled for tiering has a setting called table.datalake.freshness. It's a duration, something like 5min or 30s, and the default is 3 minutes. This single number is the most common source of confusion in the tiering service, because it does two different jobs that share the same value.

Job 1: How Often Rounds Start​

The obvious meaning. After a round completes, the coordinator schedules the next round one full freshness interval later, measured from the moment that round finished. So with 5-minute freshness, the next round becomes eligible 5 minutes after the previous round committed. The round's own duration is not subtracted.

Under the hood the scheduler computes freshness − (now − last_completion_time), which looks like it deducts elapsed time. But last_completion_time is reset to the instant the round just finished, so the subtracted term is ~0 for back-to-back rounds. That subtraction only bites after a coordinator restart, where the last completion can be well in the past and the next round may fire immediately.

The practical consequence: the effective start-to-start cadence is round_duration + freshness, not freshness on its own. A table with 5-minute freshness and 90-second rounds runs a round roughly every 6.5 minutes, not every 5.

If You're Coming From a Flink Streaming Background​

Tiering cadence is driven by freshness; it is not tied to Flink checkpoints. A common mental model from Flink-CDC-style pipelines is "data lands in the sink when a checkpoint completes". That's not quite how the tiering service works. A tiering round commits to the lake when all the round's bucket results have arrived at the commit operator, which is driven by the round's own progress rather than by external checkpoint cadence. Whatever checkpoint interval the Flink tiering job has configured doesn't bound when the lake sees new data. The round itself does.

The correctness story has two layers worth separating. Atomicity comes from the lake's own commit primitive (Paimon's snapshot commit, Iceberg's metadata swap). Consistency across attempts comes from the epoch fencing mechanism the coordinator uses, introduced in Part 1, in the heartbeat section, which rejects any commit from a stale attempt (an epoch-fencing error). So you don't need to tune Flink checkpoint settings to get correct tiering; the defaults are fine.

What you do still need is a healthy checkpoint cycle. The Flink tiering job is configured with a full-restart strategy because the commit operator is stateless: it holds the per-table bucket results it has collected so far only in memory (nothing is checkpointed), so if it fails those in-flight committables are lost and the whole job restarts to collect them fresh.

That's deliberate. It keeps the all-or-nothing commit semantics simple. So checkpoints aren't your tuning knob, but they do need to complete; treat the tiering job's checkpoint cycle as load-bearing infrastructure rather than something you can ignore.

Job 2: The Ceiling On A Round's Wall-clock Duration​

The less-obvious meaning. The same value also caps how much wall-clock time a single round of this table is allowed to consume before something gives. When freshness elapses mid-round, the Flink enumerator fires a force-finish: every log split that has started commits whatever it has read so far (advancing that bucket's offset), and buckets that hadn't started yet are skipped. The skipped buckets, and the remaining tail of any partially-read bucket, get re-read in the next round (which starts immediately, since the table is re-queued straight away).

This is an important nuance and worth slowing down on: force-finish is not a failure. It's a partial commit. On the coordinator side, the table transitions Tiering → Tiered (with a force_finished flag) → Pending. It is not marked Failed. It does take a fresh tiering epoch on the way back into the queue: the Tiered → Pending transition bumps the epoch exactly as any normal re-enqueue does, but that's ordinary re-attempt bookkeeping, not a failure recorded against the table. The lake snapshot that lands contains every bucket that tiered any data this round, each up to wherever it reached, and that snapshot is still atomic. Readers never see a partial commit. Buckets that never started just roll forward to the next round.

One exception to highlight: for the first round of a PK table, assuming a KV snapshot already exists (the normal case), that round reads the snapshot, and force-finish is suppressed for snapshot splits. (If no KV snapshot has been taken yet, the first round instead reads the changelog from the earliest offset as ordinary log splits, and force-finish applies as usual.) The reason force-finish is suppressed is that a snapshot split has to be read in full to yield the log offset that the snapshot ends at, and that offset is what anchors the next incremental round. Truncate the snapshot read part-way and you lose that offset. There's nothing for the following round to start from. So when freshness fires during a PK snapshot round, the force-finish signal is sent but the snapshot reader simply ignores it; the splits keep running until the snapshot finishes naturally. The lake commit lands whenever that happens, no matter how much longer it takes. Force-finish only meaningfully truncates work on log splits, not snapshot splits.

What The Freshness Gives You And What It Doesn't​

Freshness is a target scheduling cadence, not a guarantee of how stale the lake is. The actual lake lag is the time between when a record is written to Fluss and when it appears in the lake, and it oscillates in a sawtooth. The floor is one round_duration: a record written just before a round's cutoff lands in the lake about one round-length later. The peak is roughly freshness + 2 × round_duration: a record written just after a round's cutoff has to wait a full freshness interval for the next round to start, then that round's duration to commit. So a table with 5-minute freshness and 90-second tiering rounds gives you a lake that's between ~90 seconds and ~8 minutes behind.

So the rule of thumb to take away is: floor lag ≈ round_duration, and peak lag ≈ freshness + ~2 × round_duration, meaningfully more than the configured freshness once rounds get long. If your stakeholders say "the lake must never be more than 10 minutes behind", work backwards from the peak: with 90-second rounds, a freshness around 7 minutes keeps peak lag near 10 minutes (7 min + ~3 min). Leave yourself margin.

The Other Timeout You Should Know About​

There's a second timeout, set on the coordinator side and applied to every table regardless of its freshness. It's a 2-minute window, but the thing it measures is easy to misread. It's not a ceiling on how long a round can run; rather, it's a ceiling on how long the coordinator will wait between heartbeats from the Flink job about a given in-flight table.

The coordinator runs a background sweep every 15 seconds. For every table currently in Tiering, it checks currentTime - lastHeartbeat. If that delta exceeds 2 minutes, the table is fenced and transitioned through Failed back to Pending, epoch bumped, any uncommitted lake files orphaned. The lastHeartbeat timestamp gets refreshed on every heartbeat from the Flink job that mentions the table, which by default is every 30 seconds (controlled by tiering.poll.table.interval).

So in a healthy Flink job, the 2-minute timer never fires for round duration. The job heartbeats four times in every 2-minute window, each heartbeat lists the in-flight tables, the coordinator's lastHeartbeat stays fresh, and a single round can keep running for an hour without the coordinator ever raising an eyebrow. The 2-minute window only catches Flink jobs that have genuinely stopped heartbeating; GC pause long enough to look like unavailable, network partition, JobManager crash, that kind of thing.

The Two Ceilings, Side By Side​

Freshness is set per table, fully tunable, and triggers a force-finish on log splits when it fires mid-round. For PK snapshot splits, force-finish is ignored and those splits run to completion regardless, because the snapshot read has to finish to produce the log offset that anchors the next incremental round. The 2-minute coordinator window is hardcoded and applies to every table, but it measures heartbeat liveness, not round duration. A healthy Flink job that heartbeats every 30 seconds can spend hours on a single round without ever triggering it. The 2-minute window is the coordinator's safety net for a job that has actually stopped responding, which is the only failure mode that needs it.

Multiple Tables: How The Queue Actually Works​

Up until now we've talked about one table. Real deployments have many. So what happens when you have, say, three tables (two log tables and one PK table) all sharing a single tiering Flink job?

The coordinator keeps one FIFO queue of pending tables. When a Flink job asks for work via heartbeat, the coordinator pops the table at the front of the queue and hands it over. The Flink job works that table's buckets in parallel and asks for the next table once its splits are all handed out. The coordinator gives out work one table per request. As we'll see, a job's spare readers can start a second table's reads before the first table's commit lands; what stays strictly one-at-a-time is the commit.

This means the effective freshness for any single table is not just its own configured freshness. It's also a function of where it sits in the queue and how long the tables ahead of it take.

A Concrete Walkthrough: Three Tables, One Job​

Here's the setup. Three tables sharing one Flink tiering job that's running with parallelism 16 (so up to 16 readers can run in parallel at any moment). Each table has its own bucket count, freshness target, and expected per-round duration:

Each table generates one split per bucket per round, so a clicks round produces 16 splits, an orders round produces 8 splits, and a customers round produces 12 splits. When the Flink job processes clicks, all 16 of its readers are busy on it at once. orders has only 8 buckets, so at most 8 readers can work on orders itself, but the other 8 don't idle: once orders' splits are all assigned and none are left pending, the enumerator pulls the next queued table and hands its splits to those readers, which start reading ahead. Same with customers (12 buckets): the 4 spare readers pick up whatever is next in the queue. The bucket count is a per-table cap on intra-table parallelism (how many readers one table's round can use), not a cap on how busy the job is. The spare readers pipeline onto the next table (detailed in Reads Pipeline: Commits Serialize, below). What stays strictly one-table-at-a-time is the commit, not the reads.

What Determines Round Duration?​

Before we walk through the example, let's pin down what "30 seconds" or "100 seconds" actually means, because those numbers aren't magic. They emerge from three concrete inputs that every round has to negotiate with:

1. How much data the round has to read. For a log table, this is the volume of writes since the last commit, roughly write_rate × time_since_last_commit. The longer the gap between rounds, the more data each round has to move. For a PK table's first round, the data is the entire KV state; every key in the table, regardless of how long it's been since the last anything. For a PK table's incremental rounds (round 2 and beyond), it's just the changelog records since the last snapshot offset, which is typically much smaller than the full state.
2. How parallel the work is. Bucket count bounds the parallelism within a single round. With 16 buckets, the work is split 16 ways. With 4 buckets, it's split 4 ways, no matter how many Flink readers you have. The Flink job's parallelism setting only matters up to the bucket count of the table currently being tiered for that table's own splits; any extra readers don't wait around: they pick up the next queued table's splits (more on that in a moment).
3. How fast each reader can move data. Bounded by network throughput from the Fluss tablet servers, deserialization speed, and write bandwidth to the lake (S3, GCS, etc.). Real-world numbers are usually a few tens of MB/s per reader.

Putting it together: round_duration ≈ data_to_read / (bucket_count × per_reader_throughput). For our clicks table, which has been collecting writes for ~1 minute, has 16 buckets, and each reader pulls maybe 30 MB/s, a round of a few hundred MB of new clicks events takes roughly 30 seconds. For customers' first round, the full KV state, let's say a few GB, 12 readers at 30 MB/s would take around 100 seconds. The exact numbers depend on your workload, but the shape of the formula is what matters: more buckets shortens a round, more data lengthens it.

One important thing this formula leaves out: the slowest bucket determines when a round can commit, because the commit operator waits for all of them. If your bucketing key has bad distribution and one bucket holds 3× the data of the others, that one bucket sets the round duration: the other 15 readers finish their splits early and move on to the next queued table, but this table's commit still waits on that one straggler. The shuffle on split assignment doesn't fix this; it just makes the bad bucket land on a different reader each round.

A newly created table doesn't tier immediately; it waits one freshness interval before its first round (the coordinator schedules it: New → Scheduled → Pending). So picture T=0 as the moment all three tables have passed that initial wait and are sitting in the coordinator's pending queue, each awaiting its first-ever round. They entered the queue as each table's first scheduling timer fired, in ascending freshness order, so it reads [clicks, orders, customers].

The Flink job's first heartbeat asks for work and gets clicks. The enumerator generates 16 splits (one per bucket), shuffles them, and assigns one to each of the 16 readers. All readers work in parallel; clicks finishes at T=30s.

At T=30s, the Flink job asks for more work. It gets orders. The enumerator generates 8 splits and assigns them to 8 readers. The other 8 readers don't sit idle: with orders' 8 splits assigned and nothing left pending, the enumerator immediately pulls the next table and those readers begin reading ahead (this is the read pipelining we detail below). What we're tracking in this walkthrough is the commit order, and orders is the table committing next: it tiers for about 90 seconds and its commit lands at T=120s, comfortably inside its 2-minute freshness window.

Now customers is next in the commit order. The enumerator generates 12 splits (since this is round 1 of a PK table, these are snapshot splits reading the full KV state). 12 readers can work on its splits at once; the 4 spare readers, again, pick up whatever's next in the queue rather than idling. The round takes roughly 100 seconds, and customers' commit lands at T=220s, well inside the configured 5-minute freshness.

But notice one thing here. clicks committed at T=30s; with a 1-minute freshness, its next round was scheduled for T=90s, so it's been waiting in the queue since T=90s. Its reads may well start earlier (spare readers can pick up clicks round 2 while customers is still being read), but its commit can't land until orders and customers have committed ahead of it. So clicks' round-2 commit doesn't land until around T=220s, putting its actual lake lag near 3.2 minutes despite being configured for 1-minute freshness. The fact that clicks uses all 16 readers when it does run didn't help. The constraint isn't how fast any one table reads; it's that the job dispatches one table at a time off the FIFO queue and commits one table at a time through a single-parallelism operator. Here clicks round 2 is re-queued only at T=90s, behind orders and customers, so it's both dispatched and committed last. The configured freshness value is the target; the queue is the constraint.

There's also a sizing lesson here, but it's subtler than "don't over-provision." It's tempting to think that setting Flink parallelism to 16 wastes slots whenever a table has fewer than 16 buckets (8 for orders, 12 for customers). It mostly doesn't: because spare readers pipeline onto the next queued table, those slots get spent reading ahead as long as there's a backlog. Parallelism above your largest table's bucket count isn't automatically wasted: with several tables queued, the extra readers stay busy on other tables' reads. It only goes to waste when you rarely have more than one table's worth of work in flight at once.

What parallelism can't fix is the commit path. The commit operator runs at parallelism 1 and serializes one table at a time, so no amount of extra parallelism speeds up the rate at which lake snapshots actually land.

A reasonable rule of thumb: set Flink parallelism at least as high as the largest bucket count of any single table (so your biggest table can use all its buckets in one round), and add more only if you routinely have several tables queued and want their reads to overlap. Size bucket counts at table-creation time based on each table's data volume and freshness target. And if you need more commit throughput than one job can deliver, that's the cue to run multiple tiering jobs (next section), not to crank up parallelism.

Reads Pipeline: Commits Serialize​

The diagram above shows when each table commits. The reads underneath those commits don't line up the same way; when a table doesn't have enough splits to keep all 16 readers busy, the Flink enumerator pulls the next table from the queue and the spare readers immediately start chewing through its splits. So reads overlap across tables. Commits don't.

The mechanism is simple. The enumerator maintains one shared pendingSplits list across all in-flight tables plus a set of readersAwaitingSplit. It asks the coordinator for the next table only when both pendingSplits is empty and at least one reader is idle. While clicks (16 splits) is running, that pair of conditions never holds simultaneously: all readers stay busy until the splits drain together at T=30s. But the moment orders (8 splits) is assigned, 8 readers are idle and pendingSplits is empty, so the enumerator immediately pulls customers from the queue, and those 8 idle readers start chewing through customers' snapshot splits while orders is still being read. The same pattern repeats when orders finishes: the freed readers pick up the remaining customers splits and then clicks round 2 (which has entered the queue by then), so customers and clicks round 2 overlap during phase 3.

The commits, however, can't overlap. The commit operator runs at parallelism 1 by design, processing one table's commit at a time. It commits a table the moment it has collected all of that table's bucket write results for the round, so commits land in completion order, not coordinator-queue order: a later-dispatched but faster table can reach the committer first and commit before an earlier, slower one when their reads overlap. What is guaranteed is that commits never run concurrently, lake snapshots land strictly one after another. In our example the rounds happen to complete in queue order (clicks saturates all 16 readers and finishes first, then orders, then customers), but that ordering is a property of this workload, not a promise the committer makes.

That's why queue position still dominates effective freshness, even with read overlap. clicks' round-2 reads can start while customers is still being read, but every commit is serialized through the single committer subtask, so a table's commit can't land until whatever the committer is currently working on is done. The lake-side appearance of "the round committed" is bottlenecked by the commit operator, not the readers, which is the exact mechanism the commit timeline was measuring.

The General Rule​

For N tables sharing one tiering job, the effective worst-case freshness of any single table is roughly the sum of all the tiering round durations, not the configured value of that one table. Configured freshness is more of a target; the queue is the constraint.

This is fine when your tables are similar in size and freshness. It's a disaster when you mix tiny fast tables with huge slow ones. The little ones get starved while the big one finishes.

The Escape Hatch​

There's a clean fix: run more than one tiering job. The coordinator's pending queue is a shared resource, and any Flink tiering job that's registered with the cluster can pull from it. Two jobs, two tables tier at once. Three jobs, three. The starvation pattern softens as soon as the number of concurrent jobs is at least as large as the number of latency tiers you actually care about. The next section is the operational walkthrough.

Scaling Out: Running Multiple Tiering Jobs​

The previous section left us with a clean problem statement: one Flink tiering job serializes work across all tables, so freshness for any one table is bounded by the queue depth ahead of it. The available knob for scaling out is to run multiple Flink tiering jobs against the same Fluss cluster. It works, but it's a blunter instrument than you might hope. The current implementation gives you concurrency, not isolation, and the section below walks through exactly what's in the code and what scaling out can and can't do.

The Architectural Approach​

Up to Fluss 0.6, the tiering service was a single stateful Flink job. The per-table sync offset, the last lake-committed offset for each bucket, was stored inside Flink's checkpointed state. Practically, that meant you could scale the one job up (more parallelism, more slots) but you couldn't run two jobs; they would have stomped on each other's state.

Fluss 0.7 re-architected the service to be stateless. The sync offset moved from Flink state into Fluss metadata. The coordinator owns it, durably, regardless of which Flink job most recently tiered a given table. The TieringSourceReader is now explicitly stateless and its snapshotState returns an empty list by design. All the truth about "where is this table in its tiering history" lives in Fluss, not in any particular Flink job. That single change is what makes horizontal scaling safe at all. Two jobs can both ask the coordinator "what's next?", and neither holds state that the other could clobber.

The Mechanics: One Queue, Undifferentiated Workers​

The coordinator maintains exactly one in-memory FIFO queue of pending tables; pendingTieringTables in LakeTableTieringManager. When a heartbeat comes in carrying request_table=true, the coordinator's requestTable() just pops the head: Long tableId = pendingTieringTables.poll(). There's no filter parameter, no caller identity, no routing logic. The protobuf LakeTieringHeartbeatRequest doesn't carry any "which job am I" field, because the concept doesn't exist on the coordinator side.

So from the coordinator's perspective, every Flink tiering job is indistinguishable. There's no job ID, no database filter, no notion of "this table belongs to that job". All registered jobs are undifferentiated workers reaching into the same queue, and the head of the queue goes to whoever happens to heartbeat first. If you have two jobs, two tables can be in Tiering at the same time. If you have five, five can. But you cannot pin clicks to job A; the coordinator wouldn't know how to honor that pin even if you asked.

The epoch mechanism from Part 1's heartbeat section keeps this safe regardless of how many jobs are running. Each table assignment carries a tiering_epoch stamped on it. If two jobs somehow ended up working on the same table (a rare edge case during coordinator failover, mainly), the coordinator only accepts the commit whose epoch matches its current record; the other is rejected with an epoch-fencing error. So multiple jobs running concurrently can never produce duplicate commits or corrupt lake state; the worst case is some wasted reader work that doesn't get committed.

The Earlier Example, With Two Jobs​

Let's revisit the previous walkthrough with two tiering jobs instead of one. The setup is unchanged. Three tables: clicks (1-min target, 30s round), orders (2-min target, 90s round), and customers (5-min target, 100s first PK round). At T=0 all three are sitting in the queue: [clicks, orders, customers].

T=0s. Both jobs heartbeat asking for work. Job A's heartbeat happens to land first and gets clicks. Job B's request gets orders (now head of queue). Both jobs work in parallel; the queue contains just customers.

T=30s. Job A finishes clicks, asks for next, and gets customers. Job B is still working orders.

T=90s. Job B finishes orders, asks for next. Queue is empty, so Job B is briefly idle. At the same moment, clicks' 1-minute freshness timer fires (last commit at T=30s). It enters the queue. Job B picks it up on its next heartbeat (≤30s later).

T=120s. Job B finishes clicks round 2. Effective freshness gap for clicks: about 90 seconds, against a 1-min target. Compare with ~3.2 min on one job.

T=130s. Job A finishes customers first round. Cold tier idle until the next freshness firing 5 minutes later.

The improvement is real, but notice it's stochastic, not architectural. Job A got clicks first only because its heartbeat happened to arrive first at T=0. You cannot promise "clicks always lands on the responsive job."

In practice the pairing tends to be sticky: once a job has been running short rounds it keeps becoming idle first and keeps picking up the next short round, so over time you do see something close to "fast job handles fast tables." But it's an emergent property of round timing, not a configured guarantee: if the workload shape shifts (a previously fast table grows, a slow table is dropped, a new heavy table is enabled), the pairing reshuffles on its own. Don't lean on it as if it were a guarantee.

What Scaling Out Fixes And What It Doesn't​

Scales: queue throughput (with N jobs, up to N tables can be in Tiering simultaneously); failure-domain separation (if one job crashes, the others keep tiering against the same Fluss cluster); and capacity headroom (bursts of "many tables freshly ready at once" get absorbed across multiple jobs rather than stacking up serially behind one).

Does not scale: any single table's round duration. If customers' first PK round takes 100 seconds with 12 buckets and one job, it still takes 100 seconds with two jobs, because only one job is working on customers at a time. Multiple jobs give concurrency across tables, not within one. The lever for a single round is bucket count and per-job parallelism, not job count.

Does not give you: deterministic routing. There is no current Fluss configuration that lets you say "this table goes to Job A, that one to Job B." The coordinator cannot tell jobs apart; they're identical to it. If you need hard isolation between latency tiers, the current options are limited:

Does not bypass: the 2-minute heartbeat-liveness window from the freshness section. Every job is subject to the same hardcoded check: if any one job stops heartbeating, the coordinator fences its in-flight tables and another job picks them up. Scaling out doesn't change this; it just spreads the heartbeat burden across multiple JobManagers.

What's Next?​

You now know all the dials, from per-table settings like bucket count and freshness, through the multi-table queue dynamics, to the deployment-shape choice between one tiering job and several. Everything you've read so far has been about how the system behaves. Part 3 is about what you do with it.

If you're new to Fluss, the lake-tiering story is one of those topics where every explanation seems to assume you already know how it works. This three-part walkthrough aims to bring some clarity to the confusing parts of the system, and to help you understand how it works in practice.

Part 1 builds the mental model from scratch and by the end of it you'll be able to describe, step by step, what happens between the moment a tiering timer fires and the moment a lake snapshot is committed.

Part 2 and Part 3 take that mental model and add the dials (parallelism, table kinds, freshness, multi-table behavior, scale-out) and then put it into a real production deployment (failures, pitfalls, monitoring).

Tiering Service Deep Dive, 3-parts:

• Part 1 - The Mental Model: how one tiering round actually works, from timer fire to lake commit.
• Part 2 - Tuning: per-table dials, multi-table dynamics, and scaling out.
• Part 3 - In Production: failure modes, design pitfalls, and monitoring.

Fluss is bounded in capacity by design and that's the entire reason the tiering service exists. Its speed comes from keeping data on local disks attached to a small number of machines. Local disk is finite, costs money to keep spinning, and is sized for the working set, not for a year of history. Fluss isn't your archive, it's the freshest layer of your platform. So old data has to leave, on a schedule, into storage designed for the long haul. That's the job of the tiering service.

Meanwhile, you have analysts and batch jobs and ML/AI pipelines who don't need millisecond freshness, but they need to query a year of orders, run a join over a quarter of clicks, train a model on six months of sessions. That's the job of the data Lakehouse. Parquet files on S3, organized as open table formats (Iceberg, Paimon, Hudi, Lance), queryable by Spark, StarRocks, DuckDB, Trino, Flink-batch or even just Python.

So you have two storage layers: the hot one (Fluss) and the cold one (the Lakehouse), exposed as one single table abstraction with different freshness layers, and you need something that moves data from one to the other on a regular schedule, in a way that's correct, atomic, and survives failures. That something is the tiering service.

The tiering service does three things at once.

1. It keeps the Lakehouse fresh: within some configurable target like "never more than two minutes behind".
2. It lets Fluss reclaim space: once data is safely in the Lakehouse, Fluss can delete its own copy.
3. And it does both atomically: the Lakehouse either sees a complete update or no update at all, never a half-written mess.

The tiering service is just a stateless Flink (Spark support is also WIP) streaming job that runs continuously. It reads from Fluss, writes Parquet files to S3, and tells the Fluss coordinator "I committed this offset range, you can release it now."

Everything else in this article is about the details, but if you remember "it's a stateless Flink job that copies data, on a schedule, in atomic batches," you've got the shape of it.

The Three Processes And What Each One Does​

Three processes are involved in tiering. They live in different places, they have different responsibilities, and they almost never overlap. Knowing which is which is half the battle of debugging anything.

The Coordinator​

The Fluss coordinator is the brain of the cluster that handles metadata, leader election, and cluster state. Tiering is one of its many responsibilities. It keeps a list of which tables need to be tiered, when each was tiered last, and which one should go next. It does not move any data itself; it decides what runs and tracks what's in flight.

The Flink Tiering Job​

This is a stateless Flink streaming job that you deploy. It runs forever. Every so often, it calls the coordinator and asks "what should I work on next?". When the coordinator hands it a table, the Flink job reads that table's data from the Fluss tablet servers, writes it out as Parquet files in the lake, and commits a lake snapshot. Then it reports back, "done, you can mark this table as tiered" and asks for the next one.

The Fluss Tablet Servers​

These are the Fluss processes that actually hold your data. The tiering job reads from them but never writes to them. They mostly don't know they're being tiered, they just serve reads.

The Data Lakehouse​

S3, GCS, or whatever object store you're using. The open table format (Paimon, Iceberg, Hudi or Lance) defines how files and metadata are laid out so that query engines can read them. The tiering job writes here. Nobody else touches it during tiering.

Why Split It This Way? Separating the "who decides" from the "who moves" is a deliberate design choice. The coordinator already manages cluster state. Making it also responsible for scheduling tiering is a small addition. Putting the actual data movement in Flink means you get all of Flink's benefits for free, parallelism, checkpointing, restart-from-failure, without having to build any of it from scratch.

The Heartbeat​

The coordinator and the Flink tiering job are different processes on different machines. They need a way to talk. They have exactly one channel: a single RPC called the heartbeat, sent by the Flink job to the coordinator on a configured interval (tiering.poll.table.interval, default 30 seconds). Every piece of communication between them happens inside this one call.

That makes the heartbeat a four-purpose message:

The Flink job's heartbeat carries:

1. What it's currently working on: the list of tables it's tiering right now
2. What it just finished: tables whose Lakehouse commits succeeded since the last heartbeat
3. What failed: tables whose tiering blew up for some reason, and
4. "Give me another": a boolean flag asking for the next table from the queue. The coordinator's response carries either a new table assignment or a "nothing for you right now" message.

Two things from this design that are worth remembering.

First, the coordinator only learns about in-progress work at heartbeat boundaries. The heartbeat cadence is for polling, "anything new for me?" Completion is different: when a table finishes (or fails), the enumerator triggers an immediate heartbeat rather than waiting for the next tick. So the worst-case lag of "up to 30 seconds" applies to mid-round status, not to commit announcements.

Second, every message has an "epoch" number stamped on it. Each time a table starts a new tiering attempt, its epoch goes up by one. If a Flink job tries to report success with a stale epoch (because the coordinator already gave up on that attempt and handed the work to someone else), the coordinator just ignores the message. This is how the system stays consistent even when things go sideways and we'll come back to it in Part 3, in the failure-modes section.

The Life Of A Table: Four States, Walked Through​

Every table that's enabled for lake tiering goes through the same lifecycle. The coordinator tracks each table's current state. For our purposes, four states are enough to keep in your head:

Note: The actual source code uses seven state names: NEW, INITIALIZED, SCHEDULED, PENDING, TIERING, TIERED, FAILED. Here we collapse them into four pedagogical states to keep the mental model small.

NEW is only used for the very first time a lake-enabled table is created, and INITIALIZED is only used for tables the coordinator rediscovers after a restart. Both transition into SCHEDULED immediately, so we ignore them here. FAILED is the unhappy-path state we'll come back to in Part 3.

WAITING. The table has been tiered recently and the freshness timer is counting down. Nothing to do.

READY. The timer has fired. The table is now in the coordinator's pending queue, waiting for a Flink job to pick it up on the next heartbeat. It might wait a few seconds, or several minutes if there are many other tables ahead of it.

TIERING. A Flink job has picked the table up and is reading its data, writing Parquet files to the Lakehouse, and aiming for a commit. The coordinator has started a clock; if this takes too long, it'll time out.

DONE. The Lakehouse commit landed. The coordinator records "this table was last tiered at time T" and immediately schedules the next round. The table goes back into WAITING.

The happy path is just WAITING → READY → TIERING → DONE → WAITING → ... forever. The unhappy path, which we'll cover in Part 3, in the failure-modes section, is when TIERING ends in failure (timeout, reader crash, anything) and the table cycles back to READY without ever reaching DONE.

A Complete Tiering Round, Step By Step​

Three concepts are important to understand when you're reading this:

• Bucket: A horizontal partition of a Fluss table. If a table has 16 buckets, every record is hashed into one of the 16 by some key. Buckets are the unit of parallelism for both writes (different writers can land in different buckets) and reads.
• Split: A unit of work for the Flink tiering job. Each split says "read offsets X to Y of bucket B". During a tiering round, the planner generates one split per bucket of the table being tiered.
• Reader: A Flink subtask that processes splits. If you set the Flink job's parallelism to 16, you have 16 readers. Each reader processes one split at a time and asks for another when it's done.

Let's walk through a single tiering round with a concrete example.​

We have one table, called orders, with four buckets. It's a log table (we'll talk about what that means in Part 2, when we cover table kinds. For now, just think "an append-only stream of records"). Freshness is configured to five minutes. The Flink tiering job is up and running.

Here's what happens:

T+0s the timer fires.​

Five minutes have passed since the last tiering round for orders. The coordinator transitions the table from WAITING to READY and pushes it onto the back of the pending queue. The act of entering the pending queue is also what increments the table's tiering epoch, to (say) 7, so every fresh attempt is stamped at enqueue time, before any job has picked it up. Right now the queue contains just orders.

T+0s the Flink job sends its heartbeat.​

The job is idle, there is no work in progress, so its heartbeat says "give me something". The coordinator pops orders off the queue, transitions it to TIERING, and replies "your next table is orders, epoch 7". The epoch was already set when the table entered the queue. requestTable() just reads it and hands it back, it doesn't bump it again.

T+0s the Flink job plans the work.​

The job's enumerator (the part of a Flink source that decides what each parallel reader will do) asks the Fluss tablet servers: "what's the latest offset for each bucket of orders right now?" Let's say the answers come back as [bucket-0: 1,000; bucket-1: 1,200; bucket-2: 950; bucket-3: 1,100]. The enumerator also looks up where the last tiering round left off; the "last committed lake offset", let's say [bucket-0: 990; bucket-1: 1,190; bucket-2: 945; bucket-3: 1,090]. The difference is the work for this round.

T+0s splits are created.​

The enumerator builds four splits, one per bucket. Each split says "read from offset X to offset Y of bucket B". It shuffles them randomly (so no reader always gets the heaviest bucket) and assigns one split to each of the four Flink readers running in parallel.

T+0s to T+60s the readers work.​

Each reader streams its assigned offset range from a Fluss tablet server and writes the records as Parquet files into a staging area in the lake. The buckets process independently and at different speeds. When a reader finishes its split, it emits a small message saying "bucket B is done, here are the files I wrote, the last offset I read was Y".

T+60s the commit operator collects all four results.​

A dedicated Flink operator called the commit operator sits at the end of the job. In the normal path it waits for results from all four buckets, then commits them together as a single atomic lake snapshot. This gives you a clean, all-or-nothing update to the lake table. Once it has all four, it calls into the open table format and writes a snapshot that includes all the new Parquet files plus the updated per-bucket offsets. (Force-finish, which we'll meet in Part 2, in the freshness section, is the one exception; the commit operator still waits to hear back from every bucket, but readers that ran out of time emit an empty result, and the commit only includes the buckets that did produce data. Anything skipped is picked up in the next round.)

One detail that's easy to miss: the lake snapshot is durable at T+60s, but the coordinator hasn't transitioned the table to DONE yet. That happens on the next heartbeat from the enumerator.

T+60s the commit operator records the lake snapshot back to Fluss.​

Now that the lake snapshot is durable, the commit operator (TieringCommitOperator) records the new snapshot offsets into Fluss metadata through the coordinator. It delegates this to a helper, FlussTableLakeSnapshotCommitter, which implements a two-phase commit (prepare + commit) against the coordinator gateway rather than RPC'ing tablet servers directly. This lets the cluster know that data up to those offsets is safely in the lake and can be aged out of Fluss's hot storage when the time comes. Tablet servers learn about the new tier point through the standard metadata-propagation path, not via a direct RPC from the committer.

T+60s the commit operator notifies the enumerator.​

Immediately after the commit, the commit operator sends a FinishedTieringEvent back to the source enumerator. As we said earlier, the enumerator doesn't wait for the next 30-second poll tick. It fires an out-of-band heartbeat right away, carrying orders in the finished_tables list. The coordinator validates the epoch (still 7, all good), records the completion time, transitions orders to DONE, and immediately schedules the next round.

Scheduling the next round.​

The freshness timer is computed from when the round completed, not from when it started. The coordinator does delay = freshness − (now − lastTieredTime), and lastTieredTime is set to the moment the completion heartbeat is processed. So with a five-minute freshness, the next round fires roughly five minutes after the commit, i.e. at about T+360s. This is the (intentional) reason the lake always lags by at least one full freshness interval plus one round duration.

That's it. Round complete. The lake now contains orders data up to the offsets that were current at T+0s. In another ~five minutes, the whole process repeats.

The most subtle point in this section: The "stopping offset" is decided at planning time, not at reading time. When the enumerator asks the tablet servers "what's your latest offset?" at T+0s, those answers freeze. Any records written to Fluss after T+0s but before the readers actually start working are not part of this round; they'll be picked up in the next one. This is correct behavior, but it means the lake always lags by at least one round, even if your freshness is configured aggressively.

Next Up​

You've now got the mental model, the processes, the heartbeat conversation, the lifecycle, and what happens during one full round. The next part takes that round and shows what changes when you start tuning the dials, and then what changes again when you stop thinking about one table at a time.

Apache Fluss stores data in three places: local disk on the tablet server, remote object storage like S3, and the lakehouse. Which place holds which data at any given moment, and what is responsible for moving it between them, is the foundation everything else rests on. Your capacity plan depends on it. Your latency targets depend on it. Your disaster-recovery story depends on it. So does your ability to predict, in advance, that a particular configuration change is going to fill up local disk a week later.

This post walks through that layering. We'll cover what each tier holds, the two background tasks that move data between them, what changes for primary-key tables, and how recovery actually works when a tablet server loses its disk.

By the end you should be able to look at a Fluss deployment and say, for any given record, where it lives right now and where it will live in an hour.

The Three-Tier Storage Hierarchy​

Tier 1 is local disk on the tablet server. It holds the hot data: recent log segments, the full live RocksDB state for every primary-key table, and a staging view of the most recent KV snapshots (hard links to live SST files while uploads are in flight). Reads from this tier are in milliseconds.

Tier 2 is remote object storage (S3, GCS, or similar), used for two distinct purposes that share the same remote.data.dir filesystem.

• First: older log segments uploaded by the remote-log tiering task in Fluss's native binary format, which extends local retention without growing local disk.
• Second: durable KV snapshots for every primary-key table, uploaded periodically so that a tablet server can recover after disk loss.

Remote log storage is enabled by default. It's controlled by remote.log.task-interval-duration (default 1min), and is only disabled when that value is set to 0. KV snapshot upload is independent of remote-log tiering and is governed by kv.snapshot.interval (default 10min).

Tier 3 is the lakehouse. Paimon, Iceberg, or Lance are holding data in analytical file formats queryable by any engine. Hudi support is in active development under FIP-24. Reads from the lakehouse cost seconds.

A Note On Single-Copy Storage​

A Fluss table is a single logical abstraction across all three tiers, each holding data at a different freshness level. Local disk has the hot, most recent data; remote object storage extends retention beyond what fits locally; the lakehouse holds the analytical projection. Across those tiers, each record has one home at a time and no tier permanently holds a second copy of what another tier already owns.

There is one temporary exception: when lakehouse tiering is enabled, a remote log segment is only deleted once both its TTL has expired and the lakehouse has ingested it. That's a safety net against lakehouse lag, and it creates a bounded transition window where the same data exists in both Tier 2 and Tier 3, governed by table.log.ttl (default 7 days). Shorten the TTL if minimizing that overlap matters more to you than a long lakehouse catch-up window.

Once the lakehouse catches up, the remote copy is removed and the overlap closes.

Log Tables on Local Disk​

A log table on disk is a sequence of log segments. Each segment is a .log file holding raw records alongside a small set of companion files: a .index offset index and a .timeindex time index for fast seek, plus per-segment writer-state snapshots used for idempotent producers. The active segment is open for appends; every other segment is immutable and named by its starting offset.

What happens to those sealed segments is governed by two retention controls.

• table.log.ttl (default 7 days) is the global retention contract for the log. It defines the maximum age of any log data in the table, regardless of which tier it currently lives on.
• table.log.tiered.local-segments (default 2) is a count-based floor for local disk, only meaningful when remote-log tiering is on. The remote-log task keeps at least this many recent segments on local disk after upload, so consumers reading near the head don't pay an S3 round-trip for the freshest data.

A segment becomes a candidate for upload the moment it is sealed and its records are below the high watermark (i.e., committed/acked). Sealing happens when the active segment hits its size threshold, fills the offset or time index, or can no longer encode records as relative offsets, and then rolls over: Fluss closes the current active segment (which becomes immutable) and opens a new one for subsequent writes. The freshly-closed segment is now something the remote-log task can pick up on its next pass. This is the same model Kafka uses for tiered storage, and it has the same operational consequence: data sitting in the active segment lives only on the local Fluss server until rollover, which means the active segment's size threshold sets an upper bound on how fresh remote-tier data can be. Anything newer than the current rollover is local-only.

What remote-log Tiering Actually Does​

By default, remote-log tiering is on (remote.log.task-interval-duration=1min) and TTL is 7 days. The remote-log task does three things on each pass:

1. Uploads newly-sealed segments to S3.
2. Advances the local log's remoteLogEndOffset, which causes the local log to trim every sealed segment now in S3, keeping at least table.log.tiered.local-segments recent ones.
3. Deletes S3 segments past TTL.

Local disk is bounded primarily by the count-based floor, usually a handful of recent segments. The TTL value applies most visibly on the S3 side, because S3 is where data lives the longest.

Disabling Remote Tiering​

Setting remote.log.task-interval-duration=0 opts out of Tier 2 entirely, but this comes with an additional consequence: it also disables the scheduled cleanup task itself, because that task is what runs both the upload and the segment-deletion paths. With the task disabled, nothing trims local segments. There is no automatic fallback to the lakehouse on the write path.

The end result is unbounded local-disk growth. Eventually the tablet server runs out of disk and write batches start failing with storage exceptions.

Remote Tiering and Lakehouse Tiering Are Different Features​

These two features are frequently conflated, which is fair because the names suggest a relationship, but they solve different problems and produce different output.

Remote tiering is about disk economics on the tablet server. It copies raw log segments in Fluss's native binary format to S3, extending local retention without growing local disk. The tablet server can then read from S3 when a consumer requests an offset that has been trimmed locally. It's managed entirely server-side, by a background task. As a side effect, it's also the only mechanism that trims local log segments.

Lakehouse tiering is about analytical access. It converts Fluss data into lakehouse-native formats, like ORC, Parquet, Lance and writes them to the lakehouse via an external Flink job (the Tiering Service). The output is queryable by Spark, Trino, and Flink independently of Fluss.

These are complementary layers. You can run any combination of them. When both are enabled, the lakehouse confirmation acts as an additional safety gate on top of TTL-based S3 deletion: a remote log segment is not expired until TTL has passed and the lake has confirmed it.

Primary-Key Tables​

A log table has one thing on disk: the log. A primary-key table has three. They serve different roles, they live in different places, and they fail in different ways. Operating primary-key tables without seeing them as three distinct structures is one of the faster routes to a confusing production incident.

Structure 1: Live RocksDB Store​

This is the current state of the table. One entry per primary key, always up to date, sitting on the tablet server's local disk inside a RocksDB instance. Every point lookup reads from here. Every upsert merges into here. The live store is created when the tablet opens and deleted only when the table is dropped.

Nothing moves the live store. There is no setting that puts it on S3, in the lakehouse, or anywhere else. RocksDB on local disk is where the work happens, and that's the only place it can happen.

The role to understand here is "what serves traffic," not "what is durably stored." The live store is what serves traffic. What survives a disk loss is the snapshot in remote storage, which is Structure 2. Two roles, two copies, related data. You need local disk for the full merged state of every bucket the tablet server is responsible for; the lakehouse cannot stand in for this, and the tablet server doesn't read PK state from the lake under any circumstance.

Structure 2: KV Snapshots​

Every ten minutes by default (kv.snapshot.interval=10min), the tablet server takes a snapshot of the live RocksDB and writes it to remote storage. This is the system's only durable record of the table's merged state at a point in time. If the tablet server's local disk evaporates, recovery begins from the most recent snapshot and then replays the changelog forward from that snapshot's offset to reach the present. The two-stage process described in the Recovery section below.

Step one happens locally and completes immediately. The tablet server hard-links the current RocksDB SST files into a staging directory. No bytes are copied, just new pointers to existing files. This is what lets a snapshot start instantly regardless of how large the table is, because nothing is being duplicated on disk.

Step two is the one that actually moves data. Those files, plus a bit of metadata, get uploaded to remote storage. The remote copy is the durable one; the local staging directory is there so the uploader sees a frozen, consistent view of the files while RocksDB keeps writing and compacting underneath it. The snapshot is considered durable once the upload finishes.

Fluss keeps the last two snapshots in remote storage by default. When a new snapshot supersedes an old one, the old one is deleted, with one guard: if anything (most commonly a long-running lakehouse tiering job on its first round) is still reading the older snapshot, a lease prevents the cleanup from removing it underneath the reader. This sounds like a detail, and it is most of the time. It becomes load-bearing the first time a large primary-key table takes longer to tier than the gap between snapshots, and the lease is what keeps the system from racing itself.

Structure 3: The Changelog​

Every upsert and every delete also gets appended to a log, in the order it happened. This log behaves exactly like a regular log table on disk, same retention rules, same tiering to remote storage, same handoff to the lakehouse.

Two things make the changelog different from the rest of the primary-key table.

It grows with the number of writes, not the number of unique keys. A primary-key table that updates the same 100 keys ten million times has a small live store and an enormous changelog. RocksDB collapses by key; the log does not. This is what makes the changelog useful as a CDC feed · downstream consumers see every change in order, not just the latest value.

Deleting old changelog segments has no effect on the live store. The live store is complete on its own; it doesn't need the log to know the current value of any key. The log is there for replay (when a tablet needs to recover) and for downstream feed (when something is reading change events). It is not a place where state lives.

Note: This is a simplified version of the changelog for illustrative purposes.

Recovery: Independent Tracks, Coupled Outcomes​

The snapshot upload track and the log upload track look independent from a configuration standpoint. Separate settings, separate schedulers, separate remote subdirectories. They are not independent when you actually need to recover from disk loss.

Recovery on a fresh tablet server works in two stages. The snapshot brings the live state up to whatever point it was taken at. The changelog then replays every change since that point to catch up to the current moment.

If remote-log tiering is off, that changelog tail lives only on the failed tablet server's local disk, which is the disk you just lost. The snapshot, however durably stored, can only restore the state as of its own offset. Everything written since then is gone.

The two upload tracks are independent on the way in. The recovery story stitches them back together on the way out, and breaks if either piece is missing.

Standby Replicas​

Standby replicas are a new feature. The election machinery that designates a follower as the standby has landed (#2828). The hot-standby path that keeps the standby's RocksDB current with the leader, and the bootstrap path that downloads the latest KV snapshot when a fresh server becomes a standby, are tracked by PR #2835. The description below reflects the target behavior once those land.

Everything described so far is the cold-start path; the one that runs when no other copy of a bucket is still alive. Most production recoveries aren't cold restarts.

Fluss replicates each bucket across multiple tablet servers: one leader handling writes, plus followers continuously tailing the same log. One of those followers is the designated standby, the replica the controller will promote on leader failure, and the one that maintains a live RocksDB kept current with the leader in near real time.

When the leader fails, the controller promotes the standby. The standby's live RocksDB is already current, so traffic resumes in seconds, with no S3 download and no log replay. The snapshot path still matters, it's the safety net when an entire replica set is lost at once, when a bucket gets reassigned to a brand-new tablet server, or when a fresh follower is bootstrapping into the cluster. But that path is the fallback, not the everyday failure handler.

This refines the framing of remote storage. Calling it the recovery substrate and the durability floor was accurate. It just isn't the recovery path you exercise most often in healthy production. The everyday path is one replica picking up where another left off, which is precisely why running with replication factor 1 in production is a bad idea, however durable your snapshots are.

Combining Tiers​

There are four ways to combine remote-log tiering and Lakehouse tiering. Three are useful; one isn't.

The primary-key snapshot track is orthogonal to all of this. It runs on its own cadence (kv.snapshot.interval, default 10 minutes), writes to its own remote subdirectory (/kv), and is what makes primary-key tables recoverable after disk loss. Disabling remote-log tiering does not disable KV snapshot upload. Three independent tracks, three independent config keys · the configuration vocabulary does not make this obvious, but the runtime behavior does.

Closing Thoughts​

Fluss's storage layer is structurally simple -- three tiers, two background tasks -- and the simplicity is what makes it easy to misread.

• Tier 1 looks like the tier that matters, because it's the only one on the live query path.
• Tier 2 looks like an implementation detail, because it's "just S3".
• Tier 3 looks like a destination, because it's the lakehouse.

Each shortcut is wrong in a way that only becomes visible after you've configured something based on it.

The model here is: three tiers with three different jobs, two background tasks that should be reasoned about independently, a small set of defaults deliberately tuned for production.

Disabling those defaults is almost always the wrong move. Tuning them to your workload is almost always the right one. 🌊

TL;DR:

Apache Kafka's "column pruning" is actually pseudo-pruning. All fields still cross the network, and clients discard unwanted ones after the fact. Apache Fluss redesigns the storage format, server-side read path, and write-side batching strategy from the ground up with Arrow IPC columnar storage, zero-copy server-side pruning, and client-side pre-shuffle batching. The result: pruning 90% of columns yields a 10x read throughput improvement, with performance scaling linearly with the pruning ratio.

What Is Column Pruning? Why Does It Matter?​

In real-time big data processing, data often contains dozens or even hundreds of fields, yet downstream applications typically only need a small subset. Column pruning means reading only the columns a query requires, skipping irrelevant ones to reduce disk I/O, network transfer, and computation overhead.

For example, consider a table with columns a, b, c, d and the following query:

SELECT a FROM t WHERE b > 5;

This query only uses columns a and b. Columns c and d are entirely unnecessary. If the storage engine can skip c and d during reads, it dramatically reduces the data that needs to be read from disk and sent over the network. That is column pruning.

Column pruning has long been widely adopted in batch processing systems (e.g., Hive, Spark SQL with Parquet or ORC) with significant results. But in the streaming domain, it has always been a hard problem because mainstream messaging systems like Kafka have no native concept of columnar storage.

In production, a wide table may have hundreds of columns, while a downstream query typically uses only 5–10. In such cases, column pruning can reduce data transfer by an order of magnitude.

Why Can't Apache Kafka Do Column Pruning?​

The Kafka Broker Has No Schema Awareness​

Apache Kafka's broker treats records as opaque byte arrays and has no understanding of the data structure within. Schema parsing relies on client-specified formats such as JSON or Avro. This means that even if a consumer tells the broker "I only need columns a and b," the broker cannot understand or act on that request.

Kafka Uses Row-Based Storage​

Kafka stores each message as a contiguous byte sequence containing all fields. Because all fields are packed together, there is no way to skip a subset of columns during a fetch. The entire message must be read and transferred.

"Pseudo-Pruning": Pruning Happens on the Client, Not the Server​

The current Flink and Kafka column pruning implementation runs entirely at the Flink Source (client side). The Kafka broker still sends the full record to the consumer, which then discards irrelevant columns through field mapping during deserialization.

Figure 1: Kafka's "Pseudo-Pruning" -- pruning happens on the client side, not saving any network I/O

As shown above, the entire pruning process can be summarized as: the Kafka broker sends full records to the Flink Source, which prunes fields row by row during deserialization, then emits the pruned output.

The most expensive step, the network transfer from broker to consumer, is not optimized at all. All columns are still transmitted over the network; pruning merely filters them out at the destination.

Core Challenges of Column Pruning in Streaming Storage​

Introducing column pruning into streaming storage is fundamentally different from doing so in traditional OLAP or data lake systems. OLAP systems process bounded data and can perform extensive sorting and encoding optimizations after writes are complete. Streaming storage, by contrast, faces continuous-append and real-time consumption workloads with strict latency and throughput requirements. This creates three distinct challenges.

Challenge 1: Column Pruning vs. Streaming Writes, a Fundamental Storage Layout Conflict​

Efficient column pruning requires columnar storage, where data for each column is stored contiguously so that irrelevant columns can be skipped entirely during reads. But streaming storage systems are inherently row-append-based: data arrives record by record, each containing all fields, and must be persisted promptly. This creates a fundamental contradiction: data arrives in rows but must ideally be stored in columns.

During Fluss's evolution, we explored a compromise: the Indexed Row format. It is a row-based format with field-offset indexes in each record's header, allowing direct jumps to target fields without parsing the entire record. The results were disappointing. Indexed Row is still fundamentally row storage: all fields within each record are packed together, so reads still require row-by-row scanning. Its performance gains in column pruning scenarios were minimal, and this approach was ultimately a dead end.

The real solution is to adopt a true columnar storage format. The most widely adopted columnar file format in batch analytics is Parquet. However, Parquet is designed entirely for offline batch processing and poses a fundamental problem for streaming reads. Parquet requires all row groups to be written before the file footer (which contains the column offset index) can be emitted, meaning a Parquet file can only be consumed after it is completely closed. This makes sub-second streaming latency impossible.

Figure 2: The Streaming Read Dilemma of Parquet -- batch write vs. frequent close

This is the first fundamental challenge of column pruning in streaming storage: existing columnar formats are designed for batch processing and cannot simultaneously satisfy low-latency streaming consumption and efficient columnar storage.

Challenge 2: Eliminating Server-Side Pruning Overhead​

How can the server perform column pruning without incurring additional computation and memory overhead?

The traditional approach is a full deserialize-prune-reserialize cycle on the server: upon receiving a read request, the server deserializes all data from disk into memory, performs column pruning in memory, then reserializes and sends the pruned data to the consumer. This is feasible for low-frequency query systems, but a streaming storage server is a shared resource serving many concurrent consumers. Each read request that triggers a full decode-prune-encode cycle adds 2–5x memory amplification under load.

Figure 3: Overhead of Traditional Server-Side Column Pruning -- deserialize, prune, re-serialize

This is the second core challenge: a streaming storage server needs a zero-overhead column pruning approach: no data parsing, no extra memory allocation, just metadata and byte-offset manipulation.

Challenge 3: The Tension Between Low-Latency Writes and Columnar Batching​

Columnar storage efficiency depends heavily on batch size. Larger batches mean more contiguous data per column, better compression ratios, and more efficient bulk reads. But streaming storage has strict write latency requirements (typically sub-100 ms), making indefinite batching impractical.

The problem is compounded by data distribution. In a typical Flink write scenario, multiple Sink operator instances distribute records across all Buckets via round-robin. With N Sink instances and M Buckets, this creates N×M connections, each carrying a small fraction of the total data.

Figure 4: Round-Robin distribution causes fragmented small batches

Such fragmented small batches are devastating for columnar storage: too few rows per column render compression algorithms nearly useless, column contiguity is lost, and storage efficiency can degrade below even row storage. This is the third core challenge: the fundamental tension between streaming storage's low-latency write requirements and columnar storage's dependence on large batches, further amplified by round-robin data distribution.

How Does Fluss Implement Column Pruning?​

With the above challenges understood, let's see how Apache Fluss addresses them. Column pruning is designed as a core feature from the ground up, with end-to-end solutions spanning the storage format, the server-side read path, and the client-side write pipeline. Compared to Kafka's pseudo-pruning, Fluss achieves three key breakthroughs:

Key Design 1: Streaming Columnar Storage Format with Apache Arrow​

Challenge 1 surfaced two requirements: the storage format must be columnar (unlike Kafka's row storage), and column metadata must be inlined with each batch (unlike Parquet's file-level footer). Fluss chose the Apache Arrow IPC Streaming Format as the on-disk format for log segments, satisfying both requirements simultaneously.

A Fluss Log Segment consists of sequentially appended RecordBatches, each ranging from 100 KB to several MB. Each RecordBatch has two parts:

• Fluss Header: contains magic, schemaId, batchLen, rowCount, and other fields for quick location and validation.
• Arrow RecordBatch: follows the standard Arrow IPC format, with a leading metadata section (FlatBuffers-encoded, recording each column's byte offset, length, and FieldNode) followed by contiguous column Buffer data.

Figure 5: RecordBatch Layout in Fluss Log Segment

This design addresses both challenges directly. Because data for the same column is stored contiguously within each RecordBatch, reading a specific column can skip all other columns' disk regions entirely, with no row-by-row scanning. And because each RecordBatch carries its own complete Arrow metadata, it can be consumed independently as soon as it is written, with no need to wait for file closure. Streaming consumers can read batch by batch with millisecond-level latency.

Key Design 2: End-to-End Zero-Copy Column Pruning​

Fluss leverages Arrow's buffer layout to achieve true end-to-end zero-copy column pruning: target column data travels from disk to the network without ever entering user space.

When the server receives a column-pruning read request, the processing flow is:

1. Read Arrow metadata: The RecordBatch header records each column's byte offset and length within the file.
2. Reconstruct the buffer descriptor: Build new metadata referencing only the target columns' FieldNodes and Buffers, without reading any column data from disk.
3. Send directly to the network: Transfer the new metadata and the target columns' raw byte ranges from the page cache to the network interface card (NIC), bypassing user space entirely.

Figure 6: Fluss Zero-Copy Column Pruning -- from disk to NIC without user-space data parsing

Throughout this process, the server neither deserializes nor reserializes any data. It reassembles buffer references based on Arrow metadata and transfers the target columns' bytes directly to the network. This is the fundamental reason Fluss's column pruning performance scales linearly with the pruning ratio.

Key Design 3: Client-Side Pre-Shuffle for Large Batches​

To address the batching dilemma from Challenge 3, Fluss performs a pre-shuffle (client-side data partitioning) that consolidates records destined for the same bucket before they are sent to the server, transforming fragmented small batches into large contiguous ones. Fluss implements three strategies for this:

Sticky Bucket Assigner: For data without an explicit Bucket Key, Sinks can freely choose which Bucket to write to. The Sticky Bucket Assigner keeps writing to one Bucket until the batch is full (default: 2 MB), then switches to the next Bucket. This maximizes the number of rows per batch before any Bucket transitions.

Figure 7: Sticky Bucket Assigner -- fill one Bucket before switching to the next

Dynamic Shuffle Sink: For partitioned tables with many partitions (e.g., hundreds), the write-side buffer is spread thinly across all partitions, causing each partition's batch to hold as little as a single row. Dynamic Shuffle Sink analyzes partition traffic at runtime and allocates Sink parallelism proportionally, so each Sink instance writes to as few partitions as possible, maximizing per-partition batch size.

Figure 8: Dynamic Shuffle Sink -- allocate Sinks by partition traffic weight

Bucket Shuffle: When a table defines a Bucket Key, each record must be routed to its corresponding Bucket, forcing Sinks to fan out to all Buckets simultaneously. Bucket Shuffle performs the partitioning on the Flink side using the Bucket Key, concentrating data for the same Bucket at a single Sink operator instance before it reaches Fluss, eliminating the fan-out fragmentation.

Figure 9: Bucket Shuffle -- each record must be written to its corresponding Bucket

Together, these strategies ensure that within the target write latency (default: 100 ms), each batch contains enough rows that same-column data is stored contiguously on disk and can be fetched in efficient bulk reads.

Putting It All Together​

Fluss's column pruning is not a single-point optimization but an end-to-end solution spanning the on-disk storage format, the server-side read path, and the client-side write pipeline.

Figure 10: How Fluss solves the Core Challenges of Column Pruning in Streaming Storage

Because each layer addresses exactly its corresponding challenge, column pruning performance scales linearly with the pruning ratio. Pruning 50% of columns yields approximately 2x read throughput improvement; pruning 90% of columns yields approximately 10x.

Performance​

Read throughput improves linearly with the column pruning ratio. When 90% of columns are pruned, read throughput increases by 10x. Because pruning happens server-side, the pruned columns are never read from disk and never transmitted over the network, so the throughput gain is roughly proportional to the pruning ratio. Kafka's throughput, by contrast, remains constant regardless of how many columns the consumer requests.

Figure 11: Column Pruning Streaming Read Performance -- Fluss vs. Kafka

Test Setup​

The benchmark uses the Open Message Benchmark (OMB) dataset, which is widely used in the messaging systems community. The schema has 20 columns covering Integer, Long, and String types, representing a typical wide-table read scenario. Each record is 2 KB. For the test, 100 GB of data was pre-loaded onto the server with writes stopped, and remote tiering was disabled, so all reads are served entirely from the TabletServer's local disk.

Results​

Figure 12: Fluss vs. Kafka -- Column Pruning Read TPS Comparison

Kafka's throughput remains flat at 117K records/s across all pruning ratios because the broker performs no server-side column pruning. Fluss's throughput improves approximately linearly with the pruning ratio, confirming the effectiveness of zero-copy server-side pruning.

References​

• Apache Fluss
• Apache Arrow IPC Streaming Format
• Sticky Bucket Assigner (Source Code)
• Dynamic Shuffle Sink for Partitioned Tables
• Bucket Shuffle PR
• Open Message Benchmark (OMB)

If you found this interesting, consider exploring the Apache Fluss documentation or giving the project a star on GitHub. Community contributions, feedback, and bug reports are always welcome.

Every autumn in China, social media floods with posts about "The First Cup of Milk Tea in Autumn." With a tap on their phone, consumers expect their order delivered within 30 minutes. That effortless experience is no accident: it is the result of Taobao Instant Commerce making thousands of data-driven decisions every second.

Taobao Instant Commerce has scaled from a single-category food delivery service into a high-frequency platform spanning fresh produce, consumer electronics (3C), and beauty products. It operates under two very different modes: steady high-frequency daily transactions, and explosive traffic surges during promotional events where order volumes can multiply within minutes. Both demand the same thing: real-time responsiveness across hundreds of millions of SKUs.

Real-time is not a nice-to-have here; it is the lifeline for three critical functions:

• Operations: Refresh conversion rates and funnels within 30 seconds.
• Algorithms: Order prediction models must iterate at minute-level granularity.
• Quality Assurance: Canary release anomalies must be detected within seconds and trigger instant alerts.

The existing pipeline (built on Kafka, Flink, Paimon, and StarRocks) handled this at one scale.

Note: In Alibaba's internal infrastructure, TT (TimeTunnel) is the internal equivalent of Apache Kafka — a high-throughput distributed message queue. Throughout this post, "Kafka" refers to TT in the Taobao Instant Commerce context. But as the business grew, three fundamental bottlenecks emerged: unbounded state growth from stream joins, mounting complexity in building multi-stream denormalized tables, and excessive resource consumption from lakehouse synchronization. Together they formed an impossible triangle: no matter how the team tuned the system, latency, consistency, and cost could not all be optimized at once.

Fluss broke this impasse. By replacing the fragmented stream-batch architecture with a unified storage layer, its features (Delta Join, Partial Update, Streaming-Lakehouse Unification, Column Pruning, and Auto-Increment Columns) systematically eliminated all three bottlenecks and fundamentally reshaped how Taobao Instant Commerce handles real-time decision-making at scale.

The "Impossible Triangle" Dilemma: Business Pain Points & Technical Challenges​

The real-time data system of Taobao Instant Commerce must reliably process ultra-large-scale, high-concurrency data streams. It must also build real-time denormalized tables (wide tables) that aggregate metrics across multiple business domains, and support core pipelines such as associating page view streams with order streams, as well as canary release monitoring. This imposes extreme requirements on latency, consistency, cost-efficiency, and system scalability.

Under the traditional technology stack, challenges across these four dimensions were deeply intertwined, ultimately forming the same trilemma of latency, consistency, and cost that the business had already identified.

The root cause lies in the stream-batch separation at the storage layer: Kafka serving as the message queue, represents the "stream" abstraction, while Paimon, as the lakehouse format, represents the "batch" abstraction. Bridging these two relies heavily on numerous Flink ETL jobs acting as a glue layer. Each additional layer of glue compounds latency, increases costs, and heightens the risk to data consistency.

Four Core Business Issues​

Three Core Pain Points of the Traditional Tech Stack​

The traditional pipeline based on Kafka, Flink and Paimon exposed three critical pain points under Taobao Instant Commerce’s massive scale, high concurrency, and strict real-time requirements. These issues became the core focus areas for the subsequent Fluss implementation:

• Surging Memory Pressure from Dual-Stream Joins: Kafka lacks native support for dimension table lookups. Consequently, order information had to be fully loaded into Flink State. During promotional events, over 100 million orders caused the state size of single jobs to surge to hundreds of GBs, leading to Checkpoint timeouts and frequent task failures.
• Fan-Out Pipeline Complexity in Wide Table Construction: The product-store analysis wide table depended on 5+ upstream systems. Using Kafka required writing to a new Topic after every Join operation, resulting in a sprawling, tightly coupled pipeline with high operational and maintenance costs.
• Core Resource Drain from Lakehouse Synchronization: Each table synced to Paimon required maintaining an independent Flink consumption job. During peak events, dozens of these "data transfer" jobs ran simultaneously, competing with core computing tasks for resources and causing overall performance degradation.

Architecture Selection Logic: From "Compute-Layer Stitching" to "Storage-Layer Unification"​

Guided by the impossible triangle dilemma described above, we established a core principle for our technology selection: the breakthrough lies not in the compute layer, but in the storage layer. The traditional architecture, relying on the combination of Kafka, Flink and Paimon as the open table format, was essentially a stitched model characterized by stream-batch separation. As the compute engine, Flink was forced to assume numerous glue responsibilities: maintaining TB-scale State, scheduling dozens of data transfer jobs, and handling memory bloat from dual-stream joins. This not only drove up resource costs but also made data consistency difficult to guarantee.

Fluss’s design philosophy directly addresses this pain point through Stream-Batch Unification at the storage layer. It is not merely another message queue; rather, it converges streaming consumption and batch querying into a single distributed storage kernel. Streaming and batch operations require no data replication, no format conversion, and no additional ETL jobs for data movement. This fundamental shift in underlying architecture directly eliminates the three primary sources of latency, cost, and consistency risks inherent in traditional pipelines, providing a unified foundation for the implementation of subsequent core features.

Fluss Core Scenario Implementation​

Taobao Instant Commerce designed a three-layer architecture underpinned by five core Fluss capabilities, each mapped to a specific business need. By establishing Fluss as the unified data foundation, this architecture covers the entire pipeline, from data ingestion and stream-batch processing to lakehouse storage and service delivery, eliminating the pain points described above.

Three-Layer Overall Architecture​

The real-time data system of Taobao Instant Commerce, powered by Fluss, is structured into three layers: the Data Source Layer, the Fluss Storage & Compute Layer, and the Business Service Layer. Built on the core philosophy of Stream-Batch Unification, this architecture enables Write Once, Read Anywhere data reuse, completely eliminating the stream-batch separation and glue-layer ETL jobs inherent in traditional pipelines.

• Data Source Layer: Integrates all business data sources (including order streams, visit streams, product dimension tables, and inventory/price streams) and uniformly writes them into Fluss’s Log Tables or KV Tables, adapting to the specific ingestion characteristics of each data source.
• Fluss Storage & Compute Layer: As the core layer, it achieves unified data storage via Fluss’s Log/KV Tables. It leverages features like Delta Join, Partial Update, and Streaming-Lakehouse Unification (Auto-Tiering) to handle real-time computation, multi-table joins, wide table construction, and lakehouse synchronization. Crucially, it provides stateless computing support for Flink, significantly reducing state management overhead.
• Business Service Layer: Leveraging a unified data view from both Fluss and lakehouse storage (Paimon), this layer delivers data services to operational real-time dashboards, algorithmic order prediction models, canary monitoring systems, and product-store analysis systems, enabling decision support within seconds.

Five Core Scenarios and Feature Combination Matrix​

The following sections walk through five production scenarios where Fluss was applied, covering the architectural decisions, implementation details, and measured results for each.

Scenario 1: Real-Time Conversion Rate and Order Attribution: Solving Unbounded State Growth in Dual-Stream Joins with Delta Join​

Conversion Rate (defined as the ratio of purchasing users to visiting users) is a core metric for measuring conversion efficiency in Taobao Instant Commerce. Its calculation relies on the real-time join of user visit streams with order streams, directly enabling operational decisions within seconds.

Pain Points of Traditional Solutions​

In the legacy architecture, a Flink Regular Join must continuously accumulate all unexpired events in its local state to match orders with browsing records. This creates an architectural dilemma: extending the conversion observation window incurs linearly growing storage costs.

• Unbounded State Growth & Compaction Pressure: During promotional events, hundreds of billions of access logs must be retained within long time windows. The total state for a single job grew past TB levels, leading to severe RocksDB compaction backlogs and extremely long checkpoint durations.
• I/O Saturation & Latency Degradation: Amplified State read/write operations saturated disk I/O. Real-time attribution latency degraded from seconds to minutes.
• Checkpoint Failures & High RTO: Checkpoint durations deteriorated from minutes to 10–15 minutes, frequently causing timeouts and failures.
• Limited Attribution Window: The effective association time window was constrained by local storage capacity, making it impossible to support long-cycle "Browse-to-Order" conversion attribution.

Architecture Based on Fluss Delta Join​

To address the pain points of traditional architectures, our solution upgrades the conventional Flink dual-stream join to a Delta Join. This approach offloads the full volume of dual-stream data from Flink State to Fluss KV Tables (backed by RocksDB). By implementing bidirectional Prefix Lookup (where incoming records from either stream trigger a lookup in the opposite KV Table), we achieve a near-stateless dual-stream join. Furthermore, query performance is enhanced through MurmurHash-based bucket routing and column pruning.

• Dual KV Table Ingestion with Composite Keys: The real-time order stream is written to a Fluss KV Table with PRIMARY KEY (user_id, ds, order_id) and bucket.key = ‘user_id’. Simultaneously, real-time traffic visit logs are written to a Fluss KV Table with PRIMARY KEY (user_id, ds) and bucket.key = ‘user_id’. Both tables use a composite primary key design where the join key (user_id) serves as the bucket.key and is a strict prefix of the primary key, enabling efficient Prefix Lookup operations.
• Stateless Bidirectional Join via Delta Join Operator: Leveraging the Delta Join operator in Flink 2.1+, we implement bidirectional association: incoming orders trigger lookups in the visit log table, and incoming visits trigger lookups in the order table. Flink calculates the target bucketId using MurmurHash(bucket.key) to route RPC requests precisely to the single Fluss TabletServer holding that specific bucket’s RocksDB instance. This eliminates the need for broadcast scans or full-table scans. The Flink operator itself retains only minimal transient state (for asynchronous lookup queues), completely removing dependency on large-scale Join State.
• Real-Time Attribution & Lakehouse Sync: The joined results are written to a Fluss wide table and automatically synchronized to Paimon via the Streaming-Lakehouse Unification mechanism. The data is finally served to StarRocks and real-time BI dashboards, achieving metric refresh latency within 30 seconds.

Core DDL and Join SQL​

-- Note: ${xxx} represents environment-specific parameters. 
-- Actual deployment values should be evaluated based on cluster scale and data volume.

-- Order Stream (Right Table in Join)
CREATE TABLE `fluss_catalog`.`db`.`order_ri` (
    order_id STRING,
    user_id STRING,
    order_time STRING,
    pay_amt STRING,
    ds STRING,
    PRIMARY KEY (user_id, ds, order_id) NOT ENFORCED
) PARTITIONED BY (ds)
WITH (
    'bucket.key' = 'user_id',
    'bucket.num' = '${BUCKET_NUM}',
    'table.auto-partition.enabled' = 'true',
    'table.auto-partition.key' = 'ds'
);

-- Visit Stream (Left Table in Join)
CREATE TABLE `fluss_catalog`.`db`.`log_page_ri` (
    user_id STRING,
    first_page_time STRING,
    pv BIGINT,
    ds STRING,
    PRIMARY KEY (user_id, ds) NOT ENFORCED
) PARTITIONED BY (ds)
WITH (
    'bucket.key' = 'user_id',
    'bucket.num' = '${BUCKET_NUM}',
    'table.auto-partition.enabled' = 'true',
    'table.auto-partition.key' = 'ds'
);

-- Sink Table
CREATE TABLE `fluss_catalog`.`db`.`log_page_ord_detail_ri`
(
    `user_id`       STRING COMMENT 'User ID',
    `order_id`      STRING COMMENT 'Order ID',
    `first_page_time` STRING COMMENT 'First Visit Time',
    `order_time`    STRING COMMENT 'Order Placement Time',
    `pv`            BIGINT COMMENT 'Page Views (Visit Count)',
    `pay_amt`       STRING COMMENT 'Payment Amount',
    `is_matched`    STRING COMMENT 'Match Status',
    `ds`            STRING COMMENT 'Date Partition',
   PRIMARY KEY (user_id, order_id, ds) NOT ENFORCED
)
COMMENT 'Attribution Result Table'
PARTITIONED BY (ds)
WITH (
    'bucket.key' = 'user_id,order_id',
    'bucket.num' = '${BUCKET_NUM}',
    'table.auto-partition.enabled' = 'true',
    'table.auto-partition.key' = 'ds',
    'table.datalake.enabled' = 'true', -- Enable Streaming-Lakehouse Unification
    'table.datalake.freshness' = '3min' -- Controls Lakehouse Table Freshness
);

Subsequently, the Lakehouse Tiering Service continuously syncs data from Fluss to Paimon. The table.datalake.freshness parameter controls how frequently Fluss writes data to the Paimon table. By default, the data freshness is set to 3 minutes.

The implementation of the Delta Join is as follows:

CREATE TEMPORARY VIEW matched_page_ord_detail AS
SELECT
    l.ds,
    l.user_id,
    l.first_page_time,
    l.pv,
    o.order_id,
    o.order_time,
    CASE WHEN o.order_time >= l.first_page_time THEN '1' ELSE '0' END AS is_matched
FROM `fluss_catalog`.`db`.`log_page_ri` l
    LEFT JOIN `fluss_catalog`.`db`.`order_ri` o
        ON l.user_id = o.user_id AND l.ds = o.ds

Fluss supports two TTL (Time-To-Live) mechanisms:

• Changelog Expiration: Controlled by table.log.ttl (default: 7 days).
• Data Partition Expiration: Controlled by table.auto-partition.num-retention (default: 7 partitions).

Measured Results​

The Fluss Delta Join solution enables stateless dimension table joins, completely eliminating the unbounded state growth problem and delivering order-of-magnitude improvements in core metrics:

Scenario 2: Real-Time Product-Store Wide Table: Efficient Column-Level Updates via Partial Update​

In Taobao Instant Commerce’s traffic operations ecosystem, the Real-time Product-Store Wide Table (a denormalized table that combines metrics from multiple behavioral data sources into a single queryable record) is the most critical data asset in the traffic domain. This table must aggregate four core behavioral streams (impressions, clicks, visits, and orders) in real time, producing key metrics across both "Product" and "Store" dimensions. These include UV/PV counts for each event type (where UV = unique users and PV = total event count), unique orders, and GMV. The table directly powers real-time traffic dashboards, ROI analysis, algorithmic feature engineering, and decision-making during live promotional events.

However, under extreme scale and concurrency, traditional real-time aggregation models based on multi-stream joins face severe challenges: excessive state accumulation, high pipeline coupling, and poor fault tolerance.

Fluss addresses these issues by leveraging Partial Update. It refactors the complex multi-stream real-time join aggregation model into a streamlined architecture of independent multi-stream writes with automatic server-side column-level merging. This transformation simplifies the real-time wide table construction pipeline and delivers a leap in performance.

Pain Points of Traditional Solutions​

The legacy solution constructed the wide table using multi-stream Flink joins over Kafka topics, resulting in a tightly coupled fan-in architecture. Its core pain points were as follows:

• Full-Row Write I/O Waste: Even if only a single behavior metric (e.g., "clicks" or "impressions") is updated, the system must write all 100+ columns of the full row. This causes severe redundancy in both network and storage I/O.
• High Operational Complexity: The architecture resembles a fan-out pipeline, requiring N upstream jobs and M join jobs to be maintained in parallel. Adding a new field requires modifying multiple jobs, significantly increasing operational overhead.
• Latency Coupling (Straggler Problem): The write latency of the wide table is determined by the slowest upstream stream. A delay in any single data stream causes the entire row update to be delayed, leading to inconsistent real-time visibility.
• Unbounded State Growth and No Live Schema Updates: To match data across different behavioral streams for conversion rate calculations, Flink must cache large amounts of intermediate state locally. This leads to TB-scale unbounded state growth, Checkpoint timeouts, and risk of out-of-memory (OOM) errors. Furthermore, the tightly coupled join logic means the system cannot be updated while running: any schema change, scaling operation, or model update requires a full service restart and data reprocessing.

Architecture Based on Fluss Partial Update​

To address the aforementioned pain points, our solution utilizes Fluss KV Tables as the unified storage for a multi-dimensional wide table (spanning Product and Store dimensions). By leveraging Partial Update, we achieve column-level updates that completely decouple the ingestion of diverse behavior streams such as impressions, clicks, visits, and orders. Each Flink job is responsible solely for aggregating its specific behavioral metrics and writing to the target columns, eliminating the need for complex multi-stream joins. The field-level merging is automatically handled by the RowMerger on the Fluss server side. The core architectural workflow is as follows:

• Independent Aggregation & Ingestion of Multiple Streams: Five or more upstream behavior streams (impressions, clicks, visits, add-to-cart, orders) run as independent Flink jobs. Each job consumes only a single type of behavior log, pre-aggregates data by "Product" or "Store" dimension, and writes exclusively to the corresponding metric columns in the wide table. This approach thoroughly eliminates inter-stream dependencies.
• Server-Side Intelligent Column-Level Merging: Fluss leverages the Partial Update feature, utilizing BitSet bitmap technology to identify target columns in write requests. Within the storage engine, only the target columns are read and updated (via accumulation or overwrite), while non-target columns retain their historical values in RocksDB. This avoids transmitting redundant data over the network or in memory, achieving microsecond-level merging.
• Streaming-Lakehouse Unification with Auto-Tiering: The merged, up-to-date wide table data provides low-latency point queries and streaming consumption capabilities in real-time. Simultaneously, the Auto-Tiering mechanism asynchronously syncs data to the Paimon lakehouse. This supports offline T+1 (next-day batch) validation, historical backtracking, and AI model training without requiring additional ETL development for data movement.
• Online Schema Evolution: When new statistical metrics are required, business teams simply execute an ALTER TABLE ADD COLUMN DDL statement and deploy a new write job for that specific column. This takes effect immediately without downtime or restarting existing jobs (e.g., for impressions or orders), enabling agile iteration even during peak promotional events.

Core DDL and Per-Job Write Implementation​

CREATE TABLE shop_stat_wide (
    shop_id    BIGINT NOT NULL,
    stat_date  STRING NOT NULL,
    stat_hour  STRING NOT NULL,
    exp_cnt    BIGINT,  -- Exposure Count
    page_cnt   BIGINT,  -- Page View Count
    PRIMARY KEY (shop_id, stat_date, stat_hour) NOT ENFORCED
) WITH (
    'bucket.num' = '${BUCKET_NUM}',
    'bucket.key' = 'shop_id'
);

-- Write Exposure Data
INSERT INTO shop_stat_wide (shop_id, stat_date, stat_hour, exp_cnt)
SELECT
    shop_id,
    stat_date,
    stat_hour,
    SUM(exp_cnt) AS exp_cnt
FROM exposure_log
GROUP BY shop_id, stat_date, stat_hour;

-- Write Visit Data
INSERT INTO shop_stat_wide (shop_id, stat_date, stat_hour, page_cnt)
SELECT
    shop_id,
    stat_date,
    stat_hour,
    SUM(page_cnt) AS page_cnt
FROM page_log
GROUP BY shop_id, stat_date, stat_hour;

Measured Results​

The Fluss Partial Update solution enables column-level updates and decoupled writes for wide tables, delivering significant improvements in core metrics while reducing operational costs.

Scenario 3: Real-Time Order Estimation: Efficient Data Fusion via Fluss KV Lake-Stream Unification​

Real-time order forecasting serves as a critical support pillar in Taobao Instant Commerce’s algorithmic decision-making framework. During promotional events, the system iterates the prediction model every minute to accurately project the total daily Gross Merchandise Volume (GMV), thereby enabling dynamic adjustments to subsidy strategies and inventory allocation. This scenario imposes extreme challenges on the data pipeline: model inputs must deeply integrate real-time cumulative order streams (high-frequency incremental data) with historical baseline data from corresponding periods (massive static data), demanding exceptionally high standards for both data consistency and low latency.

Architecture Based on Fluss Lake-Stream Unification​

The current implementation leverages Fluss KV Tables as a unified storage layer for both streaming and batch processing. By integrating Streaming-Lakehouse Unification, it enables automated synchronization to the data lake. Since both streaming and batch operations rely on the same KV Table, data consistency is inherently guaranteed. This approach eliminates the need for independent data migration jobs, ensuring low-latency synchronization of prediction results.

Core Architecture Workflow:

• Real-time Accumulation: The real-time order stream is written into the Fluss KV Table, enabling the calculation of rolling cumulative order counts on a per-minute basis via stream reads.
• Efficient Stream-Batch Fusion: A Flink UDF directly reads both real-time data (via stream read) and historical baseline data (via batch read / Snapshot Lookup) from the Fluss KV Table, achieving efficient fusion of streaming and batch data.
• Low-Latency Lake Sync: Model prediction results are written back to the Fluss KV Table. With automatic Tiering enabled, these results are synchronized to Paimon with low latency (with configurable snapshot intervals).
• Unified Data Consistency: Downstream algorithm models and operational dashboards read real-time prediction results from the Fluss KV Table and historical prediction results from Paimon, ensuring end-to-end data consistency across the entire pipeline.

Core DDL and Write Implementation​

-- Note: ${xxx} represents environment-specific parameters. 
-- Actual deployment values should be evaluated based on cluster scale and data volume.

-- 1. Real-time Order Minute-Accumulation Stream
CREATE TABLE `fluss_catalog`.`db`.`order_dtm`
(
    `ds` STRING COMMENT 'Day Partition',
    `mm` STRING COMMENT 'Minute',
    `order_source` STRING COMMENT 'Order Source',
    `order_cnt` BIGINT COMMENT 'Order Count',
    PRIMARY KEY (ds, mm, order_source) NOT ENFORCED
)
    COMMENT 'Real-time Order Minute-Accumulation Stream'
PARTITIONED BY (ds)
WITH (
  'bucket.key' = 'mm,order_source',
  'bucket.num' = '${BUCKET_NUM}',
  'table.auto-partition.enabled' = 'true',
  'table.auto-partition.time-unit' = 'day',
  'table.auto-partition.key' = 'ds',
  'table.auto-partition.num-precreate' = '0', -- Number of future partitions to pre-create.
  'table.auto-partition.num-retention' = '2', -- Retain the last N partitions; automatically delete older ones.
  'table.replication.factor' = '${REPLICATION_FACTOR}',
  'table.log.arrow.compression.type' = 'zstd'
);

-- 2. Order Forecast Stream (Fluss Streaming-Lakehouse Unification)
CREATE TABLE `fluss_catalog`.`db`.`order_forecast_mm`
(
    `ds` STRING COMMENT 'Date Partition (Format: yyyymmdd)',
    `mm` STRING COMMENT 'Minute (Format: HH:mm or HHmm)',
    `order_source` STRING COMMENT 'Order Source',
    `cumulative_order_cnt` BIGINT COMMENT 'Cumulative Order Count up to the current minute',
    `forecast_daily_order_cnt` STRING COMMENT 'Predicted Total Daily Order Volume (24h) based on historical ratios',
    PRIMARY KEY (ds, mm, order_source) NOT ENFORCED
)
    COMMENT 'Order Forecast Stream'
PARTITIONED BY (ds)
WITH (
  'bucket.key' = 'mm,order_source',
  'bucket.num' = '${BUCKET_NUM}',
  'table.auto-partition.enabled' = 'true',
  'table.auto-partition.time-unit' = 'day',
  'table.auto-partition.key' = 'ds',
  'table.auto-partition.num-precreate' = '0',
  'table.auto-partition.num-retention' = '2',
  'table.replication.factor' = '${REPLICATION_FACTOR}',
  'table.log.arrow.compression.type' = 'zstd',
  'table.datalake.enabled' = 'true', -- Enable Streaming-Lakehouse Unification
  'table.datalake.freshness' = '30s' -- Controls Lakehouse Table Freshness
);

-- 3. Real-time Incremental & Offline Feature Fusion: 
-- Uses Flink Temporal Join to associate the Fluss real-time stream with the Paimon historical baseline table, 
-- and performs prediction calculations via a UDF.
CREATE TEMPORARY FUNCTION predict_hour_gmv AS 'com.example.flink.udf.FutureHourlyForecastUDF';

CREATE TEMPORARY VIEW trd_item_order AS
SELECT
    ds,
    mm,
    order_source,
    order_cnt,
    PROCTIME() AS proc_time
FROM `fluss_catalog`.`db`.`order_dtm`;

INSERT INTO `fluss_catalog`.`db`.`order_forecast_mm`
SELECT
    t1.ds,
    t1.mm,
    t1.order_source,
    t1.order_cnt AS cumulative_order_cnt,
    predict_hour_gmv(
            COALESCE(t1.order_cnt, 0),
            t1.mm,
            t2.minute_ratio_feat
    ) AS forecast_hour_order_cnt
FROM
    trd_item_order t1
        LEFT JOIN paimon_catalog.db.order_mm_his_str /*+ OPTIONS(
    'scan.partitions' = 'max_pt()',
    'lookup.dynamic-partition.refresh-interval' = '1 h'
) */
FOR SYSTEM_TIME AS OF t1.proc_time AS t2 ON t1.order_source = t2.channel_id;

Measured Results​

The Fluss KV Streaming-Lakehouse Unification solution achieves unified source integration for streaming and batch data, eliminating independent data migration jobs and delivering significant improvements in core metrics:

Scenario 4: Canary Monitoring: Efficient Log Processing via Log Tables and Column Pruning​

In the context of Taobao Instant Commerce’s high-frequency iteration cycle, frontend telemetry data covering the full user journey (from app launch, site entry, impressions, traffic redirection, store visits, cart additions, and orders through to fulfillment) serves as the core basis for evaluating release quality, monitoring marketing effectiveness, and optimizing search and recommendation strategies. To mitigate release risks, canary releases have become the standard procedure, with canary monitoring acting as the critical "safety valve." This process aims to detect data collection anomalies or business logic defects within seconds by real-time analysis of discrepancies between canary traffic and baseline traffic. However, facing TB-scale log throughput and millisecond-level alerting requirements, the traditional "Kafka + Flink" architecture has gradually revealed bottlenecks such as severe I/O waste, redundant pipelines, and data silos across multiple sources.

Fluss has refactored the data foundation for canary monitoring by leveraging:

• High-throughput writes via Log Tables;
• Server-side column pruning for I/O optimization;
• Automatic archiving through Streaming-Lakehouse Unification.

This transformation upgrades the architecture from reactive firefighting to proactive governance.

Pain Points of Traditional Solutions​

The legacy architecture utilized Kafka as the message queue, Flink for real-time consumption and computation, and Paimon for storing detailed data. However, as data volumes surged and business complexity increased, this architecture exposed several core pain points:

Architecture Based on Fluss Log Table and Column Pruning​

To address the aforementioned pain points, Taobao Instant Commerce underwent a three-stage architectural evolution, ultimately establishing a unified log governance solution centered on Fluss:

• Phase 1: Introduced Fluss for a specific client’s data to validate the feasibility of cost optimization and improvements in data freshness.
• Phase 2: Integrated data from all client endpoints. While this achieved unified data ingestion, computational tasks remained redundant due to difficulties in reusing the intermediate layer.
• Phase 3: Leveraged Fluss’s server-side column pruning to eliminate invalid I/O, and used automatic Tiering to merge monitoring and synchronization pipelines. This completely resolves the issue of redundant computations.

The new architecture establishes Fluss Log Tables as the single source of truth, creating an efficient write once, multi-dimensional reuse data loop:

• Unified High-Throughput Ingestion: Multi-source canary logs are uniformly written into Fluss Log Tables. Leveraging the Append-Only nature of Log Tables, the system perfectly accommodates high-concurrency write scenarios for massive volumes of logs.
• Server-Side Column Pruning: During Flink consumption, Projection Pushdown pushes the required columns down to the Fluss server side. The server reads and returns only the specified 30 core fields, blocking 67% of invalid network transmission at the source.
• Automated Archiving via Streaming-Lakehouse Unification: By simply enabling the table.datalake.enabled configuration, the system automatically and asynchronously synchronizes real-time data to Paimon. This eliminates the need for deploying separate Flink archiving jobs. Real-time monitoring and offline analysis share the same data pipeline, achieving seamless stream-batch data fusion.
• Multi-Client Reuse & Within-Seconds Alerting: Real-time monitoring tasks directly consume the lightweight, pruned data stream, feeding BI dashboards to achieve within-seconds anomaly alerting. Offline analysis queries Paimon tables directly, ensuring inherent consistency between streaming and batch data.

Core DDL and Write Implementation​

-- Note: ${xxx} represents environment-specific parameters. 
-- Actual deployment values should be evaluated based on cluster scale and data volume.

CREATE TABLE `fluss_catalog`.`db`.`log_page_ri`
(
    `col1` STRING,
    `col2` STRING,
    `col3` STRING,
    `col4` STRING
    -- ....... Intermediate fields omitted
    ,ds STRING
)
    COMMENT 'Page View Log Stream'
PARTITIONED BY (ds) -- Partition key defined above
WITH (
  'bucket.num' = '${BUCKET_NUM}',
  'table.auto-partition.enabled' = 'true',
  'table.auto-partition.time-unit' = 'day',
  'table.auto-partition.key' = 'ds',
  'table.auto-partition.num-precreate' = '0', 
  'table.auto-partition.num-retention' = '1',
  'table.log.ttl' = '${LOG_TTL}',
  'table.replication.factor'='${REPLICATION_FACTOR}',
  'table.log.arrow.compression.type' = 'zstd',
  'table.datalake.enabled' = 'true', -- Enable Streaming-Lakehouse Unification
  'table.datalake.freshness' = '3min' -- Control lake table data freshness
)
;

-- Monitoring Query (Reads only 30 core columns to trigger server-side column pruning)
INSERT INTO `fluss_catalog`.`db`.ads_gray_monitoring_ri /*+ OPTIONS('client.writer.enable-idempotence'='false') */
SELECT
    col1,
    col2,
    col3,
    col4,
    -- ......... Intermediate fields omitted
    ds
FROM `fluss-ali-log`.`db`.`log_page_ri`
WHERE KEYVALUE(`args`, ',', '=', 'release_type') = 'grey' -- Canary release flag
;

-- Flink Cube Aggregation
INSERT INTO `fluss_catalog`.`db`.ads_gray_monitoring_permm_ri /*+ OPTIONS('client.writer.enable-idempotence'='false') */
SELECT
    col1,
    col2,
    col3,
    col4,
    count(*) AS pv,
-- ......... Intermediate fields omitted

Measured Results​

Scenario 5: Global Real-Time UV Statistics: Auto-Increment Columns and Aggregation Tables for Unique User Counting​

In Taobao Instant Commerce’s refined operational framework, unique visitor (UV) counts across the full user journey (covering impressions, visits, clicks, and orders) are core metrics for measuring traffic quality and conversion efficiency. In the original architecture, a hybrid stack of Flink + Hologres + Paimon + StarRocks was used:

• Hologres (Alibaba Cloud’s real-time OLAP database) stored user mapping tables to handle ID resolution;
• Flink constructed Bitmaps via custom user-defined functions (UDFs) and wrote them to Paimon;
• StarRocks accelerated queries using materialized views.

However, under ultra-large-scale data volumes, this architecture exposed three critical pain points:

• Lookup hotspots and latency spikes in Hologres;
• Backpressure from unbounded Flink state growth;
• High data synchronization latency from Paimon’s compaction model.

These issues made it impossible to achieve within-seconds real-time monitoring during promotional events. The introduction of Fluss marks a key iteration of this architecture:

• It replaces Hologres with Fluss KV Tables for low-latency ID mapping;
• It replaces Flink state with Fluss Aggregation Tables for stateless local aggregation;
• It replaces the independent Paimon synchronization pipeline with Auto Tiering for within-seconds sync.

This solution achieves an architectural upgrade from a fragmented, high-latency multi-component stack to a unified, low-latency Streaming-Lakehouse architecture.

Pain Point of the Old Approach​

The pre-iteration architecture suffered from the following critical bottlenecks:

• Hologres Lookup Hotspots and Network Bottlenecks: Flink issued massive volumes of RPC requests to Hologres. Under high concurrency, this led to data skew and hotspots, causing lookup latency to degrade to several seconds.
• Unbounded Flink State Growth and Checkpoint Failures: Flink UDFs were required to maintain TB-scale Bitmap state. This resulted in excessively long Checkpoint durations (>10 minutes), frequent timeouts, and poor job stability.
• Paimon Synchronization Latency: Paimon compaction introduced minute-level latency, leaving downstream StarRocks Materialized Views with stale data. This made it impossible to meet the requirements for within-seconds monitoring.

Architecture Based on Fluss Auto-Increment Columns and Aggregation Tables​

To address the three major pain points of the traditional architecture, we leveraged Fluss’s core design with a strategy centered on "mapping sparse business IDs to dense integer IDs for bitmap computation." Through techniques such as full pre-loading, dense integer ID generation, sharded parallelism, and 32-bit bitmap optimization, we achieved high-performance real-time unique visitor counting.

• Auto-Increment KV Mapping Layer: Using Flink’s built-in dimension table lookup, we built real-time ID resolution capabilities:
  • Lock-Free Segment Allocation: Fluss requests ID segments in batches and caches them locally. This lock-free, thread-level allocation avoids lock contention and GC pauses during concurrent writes, significantly improving ID generation throughput.
  • Query-Then-Insert: In dimension table Lookup scenarios, if a business ID is not found, Fluss automatically triggers an INSERT and returns the new auto-increment ID. This mechanism merges the traditional "check-then-write" two-step interaction into a single request, effectively reducing network traffic and cluster pressure.
  • Batching & Pipeline Optimization: By combining Flink’s asynchronous Lookup batch request mechanism with Fluss client-side Pipeline writes, massive discrete RPCs are aggregated and processed in parallel. This design reduces interaction overhead and further optimizes overall system throughput.
  • Full Pre-loading & Dense ID Generation: Before going live, billions of historical business IDs were batch-imported into the Fluss KV Table to generate unique 64-bit integer IDs using the auto-increment column. This eliminates cross-cluster RPC latency for existing users (100% cache hit rate), while new users are assigned IDs on first appearance via INSERT.
• Sharded Parallelism: By taking the modulus of the physical ID, all users are uniformly distributed across N logical shards, enabling localized parallelism for global deduplication tasks. Since behaviors from the same user strictly fall into the same shard, data volume across shards remains absolutely balanced, completely avoiding hotspot skew caused by patterns in business ID ranges.
• 32-Bit ID Compression: When the total number of physical IDs is below 4 billion, 64-bit IDs can be downcast to 32-bit integers, enabling the use of the rbm32 (32-bit RoaringBitmap) aggregation function. Compared to rbm64, this reduces memory usage by over 50% and significantly improves bitwise operation efficiency and CPU cache utilization.

Dual-Path Aggregation Architecture​

Based on the aforementioned preprocessing, the team implemented two parallel data pipelines:

• Option A (Fluss Aggregation Table Pre-Aggregation): The server side performs local Bitmap merging keyed by (ds, hh, mm, shard_id), allowing Flink to consume only the already-merged results. This offloads the computational work to the storage layer and is ideal for extreme real-time scenarios such as live event war rooms, achieving sub-second latency.
• Option B (Fluss Auto Tiering + StarRocks Native Deduplication): Detailed data is synchronized to Paimon within seconds, with StarRocks handling the final deduplication. This approach suits flexible multi-dimensional analysis (such as daily operational dashboards) and supports ad-hoc queries across arbitrary dimensions.

Both paths share the same ID mapping and sharding logic, allowing the team to freely switch between them based on business requirements for real-time performance versus query flexibility.

Core DDL and Write Implementation​

User Mapping Table (with Auto-Increment Column Enabled)​

-- Note: ${xxx} represents environment-specific parameters. 
-- Actual deployment values should be evaluated based on cluster scale and data volume.

CREATE TABLE `fluss_catalog`.`db`.`user_mapping_table` (
  user_id BIGINT COMMENT 'Original User ID',
  uid_int64 BIGINT COMMENT 'Auto-incremented Globally Unique Integer ID', 
  update_time TIMESTAMP(3) COMMENT 'Update Time',
  PRIMARY KEY (user_id) NOT ENFORCED
) WITH (
  'bucket.key' = 'user_id',
  'bucket.num' = '${BUCKET_NUM}',
  'auto-increment.fields' = 'uid_int64', -- Enable auto-increment column
  'table.log.arrow.compression.type' = 'zstd'
);

New User Discovery and Mapping Ingestion​

INSERT INTO `fluss_catalog`.`db`.`user_mapping_table` (user_id) 
SELECT /*+ REPLICATED_SHUFFLE_HASH(t2) */ 
  t1.user_id
FROM `fluss_catalog`.`db`.`log_exp_ri` t1
LEFT JOIN `fluss_catalog`.`db`.`user_mapping_table` 
 /*+ OPTIONS(
    'lookup.insert-if-not-exists' = 'true',
    'lookup.cache' = 'PARTIAL',
    'lookup.partial-cache.max-rows' = '500000',
    'lookup.partial-cache.expire-after-write' = '1h',
    'lookup.partial-cache.cache-missing-key' = 'false'
) */
FOR SYSTEM_TIME AS OF PROCTIME() AS t2 
ON t1.user_id = t2.user_id
GROUP BY t1.user_id;

Implementation of Solution A: Fluss Aggregation Table + Flink Global Merge​

Core Idea: Use the Fluss Aggregation Table to perform partial Bitmap merging on the server side, so Flink only needs to consume the pre-aggregated results for a final global merge.

Fluss DDL (Aggregation Table)

-- Note: ${xxx} represents environment-specific parameters. 
-- Actual deployment values should be evaluated based on cluster scale and data volume.
CREATE TABLE IF NOT EXISTS `fluss_catalog`.`db`.`ads_uv_agg` (
    ds STRING COMMENT 'Date partition yyyyMMdd',
    hh STRING COMMENT 'Hour HH',
    mm STRING COMMENT 'Minute mm',
    shard_id INT COMMENT 'Shard ID',
    uv_bitmap BYTES COMMENT 'Minute-level user bitmap',
    PRIMARY KEY (ds, hh, mm, shard_id) NOT ENFORCED
) 
PARTITIONED BY (ds)
WITH (
    'bucket.key' = 'shard_id',
    'bucket.num' = '${BUCKET_NUM}',
    'table.merge-engine' = 'aggregation',  -- Enable aggregation engine; otherwise, Bytes fields cannot be automatically merged
    'fields.uv_bitmap.agg' = 'rbm32', -- Specify that the Bytes field uses the rbm32 algorithm for Union aggregation
    'table.auto-partition.enabled' = 'true',
    'table.auto-partition.time-unit' = 'day',
    'table.auto-partition.key' = 'ds'
);

Fluss DML (Aggregation Table)

-- Note: N is a power of 2 and should be dynamically configured based on cluster scale.
INSERT INTO fluss_catalog.db.ads_uv_agg
SELECT
    t1.ds,
    t1.hh,
    t1.mm,
    CAST(t2.uid_int64 % N AS INT)                                    AS shard_id,
    BITMAP_TO_BYTES(BITMAP_BUILD_AGG(CAST((t2.uid_int64 / N) AS INT))) AS uv_bitmap
FROM `fluss_catalog`.`db`.`log_exp_ri` t1
LEFT JOIN `fluss_catalog`.`db`.`user_mapping`
    /*+ OPTIONS('lookup.cache' = 'PARTIAL', 'lookup.partial-cache.max-rows' = '500000') */
    FOR SYSTEM_TIME AS OF PROCTIME() AS t2
ON t1.user_id = t2.user_id
GROUP BY t1.ds, t1.hh, t1.mm, CAST(t2.uid_int64 % N AS INT);

Implementation of Solution B: Fluss Auto Tiering + StarRocks Native Deduplication​

Core Idea: Fluss acts as a high-throughput data pipeline, synchronizing data to Paimon via Auto Tiering. StarRocks then handles the final deduplication using its native OLAP engine.

Fluss DDL (Log/KV Table with Tiering):

CREATE TABLE fluss_catalog.db.ads_uv_detail (
    ds STRING,
    -- ....... Intermediate fields omitted
    shard_id INT,
    uid_int32 INT,
    PRIMARY KEY (ds, shard_id, uid_int32) NOT ENFORCED
) WITH (
    'table.datalake.enabled' = 'true', -- Enable Streaming-Lakehouse Unification
    'table.datalake.freshness' = '10s' -- Second-level synchronization
);

Fact Table Denormalization (Generating Shard ID and Bitmap Index)

-- Note: N is a power of 2 and should be dynamically configured based on cluster scale.
INSERT INTO fluss_catalog.db.ads_uv_detail
SELECT
  t1.ds,
  -- ....... Intermediate fields omitted
  CAST(t2.uid_int64 % N AS INT) AS shard_id,      -- Shard ID, used for parallel aggregation
  CAST((t2.uid_int64 / N) AS INT) AS uid_int32   -- Compressed ID, used for Bitmap
FROM `fluss_catalog`.`db`.`log_exp_ri` t1
LEFT JOIN `fluss_catalog`.`db`.`user_mapping` 
  /*+ OPTIONS('lookup.cache' = 'PARTIAL','lookup.partial-cache.max-rows' = '500000','lookup.partial-cache.expire-after-access' = '2min') */
  FOR SYSTEM_TIME AS OF PROCTIME() AS t2 
ON t1.user_id = t2.user_id;

Downstream StarRocks Precise Deduplication Query

-- Note: ${xxx} in the text represents environment-specific parameters. Actual deployment requires evaluation based on cluster scale and data volume.
-- Create a materialized view to accelerate queries
CREATE MATERIALIZED VIEW mv_global_uv AS
SELECT 
  t.ds,
  SUM(t.rb_partial) as rbm_user_count -- Globally merge partial Bitmaps from each shard
FROM (
  SELECT 
    ds,
    shard_id, 
    BITMAP_UNION_COUNT(TO_BITMAP(uid_int32)) as rb_partial -- Partial Bitmap count
  FROM 
    paimon_catalog.db.ads_uv_detail
  WHERE 
    ds = '${date}'
  GROUP BY 
    ds, shard_id
) t
GROUP BY t.ds;

Features: Decoupled architecture; StarRocks bears the computational load and supports flexible ad-hoc queries.

Measured Results​

Compared to the legacy solution, both new approaches achieve significant optimizations, but with different focuses:

• If the business prioritizes extreme real-time performance and has fixed aggregation logic, Solution A is recommended, as it pushes the computational load down to the storage layer to the greatest extent.
• If the business requires flexible multi-dimensional analysis and can tolerate a few seconds of latency, Solution B is recommended, as it fully leverages StarRocks' OLAP capabilities and results in a simpler, more general-purpose architecture.

By adopting these two solutions, Taobao Instant Commerce successfully achieved real-time, accurate, and cost-effective end-to-end unique visitor (UV) metrics at massive scale. This approach retains StarRocks' query strengths while resolving the upstream pipeline performance bottlenecks through Fluss.

Fluss Implementation Best Practices and Future Roadmap​

During the rollout of Fluss, Taobao Instant Commerce progressed from single-feature pilots to full pipeline implementations combining multiple Fluss features. This process has produced practical, reusable patterns while also shaping a clear technical roadmap aligned with both business needs and upstream community development.

Key Lessons Learned​

• Prioritize DDL Standards: When creating wide tables, plan the responsible jobs for each column in advance. All non-primary key columns must be declared as NULLABLE, and writers should specify only the target columns. This standard must be integrated into the team’s table creation workflow to prevent update failures caused by improper field configurations.
• Precise Bucketing Strategy: An insufficient number of buckets can lead to excessive data volume in a single bucket, causing read amplification in RocksDB.
• Maximize Column Pruning Benefits: Business SQL queries should explicitly specify the required columns to avoid SELECT *. This is particularly important in scenarios involving log wide tables and dimension table joins, allowing Fluss to fully leverage its I/O savings advantages.
• Phased Migration: There is no need to replace the entire traditional pipeline at once. Start by replacing large-state two-stream joins with Delta Joins, then gradually expand to Partial Update wide tables and lake-stream integration. Each step should deliver independent, quantifiable benefits, and should be promoted broadly only after its effectiveness has been verified.

Future Roadmap​

In the future, the overall data architecture will gradually unify computational logic through Flink SQL, evolve into a unified lake-stream storage foundation based on Fluss and Paimon, and build a unified query acceleration layer leveraging StarRocks. The ultimate goal is to achieve one codebase, one dataset, and one metric definition, serving both real-time applications and offline analytics simultaneously, thereby thoroughly resolving data inconsistencies and high maintenance costs caused by the separation of stream and batch processing.

UV (Unique Visitors) measures the count of distinct users who visited a page or triggered an event within a given time window — unlike PV (Page Views), which counts every request regardless of who made it. For any product or platform, accurate real-time UV statistics across dimensions like channel, city, date, and hour are a core analytical requirement. The full combination of four dimensions means 16 grouping methods; when the dimension count increases to seven, the number of possible groupings reaches 128.

How can multi-dimensional deduplication be both accurate and flexible while maintaining real-time performance? Behind this challenge lie two very different computing paradigms: direct deduplication of raw data, or set operations based on bitmaps.

Problems with Traditional Deduplication Schemes​

Direct deduplication of raw data (e.g. COUNT(DISTINCT user_id)) is the most intuitive and accurate deduplication method: maintain a hash set for each dimension combination and record all user IDs that have appeared under the group.

This approach works well with smaller data volumes and fewer dimensions, but as business scale grows, three core limitations emerge.

Performance increases linearly with the amount of data. Each dimension combination needs to independently store all user identities that have occurred. When daily page view logs reach billions and distinct users number in the tens or hundreds of millions, each group's hash set demands significant memory and CPU for insertion and lookup. Total compute cost scales with the product of dimension combinations and data volume, which is especially noticeable during peak periods.

Multidimensional extensibility is limited. If an analyst needs UV by Channel x City and Channel x Date, the traditional scheme must calculate two separate deduplication results. The existing Channel x City result cannot be rolled up to get the Channel-level UV, because user sets across cities may overlap and simple summation leads to double counting. With each additional dimension, the computational task multiplies and existing intermediate results cannot be reused.

Insufficient query flexibility. Pre-aggregation schemes must fix dimension combinations in advance. When a business team needs a new dimension on the fly (such as adding device type), adjusting the entire data processing pipeline is often required instead of simply modifying a query.

These three limitations share a root cause: storing and computing on raw user identities at every dimension combination does not scale.

Put it another way: what if instead of storing the raw ID of each user, we represented the entire user set as a compact structure — one bit per user?

In the bitmap scheme, deduplication count equals counting the number of 1s in the bitmap (popcount), merging two user sets equals a bitwise OR, and cross-dimensional analysis equals a bitwise AND. All operations are CPU-native bit operations, blisteringly fast.

More importantly, bitmaps naturally support roll-up: the bitmap of Channel = app, City = Shanghai combined with OR with the bitmap of Channel = app, City = Beijing yields the user set for Channel = app — no need to re-scan the original data.

This is a fundamental difference between the two computing paradigms: direct deduplication requires independent calculations for each dimension combination, while the intermediate results of bitmap deduplication are naturally composable.

But the raw bitmap has a problem: if the user ID space is sparse (such as UUID or hash values), the bitmap will be very large and waste memory. A compressed bitmap format that works efficiently on sparse data is needed. This is where RoaringBitmap comes in.

How RoaringBitmap Works​

The core idea of a bitmap is simple: use an array of bits to represent a collection of integers. If the user ID is k, set the k-th position to 1. The deduplication count is the number of 1s, and set merging is a bitwise OR. These operations map directly to CPU instructions, which is extremely efficient.

However, the size of a raw bitmap depends on the maximum ID value. If the ID space is a 32-bit integer (about 2.1 billion), a full bitmap takes 256 MB. When the ID distribution is sparse, most bits are 0, and almost all memory is wasted.

RoaringBitmap solves this with a layered, adaptive compression strategy. It divides the 32-bit integer space into 65,536 partitions by the upper 16 bits, and each partition holds up to 65,536 values (the lower 16 bits), called a Container. Each container automatically selects the optimal storage format based on its data density:

• ArrayContainer: When the number of elements is less than 4,096, values are stored in a sorted array of 16-bit integers. For sparse data, this saves tens of times more space than a raw bitmap.
• BitmapContainer: When the number of elements reaches 4,096 or more, the container switches to a raw 8 KB bitmap. For dense data, bitmaps are more compact than arrays.
• RunContainer: When data presents continuous interval characteristics (such as IDs 100–500), values are stored using run-length encoding, recording only the start and end of each run.

This adaptive strategy allows RoaringBitmap to remain efficient across a variety of data distributions: as compact as an array when sparse, as fast as a bitmap when dense, and as economical as run-length encoding when continuous. In practice, its memory footprint is usually only one-tenth to one-hundredth of the original hash set.

For deduplication scenarios, RoaringBitmap set operations map directly to analytical requirements: OR merges user sets, AND performs cross-dimensional analysis, ANDNOT performs exclusion analysis, and cardinality returns the deduplication count. All operations are accelerated by SIMD instructions and can be completed in milliseconds even on hundreds of millions of records.

One prerequisite: the input to RoaringBitmap must be an integer, and the denser the ID space, the better the compression and efficiency. If the original user ID is a string (such as a UUID or phone number), a dictionary table is needed to map sparse IDs to dense integer IDs.

How Apache Fluss Enables RoaringBitmap Deduplication​

A RoaringBitmap-based deduplication scheme requires three things: a dense integer ID space, bitmap merging at the storage layer, and native bitmap aggregation functions. Apache Fluss provides all three as built-in capabilities.

Auto-Increment Columns: Zero-Code Dictionary Tables​

The auto-increment column feature of Fluss lets you declare a field directly in the table definition. When a new user is written for the first time, Fluss automatically assigns an incrementing integer ID. No external ID service or historical data migration is required.

CREATE TABLE uid_dictionary (
    user_id STRING,
    uid INT,
    PRIMARY KEY (user_id) NOT ENFORCED
) WITH (
  'auto-increment.fields' = 'uid'
);

A single INSERT statement is all that is needed to map sparse identities to dense integers. int32 covers about 2.1 billion users, which is sufficient for most business scenarios. The generated ID is unique and monotonically increasing, which is naturally suited to RoaringBitmap's compression strategy.

Aggregate Merge Engine: Merge on Write​

The Aggregation Merge Engine of Fluss allows you to define aggregation rules for each column. When rows with the same primary key arrive, Fluss automatically merges them on the server side instead of overwriting or appending.

This means the compute layer only needs to send raw events and does not need to maintain any aggregated intermediate state. With exactly-once semantics based on Flink Checkpoint, the write path is compact and reliable.

For append-only aggregation operations such as bitmap union (UV deduplication), merging is naturally idempotent during writing. Adding the same uid to a bitmap multiple times does not change the result. In such scenarios, Flink jobs are only responsible for forwarding — no complex UDAFs or retraction messages are needed. For scenarios involving data retraction (such as order cancellation), rollback support for the aggregation merge engine is currently under development and is expected in a future release.

Built-in rbm32 / rbm64 Aggregate Functions​

Fluss supports RoaringBitmap as a first-class aggregation type. rbm32 (32-bit) and rbm64 (64-bit) perform native bitmap union on write.

CREATE TABLE uv_agg (
    channel   STRING,
    city      STRING,
    ymd       STRING,  -- format: YYYYMMDD
    hh        STRING,  -- format: HH (00-23)
    uv_bitmap BYTES,
    pv        BIGINT,
    PRIMARY KEY (channel, city, ymd, hh) NOT ENFORCED
) WITH (
    'table.merge-engine'    = 'aggregation',
    'fields.uv_bitmap.agg' = 'rbm32',
    'fields.pv.agg'        = 'sum'
);

Note: the quickstart section extends this table with 'table.datalake.enabled' = 'true' and 'table.datalake.freshness' = '30s' to enable lake tiering for batch queries. The core deduplication behaviour is identical.

Each event carries a single-element bitmap, which Fluss merges into the existing bitmap of the corresponding primary key on the server side. The deduplication logic is entirely handled by the storage layer without requiring any custom merge code.

Putting It Together: The Full Data Flow​

These three capabilities are interlinked: the auto-increment column provides a dense ID (the prerequisite for bitmap deduplication), the aggregation engine merges on write (deduplication is completed in the storage layer), and rbm32 makes bitmap merging a native storage operation (no custom logic needed).

The data flow is straightforward: when an event arrives, Flink obtains the dense uid by querying the dictionary table via a lookup join, calls RB_BUILD(uid) to build a single-element bitmap, and writes it to the aggregation table. Fluss automatically executes bitmap union on the server side. At query time, RB_CARDINALITY() extracts the UV count directly.

The responsibilities are clearly separated: Flink handles mapping, joining, and forwarding without holding long-lived state; Fluss holds the persistent merged result; queries read the merged bitmap and extract the count. Deduplication happens at write time — the query side requires no aggregation at all.

Streaming Lakehouse Unification​

When data volumes grow to require analysis across longer time windows, this architecture can be seamlessly extended through Fluss's Lake-Stream integration capabilities. In a daily partitioning scenario, each day's UV data is stored as a bitmap partition. Historical partitions (e.g., older than 30 days) can be offloaded to Data Lake storage (Paimon or Iceberg), while recent partitions remain in Fluss for real-time updates.

To count UV for an entire year, you only need to OR the 365 daily bitmap partitions and compute the cardinality. Regardless of whether data is in Fluss or the data lake, the bitmap union semantics are identical. This architecture balances storage cost and query flexibility, enabling accurate deduplication analysis across any time span.

End-to-End Quickstart​

This section walks through the complete workflow with a minimal working example you can run locally.

Environment Preparation​

Requirements: Docker and Docker Compose.

1. Create a working directory and download dependencies

mkdir fluss-rbm-quickstart && cd fluss-rbm-quickstart
mkdir lib

# Download Fluss Flink connector
curl -fL -o lib/fluss-flink-1.20-0.9.0-incubating.jar \
  https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-1.20/0.9.0-incubating/fluss-flink-1.20-0.9.0-incubating.jar

# Download RoaringBitmap UDF (provides RB_BUILD, RB_CARDINALITY, RB_OR_AGG, etc.)
curl -fL -o lib/flink-roaringbitmap-0.2.0.jar \
  https://github.com/flink-extended/flink-roaringbitmap/releases/download/v0.2.0/flink-roaringbitmap-0.2.0.jar

# Download Paimon and Fluss Lakehouse dependencies (for data tiering and batch queries)
curl -fL -o lib/paimon-flink-1.20-1.3.1.jar \
  https://repo1.maven.org/maven2/org/apache/paimon/paimon-flink-1.20/1.3.1/paimon-flink-1.20-1.3.1.jar

curl -fL -o lib/fluss-lake-paimon-0.9.0-incubating.jar \
  https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-paimon/0.9.0-incubating/fluss-lake-paimon-0.9.0-incubating.jar

curl -fL -o lib/fluss-flink-tiering-0.9.0-incubating.jar \
  https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-tiering/0.9.0-incubating/fluss-flink-tiering-0.9.0-incubating.jar

curl -fL -o lib/flink-shaded-hadoop-2.8.3-10.0.jar \
  https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar

The UDF JAR is a self-contained fat JAR. Placing it in Flink's lib/ directory makes all functions available without additional dependencies. The source code is available in the flink-roaringbitmap repository. Paimon-related JARs enable the Lakehouse Tiering feature of Fluss, where data automatically tiers from Fluss into Paimon, supporting ORDER BY and GROUP BY aggregated batch queries.

2. Create docker-compose.yml

services:
  zookeeper:
    restart: always
    image: zookeeper:3.9.2

  coordinator-server:
    image: apache/fluss:0.9.0-incubating
    entrypoint: ["sh", "-c", "umask 0000 && exec /docker-entrypoint.sh coordinatorServer"]
    depends_on:
      - zookeeper
    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        bind.listeners: FLUSS://coordinator-server:9123
        remote.data.dir: /tmp/fluss-remote-data
        datalake.format: paimon
        datalake.paimon.metastore: filesystem
        datalake.paimon.warehouse: /tmp/paimon
    volumes:
      - shared-warehouse:/tmp/paimon
      - shared-remote-data:/tmp/fluss-remote-data

  tablet-server:
    image: apache/fluss:0.9.0-incubating
    entrypoint: ["sh", "-c", "umask 0000 && exec /docker-entrypoint.sh tabletServer"]
    depends_on:
      - coordinator-server
    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        bind.listeners: FLUSS://tablet-server:9123
        data.dir: /tmp/fluss/data
        kv.snapshot.interval: 30s
        remote.data.dir: /tmp/fluss-remote-data
        datalake.format: paimon
        datalake.paimon.metastore: filesystem
        datalake.paimon.warehouse: /tmp/paimon
    volumes:
      - shared-warehouse:/tmp/paimon
      - shared-remote-data:/tmp/fluss-remote-data

  jobmanager:
    image: flink:1.20
    ports:
      - "8083:8081"
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
    entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/flink/lib && exec /docker-entrypoint.sh jobmanager"]
    volumes:
      - ./lib:/tmp/lib
      - shared-warehouse:/tmp/paimon
      - shared-remote-data:/tmp/fluss-remote-data

  taskmanager:
    image: flink:1.20
    depends_on:
      - jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 10
        taskmanager.memory.process.size: 4096m
        taskmanager.memory.framework.off-heap.size: 256m
    entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/flink/lib && exec /docker-entrypoint.sh taskmanager"]
    volumes:
      - ./lib:/tmp/lib
      - shared-warehouse:/tmp/paimon
      - shared-remote-data:/tmp/fluss-remote-data

volumes:
  shared-warehouse:
  shared-remote-data:

3. Start the environment

docker compose up -d

4. Start the Lakehouse Tiering Service

The tiering service continuously transfers data from Fluss to Paimon to support batch queries. Start it in the background before writing data:

docker compose exec -d jobmanager \
  /opt/flink/bin/flink run \
    -D execution.checkpointing.interval=30s \
    /opt/flink/lib/fluss-flink-tiering-0.9.0-incubating.jar \
    --datalake.bootstrap.servers coordinator-server:9123 \
    --datalake.format paimon \
    --datalake.paimon.metastore filesystem \
    --datalake.paimon.warehouse /tmp/paimon

5. Open the SQL Client

docker compose exec jobmanager ./bin/sql-client.sh

Create a Catalog and Register the UDFs​

CREATE CATALOG fluss_catalog WITH (
  'type'              = 'fluss',
  'bootstrap.servers' = 'coordinator-server:9123'
);

USE CATALOG fluss_catalog;

Register the RoaringBitmap UDFs. The JAR contains three functions: RB_BUILD wraps a single integer as a bitmap; RB_CARDINALITY extracts the deduplication count from a bitmap; RB_OR_AGG executes bitmap OR aggregation for roll-up queries. The deduplication merge at write time is performed by Fluss's rbm32 aggregation — these UDFs handle bitmap construction on the Flink side and analysis on the query side.

CREATE TEMPORARY FUNCTION RB_BUILD
  AS 'org.apache.flink.udfs.bitmap.RbBuildFunction';
CREATE TEMPORARY FUNCTION RB_CARDINALITY
  AS 'org.apache.flink.udfs.bitmap.RbCardinalityFunction';
CREATE TEMPORARY FUNCTION RB_OR_AGG
  AS 'org.apache.flink.udfs.bitmap.RbOrAggFunction';

Note: These functions are currently provided by the external flink-roaringbitmap library. Once FIP-37: Native RoaringBitmap Integration for Apache Fluss is completed, they will be built directly into Fluss — no external JAR or manual UDF registration will be required.

Create Dictionary and Aggregation Tables​

-- Dictionary table: sparse user_id -> dense uid
CREATE TABLE uid_dictionary (
  user_id STRING,
  uid     INT,
  PRIMARY KEY (user_id) NOT ENFORCED
) WITH (
  'auto-increment.fields' = 'uid'
);

-- Aggregation table: bitmap is automatically merged on write
CREATE TABLE uv_agg (
  channel    STRING,
  city       STRING,
  ymd        STRING,  -- format: YYYYMMDD
  hh         STRING,  -- format: HH (00-23)
  uv_bitmap  BYTES,
  pv         BIGINT,
  PRIMARY KEY (channel, city, ymd, hh) NOT ENFORCED
) WITH (
  'table.merge-engine'       = 'aggregation',
  'fields.uv_bitmap.agg'    = 'rbm32',
  'fields.pv.agg'           = 'sum',
  'table.datalake.enabled'  = 'true',
  'table.datalake.freshness' = '30s'
);

Write Sample Data​

-- Simulate page browsing events: users appear across channels, cities, and time periods
CREATE TEMPORARY VIEW page_views AS
SELECT * FROM (
  VALUES
    ('user_1', 'app',          'Shanghai', '20260301', '10'),
    ('user_1', 'app',          'Shanghai', '20260301', '10'),
    ('user_2', 'app',          'Shanghai', '20260301', '10'),
    ('user_2', 'app',          'Shanghai', '20260301', '14'),
    ('user_3', 'app',          'Beijing',  '20260301', '10'),
    ('user_1', 'app',          'Beijing',  '20260301', '14'),
    ('user_3', 'web',          'Beijing',  '20260301', '10'),
    ('user_4', 'web',          'Shanghai', '20260301', '14'),
    ('user_2', 'app',          'Shanghai', '20260302', '09'),
    ('user_5', 'app',          'Beijing',  '20260302', '09'),
    ('user_4', 'mini_program', 'Beijing',  '20260302', '11'),
    ('user_1', 'mini_program', 'Shanghai', '20260302', '11')
) AS t(user_id, channel, city, ymd, hh);

-- Populate the dictionary table
INSERT INTO uid_dictionary (user_id)
SELECT DISTINCT user_id FROM page_views;

-- Build bitmaps and write to the aggregation table
INSERT INTO uv_agg
SELECT
  pv.channel,
  pv.city,
  pv.ymd,
  pv.hh,
  RB_BUILD(d.uid) AS uv_bitmap,
  CAST(1 AS BIGINT) AS pv
FROM page_views AS pv
JOIN uid_dictionary AS d ON pv.user_id = d.user_id;

Each event is written as one row carrying a single-element bitmap. Fluss merges it server-side via rbm32 union into the existing bitmap of the corresponding primary key. Flink jobs are only responsible for mapping and forwarding — they hold no aggregation state, require no GROUP BY, and do not process retraction messages.

Query Deduplication Results​

After writing the data, wait approximately 60 seconds for the data to tier into Paimon. For PK tables, the tiering service uses the KV snapshot as a synchronization checkpoint — it reads the snapshot state first, then replays subsequent CDC events from that point. The wait covers one KV snapshot period (30 seconds) plus one tiering checkpoint period. Switch to batch mode to query the tiered data. Batch mode supports ORDER BY and does not involve streaming retraction semantics.

SET 'execution.runtime-mode' = 'batch';
SET 'sql-client.execution.result-mode' = 'tableau';

SELECT
  channel, city, ymd, hh,
  RB_CARDINALITY(uv_bitmap) AS uv,
  pv
FROM uv_agg
ORDER BY channel, city, ymd, hh;

Result:

+--------------+----------+----------+----+----+----+
| channel      | city     | ymd      | hh | uv | pv |
+--------------+----------+----------+----+----+----+
| app          | Beijing  | 20260301 | 10 |  1 |  1 |
| app          | Beijing  | 20260301 | 14 |  1 |  1 |
| app          | Beijing  | 20260302 | 09 |  1 |  1 |
| app          | Shanghai | 20260301 | 10 |  2 |  3 |
| app          | Shanghai | 20260301 | 14 |  1 |  1 |
| app          | Shanghai | 20260302 | 09 |  1 |  1 |
| mini_program | Beijing  | 20260302 | 11 |  1 |  1 |
| mini_program | Shanghai | 20260302 | 11 |  1 |  1 |
| web          | Beijing  | 20260301 | 10 |  1 |  1 |
| web          | Shanghai | 20260301 | 14 |  1 |  1 |
+--------------+----------+----------+----+----+----+

Note the (app, Shanghai, 20260301, 10) row: user_1 appeared twice and user_2 appeared once, yielding uv = 2. The three single-element bitmaps were merged by Fluss into a deduplicated set containing two distinct users. pv = 3 is the raw event count from the sum aggregation. The query side requires no aggregation — results are ready at write time.

Multi-Dimensional Roll-Up Queries​

The preceding query reads the finest-granularity pre-aggregation results (channel × city × ymd × hh). The real power of bitmaps is supporting flexible roll-up: performing OR operations across fine-grained bitmaps yields coarse-grained deduplication counts without returning to the original data.

Roll up by channel, overall UV across all cities, dates, and hours:

SELECT
  channel,
  RB_CARDINALITY(RB_OR_AGG(uv_bitmap)) AS uv,
  SUM(pv) AS pv
FROM uv_agg
GROUP BY channel;

+--------------+----+----+
| channel      | uv | pv |
+--------------+----+----+
| app          |  4 |  8 |
| mini_program |  2 |  2 |
| web          |  2 |  2 |
+--------------+----+----+

user_1 visited the app channel from both Shanghai and Beijing at multiple times. RB_OR_AGG combines all fine-grained bitmaps under the same channel into a single deduplicated set — each user is counted only once. The app channel has four distinct users (1, 2, 3, 5), not a simple sum of per-group UV counts.

Roll up by date, daily UV across all channels and cities:

SELECT
  ymd,
  RB_CARDINALITY(RB_OR_AGG(uv_bitmap)) AS uv,
  SUM(pv) AS pv
FROM uv_agg
GROUP BY ymd;

+----------+----+----+
| ymd      | uv | pv |
+----------+----+----+
| 20260301 |  4 |  8 |
| 20260302 |  4 |  4 |
+----------+----+----+

Active users are users 1–4 on March 1 and users 1, 2, 4, and 5 on March 2. Simply summing per-group UV counts would result in double counting, but the bitmap OR operation gives accurate results.

Roll up by channel x date, a common dashboard view:

SELECT
  channel,
  ymd,
  RB_CARDINALITY(RB_OR_AGG(uv_bitmap)) AS uv,
  SUM(pv) AS pv
FROM uv_agg
GROUP BY channel, ymd;

+--------------+----------+----+----+
| channel      | ymd      | uv | pv |
+--------------+----------+----+----+
| app          | 20260301 |  3 |  6 |
| app          | 20260302 |  2 |  2 |
| mini_program | 20260302 |  2 |  2 |
| web          | 20260301 |  2 |  2 |
+--------------+----------+----+----+

Global UV, total distinct users across all dimensions:

SELECT
  RB_CARDINALITY(RB_OR_AGG(uv_bitmap)) AS total_uv,
  SUM(pv) AS total_pv
FROM uv_agg;

+----------+----------+
| total_uv | total_pv |
+----------+----------+
|        5 |       12 |
+----------+----------+

Five distinct users generated a total of 12 page views. One query, one full bitmap OR — no need to re-scan the original events.

All roll-up queries follow the same pattern: GROUP BY the target dimensions, apply RB_OR_AGG on the bitmap column, and use RB_CARDINALITY to extract the count. The finest-granularity pre-aggregated bitmap is the building block for all coarser-grained analysis.

In a production environment, replace the bounded page_views view with a streaming data source and add the /*+ OPTIONS('lookup.insert-if-not-exists' = 'true') */ hint to automatically register new users and enable a real-time bitmap write pipeline. The storage-side schema remains identical.

Clean Up the Environment​

After exiting the SQL Client, run the following to stop and remove all containers:

docker compose down -v

Beyond UV: Other Use Cases​

This pattern is not limited to UV counting. It extends naturally to a variety of analytical scenarios. The core structure is consistent: choose the appropriate primary key granularity, and bitmap merging is handled by the storage layer at write time.

Real-time multi-dimensional UV analysis. The main scenario in this post. Accurate UV statistics based on any combination of dimensions such as channel, city, date, and hour, with flexible dimension roll-up via bitmap union. Applicable to traffic dashboards, advertising effectiveness analysis, and activity monitoring.

User retention analysis. Store active user bitmaps at the granularity of user × date. The retained user set is obtained by AND-ing bitmaps from different dates. For example, the cardinality of Day1_bitmap AND Day7_bitmap gives the 7-day retention count.

Funnel analysis. Store an aggregation table for each funnel step, with bitmaps recording the set of users who reached that step. Conversion rates are computed by AND-ing bitmaps of adjacent steps. Since bitmaps support set difference (ANDNOT), you can further analyze the characteristics of users who dropped off at each step.

Audience segmentation and user profiling. Build user attribute labels (such as gender, province, and preference category) as bitmaps. Multi-label combination queries are converted into bitmap AND/OR/ANDNOT operations, enabling selection of tens of millions of users in milliseconds. This is a high-frequency scenario in advertising DMP and recommendation systems.

Getting Started​

If you are evaluating real-time multi-dimensional exact deduplication schemes, Fluss's RoaringBitmap capability is a worthwhile starting point. The architecture follows a clear three-step pattern:

1. Map sparse user identities to dense integers using an auto-increment uid_dictionary table.
2. Deduplicate at write time using an rbm32 aggregation table — no stateful Flink jobs, no complex UDAFs.
3. Query directly using RB_CARDINALITY() for point reads or RB_OR_AGG() + RB_CARDINALITY() for any roll-up combination.

The result is a system where deduplication is handled entirely by the storage layer, and the query side reads pre-computed results with no additional aggregation overhead.

rbm32 / rbm64 aggregation and the auto-increment dictionary scheme are available from Apache Fluss 0.9. See the Fluss documentation for the full configuration reference.

Join the Community​

Apache Fluss is an open-source project under active development within the Apache Software Foundation.

• GitHub: github.com/apache/fluss
• Documentation: fluss.apache.org

If you maintain a data system that only speaks Java, you will eventually hear from someone who doesn't. A Python team building a feature store. A C++ service that needs sub-millisecond writes. An AI agent that wants to call your system through a tool binding. They all need the same capabilities (writes, reads, lookups) and none of them want to spin up a JVM to get them.

Apache Fluss, streaming storage for real-time analytics and AI, hit this exact inflection point. The Java client works well for Flink-based compute, where the JVM is already the world you live in. But outside that world, asking consumers to run a JVM sidecar just to write a record or look up a key creates friction that compounds across every service, every pipeline, every agent in the stack.

We could have written a separate client for each language. Maintain five copies of the wire protocol, five implementations of the batching logic, five sets of retry semantics and idempotence tracking. That path scales linearly with languages and ends predictably: the Java client gets features first, the Python client gets them six months later with slightly different edge-case behavior, and the C++ client is perpetually "almost done."

We took a different path and tried to leverage the lessons of the great.

The librdkafka Model​

If you've worked with Kafka clients outside of Java, you've probably used librdkafka without knowing it. It's a single C library that powers confluent-kafka-python, confluent-kafka-go, and others. One core handles the wire protocol, batching, memory management, and delivery semantics. Each language binding is a thin wrapper, a glue on top of a battle-tested engine.

The model is elegant because it inverts the usual maintenance equation. Instead of N full client implementations that diverge over time, each developing its own bugs, its own subtle behavioral differences, its own backlog of features the Java client has but the Python client doesn't yet, you get one implementation and N thin bindings that stay in sync by construction. A bug gets fixed once, and every language picks it up on the next build.

The deeper benefit is correctness, not just code reuse. When you maintain three separate implementations of a client protocol, behavioral drift is inevitable. Edge cases in retry logic, subtle differences in how backpressure kicks in, inconsistencies in how idempotent writes handle sequence numbers. These are the bugs that don't show up in unit tests but surface in production under load, and they surface differently in each language.

We built fluss-rust on this same idea. A single Rust core implements the full Fluss client protocol (Protobuf-based RPC, record batching with backpressure, background I/O, Arrow serialization, idempotent writes, SASL authentication) and exposes it to three languages:

• Rust: directly, as the fluss-rs crate
• Python: via PyO3, the Rust-Python bridge
• C++: via CXX, the Rust-C++ bridge

To give a sense of proportion: the Rust core is roughly 40k lines, while the Python binding is around 5k and the C++ binding around 6k. The bindings handle type conversion, async runtime bridging, and memory ownership at the language boundary, but all the protocol logic, batching, Arrow codec, and retry handling live in the shared core.

Why Rust and Not C​

C would have been the obvious choice. librdkafka already proves the model works at enormous scale, and the C ABI is the universal language of foreign function interfaces.

We chose Rust, and the reason is specific rather than philosophical: compile-time safety is a force multiplier for a small team maintaining a shared core that multiple languages depend on.

Getting memory safety right in C means manual lifetime tracking, careful code review, and years of experience knowing where the subtle bugs hide. Rust gives the same zero-overhead profile (no garbage collector, no runtime) but checks those invariants at compile time instead.

To make this concrete: the Fluss write path moves ownership from the caller through a concurrent map, into a background event loop, and back out to futures the caller may or may not still be holding. In C, getting that right is a matter of discipline, and getting it wrong means segfaults that only reproduce under production load. In Rust, the borrow checker and the Send/Sync traits catch those problems before the code ever runs.

We're not alone here. Polars, Apache OpenDAL, and delta-rs all chose Rust as a shared core with language bindings on top. Fluss's Rust SDK sits in that lineage.

Relationship with the Java Client​

The Java client remains the primary integration point for Flink, powering the SQL connector, the DataStream API, and the tightest path for JVM-based streaming compute. If your workload is Flink reading from and writing to Fluss, that's still the right client to use.

fluss-rust isn't trying to replace it. It exists for the consumers the Java client was never designed to serve: Python pipelines, C++ services, Rust applications, and anything else that doesn't want a JVM in the process. Both clients talk the same wire protocol and get the same server-side behavior. Most teams will end up using both, the Java client for Flink and fluss-rust for everything around it.

What the Rust Core Covers​

The Rust core implements the complete Fluss client protocol. Here's how the pieces fit together.

When you write a record, the call is synchronous: the record gets queued into a per-bucket batch without touching the network. A background sender task picks up ready batches and ships them as RPCs to the responsible TabletServers. This follows the same pattern as both the Fluss Java client and Kafka producers.

The caller gets back a WriteResultFuture. Await it to block until the server confirms, or drop it for fire-and-forget. The write proceeds through the background sender regardless, with the same retries and server-side durability (acks=all by default). In high-throughput pipelines you typically don't await every write individually, but collect futures for a batch of records and await them together before committing your source offset.

Batches ship automatically when they fill up or after a short timeout (100ms by default), so flush() isn't needed for data to reach the server. It's there for when you need to confirm that everything in flight has landed. If the write buffer fills up, new writes block until space frees up rather than silently consuming unbounded memory.

Fluss has two table types (primary key tables and log tables), and the Rust core has a writer for each. UpsertWriter handles keyed writes: full upserts, deletes, and partial updates where you send only the columns that changed. AppendWriter handles append-only log writes and can also accept Arrow RecordBatch directly if you already have columnar data. Both support idempotent delivery.

For reads, LogScanner provides streaming consumption with offset tracking. You can push column projection down to the server, so if you only need three columns out of twenty, only those three travel over the network. Each record also carries changelog metadata (offset, timestamp, and change type like AppendOnly, Insert, UpdateBefore, UpdateAfter, Delete), which is how Fluss exposes its CDC semantics to consumers outside of Flink.

Lookuper handles the other access pattern: point queries against primary key tables. You encode a key, and the client resolves the bucket, finds the leader, and returns the row. The response is compact binary, and the row is only deserialized when you first access a field.

The core also covers admin operations (creating and dropping databases and tables, schema management) and SASL/PLAIN authentication at the connection level.

Arrow on the Wire​

Arrow deserves its own section because it's the architectural decision that makes the multi-language model work end to end, not just at the API level but down to the bytes on the wire.

Fluss transmits data as Arrow IPC, compressed with ZSTD by default. When you write an Arrow RecordBatch, it goes straight into the wire format. When you scan, the response comes back as Arrow RecordBatch. The data stays in Arrow throughout, which means there's no serialization boundary between the Rust core and the caller.

This matters most at the language boundary. The Python binding already supports full Arrow interop in both directions: poll_arrow() returns scan results as a PyArrow Table, and write_arrow_batch() accepts a PyArrow RecordBatch for writes. Both cross the Rust-Python boundary without copying, because arrow-pyarrow shares the underlying memory buffers. A scan result goes straight from the Rust core into PyArrow, and from there into Pandas, Polars, or DuckDB with no conversion step.

On the C++ side, the Arrow C Data Interface handles the same zero-copy handoff for callers that export or import Arrow arrays.

Looking ahead, Arrow also makes Apache DataFusion integration straightforward. DataFusion's table providers already expect Arrow, so wiring fluss-rust as a data source is a natural extension.

What This Looks Like in Practice​

Suppose a Flink job consumes CDC from Postgres, computes user features, and writes them into a Fluss primary key table. A Python scoring service needs to look up those features before running a model. With the Python binding, that's a few lines:

from fluss import FlussConnection, Config, TablePath

conn = await FlussConnection.create(Config({"bootstrap.servers": "fluss:9123"}))
table = await conn.get_table(TablePath("analytics", "user_features"))

lookuper = table.new_lookup().create_lookuper()

result = await lookuper.lookup({"user_id": request.user_id})
score = model.predict(result)

The lookup goes through the same Protobuf RPC and hits the same KV store on the TabletServer that the Java client would use. The Python service just doesn't need a JVM to get there.

On the write side, consider an IoT gateway written in C++ that pushes sensor readings into a Fluss log table. It can't afford to block on each record, so it queues them and lets the Rust core handle batching and delivery:

fluss::AppendWriter writer;
table.NewAppend().CreateWriter(writer);

for (const auto& event : events) {
    fluss::GenericRow row;
    // ... populate row from event ...
    writer.Append(row);               // queued in Rust, sent automatically
}

When the Caller Is a Machine​

The examples above involve humans writing Python or C++ code. But increasingly, the caller isn't a human at all. AI agents interact with data infrastructure through tool calls, and the way they use a client library is different from how a developer does.

An agent that needs to check a user's subscription tier or write back a recommendation doesn't read documentation or understand batching internals. It sees a tool definition: lookup(table, key), upsert(table, row), flush(). The smaller and more predictable that interface is, the more reliably the agent uses it. If you've worked with LLM tool-calling, you've seen how quickly reliability degrades as the number of functions or the complexity of their signatures grows.

This is where the single-core architecture pays off in a way we didn't originally design for. Because the Rust core hides all the protocol and batching complexity, the Python binding exposes a small set of straightforward functions. An agent calls await lookuper.lookup(key), gets a dict back, and moves on. No JVM to manage, no sidecar to health-check, just a Python function that happens to run compiled Rust underneath.

More broadly, as we discussed in What does Apache Fluss mean in the context of AI?, real-time intelligent systems need fresh features, evolving context, and continuously updated state. Fluss fits naturally as a context store for these systems, and the Rust SDK is what makes that context store accessible outside the JVM world.

How We Got Here​

We didn't build everything at once. The Rust core came first with the protocol, batching, Arrow integration, writes, and scanning. We added PyO3 for Python and CXX for C++ once those paths were stable, and features like lookups landed later as the Rust API matured. If we'd started the bindings earlier, the FFI boundary would have been designed around a half-finished API, and we'd have spent more time reshaping glue code than building features.

Async was the most involved design problem. Rust runs on Tokio, Python on asyncio, and C++ callers expect synchronous returns. We decided early that all async complexity would stay inside the Rust core: Python spawns work on the shared Tokio runtime and gets back an asyncio-compatible future, while C++ blocks on Tokio and returns when the operation completes. That meant tricky problems like WriteResultFuture drop semantics (making sure a dropped future doesn't leak memory or leave a batch stuck in the accumulator) only had to be solved once.

We also underestimated how many bugs live at the language boundary rather than in the core. The Rust integration tests against a real Fluss cluster all passed, but when we ran the same scenarios through the Python and C++ bindings, new issues appeared. On the Python side, API errors from the server weren't being propagated through PyO3 at all, so operations would fail silently. On the C++ side, the row access layer had panics that required a significant rework, and the error handling for pointer-returning FFI methods was swallowing server errors instead of surfacing them. As a result, we now run full round-trip integration tests for all three languages in CI.

What's Next​

The core read and write paths work, but there are gaps to close before the Rust SDK reaches full parity with the Java client. Complex data types (Array, Map, Row) aren't supported yet, which limits what schemas you can work with. Limit scans and batch scanning are in progress, and once those land with Python and C++ bindings, the SDK becomes usable for a wider range of analytical workloads. We also want to support subscribing to primary key table changelogs, which would let non-Flink consumers track how keyed state evolves over time.

On the infrastructure side, we're adding client metrics so operators can monitor the SDK in production, and improving Python ergonomics with async iterator support for log scanning.

Beyond feature parity, there are four directions we're especially excited about.

The first is DataFusion integration. The Rust core already produces Arrow RecordBatches, which is exactly what DataFusion's table provider interface expects. Wiring the two together would let users run SQL queries directly over Fluss data from Rust or Python, without going through Flink.

The second is a Go client. The shared-core model extends naturally to Go via CGo or a similar FFI bridge. A Go client would unlock native log ingestion for the Go ecosystem, including integrations with tools like Filebeat, and is already something Fluss users have asked for.

The third is a CLI for AI agent integration. A command-line tool built on the Rust core that can look up keys, write records, and scan tables gives AI agents a natural way to interact with Fluss through tool-calling, without importing a library or managing a runtime. It's equally useful for operators debugging in production, shell scripts, and CI/CD pipelines.

The fourth is a multiprotocol query gateway built on top of the Rust core. A Rust-based gateway could expose Fluss through Flight SQL (accessible via ADBC), REST, and the PostgreSQL wire protocol. For SQL queries it delegates to DataFusion; for lookups and log writes it calls the Rust core directly. On the project side, the community is working toward moving fluss-rust into the main Apache Fluss repository. This would unify the release process, simplify cross-repo coordination, and signal the project's long-term commitment to the multi-language SDK as a first-class part of Fluss.

If any of this is interesting to you, we welcome contributions, bug reports, and feedback.

And before you go 😊 don't forget to give some ❤️ via ⭐ on GitHub: Apache Fluss and Fluss Rust SDK

We are excited to announce the release of fluss-rust clients 0.1.0, the first official release of the Rust, Python, and C++ clients for Apache Fluss. This 0.1.0 release represents the culmination of 210+ commits from the community, delivering a feature-rich multi-language client from the ground up.

Under the hood, all three clients share a single Rust core that handles protocol negotiation, batching, retries, and Apache Arrow-based data exchange, with thin language-specific bindings on top. This was a deliberate community decision to deliver native performance and feature parity across every language from day one.

Highlights​

Support for all Fluss table types​

• Log Tables - Append-only streaming ingestion with subscription-based polling. Example uses include clickstreams, IoT sensor data, and audit logs. See log table examples (Python, C++).
• Primary Key Tables - Upsert, delete, and point lookups by key, with support for partial column updates. Example uses include product catalogs and real-time dashboards backed by data from multiple sources. See primary key table examples (Python, C++).
• Partitioned Tables - Both Log and Primary Key tables support partitioning, with partition-aware reads and writes. See the partitioned table examples (Python, C++).

All table types support idempotent writes and memory-bounded backpressure for reliable production use. The scanner includes parallel, prioritized remote segment fetching for efficient reads over tiered storage. The Python client additionally provides Polars and Pandas specific APIs for seamless integration with dataframe workflows.

Zero-copy Arrow integration​

Leverage your existing Apache Arrow ecosystem. Whether you use Polars, Pandas, DuckDB, DataFusion, or Arrow directly, the clients speak Arrow natively. Log records are represented as Arrow RecordBatches throughout the stack, so data arriving from Fluss can be handed directly to your tools without serialization or conversion.

A batch scanner mode makes this practical at scale: it returns complete RecordBatches to the caller, ready for immediate use. Python users can pass batches straight to Polars or Pandas, C++ applications can feed them into DuckDB, and Rust tools benefit from the same zero-copy path. The Java client does not offer this today because Flink and Spark convert data into their own internal row formats on ingestion. For the native ecosystem, skipping that conversion is where the performance wins are.

Direct reads from tiered storage​

Fluss tiers older log segments to remote storage. When you need to replay or backfill historical data, fluss-rust reads those segments directly from the object store rather than routing them back through the server. This keeps replay workloads off the serving path and avoids paying for data to round-trip through the cluster.

Supported storage backends (enabled via feature flags):

• storage-fs — Local filesystem (default)
• storage-s3 — Amazon S3
• storage-oss — Alibaba Object Storage Service

A priority-queue-based prefetching system with configurable concurrent downloads keeps sequential scans from stalling on object store round trips.

Other Features​

The release also includes SASL/PLAIN authentication across all three clients, a comprehensive admin API for database and table management, fire-and-forget write batching with configurable bucket assignment strategies, column projection, and more.

Getting Started​

• Rust: fluss-rs installation guide.
• Python: pyfluss installation guide.
• C++: fluss-cpp installation guide.

What's Next​

This is the first release of fluss-rs, pyfluss and fluss-cpp, and the community is actively working on expanding capabilities. Areas of future development include additional language bindings, additional storage backends, enhanced compression support, and expanded ecosystem integrations.

Thank You, Contributors​

This release would not have been possible without the efforts of our contributors. Thank you to everyone who submitted code, reported issues, reviewed pull requests, and helped shape the project:

AlexZhao, Andrea Bozzo, Anton Borisov, Arnav Panjla, Chase Naples, Evan, Jark Wu, Junbo Wang, Junfan Zhang, Kaiqi Dong, Karan Pradhan, Keith Lee, Kelvin Wu, Miao, Nikhil Negi, Pavlos-Petros Tournaris, Prajwal Banakar, SkylerLin, Yang Guo, Yuxia Luo, naivedogger, and tison.

Getting Involved​

The Apache Fluss community welcomes contributions!

• Client GitHub: https://github.com/apache/fluss-rust
• Client Documentation: https://clients.fluss.apache.org
• Website: https://fluss.apache.org/
• Mailing List: https://fluss.apache.org/community/welcome/#mailing-lists
• Slack: https://apache-fluss.slack.com

Apache Fluss (Incubating) started as streaming storage for real-time analytics, built to work closely with stream processors like Apache Flink. Its focus has always been on freshness, efficient analytical access, and continuous data, making fast-changing streams directly usable without forcing them through batch-oriented systems or log-only pipelines.

Over the last year, Fluss has expanded beyond this original framing. You’ll now see it described as streaming storage for real-time analytics and AI. This change reflects how data systems are being used today: more workloads depend on continuously updated data, low-latency access to evolving state, and the ability to reason over context as it changes.

In this context, “AI” does not mean training or serving models inside Fluss. It refers to the class of intelligent systems that rely on fresh features, evolving context, and real-time state to make decisions continuously. Whether those systems use traditional machine learning models, newer AI techniques, or a combination of both, they all depend on the same data foundations.

This shift explains the recent evolution of Apache Fluss. Investments in stateless compute, richer data types with zero-copy schema evolution, and vector support through Lance were driven by a single question:

What does a data foundation need to look like to support real-time intelligent systems reliably at scale?

The rest of this post answers that question. We’ll explain what AI means when viewed through the lens of Apache Fluss, and why a streaming-first foundation for features, context, and state is central to building the next generation of intelligent systems.

What We Mean by “Real-Time Intelligent Systems”​

In this post, when we talk about real-time intelligent systems, we are not referring to a specific class of models or a particular AI technique. We are describing a category of systems defined by how they behave over time and how they interact with data that is constantly changing.

At a minimum, these systems continuously ingest live data: events, updates, interactions, signals, from the world around them. The data never really “stops,” and the system is expected to keep up, processing information as it arrives rather than waiting for periodic batch windows.

They also maintain state that evolves over time. This state might represent user profiles, counters, risk scores, preferences, embeddings, or derived features. Crucially, this state is not static; it is updated incrementally as new data arrives and as past information becomes less relevant or is refined.

On top of that, real-time intelligent systems make decisions repeatedly, not once per day or once per job run. They score, rank, filter, recommend, or trigger actions continuously, and their outputs are expected to adapt as conditions change. The feedback loop between data, state, and decisions is tight and ongoing.

From the perspective of a streaming storage system, this framing matters. The intelligence may come from ML models, rules, LLMs, or agents, but the system itself is fundamentally about data plus state over time. Fresh data, historical context, and continuously updated state are the real inputs and enabling that combination efficiently is where the data foundation becomes critical.

Where Fluss Came From: Streaming Storage for Real-Time Analytics​

To understand why Apache Fluss is evolving in the direction it is today, it helps to look at the architecture patterns it was originally designed to address. Traditional streaming systems were excellent at moving events quickly, but far less effective at making those events easily queryable once they were in motion.

A common architecture emerged as a result: producers write to Kafka, stream processors like Flink consume and process the data, state lives inside the compute layer, and results are pushed out to multiple downstream systems. Analytical queries usually happen later, against an OLAP system or a lakehouse populated asynchronously.

Over time, this led to a familiar but complex stack. Kafka handled transport, Flink handled computation, an operational database stored the latest state, an OLAP engine supported analytics, and a lakehouse stored historical data. Each component solved a real problem, but the same data had to be copied, transformed, and re-ingested at every layer.

This duplication came at a cost. Pipelines became harder to reason about, data freshness varied across systems, and infrastructure costs grew as every layer maintained its own storage and indexing strategy. Querying what’s happening now often meant stitching together multiple systems that were never designed to work as a single whole.

Apache Fluss started as a response to this fragmentation. The core idea was to ingest streaming data once, store it durably, and make it directly queryable in near real time using analytical access patterns. Instead of treating streams as transient transport, Fluss treats them as a durable dataset, one that can serve both continuous processing and real-time analytics without constant data movement.

The Last Year: Three Big Investments That Changed What Fluss Can Be​

Apache Fluss is an open-source project, and its evolution over the last year reflects very deliberate architectural choices rather than a collection of disconnected features. The work we invested in during this period was guided by a single question:

What kind of data foundation is needed for systems that are continuous, stateful, and expected to evolve over time?

Those investments converged around three major areas. Each one addressed a recurring limitation we observed in real-world streaming architectures, and each one pushed Fluss beyond its original role as “just” streaming storage for real-time analytics. Together, they define the next stage of what Fluss can support.

The first investment focused on how compute and state interact. The second addressed how real systems evolve at the data model level. The third extended Fluss into the vector domain, which has become foundational for many intelligent workloads. While these areas may seem independent at first glance, they reinforce each other at the system level.

Importantly, these changes also influenced how Fluss fits into the broader lakehouse ecosystem. We strengthened its role as a durable, queryable layer that can interoperate with engines like Flink, Spark, DuckDB and soon more; and integrate cleanly with open table formats such as Apache Iceberg, Paimon, and Lance, rather than sitting outside the lakehouse as a special-purpose system.

What follows is a closer look at each of these investments, why they matter individually, and how they collectively move Fluss toward becoming a long-term foundation for real-time intelligent systems.

Stateless Compute (Zero-State Streaming)​

One of the clearest directions in modern streaming architectures is the separation of compute and state. Over the last year, we invested heavily in stateless stream processing, driven by the idea that compute should be lightweight and replaceable, while state should be durable and externalized.

In many traditional streaming systems, state lives inside the stream processor itself. This tightly couples long-lived state with the compute runtime, making recovery slow, scaling expensive, and operational complexity high. Restarting or resizing a job often means rebuilding large amounts of state before the system becomes useful again.

With stateless compute, stream processors focus purely on processing, while durable state is managed externally in Fluss. Compute becomes disposable, while state remains stable and queryable, independent of any single job or runtime.

This separation enables faster recovery, elastic scaling, and significantly simpler operations. It also leads to stronger RTO and RPO characteristics for stateful pipelines, because state is no longer trapped inside ephemeral compute containers.

For real-time intelligent systems, this matters deeply. These systems are continuous by nature and depend on ever-evolving state. Making compute stateless allows teams to evolve logic, scale workloads, and recover from failures without destabilizing the intelligence built on top of that state.

You can find out more here.

Complex Data Types and Zero-Copy Schema Evolution​

Real systems do not stand still, and neither do their data models. Over time, new fields are added, existing structures become more nested, and records that started simple accumulate richer context. If schema evolution is painful, innovation slows or technical debt piles up.

To address this, we invested in support for complex data types and zero-copy schema evolution. This allows Fluss to handle rich, nested structures without forcing users to flatten their data or redesign pipelines every time requirements change.

Zero-copy schema evolution means schemas can evolve without rewriting existing data or triggering large-scale migrations. A record can grow from a few scalar fields to include nested structures, contextual attributes, or even embeddings, while older data remains valid and accessible.

This capability is especially important in environments where Fluss acts as both a streaming store and part of a broader lakehouse architecture. By aligning with open formats and improving interoperability with systems like Apache Iceberg, Fluss can participate in analytical workflows without imposing rigid schema constraints.

For intelligent systems, this flexibility is essential. Features, profiles, and contextual records evolve rapidly, and the data foundation must keep up without breaking pipelines or forcing expensive reprocessing.

Note: Schema changes are also propagated downstream to open table formats like Apache Iceberg, Paimon, and Lance, ensuring that data remains accessible and usable across different systems and use cases.

Vectors and Lance​

Modern intelligent systems increasingly rely on vector embeddings to represent unstructured data such as text, images, audio, and video. These embeddings enable similarity-based queries that are central to use cases like semantic search, recommendation, and content discovery.

Traditionally, vectors are stored in specialized vector databases, separate from streaming systems and lakehouses. That separation introduces yet another silo, along with additional ingestion pipelines and consistency challenges.

Over the last year, we invested in bringing vector support closer to Fluss through integration with Lance. This allows vector data to live alongside structured and streaming data, rather than being isolated in a separate system.

By colocating vectors with the rest of the data foundation, Fluss can support hybrid workloads where structured attributes, streaming signals, and embeddings are all part of the same logical dataset. This also aligns with ongoing lakehouse work, making it easier to integrate vector-aware workloads with analytical engines and table formats like Iceberg.

For real-time intelligent systems, this reduces friction significantly. When vectors are no longer a separate silo, building end-to-end pipelines, from ingestion to feature generation to retrieval, becomes simpler, more consistent, and easier to operate at scale.

You can find a quickstart tutorial here.

The “Aha” Moment: Real-Time Feature Store and Zero Drift​

As these capabilities came together, we started noticing a recurring pattern: teams were increasingly asking for real-time feature store behavior, even when they weren’t explicitly building a “feature store.” The need kept emerging organically once streaming data, durable state, and low-latency access converged in a single system.

At its core, a feature store exists to compute and serve features, the structured inputs consumed by machine learning models. These features might represent recent activity, aggregated behavior, derived metrics, or continuously updated scores that capture how an entity changes over time.

The reason feature stores exist at all is a well-known failure mode in production ML systems. Training data is often computed offline using batch pipelines, while inference data is computed online using streaming or request-time logic. Over time, these two paths drift apart in logic, timing, or semantics.

This divergence is commonly referred to as training–serving skew. It’s subtle, hard to detect early, and is responsible for a large number of models underperforming in production despite looking correct during training and evaluation.

Once you view the problem through a data foundation lens, the implication becomes clear: eliminating drift is less about adding more tooling and more about ensuring that training and serving are built on the same underlying data, with the same semantics.

Why Fluss as a Feature Store​

Because Fluss stores streaming data durably, it can act as both a historical record and a source of continuously updated state, without splitting the data across separate systems.

From the same dataset, Fluss can support historical scans used for training, as well as latest-state access used for online inference. The data does not need to be duplicated, re-materialized, or reshaped into two different pipelines.

This unification is the critical shift. When training and serving operate on the same dataset, with the same schema and update semantics, drift becomes dramatically easier to avoid. The system enforces consistency by construction rather than by convention.

In practice, this means feature computation logic can be shared or reused, and the gap between offline and online feature views shrinks. The distinction still exists at the access level, but not at the data foundation level.

This is why feature-store-like use cases naturally emerge once you build a shared, durable streaming foundation. Fluss doesn’t start as a feature store, but it removes the architectural reasons that feature stores had to exist as separate systems in the first place.

From Feature Stores to Context Stores​

Traditional feature stores focus narrowly on model inputs, structured, derived attributes designed to be stable and predictable. That framing works well for many ML pipelines, but it only captures part of what real-time intelligent systems actually need.

In practice, these systems depend on context, not just features.

Context includes structured features, but also the latest entity state, recent event sequences, historical data for reconstruction, and increasingly, embeddings for semantic understanding.

A context store is therefore broader than a feature store. It provides multiple perspectives on the same underlying data, depending on whether the system is training a model, making a real-time decision, or debugging past behavior.

Apache Fluss fits this model naturally because it can expose multiple views over the same durable streaming dataset. The same data can serve as a feature table, a state store, an event log, or a historical archive, depending on how it is accessed.

This flexibility matters because real-time intelligent systems rarely draw a clean line between features and context. They blend both, and the data foundation needs to support that blend without forcing artificial separation.

Multi-Modal Data and Unified Access Patterns​

Modern intelligent systems work with more than just rows in a table. They combine structured records, semi-structured events, unstructured content, and vector representations derived from that content.

A single real-time decision might depend on a user’s latest profile, their recent actions, the text they just submitted, and embeddings representing both the query and previously consumed content. In many architectures, each of these elements lives in a different system.

Fluss aims to make this feel like a single, coherent foundation by supporting multiple access patterns over the same data. These patterns align with how intelligent systems actually consume information.

Row-oriented access retrieves the latest state for a specific key and is essential for real-time decisions. Column-oriented access scans attributes across many entities and underpins analytics and model training. Vector access enables semantic similarity search and retrieval.

By converging row, column, and vector access on a shared foundation, Apache Fluss reduces system sprawl and conceptual overhead. The intelligence still lives in the models and applications, but the data they depend on finally lives in one place.

Virtual Tables for Decision Tracking and Auditability​

One of the most underestimated requirements of real-time intelligent systems is auditability. When systems make decisions continuously and automatically, it’s no longer enough to know what happened; you need to understand why it happened and what the system knew at the time.

In practice, this means being able to answer questions such as what decision was made, what the system’s state looked like at that moment, which signals influenced the outcome, and how that state evolved leading up to the decision. These questions matter for debugging, trust, and increasingly for regulatory and compliance reasons.

This is where changelogs and virtual tables become foundational. Instead of treating state as something hidden inside a running system, they make state transitions explicit and durable. Every update becomes data that can be queried, replayed, and inspected.

Virtual tables built on changelogs allow systems to treat decisions and state evolution as first-class citizens. Rather than capturing only final outcomes, the system records how it arrived there, step by step, as the data changed.

For real-time intelligent systems, this shifts auditability from an afterthought to a built-in property of the architecture.

Changelogs in Action: Tracking AI Decisions Over Time​

Consider a real-time system that decides whether to approve or decline transactions. For each user, it maintains a continuously updated decision context containing fields such as risk score, recent activity velocity, total spend, and account state.

Applications compute this context and emit it as a changelog table, where each update is expressed as a semantic change rather than a full overwrite. Changelog events follow a simple model: inserts, updates expressed as retractions plus new values, and deletions.

For example, imagine the following changelog sequence for user_id = 123:

+I  user_id=123, risk_score=0.12, velocity_10m=1, total_spend_24h=35
-U  user_id=123, risk_score=0.12, velocity_10m=1, total_spend_24h=35
+U  user_id=123, risk_score=0.47, velocity_10m=5, total_spend_24h=120
-U  user_id=123, risk_score=0.47, velocity_10m=5, total_spend_24h=120
+U  user_id=123, risk_score=0.83, velocity_10m=9, total_spend_24h=240

Each update captures how the user’s risk profile evolves as new events arrive, rather than hiding those transitions behind a mutable row.

Reconstructing the Decision​

At a certain point, the system evaluates a transaction and produces a decision.

Because the changelog is stored durably, this decision is no longer opaque. You can reconstruct exactly what the system’s state was at decision time and trace how it reached that state through prior updates.

This makes it possible to answer concrete questions: which signals pushed the risk score higher, when thresholds were crossed, and whether the decision logic behaved as expected under changing conditions.

Importantly, this reconstruction does not rely on logs or ad-hoc instrumentation. It falls naturally out of the data model itself.

Changelogs as an Audit Trail​

Once changelogs are treated as immutable, append-only records, they become a powerful audit trail. Every change is captured, timestamped, and associated with its update semantics and lineage.

The same approach can be applied beyond the decision state. Teams can track changes to training data versions, model predictions, feature definitions, and user profile evolution using the same underlying mechanism.

Instead of asking “why did the system do this?” and hoping logs still exist, the answer lives in the data. What changed, when it changed, how it changed, and which job or model produced the update are all preserved.

For real-time intelligent systems operating at scale, this level of transparency is not just good engineering; it is quickly becoming a baseline requirement.

Where We’re Going Next​

Apache Fluss started as streaming storage for real-time analytics, with a narrow and intentional focus on making continuously changing data queryable as it arrived. That foundation shaped the system’s core assumptions around freshness, durability, and tight integration with stream processing engines.

Over the last year, we invested in capabilities that extend Fluss beyond analytics without abandoning that original focus. Stateless compute and externalized state changed how systems scale and recover. Support for complex data types and zero-copy schema evolution made it possible for data models to grow without constant rewrites. Vector support and Lance integration brought unstructured and semantic data into the same foundation.

At the same time, the direction of the broader industry has become increasingly clear. Intelligent systems depend on shared, real-time context that spans features, state, history, and embeddings. Feature-store-like capabilities are no longer a niche requirement—they are becoming a central building block for systems that operate continuously and adapt over time.

These threads converge on a direction we are committing to: Apache Fluss as a streaming-first foundation that supports real-time analytics, feature and context access, multi-modal data, decision tracking, and explainability. All of this is built on stateless, resilient streaming applications that can evolve without destabilizing the system.

That is what “AI” means in the context of Apache Fluss. Not a model platform, and not a framework, but the data and state foundation that real-time intelligent systems depend on to operate reliably, transparently, and at scale.

🌊 We are excited to announce the official release of Apache Fluss (Incubating) 0.9!

This release marks a major milestone for the project. Fluss 0.9 significantly expands Fluss’s capabilities as a streaming storage system for real-time analytics, AI, and state-heavy streaming workloads, with a strong focus on:

• Richer and more flexible data models
• Safe, zero-downtime schema evolution
• Storage-level optimizations (aggregations, CDC, formats)
• Stronger operational guarantees and scalability
• A more mature ecosystem and developer experience

Whether you’re building unified stream & lakehouse architectures, real-time analytics, feature/context stores, or long-running stateful pipelines, Fluss 0.9 introduces powerful new primitives that make these systems easier, safer, and more efficient to operate at scale.

TL;DR: What Fluss 0.9 Unlocks​

• Zero-copy schema evolution for evolving streaming jobs
• Storage-level aggregations that further enhances zero-state processing
• Change data feed for CDC, audit trails, point-in-time recovery, and ML reproducibility
• Safer snapshot-based reads with consumer-aware lifecycle management
• Operationally robust clusters with automatic rebalancing and safer maintenance workflows
• Apache Spark integration, enabling unified batch and streaming analytics on Fluss
• First-class Azure support, allowing Fluss to tier and operate seamlessly on Azure Blob Storage and ADLS Gen2

1. Richer Data Models & Schema Evolution​

Support for Complex Data Types​

Apache Fluss 0.9 strengthens and extends support for complex data types with a focus on deep nesting, safe schema evolution, and new ML-oriented use cases.

Supported types now include ARRAY, MAP, ROW, and deeply nested structures. For example, complex schemas such as:

ARRAY<MAP<STRING, ROW<values ARRAY<FLOAT>, ts TIMESTAMP_LTZ(3)>>>

are now fully supported in production. These nested structures are handled as schema-aware rows, not opaque payloads, ensuring data correctness and type safety.

Additionally, adding new columns does not affect existing jobs after clients upgrade to 0.9. Moreover, with support for the Lance format, Fluss can be used for the ingestion of multi-modal data and vector storage. Users can now store embeddings directly in tables using ARRAY<FLOAT> or ARRAY<DOUBLE>:

CREATE TABLE documents (
  doc_id BIGINT PRIMARY KEY,
  embedding ARRAY<FLOAT>
);

This enables new use cases where Fluss acts as the source of truth for vector embeddings, which can then be incrementally consumed by vector engines to maintain ANN indexes.

You can find the complete feature umbrella here.