feat: add compact and postpone bucket writer utilities#114
Open
lucasfang wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces foundational utilities for compaction management and adds a postpone-bucket writer implementation for primary-key tables, including unit tests for the new behaviors.
Changes:
- Added a new
compact/utility layer (cancellation, future-based manager scaffolding, compact results/units/tasks, deletion-file handling + tests). - Implemented
PostponeBucketWriterto write postpone-bucket data files with special fields and commit increments (+ tests). - Added
PostponeBucketFileStoreWriteto wire postpone-bucket writing into the file-store write path.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/paimon/core/postpone/postpone_bucket_writer.h | Declares postpone-bucket batch writer and lifecycle/commit APIs. |
| src/paimon/core/postpone/postpone_bucket_writer.cpp | Implements postpone-bucket writing to rolling data files and commit draining. |
| src/paimon/core/postpone/postpone_bucket_writer_test.cpp | Covers basic, nested, multi-batch, multi-commit, empty, close, and IO-failure cases. |
| src/paimon/core/postpone/postpone_bucket_file_store_write.h | Integrates postpone-bucket writer into file-store write flow with unique prefixes. |
| src/paimon/core/compact/compact_manager.h | Defines compaction manager interface (trigger/result/cancel lifecycle). |
| src/paimon/core/compact/compact_future_manager.h | Provides shared future-based compaction manager behavior. |
| src/paimon/core/compact/compact_future_manager_test.cpp | Tests cancelled-status handling in future-based manager result retrieval. |
| src/paimon/core/compact/noop_compact_manager.h | Adds a no-op compaction manager implementation. |
| src/paimon/core/compact/noop_compact_manager_test.cpp | Tests the no-op manager’s behaviors and guardrails. |
| src/paimon/core/compact/compact_result.h | Introduces compaction result container and merge behavior. |
| src/paimon/core/compact/compact_task.h | Adds a base compaction task wrapper with metrics + logging. |
| src/paimon/core/compact/compact_unit.h | Defines a compact “unit” abstraction for compaction input selection. |
| src/paimon/core/compact/cancellation_controller.h | Adds a lightweight cancellation flag helper. |
| src/paimon/core/compact/compact_deletion_file.h | Implements generated vs. lazy deletion-file handling for compaction. |
| src/paimon/core/compact/compact_deletion_file_test.cpp | Tests deletion-file generation, merging constraints, and cleanup behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+21
to
+26
| #include <map> | ||
| #include <memory> | ||
| #include <random> | ||
| #include <string> | ||
| #include <utility> | ||
| #include <vector> |
Comment on lines
+98
to
+106
| Status DoClose() { | ||
| sequence_number_array_.reset(); | ||
| row_kind_array_.reset(); | ||
| if (writer_) { | ||
| writer_->Abort(); | ||
| writer_.reset(); | ||
| } | ||
| return Status::OK(); | ||
| } |
Comment on lines
+65
to
+67
| Status Compact(bool full_compaction) override { | ||
| return Status::NotImplemented("not implemented"); | ||
| } |
Comment on lines
+125
to
+128
| Result<CommitIncrement> PostponeBucketWriter::PrepareCommit(bool wait_compaction) { | ||
| PAIMON_RETURN_NOT_OK(Flush()); | ||
| return DrainIncrement(); | ||
| } |
Comment on lines
+59
to
+62
| Result<std::optional<std::shared_ptr<CompactResult>>> GetCompactionResult( | ||
| bool blocking) override { | ||
| return std::optional<std::shared_ptr<CompactResult>>(); | ||
| } |
Comment on lines
+245
to
+247
| ::ArrowSchema arrow_schema; | ||
| ScopeGuard guard([&arrow_schema]() { ArrowSchemaRelease(&arrow_schema); }); | ||
| PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportSchema(*write_schema_, &arrow_schema)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: No linked issue
This change adds compact management and postpone bucket writer support.
Included changes:
Compact module (
src/paimon/core/compact/):cancellation_controller.hfor task cancellation control.compact_deletion_file.hand test coverage for deletion file handling.compact_future_manager.hand test coverage for future-based compact orchestration.compact_manager.h,noop_compact_manager.hand test coverage for compact manager interface and no-op implementation.compact_result.h,compact_task.h,compact_unit.hfor compact result, task and unit abstractions.Postpone module (
src/paimon/core/postpone/):postpone_bucket_writer.h,postpone_bucket_writer.cppand test coverage for postponed bucket writing logic.postpone_bucket_file_store_write.hfor file store write abstraction in postpone flow.Tests
Test coverage included in this change:
CompactDeletionFileTestCompactFutureManagerTestNoopCompactManagerTestPostponeBucketWriterTestAPI and Format
No public API, storage format, or protocol changes.
Documentation
No documentation changes required.
Generative AI tooling
Migrate-by: Aone Copilot (Qwen3.7-Max)