Skip to content

[lake/iceberg] Add unit tests for FlussRecordAsIcebergRecord tiering wrapper#3574

Open
XuQianJin-Stars wants to merge 1 commit into
apache:mainfrom
XuQianJin-Stars:feat/iceberg-tiering-complex-types
Open

[lake/iceberg] Add unit tests for FlussRecordAsIcebergRecord tiering wrapper#3574
XuQianJin-Stars wants to merge 1 commit into
apache:mainfrom
XuQianJin-Stars:feat/iceberg-tiering-complex-types

Conversation

@XuQianJin-Stars

Copy link
Copy Markdown
Contributor

Introduce FlussRecordAsIcebergRecordTest that mirrors the coverage matrix of FlussRecordAsPaimonRowTest, focusing on behaviors specific to the tiering wrapper on top of FlussRowAsIcebergRecord:

  • All-types log-table row: dispatch of __bucket / __offset / __timestamp system columns via both positional get(pos) and name-based getField(name), alongside the existing 14 primitive/business columns delegated to the parent wrapper.
  • Primary-key table with all ChangeType values (INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE / APPEND_ONLY): asserts that offset / timestamp are refreshed on every setFlussRecord call and system columns remain consistent.
  • Field count mismatch: setFlussRecord throws IllegalStateException when Fluss row field count does not match struct.size - system.
  • Complex-type propagation via the parent converter: ARRAY<INT>, ARRAY<ARRAY<INT>>, ARRAY with nullable / empty / null variants, MAP<STRING,INT> populated / empty / null, and ROW<INT,STRING> including a null nested row.

This mirrors the guardrails present in fluss-lake-paimon's FlussRecordAsPaimonRowTest so log-table tiering into Iceberg has equivalent unit-test coverage.

Purpose

fluss-lake-iceberg's tiering wrapper FlussRecordAsIcebergRecord sits between the raw Fluss LogRecord stream and Iceberg's Record interface, injecting the three system columns (__bucket, __offset, __timestamp) on top of the field-by-field conversion done by FlussRowAsIcebergRecord. Prior to this PR, only the parent converter had unit-test coverage; the wrapper's system-column dispatch, field-count guardrail, per-record offset/timestamp refresh semantics and complex-type propagation were tested only indirectly through end-to-end IT cases. Meanwhile, the sibling fluss-lake-paimon module already carries a dedicated FlussRecordAsPaimonRowTest that exercises exactly this contract for its Paimon counterpart.

This PR closes the gap by adding an Iceberg equivalent test class that pins down the same contract, so future refactors of the Iceberg tiering path (adding new complex types, changing system-column ordering, tightening the mismatch check, etc.) will surface regressions at unit-test speed rather than IT-test speed.

Linked issue: close #xxx

Brief change log

Test-only change. No production code is touched.

  • Add fluss-lake/fluss-lake-iceberg/src/test/java/org/apache/fluss/lake/iceberg/tiering/FlussRecordAsIcebergRecordTest.java (+477 lines, 8 @Test methods):
    • testLogTableRecordAllTypes — all-types (14 primitive/business columns + 3 system columns) log-table row; asserts both positional get(pos) and name-based getField(name) for every column, including __bucket / __offset / __timestamp.
    • testPrimaryKeyTableRecordAllChangeTypes — iterates over INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE / APPEND_ONLY, calling setFlussRecord with distinct offsets/timestamps each time, and asserts the system columns are refreshed and business columns remain consistent per record.
    • testFieldCountMismatchThrows — feeds a row whose field count does not match struct.size - system and asserts IllegalStateException is thrown by setFlussRecord.
    • testArrayTypeWithIntElementsARRAY<INT> propagation through the parent converter.
    • testNestedArrayTypeARRAY<ARRAY<INT>> propagation.
    • testArrayWithNullableElementsAndEmptyAndNullArrayARRAY with nullable elements, empty array, and top-level null array.
    • testMapTypeWithStringToIntAndEmptyAndNullMapMAP<STRING,INT> populated / empty / null.
    • testNestedRowTypeROW<INT,STRING> including a top-level null nested row.

The 8 test methods together mirror the coverage matrix of fluss-lake-paimon's FlussRecordAsPaimonRowTest, adapted to the Iceberg Record / Types.NestedField API.

Tests

New tests are the deliverable of this PR. Verified locally:

  • ./mvnw -pl fluss-lake/fluss-lake-iceberg test -Dtest=FlussRecordAsIcebergRecordTest -Dspotless.check.skip — all 8 tests pass.
  • ./mvnw -pl fluss-lake/fluss-lake-iceberg test -Dspotless.check.skip — full module test suite still green (no interference with existing tests).
  • ./mvnw -pl fluss-lake/fluss-lake-iceberg checkstyle:check — 0 violations.

No production behavior change; existing tests are unchanged.

API and Format

None. This PR only adds tests under src/test/java. There are no changes to public Java API, table properties, wire format, or Iceberg on-disk format.

Documentation

None. Test-only change; no user-facing documentation impact.

…wrapper

Introduce FlussRecordAsIcebergRecordTest that mirrors the coverage matrix of FlussRecordAsPaimonRowTest, focusing on behaviors specific to the tiering wrapper on top of FlussRowAsIcebergRecord:

- All-types log-table row: dispatch of __bucket / __offset / __timestamp system columns via both positional get(pos) and name-based getField(name), alongside the existing 14 primitive/business columns delegated to the parent wrapper.

- Primary-key table with all ChangeType values (INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE / APPEND_ONLY): asserts that offset/timestamp are refreshed on every setFlussRecord call and system columns remain consistent.

- Field count mismatch: setFlussRecord throws IllegalStateException when fluss row field count does not match struct.size - system.

- Complex-type propagation via the parent converter: ARRAY<INT>, ARRAY<ARRAY<INT>>, ARRAY with nullable / empty / null variants, MAP<STRING,INT> populated / empty / null, and ROW<INT,STRING> including a null nested row.

This mirrors the guardrails present in fluss-lake-paimon's FlussRecordAsPaimonRowTest so log-table tiering into Iceberg has equivalent unit-test coverage.
@XuQianJin-Stars XuQianJin-Stars force-pushed the feat/iceberg-tiering-complex-types branch from 5c6d70f to 82e499c Compare July 4, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant