Skip to content

issue #7488 : Sorted Schema Merge transform#7489

Merged
hansva merged 1 commit into
apache:mainfrom
mattcasters:issue-7488
Jul 13, 2026
Merged

issue #7488 : Sorted Schema Merge transform#7489
hansva merged 1 commit into
apache:mainfrom
mattcasters:issue-7488

Conversation

@mattcasters

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add a Sorted Schema Merge transform (issue #7488), ported from the hop-data-vault effort.

It combines Sorted Merge and Stream Schema Merge:

  • Merges multiple pre-sorted input streams into one globally sorted output
  • Supports different schemas per input and produces a unified sparse row layout
  • Useful with Repeat Fields when combining multi-source / Data Vault satellite data into SCD2-style streams

Also includes:

  • Unit tests for schema union, sort-key resolution, and compare logic
  • User manual docs and transform icon
  • Integration test main-0106-sorted-schema-merge with a golden sparse-row data set

What was tested?

  • ./mvnw -pl plugins/transforms/sortedschemamerge spotless:apply test (5 unit tests)
  • Ran main-0106-sorted-schema-merge.hwf via hop-run against a local Hop client with the new plugin installed; unit test and golden data set validation passed

How can this be tested?

  1. Build and install the sortedschemamerge transform plugin
  2. Create a pipeline with 2+ pre-sorted streams of different layouts → Sorted Schema Merge (sort key) → Dummy
  3. Or run the transforms integration test:
    • main-0106-sorted-schema-merge.hwf / unit test 0106-sorted-schema-merge UNIT

Fixes: #7488

Port the Sorted Schema Merge transform from hop-data-vault into Hop as a
Flow transform that merges pre-sorted input streams with different schemas
into one globally sorted sparse output stream. Includes unit tests, docs,
and an integration test with a golden data set.
@hansva hansva merged commit 8b24a40 into apache:main Jul 13, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: A Sorted Schema Merge transform

2 participants