Skip to content

feat(installer): install extensions by default — REPL cross-index JOINs & MVs out of the box - #156

Merged
fupelaqu merged 6 commits into
mainfrom
feat/installer-extensions
Jul 28, 2026
Merged

feat(installer): install extensions by default — REPL cross-index JOINs & MVs out of the box#156
fupelaqu merged 6 commits into
mainfrom
feat/installer-extensions

Conversation

@fupelaqu

Copy link
Copy Markdown
Contributor

Problem

A fresh REPL install could not run cross-index JOINs at all:

  1. The softclient4es{N}-cli assembly contains no JOIN engine (zero DuckDB/Arrow classes — verified by jar inspection). JOINs live in softclient4es-arrow-extensions, discovered via the ExtensionSpi ServiceLoader.
  2. The launcher ran java -jar <assembly>, which ignores the classpath entirely — manually dropping jars into lib/ did nothing.
  3. The extensions are thin jars needing their full dependency closure (~256 jars — Apache Arrow, DuckDB native, Flight, …). "Just add a couple of jars" was never going to work.

Changes (install.sh + documentation/client/repl.md)

  • Classpath launcher: java -cp "<assembly>:lib/*" app.softnetwork.elastic.client.Cli, assembly first so it wins any conflict; Arrow --add-opens flags added automatically on Java 9+.
  • Extensions installed by default, resolved with their full dependency closure via a bootstrapped coursier launcher (single portable jar, runs on the already-required JVM):
    • community extensions (materialized views): always — any engine version (matching 0.1.x line selected for engines < 0.20), Java 8+;
    • arrow extensions (cross-index JOINs): engines ≥ 0.20, Java 11+ — a hard runtime constraint, not a policy choice: Apache Arrow 18.3.0 ships Java-11 bytecode (class-file major 55, verified on the published jars), so on Java 8 the JOIN engine cannot load. On Java 8 the installer says so and still installs MVs.
  • --no-extensions for a minimal install; VERSION file + summary report what was installed.
  • repl.md gains an Extensions section: default behaviour, manual cs fetch recipe for existing installs, the thin-jar warning, per-extension Java requirements.

Verified end-to-end (live ES 8.18.3, Docker)

Fresh install → CREATE TABLE ×2 → INSERT fixtures → SELECT … FROM jdbc_join_emp e JOIN jdbc_join_dept d ON e.dept_id = d.dept_idcorrect INNER JOIN rows (dept names populated, orphan dropped). 277 jars in lib/ (~488 MB).

Depends on / uncovered

  • SOFTNETWORK-APP/softclient4es-arrow#125 — E2E testing exposed an extension priority collision: EnforcedDqlExtension (priority 50, greedy canHandle) swallows JOIN queries before JoinExtension (also 50), silently returning wrong single-index rows. Fixed there; needs an arrow-extensions 0.2.1 release, which this installer picks up automatically (latest-version resolution). Until 0.2.1 is published, a default install still exhibits the collision.
  • Two more bugs to file separately: silent wrong JOIN results when no join extension is present (core hardening), and REPL JOIN … ORDER BY failing on qualifier loss.
  • install.ps1 (Windows) parity is a follow-up.

⚠️ Note before merge: this script is fetched from main at install time by end users — merging changes the public install behaviour immediately. Recommended order: merge arrow#125 → release arrow-extensions 0.2.1 → merge this.

🤖 Generated with Claude Code

…ss-index JOINs in the REPL)

The REPL assembly contains no JOIN engine, and the launcher's 'java -jar'
ignored lib/ entirely — so a fresh install could not run cross-index
JOINs at all, and no amount of jar-dropping could fix it.

install.sh now:
- launches via 'java -cp "<assembly>:lib/*" app.softnetwork.elastic.client.Cli'
  (assembly first so it wins any classpath conflict) with the Arrow
  --add-opens flags on Java 9+
- for SoftClient4ES >= 0.20 on Java 11+, bootstraps the coursier launcher
  and resolves softclient4es-arrow-extensions + community-extensions with
  their FULL dependency closure (~277 jars incl. Arrow/DuckDB) into lib/
- --no-extensions flag for a minimal install; VERSION and the summary
  report what was installed

documentation/client/repl.md (the README the installer ships) gains an
Extensions section: default behaviour, manual 'cs fetch' recipe for
existing installs, the thin-jar warning, and the Java 11+ constraint.

Verified end-to-end against live ES 8.18.3: fresh install, CREATE/INSERT
fixtures, SELECT…JOIN returns correct INNER JOIN rows through the REPL
(with softclient4es-arrow#125 applied; see that PR for the extension
priority collision this uncovered).

install.ps1 parity is a follow-up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fupelaqu and others added 5 commits July 27, 2026 19:23
Apache Arrow 18.x ships Java-11 bytecode (class-file major 55), so every
JOIN surface needs Java 11+ at runtime. Documented in repl.md
(prerequisites + extensions) and joins.md (requirement callout).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atrices

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ck 1.5.x (Java-11 bytecode)

Verified empirically on Zulu 8: 'java -jar <0.20.0 assembly> --help' crashes
with UnsupportedClassVersionError on LogbackServiceProvider before doing
anything — the 'Java 8+' claim was already false for the 0.20 CLI,
independent of extensions. The installer now requires Java 11 for ES 6-8
(17 for ES 9) and the docs state the real floor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…26-07-28)

jdbc/adbc driver jars and arrow-flight-sql refs move 0.2.0 -> 0.2.1;
softclient4es-community-extensions stays 0.2.0 (not re-released).
Installer usage examples showed '-v 0.2.0' where -v takes the CLI/engine
version — corrected to 0.20.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@fupelaqu
fupelaqu merged commit 552e3bf into main Jul 28, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant