Skip to content

Add varlock flatten command for monorepo Docker builds#930

Merged
theoephraim merged 5 commits into
mainfrom
flatten-command
Jul 24, 2026
Merged

Add varlock flatten command for monorepo Docker builds#930
theoephraim merged 5 commits into
mainfrom
flatten-command

Conversation

@theoephraim

@theoephraim theoephraim commented Jul 22, 2026

Copy link
Copy Markdown
Member

What

New varlock flatten command that collapses a package's @import graph into one self-contained directory (default .env-flat/), so the package can be deployed without the rest of the monorepo present, most commonly in the final stage of a Docker build.

How it works

  • Files imported from outside the package are mirrored under .env-imports/ by workspace-relative path, and @import paths are rewritten. Because both the package's own files and the imports preserve their directory structure, relative imports between copied files stay valid, and the output is position-independent.
  • Purely structural by default: values are never resolved and plugins are never executed, so it is safe to run in CI without secrets. Untouched files are copied byte-for-byte; only files with rewrites are reserialized (round-trip-faithful AST).
  • Conditionally-enabled imports (enabled=) are still copied with their conditions preserved, since production may enable imports the build machine does not.
  • .env.local / .env.*.local files are skipped by default (opt back in with --include-local) so machine-local secrets stay out of image layers.
  • @plugin() declarations in copied external files get pinned to the currently installed version, letting the standalone binary auto-install them where the sibling's node_modules is gone. Local-path plugins are vendored into the output. Package-internal plugin declarations are left untouched.
  • Imports pointing outside the workspace root warn and are left as-is.

Plugins in shell-less / distroless images

Two changes so pinned plugins actually work in minimal runtime images:

  • Native tarball extraction. Plugin auto-install previously shelled out to tar (exec("tar -xzf ...")), which spawns /bin/sh. Distroless images have neither a shell nor tar, so varlock load in the final stage failed with spawn /bin/sh ENOENT. Extraction is now native (zlib + a small tar reader that runs in both Node and the compiled Bun binary), so official @varlock/* plugins auto-install at runtime with no extra setup. Verified byte-for-byte against system tar, and end-to-end in the compiled binary with an empty PATH + fresh cache.
  • flatten --vendor-plugins. Copies each npm @plugin() package into the output (.env-plugins/), rewriting the declarations to local paths. It uses the copy already in your node_modules (no network), downloading only plugins that are not installed locally. The flattened directory then resolves with no runtime npm fetch, no shell, and no trust prompt. This is the only option that also covers third-party plugins and offline/air-gapped runtimes, which runtime auto-install cannot. Package-internal npm plugins are vendored too under this flag, since a distroless runtime has no node_modules to resolve from.

Why not resolve-and-bake

Resolving values at flatten time would bake secrets into the artifact. This keeps runtime resolution (varlock run in the container) working exactly as before; only the file layout changes. --vendor-plugins copies plugin code at build time but still never resolves values.

Docs: Docker guide rewritten around the new command (it was previously documented as "coming soon"), with a "Distroless and shell-less base images" section covering both fixes, plus a CLI reference entry and monorepos guide update.

Collapses the @import graph into a self-contained directory: external
imports are mirrored under .env-imports/ by workspace-relative path and
@import paths are rewritten. Local env files are skipped by default,
conditionally-enabled imports are still copied, npm plugins in copied
files get version-pinned, and local-path plugins are vendored.
@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

bumpy-frog

The changes in this PR will be included in the next version bump.

minor Minor releases

  • varlock 1.13.0 → 1.14.0

Bump files in this PR

Click here if you want to add another bump file to this PR


This comment is maintained by bumpy.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 22, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
varlock-website 7afe476 Commit Preview URL

Branch Preview URL
Jul 24 2026, 04:27 PM

@pkg-pr-new

pkg-pr-new Bot commented Jul 22, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/varlock@930
npm i https://pkg.pr.new/@varlock/1password-plugin@930

commit: a247689

@aFuzzyBear

Copy link
Copy Markdown

Hey Wizards, got round to getting this tested in my setup, and got some feedback...

Firstly I think this is a great clutch tool, off the bat its a winner 🔥

I tried flatten in an end-to-end container build against a real distroless production image (services/* tier, node distroless base, which has no shell, no package manager, given it matches most minimal-runtime container setups). Import resolution held up fine. Every var that failed did so for legitimate reasons (missing OIDC context, one undeclared schema ref unrelated to flatten), nothing flatten touched broke.

One real gap that was noticed was the pinned-plugin-version auto-install doesn't work in a shell-less image. varlock load in the distroless stage failed with: spawn /bin/sh ENOENT on both plugins (infisical, pass) . It reads like the auto-install path shells out unconditionally to install the pinned version, which distroless can't do. Worked around it for now by running varlock install-plugin in the builder stage (which has a shell) and copying whatever cache dir it populates across into the
production stage, but that's on us to maintain, not something flatten does for you.

which leads me to my next question: is pre-resolving/pre-installing plugins at flatten time on the roadmap, or is copying the plugin cache across stages meant to be the intended pattern for shell-less runtimes?
If it's the latter, worth a line in the Docker guide. The present guide has the auto-install pitch reads like it's meant to 'Just Work' in the final image, and this is the one place it didn't for us, given that its a simple addendum that can be added to it.

Perhaps under "Plugins in containers":

"varlock's auto-install needs a shell to spawn into in distroless, shell-less base images this will pose a significant hurdle. If you're on one of those, pre-install plugins in a stage that has a shell (varlock install-plugin <name>) and copy the resulting cache dir into the final stage yourself, same pattern as copying a dist/ build output across. flatten pins the version for you; it doesn't get you out of needing a shell somewhere in the chain to actually install it.

Otherwise I really do appreciate the effort and labour or love you lot do with this awesome project.

💚

@aFuzzyBear

Copy link
Copy Markdown

seeing the past two commits, you guys are awesome💚🔥

@philmillman

Copy link
Copy Markdown
Member

couldn't do it without you @aFuzzyBear !!

Copies each npm @plugin() package into the output (.env-plugins/) and rewrites
declarations to local paths, so the flattened dir resolves with no runtime install.
Prefers the copy already in node_modules; downloads only plugins not installed locally.
…ed rule

flatten now warns when a copied local package plugin declares runtime dependencies
(they won't travel with the copy), mirroring the npm-vendor check. Documents that
local plugins must be self-contained in the plugins guide.
@theoephraim
theoephraim merged commit bbab702 into main Jul 24, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants