ActiveJob-only k8s deployment, thin base image by carl-codeorg · Pull Request #73764 · code-dot-org/code-dot-org

carl-codeorg · 2026-07-09T21:46:24Z

Summary

This PR adds an ActiveJob-only Skaffold deployment path and restructures the k8s Docker image lineage around a slim official Ruby base image.

The image hierarchy now follows the Rails-style split:

base: runtime OS packages only, from ruby:3.2.11-slim-bookworm
build: compiler/toolchain layer used for bundle, uv, and frontend build work
runtime: non-root production image with built artifacts and no dev toolchain
dev: same base plus local development tools such as build-essential, Node, Chromium, zsh, and awscli

It also adds an activejob-only Skaffold profile that builds a smaller runtime image with the frontend build skipped and a narrower Bundler group set. This gives us a working path for ActiveJob experiments without carrying the full frontend asset build through each iteration. This PR doesn't get a cold Skaffold build to complete within the 20-minute goal, but I figured we should start committing some of these base changes first before continuing to iterate.

The changes to the base image were generated using Codex 5.5 following an OpenSpec design from Fable 5.

Details

Replaces the Ubuntu/rbenv Ruby core image with pinned ruby:3.2.11-slim-bookworm.
Removes Ruby compilation from cold Docker builds.
Adds jemalloc through libjemalloc2 and an arch-independent LD_PRELOAD path.
Uses Bundler deployment mode in runtime images while leaving the dev target mutable.
Moves bootsnap out of the development-only Gemfile group and precompiles bootsnap caches during image build.
Keeps the existing five-artifact Skaffold graph for cache behavior and context scoping.
Updates mimic fixtures and profile wiring so skaffold build -p mimic --cache-artifacts=false still validates Dockerfile changes.
Fixes runtime issues found during validation, including missing uv, missing JS runtime behavior for ActiveJob, and zsh assumptions in container commands.

Changes to the main Dashboard app

Most of this PR is scoped to the k8s directory, making it very safe in general. However, there are two changes outside of k8s:

Bootsnap moved out of dev dependencies in the Gemfile. If there is a reason we need to keep this as a dev dependency, let me know. Otherwise, it seems the modern Rails way is to use Bootsnap to speed up the boot process in production. The Rails-generated Dockerfiles do this. Summary of what we get from using bootsnap outside of dev:
- runtime image can install and use Bootsnap
- build stage can precompile Bootsnap caches
- development still gets Bootsnap
- BUNDLE_WITHOUT=development:test no longer removes a gem the image build/runtime wants
development.rb and the Prosopite initializer changed to check for Gem.loaded_specs.key?('prosopite'). This avoids situations where the development Rails config is loaded, but the prosopite gem is missing because of BUNDLE_WITHOUT=development:test being set during Docker build.

Validation

skaffold dev --trigger=manual -p activejob-only -p setup-db-minimal
skaffold build --cache-artifacts=false
skaffold build -p mimic --cache-artifacts=false
Runtime boot smoke against seeded local DB
Dev target bundle install with native-extension coverage and one dashboard spring testunit
dive size comparison recorded
openspec validate docker-rails-way-image --strict
git diff --check
./tools/hooks/pre-commit

Notes

The generic cold skaffold build --cache-artifacts=false completed, but build time is still not acceptable long-term. Measured wall clock was 8223.02s; most of the time was spent in Bundler, especially around devise_invitable. This PR accepts that as follow-up work because the new image structure is functional and compatible with the existing Skaffold workflow.

Ruby 3.2 is still past upstream EOL. This PR pins the slim image digest and makes a later Ruby 3.4 upgrade much smaller, but it does not perform that upgrade.

Image size improvements from switching to a ruby-slim base instead of Ubuntu:

Image	Size with ubuntu base	Size with ruby-slim base
full `code-dot-org`	8GB	6.6GB
`code-dot-org-core`	2.67GB	613MB
`active-job-only`	4.1GB	1.87GB

Links

Jira:

Testing story

Deployment notes

Privacy and security

Signed-off-by: Nick Lathe <nick.lathe@code.org>

Planning artifacts only, no implementation. Six OpenSpec changes covering the docker image rework (rails-way multi-stage targets, thin runtime composition), the k8s process model, secrets rotation, a devcontainer, and CI seeded-DB snapshots. Each change has proposal/design/specs/tasks and validates with `openspec validate`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

stephenliang · 2026-07-10T17:11:51Z

+
+if [ "$IMPORT_IMAGE_TO_K8S_NODE" = true ]; then
+  K8S_NODE_CONTAINER=${K8S_NODE_CONTAINER:-desktop-control-plane}
+  docker save "$IMAGE_REPO:$image_id" | docker exec -i "$K8S_NODE_CONTAINER" ctr -n k8s.io images import -


There is a bit of confusion with image ids. Here you have one format, but skaffold also produces its own on skafffold.yml L158.

stephenliang · 2026-07-10T17:12:29Z

+      value: k8s/helm/activejob-only.values.yaml
+    - op: replace
+      path: /deploy/helm/releases/0/setValueTemplates/image
+      value: "code-dot-org:{{.IMAGE_TAG_code_dot_org}}"


So this traces back to skaffold, but docker-buildx has its own tagging system. (See other comment in that file). Do we need two tags?

Actually I'm not sure about this - I'll look into it and see if this is also causing the need for the workaround to import images into k8s.

stephenliang · 2026-07-10T18:51:15Z

FYI - skaffold dev seems to fail because:

bundler: command not found: rerun Install missing gem executables with `bundle install`

I'm not getting this from a generic skaffold dev but will look into it. Did you just to skaffold dev with no options?

This is on the logs for cdo-dashboard.

stephenliang · 2026-07-10T19:02:33Z

+          --build-arg BUNDLE_JOBS=1 \
+          --build-arg BUNDLE_WITHOUT=development:test:production:adhoc:staging:levelbuilder:integration \
+          --build-arg SKIP_FRONTEND_BUILD=1
+    - op: add


This seems to produce a duplicate cdo-dashboard deployment on top of skaffold

stephenliang · 2026-07-10T19:10:40Z

+if [ "$PUSH_IMAGE" != true ] || [ "$IMPORT_IMAGE_TO_K8S_NODE" = true ]; then
+  image_id=$(docker image inspect --format '{{.Id}}' "$IMAGE" | sed 's/^sha256://')
+fi
+
+if [ "$PUSH_IMAGE" != true ]; then
+  docker tag "$IMAGE" "$IMAGE_REPO:$image_id"
+fi


I'm a little confused as to the purpose of these

The point of this was to make the image accessible to the local k8s cluster using Docker Desktop. Even with imagePullPolicy set to never it was never recognizing the locally-build image, so this code imports it into the k8s image store. I can take another stab at getting the k8s to be able to see the local Docker image store, which would allow us to get rid of this workaround.

stephenliang · 2026-07-10T19:12:07Z

If it helps scope, we can probably drop dev completely from skaffold and docker?

I don't think we want to drop it completely from both - but there is potentially a middle-ground where we use a simpler docker-compose workflow for dev. Local k8s, even the build-in Docker Desktop feature, adds a ton of overhead.

stephenliang · 2026-07-10T19:13:09Z


-# Copy in the rest of the source code
-COPY --chown=${UID} --link ./ ./
+# Temporary locals.yml used to bootstrap build-time optimizations.


Let's tag it with a followup to ensure we don't make it permanent.

I'll see if I can get an agent to fix this without the temp locals.yml.

stephenliang · 2026-07-10T19:15:08Z

What's the purpose of all these keep files?

These are for the "mimic" functionality that Seth built in for local k8s experimentation. The mimic cluster basically runs dummy images that don't have any of the real app content. You can run skaffold build -p mimic for a faster build compared to building the real Docker images. I also wondered about these keep files, and Codex says to keep them - so that the docker COPY steps continue working for directories that would be gone if there were no files for Git to store.

This is something we could look into removing if we get image sizes/build times down to the point where we can dev using the real images.

nicklathe and others added 13 commits June 29, 2026 10:52

Update main docker file to get build working

cbbda1b

Signed-off-by: Nick Lathe <nick.lathe@code.org>

wip backend only

6c41f8e

bundle args

9e8e286

WIP

92ded9a

wip working dashboard container

3aea791

cleanup

971abee

re-add pegasis static tables

21fcd35

add separate activejob worker deployment

9d20c52

working thin base image, activejob only deployment

188463b

working thin image

e0b0767

remove openspec markdown

b0ced97

Gemfile linting

c7f56a1

carl-codeorg marked this pull request as ready for review July 9, 2026 23:04

carl-codeorg requested a review from a team July 9, 2026 23:04

stephenliang reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

carl-codeorg commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Changes to the main Dashboard app

Validation

Notes

Links

Testing story

Deployment notes

Privacy and security

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

carl-codeorg commented Jul 9, 2026 •

edited

Loading