Skip to content

fix: Restore a buildable, digest-pinned base image - #1

Open
johnxie wants to merge 1 commit into
masterfrom
fix/restore-buildable-digest-pinned-base
Open

fix: Restore a buildable, digest-pinned base image#1
johnxie wants to merge 1 commit into
masterfrom
fix/restore-buildable-digest-pinned-base

Conversation

@johnxie

@johnxie johnxie commented Aug 11, 2026

Copy link
Copy Markdown
Member

Summary

v1.24.12 (406449b) is broken and has never been able to build. This restores the digest-pinned base that v1.0.3 used and that taskade/taskcade runs in production today.

Found while triaging taskade/taskcade#27963, where Dependabot proposed moving taskcade from @v1.0.3 to @v1.24.12.

Two independent breaks, both reproduced with docker build

1. The base image does not exist

GET hub.docker.com/v2/repositories/bitnami/kubectl/tags/1.24.12
  -> 404 {"message":"httperror 404: tag '1.24.12' not found"}

GET hub.docker.com/v2/repositories/bitnamilegacy/kubectl/tags/1.24.12
  -> 200  sha256:3902f1334d98672cf06f8bcfa3cf728e919a6065252d1c83ad37e9646dea70ab

Bitnami retired their free Docker Hub catalog; the images moved to bitnamilegacy.

2. Resolving it to bitnamilegacy still does not build

#6 [2/5] RUN curl -L -o /usr/local/bin/aws-iam-authenticator ...
#6 0.240 /bin/bash: line 1: curl: command not found
#6 ERROR: process did not complete successfully: exit code: 127

The bitnami base ships no curl. So even the "obvious" fix of repointing at bitnamilegacy leaves the action unbuildable.

Why nobody noticed

Container actions build at job runtime, not at tag time. Nothing built v1.24.12 until it was proposed for use.

Blast radius had taskcade#27963 merged

taskade/taskcade/.github/workflows/production.yml uses this action twice:

 push to master / release tag
        |
        v
 Docker Build -> Docker Push -> Helm Deploy
        |
        v
 [Create AWS credentials secret]   uses: taskade/kubectl@v1.24.12
        X  docker build -> base image 404 / curl not found
        v
 STEP FAILS
        |
        v
 [Register Temporal Namespace]  -- never runs
        v
 RELEASE RED

Nothing in taskcade's PR CI exercises production.yml, so it would have merged green and detonated at the next release.

Verification

$ docker build --platform linux/amd64 -t tk-final:test .
=> writing image sha256:136b21d8f33527861aa49952a644a08966d9931df29a895821065500ff11d34d

$ docker run --rm --entrypoint sh tk-final:test -c 'kubectl version --client'
Client Version: v1.23.15 ... Platform:"linux/amd64"

$ docker run --rm --entrypoint sh tk-final:test -c 'aws-iam-authenticator version'
{"Version":"0.6.2","Commit":"d72e1b46444d0efcb995a28c3846223b39bc4964"}

$ docker run --rm --entrypoint sh tk-final:test -c 'echo dGVzdA== | base64 --decode'
test

entrypoint.sh needs sh, base64 and kubectl - all present. The resulting image digest is byte-identical to a build of the v1.0.3 Dockerfile, so this is a provable zero-regression restore.

Follow-up needed after merge

Re-tag v1.24.12 onto this commit, so anyone who already picked it up is repaired. I have not moved the tag in this PR because moving a published tag is an outward-facing change and should be a deliberate call.

Not done here (deliberately)

Upgrading kubectl past the 1.23.15 in this image is a real want - that was presumably the intent behind v1.24.12. Doing it properly needs:

  • a base that has curl, or a different way to fetch aws-iam-authenticator
  • a kubectl version chosen against the live EKS control plane (skew policy is +/-1 minor), not picked arbitrarily
  • aws-iam-authenticator 0.6.2 is from 2022 and probably wants a bump at the same time
  • ideally a CI job in this repo that runs docker build on every PR, which would have caught this at authoring time

Worth its own PR. This one just stops the bleeding.


Note

Low Risk
Build-only Dockerfile change that reverts to a previously verified image; no runtime logic changes beyond fixing a failed build.

Overview
Fixes an unbuildable Dockerfile by dropping bitnami/kubectl:1.24.12 and restoring the digest-pinned gcr.io/cloud-builders/kubectl image used in v1.0.3, so docker build succeeds again (including the RUN that installs aws-iam-authenticator via curl).

Adds inline documentation explaining why the Bitnami tag is unusable (404 on Docker Hub; legacy image lacks curl) and why the base must stay immutable for production workflows that run with kubeconfig and AWS credentials.

Reviewed by Cursor Bugbot for commit d75d309. Bugbot is set up for automated code reviews on this repo. Configure here.

v1.24.12 (406449b) swapped the base from a digest-pinned gcr.io image to
`bitnami/kubectl:1.24.12`. That tag is broken twice over, and both were
reproduced with `docker build` before this revert:

  1. The image does not exist. Bitnami retired their free Docker Hub catalog
     and moved the images to bitnamilegacy:
       bitnami/kubectl:1.24.12        -> 404 tag not found
       bitnamilegacy/kubectl:1.24.12  -> 200
  2. Resolving it to bitnamilegacy does not help. The build then fails at the
     aws-iam-authenticator RUN step with `curl: command not found` - the
     bitnami base ships no curl.

So v1.24.12 has never been able to build. Container actions build at job
runtime, which is why nothing surfaced it until Dependabot proposed the bump
to taskade/taskcade (taskade/taskcade#27963). That action runs on the
production deploy path, at "Create AWS credentials secret" and "Register
Temporal taskade Namespace", and nothing in PR CI exercises those workflows -
so the bump would have merged green and failed at the next release.

This restores the digest that v1.0.3 used and that taskade/taskcade runs in
production today. Verified locally: builds clean, kubectl v1.23.15,
aws-iam-authenticator 0.6.2, sh + base64 present for entrypoint.sh. The
resulting image digest is byte-identical to a build of the v1.0.3 Dockerfile.

Digest pinning is kept deliberately and commented, since a floating tag on
this action means a third party can change what runs with our kubeconfig and
AWS credentials.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant