Skip to content

ci: make a smoke failure diagnosable, and stop deploying from a red main - #40

Open
twilson63 wants to merge 1 commit into
mainfrom
ci-diagnostics-and-deploy-gate
Open

ci: make a smoke failure diagnosable, and stop deploying from a red main#40
twilson63 wants to merge 1 commit into
mainfrom
ci-diagnostics-and-deploy-gate

Conversation

@twilson63

Copy link
Copy Markdown
Collaborator

Follow-up to the smoke failure on 922717f. Two structural holes, neither of them the crash itself.

First: was it a real regression?

No. I ran the smoke locally against the exact merged tree three times — 37/37 each. CI died around check 20. And this job had never failed before: every prior CI failure in this repo was a different job (server, module resolution, 2026-07-28).

The proximate cause was a Chrome renderer crash (page.waitForTimeout: Page crashed) after every assertion had already passed. Why it crashed is still unproven — and that's the actual problem.

Hole A — the failure artifact explained nothing

The artifact from the failing run was 247 bytes: vite.log, and not one screenshot or the video — despite the job log printing each path as it wrote them.

actions/upload-artifact skips anything under a dot-directory unless include-hidden-files is set, and scripts/.smoke is exactly that. From the action's docs: "By default, hidden files are ignored by this action to avoid unintentionally uploading sensitive information" — hidden meaning any file beginning with . or inside a directory beginning with ..

The theory predicted the observation exactly: /tmp/vite.log isn't hidden and survived; the whole dot-directory didn't. So the one step that exists to diagnose a smoke failure was silently discarding 100% of its evidence.

Audited before enabling, per the action's own warning: the smokes inject only apiKey: 'sk-test' and the empty string, and CI runs VITE_MANAGED_AUTH=false, so there's no Clerk session and no real credential can render. The path stays scoped to scripts/.smoke/** rather than broadening to a bare **.

Hole B — Render deployed from a red main

The merge gate added in 3b3ecab guards the PR path. Nothing guarded the merge itself: PR #39 shipped managed auth to production while main's smoke job was still red. Auto-deploy defaults to every commit, so CI and the deploy were racing rather than ordered.

Both services now set autoDeployTrigger: checksPass — per Render's blueprint spec, "Trigger a deploy only if the linked branch's CI checks pass." It's a blueprint field, so no new secret or deploy hook.

The trade is real and documented in the file: a red or stuck main now blocks deploys, hotfixes included. The comment names the escape hatch (manual deploy from the dashboard) so nobody is stranded mid-incident.

Deliberately NOT fixed

The renderer crash. Tuning deviceScaleFactor or dropping the video recording would be guessing at an unproven cause, and could mask it rather than fix it. Fix the evidence pipeline first; let the next occurrence say what actually happened.

Verification

Rather than trust the docs, I proved Hole A empirically: a throwaway commit forced the smoke job to fail after the scripts completed, so the if: failure() upload fired with screenshots on disk. I downloaded the artifact, confirmed the contents, then dropped the commit by force-push — which is why this branch's CI history shows one deliberate red run.

Config-only diff: 2 files, +22 lines, nothing under src/ or server/. Suite unaffected (197 gleam + 304 vitest).

One honest limit: the Render change is verified as configuration, not as observed behaviour — it takes effect on blueprint sync, which can't be confirmed from this repo. Worth eyeballing the dashboard after merge that both services show deploy-on-CI-pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CwNJQ4sAbsG9ZgoioFaSLr

Two holes the 922717f smoke failure exposed. Neither is the crash itself —
that was a renderer death on the runner, not reproducible locally (37/37,
three runs, same tree).

The artifact meant to explain a smoke failure explains nothing.
upload-artifact skips anything under a dot-directory unless
include-hidden-files is set, and scripts/.smoke is exactly that. The
artifact from the failing run was 247 bytes: vite.log, and none of the
screenshots or video, despite the job log printing each path as it wrote
them. So the one step that exists to diagnose a smoke failure silently
discarded 100% of its evidence.

Safe to enable: the path stays scoped to scripts/.smoke, the smokes run
with VITE_MANAGED_AUTH=false, and the only credentials they inject are
apiKey: 'sk-test' and the empty string. Nothing real renders on screen.

Render auto-deployed main regardless of CI. The merge gate added in
3b3ecab guards the PR path; nothing guarded the merge itself, so PR #39
shipped managed auth to production while main's smoke job was still red —
CI and the deploy racing rather than ordered. Both services now use
autoDeployTrigger: checksPass, with the manual-deploy escape hatch noted
so a red main cannot strand a hotfix.

Deliberately not touched: the renderer crash. Its cause is unproven and
tuning deviceScaleFactor or dropping the video would be guessing at it —
and might mask it. Fix the evidence pipeline first; let the next
occurrence say what actually happened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CwNJQ4sAbsG9ZgoioFaSLr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant