ci: make a smoke failure diagnosable, and stop deploying from a red main - #40
Open
twilson63 wants to merge 1 commit into
Open
ci: make a smoke failure diagnosable, and stop deploying from a red main#40twilson63 wants to merge 1 commit into
twilson63 wants to merge 1 commit into
Conversation
Two holes the 922717f smoke failure exposed. Neither is the crash itself — that was a renderer death on the runner, not reproducible locally (37/37, three runs, same tree). The artifact meant to explain a smoke failure explains nothing. upload-artifact skips anything under a dot-directory unless include-hidden-files is set, and scripts/.smoke is exactly that. The artifact from the failing run was 247 bytes: vite.log, and none of the screenshots or video, despite the job log printing each path as it wrote them. So the one step that exists to diagnose a smoke failure silently discarded 100% of its evidence. Safe to enable: the path stays scoped to scripts/.smoke, the smokes run with VITE_MANAGED_AUTH=false, and the only credentials they inject are apiKey: 'sk-test' and the empty string. Nothing real renders on screen. Render auto-deployed main regardless of CI. The merge gate added in 3b3ecab guards the PR path; nothing guarded the merge itself, so PR #39 shipped managed auth to production while main's smoke job was still red — CI and the deploy racing rather than ordered. Both services now use autoDeployTrigger: checksPass, with the manual-deploy escape hatch noted so a red main cannot strand a hotfix. Deliberately not touched: the renderer crash. Its cause is unproven and tuning deviceScaleFactor or dropping the video would be guessing at it — and might mask it. Fix the evidence pipeline first; let the next occurrence say what actually happened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CwNJQ4sAbsG9ZgoioFaSLr
twilson63
force-pushed
the
ci-diagnostics-and-deploy-gate
branch
from
July 31, 2026 00:13
eb478db to
f9ecb45
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the smoke failure on
922717f. Two structural holes, neither of them the crash itself.First: was it a real regression?
No. I ran the smoke locally against the exact merged tree three times — 37/37 each. CI died around check 20. And this job had never failed before: every prior CI failure in this repo was a different job (
server, module resolution, 2026-07-28).The proximate cause was a Chrome renderer crash (
page.waitForTimeout: Page crashed) after every assertion had already passed. Why it crashed is still unproven — and that's the actual problem.Hole A — the failure artifact explained nothing
The artifact from the failing run was 247 bytes:
vite.log, and not one screenshot or the video — despite the job log printing each path as it wrote them.actions/upload-artifactskips anything under a dot-directory unlessinclude-hidden-filesis set, andscripts/.smokeis exactly that. From the action's docs: "By default, hidden files are ignored by this action to avoid unintentionally uploading sensitive information" — hidden meaning any file beginning with.or inside a directory beginning with..The theory predicted the observation exactly:
/tmp/vite.logisn't hidden and survived; the whole dot-directory didn't. So the one step that exists to diagnose a smoke failure was silently discarding 100% of its evidence.Audited before enabling, per the action's own warning: the smokes inject only
apiKey: 'sk-test'and the empty string, and CI runsVITE_MANAGED_AUTH=false, so there's no Clerk session and no real credential can render. The path stays scoped toscripts/.smoke/**rather than broadening to a bare**.Hole B — Render deployed from a red
mainThe merge gate added in
3b3ecabguards the PR path. Nothing guarded the merge itself: PR #39 shipped managed auth to production whilemain's smoke job was still red. Auto-deploy defaults to every commit, so CI and the deploy were racing rather than ordered.Both services now set
autoDeployTrigger: checksPass— per Render's blueprint spec, "Trigger a deploy only if the linked branch's CI checks pass." It's a blueprint field, so no new secret or deploy hook.The trade is real and documented in the file: a red or stuck
mainnow blocks deploys, hotfixes included. The comment names the escape hatch (manual deploy from the dashboard) so nobody is stranded mid-incident.Deliberately NOT fixed
The renderer crash. Tuning
deviceScaleFactoror dropping the video recording would be guessing at an unproven cause, and could mask it rather than fix it. Fix the evidence pipeline first; let the next occurrence say what actually happened.Verification
Rather than trust the docs, I proved Hole A empirically: a throwaway commit forced the smoke job to fail after the scripts completed, so the
if: failure()upload fired with screenshots on disk. I downloaded the artifact, confirmed the contents, then dropped the commit by force-push — which is why this branch's CI history shows one deliberate red run.Config-only diff: 2 files, +22 lines, nothing under
src/orserver/. Suite unaffected (197 gleam + 304 vitest).One honest limit: the Render change is verified as configuration, not as observed behaviour — it takes effect on blueprint sync, which can't be confirmed from this repo. Worth eyeballing the dashboard after merge that both services show deploy-on-CI-pass.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CwNJQ4sAbsG9ZgoioFaSLr