Skip to content

fix(runtime): exit 77 when collected data cannot be written for lack of disk - #1205

Open
gastonyelmini wants to merge 2 commits into
mainfrom
out-of-disk-exit
Open

fix(runtime): exit 77 when collected data cannot be written for lack of disk#1205
gastonyelmini wants to merge 2 commits into
mainfrom
out-of-disk-exit

Conversation

@gastonyelmini

Copy link
Copy Markdown
Contributor

Problem

An integration that fills its volume currently finishes with exit code 0.

Every graph object collected is written to disk before upload, so ENOSPC there is unrecoverable — but it propagates as an ordinary error. flushDataToDisk logs Failed to write chunk to disk and rethrows, the step executor marks that one step failed and moves on to the next, which fails the same way, and the job ends "with errors" having exited cleanly.

For managed integrations that clean exit is the whole problem. The ECS state machine only reaches HandleTaskFailure when the task itself fails, so a task that ran the disk dry is never retried on a larger volume and the disk-scaling path in jupiter-integration-service can never fire. It also publishes a partial graph, which looks to the customer like their data disappeared rather than like a failed run.

Real example: integration job fc815601-295c-42d8-acd8-302b49e267c5 hit ENOSPC, exited 0, and handle-task-failure never ran — no lambda logs exist for that task.

Change

writeFileToPath and symlink — the two calls flushDataToDisk makes, and therefore the funnel every graph object write passes through — detect ENOSPC and exit with code 77, which handleTaskFailure already reads as "retry this task with a bigger disk". Every other error keeps behaving exactly as before.

The diagnostic is written to fd 2 with writeSync because process.exit does not flush pending async writes, and that line is the only record of why the task died.

Notes for review

  • This is the first process.exit in the SDK's production code. The local j1-integration CLI shares this path, where the effect is that filling the disk stops the run instead of producing a confusing stack trace.
  • OUT_OF_DISK_EXIT_CODE is exported and covered by a test, because the value is a contract with a separately deployed service.

Testing

8 new tests in src/__tests__/fileSystem.test.ts covering the write, mkdir, symlink and JSON paths, the stderr diagnostic, an unwritable stderr, and the non-ENOSPC passthrough. Full integration-sdk-runtime suite green (301/301).

…of disk

Every graph object the integration collects is written to disk before it is
uploaded, so an ENOSPC there is unrecoverable. Today it propagates as an
ordinary error: the step executor marks that step failed and moves on to the
next one, which fails the same way, and the job finishes "with errors" having
exited 0.

For managed integrations that clean exit is the whole problem. The ECS state
machine only reaches HandleTaskFailure when the task itself fails, so a task
that ran the disk dry is never retried on a larger volume — the scaling path
added to jupiter-integration-service can never fire. It also publishes a
partial graph, which looks to the customer like their data disappeared rather
than like a failed run.

Fail the process instead, with the exit code HandleTaskFailure reads as
"retry with a bigger disk". The diagnostic goes to stderr synchronously
because process.exit does not flush pending async writes, and that line is the
only record of why the task died.

writeFileToPath and symlink are the two calls flushDataToDisk makes, so this
covers every graph object write without touching the call sites.
@gastonyelmini
gastonyelmini requested a review from a team as a code owner August 7, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants