Report severed exec streams as execution_timeout, not success

[eapache-opslevel]

🤖 Authored by Claude (via Claude Code), at the direction of @eapache-opslevel.

Issues

Relates to https://gitlab.com/jklabsinc/OpsLevel/-/issues/14761.

Summary

While investigating #14761 we found repo-check jobs on large repos (e.g. a 25k-file repo, ~100 checks) that run for ~26 min, stop mid-loop around check 84/100, and are recorded as success with a missing result outcome — so the check silently produces no result and the job never retries.

Root cause on the runner side: the job is exec'd into a sleep <lifetime> pod, and the outcome is decided purely by whether StreamWithContext returns an error. When the pod is terminated (evicted / exceeds its lifetime) while the exec is streaming, the API server can close the stream without delivering a non-zero exit status, so StreamWithContext returns nil and the job is misreported as a success.

Change

• Emit a unique completion marker from the job shell's EXIT trap, and tee stdout through a small detector (markerWriter).
• If the exec returns a nil error but the marker was never seen, the shell never actually exited (stream severed) → report execution_timeout (retryable) instead of success.
• The EXIT trap fires on normal completion and explicit exit 0 (used by the RepoPropertyInferenceV1 template), but not on SIGKILL / torn-down streams — which is exactly the case we want to catch. On a set -e failure the trap also fires, but runErr is non-nil then, so we report Failed before checking the marker.
• The marker embeds the unique pod identifier so a job's own output can't spoof it.

This is fix (3) of three identified in #14761.

Changelog

• Report severed exec streams as execution_timeout instead of success
• Make a changie entry (Bugfix)

Tophatting

New unit tests cover the marker detector (single write, split-across-writes, absent, large-stream-then-marker) and the marker identity:

=== RUN   TestJobCompletionMarker_EmbedsIdentifier
--- PASS
=== RUN   TestMarkerWriter_DetectsMarkerInSingleWrite
--- PASS
=== RUN   TestMarkerWriter_DetectsMarkerSplitAcrossWrites
--- PASS
=== RUN   TestMarkerWriter_AbsentMarker
--- PASS
=== RUN   TestMarkerWriter_LargeStreamThenMarker
--- PASS
ok  	github.com/opslevel/opslevel-runner/pkg

Shell verification of the EXIT trap (marker on normal exit and explicit exit 0; failing command still exits non-zero so it's reported Failed):

case 1 (normal completion):      marker present
case 2 (explicit exit 0):        marker present
case 3 (set -e failure):         marker present, but exit_code=1 -> Failed path

🤖 Generated with Claude Code