fix(reliability): keep recovery retry time on one clock - #384
Open
seonghobae wants to merge 3 commits into
Open
Conversation
📝 WalkthroughWalkthrough작업 복구 시 재시도 예약에 복구 호출의 Changes복구 재시도 시각
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
seonghobae
marked this pull request as ready for review
August 11, 2026 11:18
seonghobae
enabled auto-merge (squash)
August 11, 2026 19:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Objective
Make stale-worker recovery use the same caller-supplied evaluation instant for both eligibility and retry scheduling.
recoverPendingJobs(now, processingLeaseTimeout)previously derived the stale boundary fromnowbut recorded stalePROCESSINGrecovery with a second wall-clockInstant.now(), so deterministic recovery/replay evidence could carry two different clock observations.Exact test-first evidence
Protected
mainremains exact55d7ae8647208e301f282350f076eeddaba61d11.17ed1f53194a464ae5e1eb24d79842b0675adbdbestablished that the stale-processing retry transition did not use the caller-supplied recovery instant.5930c53fc0ed8f40efa4b7d9909946924c6875careplaced only the second wall-clock observation with the validatednowargument. The focused recovery-clock regression passed, but full CI correctly exposed one pre-existing fixture whose syntheticrecoveryNow = Instant.now().plusSeconds(120)encoded the old two-clock behavior: the repaired worker scheduled that synthetic future retry instead of processing it immediately.90ed8990c24aa4a5ab761b954387001c32ff5124fixes the fixture rather than weakening production behavior. The stalePROCESSINGjob now derivesrecoveryNowfromstartedAt.plusNanos(1), which deterministically crosses the zero-duration stale boundary without manufacturing a future wall clock; the future-retry companion remains explicitly 30 seconds after that same recovery instant.Current exact-head acceptance
On unchanged head
90ed8990c24aa4a5ab761b954387001c32ff5124:31485575667: success; Maven test, synthetic merge compatibility, and buyer-readiness script jobs all succeeded with exact-revision verification;31485575642: success;31485575675: success;31485575605: success;55d7ae8647208e301f282350f076eeddaba61d11.The PR is Ready for ordinary independent review. Automated checks/comments/model output are evidence, not approval.
Implemented invariant
recoverPendingJobs(now, processingLeaseTimeout)now uses one explicit recovery clock for both stale eligibility and the retry transition it creates. Tests no longer depend on a hidden secondInstant.now()to turn a caller-supplied future evaluation instant into immediate work.Scope
One recovery timestamp boundary plus focused regressions only. This does not implement durable scheduling, transactional outbox, worker leases across replicas, backpressure, cancellation, OpenTelemetry export, or issue #312 completion. It does not modify stale descendant #268 or any current stacked PR.
Merge gate
Keep this exact head unchanged. Merge only when live repository protection still sees every required exact-head gate passing, zero valid unresolved findings, and the qualifying independent non-author approving review with write access required by repository policy.