Skip to content

Recover native tunnel failures safely#620

Merged
kasnder merged 6 commits into
masterfrom
codex/native-failure-recovery
Jul 11, 2026
Merged

Recover native tunnel failures safely#620
kasnder merged 6 commits into
masterfrom
codex/native-failure-recovery

Conversation

@kasnder

@kasnder kasnder commented Jul 11, 2026

Copy link
Copy Markdown
Member

What changed

  • keep the user's protection intent enabled after transient native tunnel exits
  • retry native tunnel startup with a bounded 1–16 second exponential backoff
  • stop retrying after five attempts and tear down the service cleanly to prevent restart storms
  • reset retry capacity after a stable run or an explicit user restart
  • pass native errno values to the service and provide a clear file-descriptor exhaustion error
  • avoid duplicate exit reports when epoll setup cannot proceed

Why

A transient native tunnel failure previously disabled protection immediately and required user intervention. Repeated failures also lacked a bounded recovery policy. This change automatically repairs short-lived failures while retaining a finite budget for persistent faults.

User impact

Protection now recovers automatically from transient native tunnel failures. Persistent failures still surface an error and stop after a bounded number of attempts instead of creating a restart loop.

Validation

  • ./gradlew assembleGithubDebug testGithubDebugUnitTest -x wgbridgeBuild
  • native compilation completed for arm64-v8a, armeabi-v7a, x86, and x86_64
  • focused unit coverage for backoff, exhaustion, stable-window reset, explicit reset, clock reset, and file-descriptor exhaustion detection

The bridge rebuild task was excluded from the final post-rebase run because it stalled without output; it is unrelated to this change and the existing bridge artifact was used. The same branch passed a complete assembly and unit suite, including that task, before rebasing the single implementation commit onto the current default branch.

@kasnder kasnder left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Solid recovery design, but one release-only blocker — a stale ProGuard keep rule will strip the renamed JNI callback and crash the exact path this PR hardens.

Blocking

  • app/proguard-rules.pro:38 — the keep rule still lists void nativeExit(java.lang.String);, but this PR changes the method to void nativeExit(int, java.lang.String). release has minifyEnabled = true (app/build.gradle:186), so R8 runs; the new 2-arg method has no Java caller (it's invoked by name from JNI, invisible to R8), and line 24's -keepnames only prevents renaming of surviving members, not removal — the explicit -keep block at lines 37–46 is what actually retains these callbacks. Since line 38 no longer matches the new signature, R8 can strip/rename nativeExit(int,String). At runtime jniGetMethodID(env, cls, "nativeExit", "(ILjava/lang/String;)V") (netguard.c:538) then returns NULL and the following CallVoidMethod aborts — i.e. the native-failure path this PR adds crashes in release. The PR's validation only built the debug variant (no minify), so this wasn't exercised. Fix: change line 38 to void nativeExit(int, java.lang.String);.

Should fix

  • ServiceSinkhole.java nativeExit(int,String)showErrorNotification(displayReason) fires on every native exit, before the recovery decision. A transient failure that auto-recovers within ~1s still raises a hard error notification, and for non-FD reasons it surfaces the raw technical reason string to the user. This partly contradicts the "recovers transparently" goal; consider notifying only when the budget is exhausted (NO_RETRY), or using a softer transient message.

Nits

  • session.c (epoll_ctl tun setup) — the two earlier setup failures now goto cleanup, but the epoll_ctl tun failure only sets stopping = 1 and relies on fall-through into the while (!stopping) guard. Correct today, but asymmetric and fragile if code is later inserted before the loop; a matching goto cleanup would be consistent.
  • ServiceSinkhole.javatunnelThread was made volatile here, but the recovery runnable also reads vpn (written on the command looper) without volatile/synchronization. Minor visibility gap; consider making it consistent. (unverified impact)
  • NativeFailureRecoveryPolicy.onFailureinitialDelayMs << failures is safe for the shipped config (max shift 4) but would overflow / shift past 63 for a large maxRetries; guard or document, since the class is generic.

What's good

  • JNI migration is otherwise complete and consistent: all 8 report_exit call sites (session.c ×4, ip.c ×4) updated, plus the declaration (netguard.h) and definition (netguard.c:535); jniCheckException still guards the upcall so a Java-side failure can't crash the VpnService.
  • The added goto cleanup genuinely fixes a real duplicate-report bug: previously epoll_create failure fell through and re-reported via epoll_ctl on a negative fd.
  • Recovery is properly bounded and battery-safe — 1→16s backoff, 5 attempts, then a clean stop — with a synchronized policy, cancellation on start/stop/onDestroy, and re-validation inside the runnable, so no restart storm.

Automated review by Claude Opus 4.8 (agent-assisted).

kasnder and others added 4 commits July 11, 2026 13:06
The native-failure recovery path changed ServiceSinkhole.nativeExit()
from a 1-arg to a 2-arg (int, String) method, but the R8 keep rule
still referenced the old signature. Since minifyEnabled=true for
release and the method is only looked up by name/descriptor from JNI
(jniGetMethodID with "(ILjava/lang/String;)V"), R8 could strip or
rename the new method, making CallVoidMethod abort at runtime on
native failure. Updated the keep rule to match the real signature.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nativeExit() showed a hard error notification (raw technical reason for
non-FD failures) on every native tunnel exit, before deciding whether to
auto-recover. That contradicts the transparent-recovery goal: a transient
failure that is silently retried still alarmed the user with a scary,
technical notification.

Defer showErrorNotification() to the terminal give-up path (recovery
budget exhausted / NO_RETRY), where TC is actually going down and the
user should be told why. Transient failures now recover silently. Also
skip the notification entirely when TC is disabled or intentionally
paused, since a native exit is expected there. The reason string is now
computed only where it is used.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two follow-ups from review of the native tunnel recovery path:

- session.c: the "epoll add tun" setup failure only set stopping=1 and
  relied on fall-through into the `while (!stopping)` guard, while the two
  earlier epoll setup failures goto cleanup. Add a matching goto cleanup so
  the failure path is consistent and stays correct if code is later
  inserted before the loop.

- ServiceSinkhole: mark `vpn` volatile. The native recovery runnable runs
  on the main looper and reads `vpn` for its pre-check, but `vpn` is written
  on the command looper; without volatile that cross-thread read had no
  visibility guarantee (tunnelThread was already made volatile for the same
  reason).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NativeFailureRecoveryPolicy is a generic utility: with a large maxRetries
the shift amount grows, and `initialDelayMs << failures` could overflow a
signed long (or shift past 63 and wrap), yielding a negative delay that
would make postDelayed fire immediately — a restart storm. Harmless for
the shipped config (max shift 4), but guard it since the class is generic.

Clamp the shift to Long.MAX_VALUE when it would overflow. Add a test with
a generous config asserting every backoff stays positive and non-decreasing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kasnder

kasnder commented Jul 11, 2026

Copy link
Copy Markdown
Member Author

The implementation now looks sound and the earlier release blocker is fixed. One small user-facing correction before merge: msg_native_fd_exhausted promises an automatic restart if capacity remains, but nativeExit displays it only on the terminal NO_RETRY path, and nativeError displays it without initiating recovery. Please make the text truthful for both call sites (or split the messages). Otherwise I found no blocker. Merge after #621, rebase and resolve the ServiceSinkhole overlap, then rerun CI plus a release/minified build because this changes a JNI callback keep rule.

@kasnder

kasnder commented Jul 11, 2026

Copy link
Copy Markdown
Member Author

Implemented in d3195f0: split the shared FD-failure text into a terminal recovery-exhausted message and an immediate native-error message, so neither call site promises a restart it will not perform. Focused recovery-policy tests and a minified assembleGithubRelease pass locally. Non-base locales intentionally fall back to English until the normal translation workflow catches up.

@kasnder kasnder marked this pull request as ready for review July 11, 2026 21:41
@kasnder kasnder merged commit 6e5f372 into master Jul 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant