Skip to content

Fix crash in fault injection and flaky thread affinity test#312

Merged
saxena-anurag merged 4 commits into
mainfrom
user/anusa/fault_injection_fix
Jul 7, 2026
Merged

Fix crash in fault injection and flaky thread affinity test#312
saxena-anurag merged 4 commits into
mainfrom
user/anusa/fault_injection_fix

Conversation

@saxena-anurag

@saxena-anurag saxena-anurag commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Problem 1: Crash in fault injection due to unserialized dbghelp API calls

Fault injection tests crash with FAIL_FAST_GUARD_ICALL_CHECK_FAILURE (0xc0000409) inside dbghelp!diaGetLineFromAddr when multiple threads concurrently trigger fault injection. The crash manifests as a use-after-free of internal DIA SDK COM objects (RAX = 0xfeeefeeefeeefeee, the MSVC freed-heap fill pattern).

The root cause is that _cxplat_decode_symbol calls SymFromAddr and SymGetLineFromAddr64 without synchronization. These dbghelp APIs are explicitly documented as not thread-safe, and concurrent calls corrupt their internal state.

Additionally, IMAGEHLP_LINE64.SizeOfStruct was never initialized, violating the API contract for SymGetLineFromAddr64.

Fix

  • Add a static std::mutex inside _cxplat_decode_symbol to serialize all dbghelp API calls.
  • Zero-initialize IMAGEHLP_LINE64 and set SizeOfStruct as required by the API.

Repro

The crash occurs in concurrent fault injection tests such as _test_custom_maps_concurrent_insert_delete_and_query where multiple threads simultaneously allocate memory, triggering the fault injection stack-trace capture path.

Call stack

dbghelp!diaGetLineFromAddr+0x160
dbghelp!SymGetLineFromAddr64+0x69
unit_tests!_cxplat_decode_symbol+0x17f
unit_tests!_cxplat_fault_injection::log_stack_trace+0x281
unit_tests!_cxplat_fault_injection::is_new_stack+0x2c0
unit_tests!_cxplat_fault_injection::inject_fault+0x20
unit_tests!cxplat_fault_injection_inject_fault+0x3a
unit_tests!cxplat_allocate+0x5c

Problem 2: Flaky threads test failure

The threads test intermittently fails with:

REQUIRE( old_affinity == 0 )
with expansion:
  1 == 0

This is a pre-existing flaky failure (also seen on unrelated PRs like #310) that depends on Catch2's random test ordering — it reproduces when dpcs runs before threads.

Root Cause

When KeSetSystemAffinityThreadEx is called at DISPATCH_LEVEL (e.g., via KeRevertToUserAffinityThreadEx), it only updates _usersim_group_affinity_cache but does not update _usersim_group_before_raise_irql. When KeLowerIrql subsequently restores thread affinity, it uses the stale _usersim_group_before_raise_irql value (from the original KfRaiseIrql call), overwriting the intended affinity change.

In the dpcs test:

  1. KeSetSystemAffinityThreadEx(1) — pins to CPU 0, cache becomes {Mask:1}
  2. KeRaiseIrqlToDpcLevel() — saves _usersim_group_before_raise_irql = {Mask:1}
  3. KeRevertToUserAffinityThreadEx(0) at DISPATCH — updates cache to {Mask:0} but _usersim_group_before_raise_irql remains {Mask:1}
  4. KeLowerIrql() — restores from _usersim_group_before_raise_irql, setting cache back to {Mask:1}

Fix

Update _usersim_group_before_raise_irql in KeSetSystemAffinityThreadEx when called at DISPATCH_LEVEL, so KeLowerIrql restores the correct affinity.

Validation

  • All 75 usersim tests pass without fault injection
  • All [ke] tests pass consistently with fault injection (CXPLAT_FAULT_INJECTION_SIMULATION=4)
  • The specific dpcs,threads ordering that previously failed now passes reliably

@saxena-anurag saxena-anurag changed the title Fix crash in fault injection due to unserialized dbghelp API calls Fix crash in fault injection and flaky thread affinity test Jul 7, 2026
@saxena-anurag saxena-anurag merged commit 89cd6c7 into main Jul 7, 2026
12 checks passed
@saxena-anurag saxena-anurag deleted the user/anusa/fault_injection_fix branch July 7, 2026 17:52
@github-project-automation github-project-automation Bot moved this from Todo to Done in eBPF for Windows Triage Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants