Fix crash in fault injection and flaky thread affinity test#312
Merged
Conversation
Alan-Jowett
approved these changes
Jul 6, 2026
Alan-Jowett
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem 1: Crash in fault injection due to unserialized dbghelp API calls
Fault injection tests crash with
FAIL_FAST_GUARD_ICALL_CHECK_FAILURE(0xc0000409) insidedbghelp!diaGetLineFromAddrwhen multiple threads concurrently trigger fault injection. The crash manifests as a use-after-free of internal DIA SDK COM objects (RAX =0xfeeefeeefeeefeee, the MSVC freed-heap fill pattern).The root cause is that
_cxplat_decode_symbolcallsSymFromAddrandSymGetLineFromAddr64without synchronization. These dbghelp APIs are explicitly documented as not thread-safe, and concurrent calls corrupt their internal state.Additionally,
IMAGEHLP_LINE64.SizeOfStructwas never initialized, violating the API contract forSymGetLineFromAddr64.Fix
static std::mutexinside_cxplat_decode_symbolto serialize all dbghelp API calls.IMAGEHLP_LINE64and setSizeOfStructas required by the API.Repro
The crash occurs in concurrent fault injection tests such as
_test_custom_maps_concurrent_insert_delete_and_querywhere multiple threads simultaneously allocate memory, triggering the fault injection stack-trace capture path.Call stack
Problem 2: Flaky
threadstest failureThe
threadstest intermittently fails with:This is a pre-existing flaky failure (also seen on unrelated PRs like #310) that depends on Catch2's random test ordering — it reproduces when
dpcsruns beforethreads.Root Cause
When
KeSetSystemAffinityThreadExis called at DISPATCH_LEVEL (e.g., viaKeRevertToUserAffinityThreadEx), it only updates_usersim_group_affinity_cachebut does not update_usersim_group_before_raise_irql. WhenKeLowerIrqlsubsequently restores thread affinity, it uses the stale_usersim_group_before_raise_irqlvalue (from the originalKfRaiseIrqlcall), overwriting the intended affinity change.In the
dpcstest:KeSetSystemAffinityThreadEx(1)— pins to CPU 0, cache becomes{Mask:1}KeRaiseIrqlToDpcLevel()— saves_usersim_group_before_raise_irql = {Mask:1}KeRevertToUserAffinityThreadEx(0)at DISPATCH — updates cache to{Mask:0}but_usersim_group_before_raise_irqlremains{Mask:1}KeLowerIrql()— restores from_usersim_group_before_raise_irql, setting cache back to{Mask:1}Fix
Update
_usersim_group_before_raise_irqlinKeSetSystemAffinityThreadExwhen called at DISPATCH_LEVEL, soKeLowerIrqlrestores the correct affinity.Validation
[ke]tests pass consistently with fault injection (CXPLAT_FAULT_INJECTION_SIMULATION=4)dpcs,threadsordering that previously failed now passes reliably