Fix deadlock in test_machine_a_tron_rack_integration#3695
Open
kensimon wants to merge 1 commit into
Open
Conversation
A few PR's have had the same deadlock cause issues: Our integration tests run admin_force_delete_machine (just to get some extra coverage on it) after asserting they come up as Ready, and there's occasional issues where site explorer is concurrently hitting the same endpoints at the same time. If site explorer modifies endpoint A and then endpoint B in the same transaction, but admin_force_delete_machine deletes endpoint B and then endpoint A in *its* own transaction, we get a deadlock. Make sure that any transaction that's writing to multiple explored endpoints, does so in sorted order, so that they don't deadlock.
Contributor
Summary by CodeRabbit
WalkthroughThe change makes Site Explorer persistence and admin force-delete cleanup process explored endpoint records in deterministic BMC address order. A new SQLx integration test verifies concurrent exploration and force-delete behavior across a managed host and multiple DPUs. ChangesExplored endpoint lock ordering
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/api-core/src/tests/machine_admin_force_delete.rs`:
- Around line 427-439: Update the deletion assertions in the force-delete test
after the all_done check to pass each endpoint’s expected address from the
addresses collected during exploration into validate_machine_deletion instead of
None. Keep validating both managed DPU IDs and the host ID, and ensure every
explored endpoint is asserted deleted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 135305ab-d10a-43fa-9abb-9fe3e2174044
📒 Files selected for processing (3)
crates/api-core/src/handlers/machine.rscrates/api-core/src/tests/machine_admin_force_delete.rscrates/site-explorer/src/lib.rs
Comment on lines
+427
to
+439
| let response = force_delete_task | ||
| .await | ||
| .unwrap() | ||
| .expect("force delete completes once exploration commits") | ||
| .into_inner(); | ||
| assert!(response.all_done); | ||
| for machine_id in managed_host | ||
| .dpu_ids | ||
| .iter() | ||
| .chain(std::iter::once(&managed_host.id)) | ||
| { | ||
| validate_machine_deletion(&env, machine_id, None).await; | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert that every explored endpoint was deleted.
Passing None lets this regression test succeed even if cleanup leaves host or DPU endpoint rows behind. Use the addresses already collected by the test.
Proposed fix
- validate_machine_deletion(&env, machine_id, None).await;
+ validate_machine_deletion(&env, machine_id, Some(&endpoint_addresses)).await;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| let response = force_delete_task | |
| .await | |
| .unwrap() | |
| .expect("force delete completes once exploration commits") | |
| .into_inner(); | |
| assert!(response.all_done); | |
| for machine_id in managed_host | |
| .dpu_ids | |
| .iter() | |
| .chain(std::iter::once(&managed_host.id)) | |
| { | |
| validate_machine_deletion(&env, machine_id, None).await; | |
| } | |
| let response = force_delete_task | |
| .await | |
| .unwrap() | |
| .expect("force delete completes once exploration commits") | |
| .into_inner(); | |
| assert!(response.all_done); | |
| for machine_id in managed_host | |
| .dpu_ids | |
| .iter() | |
| .chain(std::iter::once(&managed_host.id)) | |
| { | |
| validate_machine_deletion(&env, machine_id, Some(&endpoint_addresses)).await; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/api-core/src/tests/machine_admin_force_delete.rs` around lines 427 -
439, Update the deletion assertions in the force-delete test after the all_done
check to pass each endpoint’s expected address from the addresses collected
during exploration into validate_machine_deletion instead of None. Keep
validating both managed DPU IDs and the host ID, and ensure every explored
endpoint is asserted deleted.
poroh
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A few PR's have had the same deadlock cause issues: Our integration tests run admin_force_delete_machine (just to get some extra coverage on it) after asserting they come up as Ready, and there's occasional issues where site explorer is concurrently hitting the same endpoints at the same time.
If site explorer modifies endpoint A and then endpoint B in the same transaction, but admin_force_delete_machine deletes endpoint B and then endpoint A in its own transaction, we get a deadlock.
Make sure that any transaction that's writing to multiple explored endpoints, does so in sorted order, so that they don't deadlock.
Related issues
Type of Change
Breaking Changes
Testing
Additional Notes
I was able to reproduce the deadlock (
cargo test --package carbide-api-integration-tests test_machine_a_tron_rack_integration -- --exact --nocapture) pretty reliably about 50% of the time locally, and after this fix I haven't been able to reproduce it once.