Free cached memory inside the batch-size probe, not after estimation - #596
Open
avnoordenne wants to merge 1 commit into
Open
Free cached memory inside the batch-size probe, not after estimation#596avnoordenne wants to merge 1 commit into
avnoordenne wants to merge 1 commit into
Conversation
determine_max_batch_size deliberately grows batches until the GPU runs out of memory, but left PyTorch's caching allocator holding the device on the way out. TorchSim#589 added an empty_cache() for this, at the end of estimate_max_memory_scaler - after InFlightAutoBatcher._get_first_batch has already called _get_next_states(), which is where the starved allocation actually happens. Move the release into determine_max_batch_size itself, in a finally so it covers all three exits: normal completion, the early return on a caught OOM, and the re-raise on a non-OOM error. The OOM return is the path that matters in practice. Waiting until the except clause has released the exception means the traceback's frames - and the activations they pin - are already dropped when empty_cache() runs. With alchemiops installed the neighbor lists draw from Warp's separate cudaMallocAsync pool, so anything PyTorch keeps cached is unavailable to them. Probing a 16-atom molecule up to 1071 replicas left 0.03 GB free on a 24 GB card and the next neighbor-list call died with "Failed to allocate 72 bytes on device 'cuda:0'"; it now returns the same batch size with 21.24 GB free. The guard in estimate_max_memory_scaler is now redundant and is removed, so the invariant lives with the function that breaks it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
determine_max_batch_sizedeliberately grows batches until the GPU runs out of memory, but left PyTorch's caching allocator holding the device on the way out. #589 added anempty_cache()for this, at the end ofestimate_max_memory_scaler— butInFlightAutoBatcher._get_first_batchcalls_get_next_states()between the first probe andestimate_max_memory_scaler, so the starved allocation happens one step before the guard:determine_max_batch_size(first_state)— probe, leaves the card full_get_next_states()— neighbor-list calls ← fails hereestimate_max_memory_scaler()— two more probes, then fix(autobatching): recognise Warp OOM in memory estimation and free cache before the real run #589'sempty_cache()_get_next_states()again — protectedThis is only reachable with
memory_scales_with="n_edges"and alchemiops installed.n_atomsand the defaultn_atoms_x_densitycompute their scalers with pure tensor arithmetic and never build a neighbor list, so on the default path_get_next_states()makes no Warp allocations at all.Move the release into
determine_max_batch_sizeitself, in afinallycovering all three exits: normal completion, the early return on a caught OOM, and the re-raise on a non-OOM error. The OOM return is the path that matters in practice. Deferring to the outerfinallymatters: by then theexceptclause's implicitdel exchas dropped the traceback frames and the activations they pin, soempty_cache()can actually reclaim them.Measured with SevenNet
7net-0over 878 molecules on a 24 GB card,memory_scales_with="n_edges". The ladder is seeded from a 16-atom molecule and capped at the set's 22,260 atoms, so it tops out at 1071 replicas (17,136 atoms), which OOMs:RuntimeError: Failed to allocate 72 bytes on device 'cuda:0'The estimate is unchanged, and can't change:
measure_model_memory_forwardalready callsempty_cache()at the top of every rung, so each measurement already begins from a cleared cache, and the value it returns ismax_memory_allocated()— a high-water mark of allocated bytes, whichempty_cache()cannot move.The guard in
estimate_max_memory_scaleris now redundant and is removed, so the invariant lives with the function that breaks it.Summary
determine_max_batch_sizeviafinally, so every exit path hands the device back — fixesFailed to allocate N bytesfrom Warp-backed neighbor lists in_get_next_states()whenmemory_scales_with="n_edges"empty_cache()fromestimate_max_memory_scaler, whose probes now clean up after themselves