Found while building the #379 synchronised_array_update rework (PR #383).
_vector_stats and _tensor_stats (discretisation_mesh_variables.py) create temporary mesh variables named _temp_mag_{id(self)} / _temp_frob_{id(self)}. id() is a rank-local memory address, so these variable names differ across ranks. Variable creation and deletion perform collective DM operations keyed by field name, and the variables are written inside uw.synchronised_array_update — any code path that uses the NAME as a cross-rank key breaks on them. This is also why the #379 collective flush uses creation-order registration ids rather than names.
Fix direction: derive the suffix from something rank-symmetric (a per-mesh monotonically increasing counter, or the parent variable's clean_name).
Underworld development team with AI support from Claude Code
Found while building the #379 synchronised_array_update rework (PR #383).
_vector_statsand_tensor_stats(discretisation_mesh_variables.py) create temporary mesh variables named_temp_mag_{id(self)}/_temp_frob_{id(self)}.id()is a rank-local memory address, so these variable names differ across ranks. Variable creation and deletion perform collective DM operations keyed by field name, and the variables are written insideuw.synchronised_array_update— any code path that uses the NAME as a cross-rank key breaks on them. This is also why the #379 collective flush uses creation-order registration ids rather than names.Fix direction: derive the suffix from something rank-symmetric (a per-mesh monotonically increasing counter, or the parent variable's clean_name).
Underworld development team with AI support from Claude Code