Skip to content

boundary_flux 3D assembly gathers the full global boundary onto every rank (memory scaling) #408

Description

@lmoresi

Noted in the adversarial review of PR #404 (2026-07-24). The 3D _desmear path allgathers per-rank {coordinate-key: reaction} dicts and subdivided-triangle sets, so every rank materializes the entire global boundary as Python objects: measured ~472 B per subtriangle → ≈1.9 GB per rank for the triangle dicts alone on a 1M-node P2 surface (~2M subtriangles), plus reactions and transient pickled allgather payloads — and the full-traction path repeats the assembly and both allgathers once per vector component (3x).

Acceptable for coarse-mesh post-processing (the current use); a blocker for production 3D resolutions. Fix direction: keep the assembly distributed (owned-node partition of the key space, reduce-scatter instead of allgather), or assemble the lumped mass as a PETSc boundary vector the way the 2D line-mass path could also be expressed.

Related: #157 (stress projection memory) — same 'postprocessing must not hold the world on one rank' family.

Underworld development team with AI support from Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions