feat: add CUDA graph debug dumps#88
Conversation
🤖 Augment PR SummarySummary: Adds optional MUSA/CUDA-graph debug “dot” dumping support to help diagnose captured graph structure. Changes:
Technical Notes: Debug dumping is opt-in via env var, errors are surfaced as warnings, and the dump path is prepared (created) up-front during patch initialization. 🤖 Was this summary useful? React with 👍 or 👎 |
| _cuda_graph_debug_dump_dir = None | ||
| return None | ||
|
|
||
| dump_dir = os.path.expanduser(path_setting) |
There was a problem hiding this comment.
src/torchada/_patch.py:404: _cuda_graph_debug_dump_dir caches dump_dir exactly as returned by expanduser(), which can still be a relative path (e.g. ./graph_dumps); if the process later changes CWD, dumps may be written to an unexpected location.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| self._debug_enabled = False | ||
|
|
||
| # Create the original graph instance | ||
| self._wrapped = _original_graph_class( |
There was a problem hiding this comment.
src/torchada/_patch.py:506: GraphWrapper doesn’t retain the graph_obj passed into __init__ and instead re-discovers it via musa_graph/cuda_graph attributes on the wrapped context manager; if those internals differ across torch/torch_musa versions, debug enable/dump could silently no-op.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
Summary
Adds an optional CUDA graph debug dump path for MUSA graph capture:
TORCHADA_CUDA_GRAPH_DEBUG_DUMP_PATHonce during CUDA graph patch setup and cache the dump directory.graph.enable_debug_mode()before graph capture when the dump directory is configured.graph.debug_dump(...)after a successful graph capture and write timestamped files namedgraph_<time_ns>.dot.Validation
Result:
The warnings are from
torch_musadebug dumping and confirm the generated dot path: