Skip to content

feat: add CUDA graph debug dumps#88

Open
popsiclexu wants to merge 1 commit into
MooreThreads:mainfrom
popsiclexu:xzx/graph-debug
Open

feat: add CUDA graph debug dumps#88
popsiclexu wants to merge 1 commit into
MooreThreads:mainfrom
popsiclexu:xzx/graph-debug

Conversation

@popsiclexu

@popsiclexu popsiclexu commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an optional CUDA graph debug dump path for MUSA graph capture:

  • Read TORCHADA_CUDA_GRAPH_DEBUG_DUMP_PATH once during CUDA graph patch setup and cache the dump directory.
  • Call graph.enable_debug_mode() before graph capture when the dump directory is configured.
  • Call graph.debug_dump(...) after a successful graph capture and write timestamped files named graph_<time_ns>.dot.
  • Document the environment variable in English and Chinese READMEs.
  • Add a real MUSA GEMM graph capture test that verifies a dot file is emitted and removes the dump file afterward.

Validation

TORCH_DEVICE_BACKEND_AUTOLOAD=0 PYTHONPATH=src python -m pytest tests/test_cuda_patching.py::TestCUDAGraph -q

Result:

12 passed, 2 warnings in 9.93s

The warnings are from torch_musa debug dumping and confirm the generated dot path:

/tmp/pytest-of-root/pytest-1/test_graph_context_manager_deb0/graph_dumps/graph_1783672385347873460.dot

@popsiclexu popsiclexu marked this pull request as ready for review July 10, 2026 08:51
@augmentcode

augmentcode Bot commented Jul 10, 2026

Copy link
Copy Markdown
🤖 Augment PR Summary

Summary: Adds optional MUSA/CUDA-graph debug “dot” dumping support to help diagnose captured graph structure.

Changes:

  • Introduces TORCHADA_CUDA_GRAPH_DEBUG_DUMP_PATH to enable graph debug dumps and caches the configured dump directory during patch setup.
  • Extends the torch.cuda.graph context-manager wrapper to call enable_debug_mode() before capture and debug_dump(...) after a successful capture.
  • Generates timestamped graph_<time_ns>.dot filenames to avoid overwriting across repeated captures.
  • Documents the new environment variable in both English and Chinese READMEs.
  • Adds a MUSA-only GEMM capture test that asserts a dot file is emitted and cleans it up afterward.
  • Includes minor test formatting cleanups (path join / imports) with no functional change.

Technical Notes: Debug dumping is opt-in via env var, errors are surfaced as warnings, and the dump path is prepared (created) up-front during patch initialization.

🤖 Was this summary useful? React with 👍 or 👎

@augmentcode augmentcode Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 2 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread src/torchada/_patch.py
_cuda_graph_debug_dump_dir = None
return None

dump_dir = os.path.expanduser(path_setting)

@augmentcode augmentcode Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

src/torchada/_patch.py:404: _cuda_graph_debug_dump_dir caches dump_dir exactly as returned by expanduser(), which can still be a relative path (e.g. ./graph_dumps); if the process later changes CWD, dumps may be written to an unexpected location.

Severity: medium

Fix This in Augment

🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.

Comment thread src/torchada/_patch.py
self._debug_enabled = False

# Create the original graph instance
self._wrapped = _original_graph_class(

@augmentcode augmentcode Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

src/torchada/_patch.py:506: GraphWrapper doesn’t retain the graph_obj passed into __init__ and instead re-discovers it via musa_graph/cuda_graph attributes on the wrapped context manager; if those internals differ across torch/torch_musa versions, debug enable/dump could silently no-op.

Severity: medium

Fix This in Augment

🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant