run_orchestrator: pre-load gpu_vendor from manifest to fix SLURM login node runs#152
Open
i-kosarev wants to merge 1 commit into
Conversation
…n node runs
On SLURM login nodes (no local GPU), _execute_local() calls
_init_runtime_context() which triggers GPU detection → RuntimeError
("Unable to determine gpu vendor") → exit 3 before skip_gpu_arch or
any model filtering can run.
The build phase already records the target GPU vendor in the manifest
context ("gpu_vendor": "AMD"). Pre-load it into additional_context
before _init_runtime_context() so init_gpu_context() sees "gpu_vendor"
in ctx and skips the local detection call. Also pre-populate GPU
docker_env_vars with placeholder values (-1 / empty) so the subsequent
per-field detection calls (MAD_SYSTEM_NGPUS, MAD_SYSTEM_GPU_ARCHITECTURE,
etc.) are also skipped; actual values are set correctly inside the
container where the GPU is actually present.
Fixes SLURM_LOGIN run phase for Docker-based models (e.g.
sglang_disagg_deepseek-r1) that are deployed to GPU compute nodes via
SLURM but whose run-phase madengine invocation happens on the login node.
Co-Authored-By: Claude <noreply@anthropic.com>
i-kosarev
requested review from
Cemberk,
Rohan138,
coketaste,
gargrahul and
leconcio
as code owners
July 8, 2026 16:30
There was a problem hiding this comment.
Pull request overview
This PR aims to prevent runtime GPU auto-detection from failing on SLURM login nodes (where no local GPU is present) by reusing GPU vendor information already recorded in the build manifest, allowing runs to proceed far enough to be dispatched to GPU compute nodes. It also reduces the risk of node-info commands hanging by bounding package-manager queries.
Changes:
- Pre-load
gpu_vendor(and related GPU docker env vars) from the build manifest intoadditional_contextbefore runtime context initialization. - Add a 10-second timeout to host package-manager queries in
_show_node_info().
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+532
to
+553
| manifest_context = manifest.get("context", {}) | ||
| if not self.additional_context: | ||
| self.additional_context = {} | ||
| if "gpu_vendor" in manifest_context and "gpu_vendor" not in self.additional_context: | ||
| vendor = manifest_context["gpu_vendor"] | ||
| self.additional_context["gpu_vendor"] = vendor | ||
| # Pre-populate GPU docker_env_vars so init_gpu_context() skips the per-field | ||
| # detection calls (get_system_ngpus etc.) that fail on GPU-less login nodes. | ||
| # Use -1 / empty placeholders; the actual values are set inside the container. | ||
| if "docker_env_vars" not in self.additional_context: | ||
| self.additional_context["docker_env_vars"] = {} | ||
| denv = self.additional_context["docker_env_vars"] | ||
| if "MAD_GPU_VENDOR" not in denv: | ||
| denv["MAD_GPU_VENDOR"] = vendor | ||
| if "MAD_SYSTEM_NGPUS" not in denv: | ||
| denv["MAD_SYSTEM_NGPUS"] = -1 | ||
| if "MAD_SYSTEM_GPU_ARCHITECTURE" not in denv: | ||
| denv["MAD_SYSTEM_GPU_ARCHITECTURE"] = manifest_context.get("gpu_arch", "") | ||
| if "MAD_SYSTEM_HIP_VERSION" not in denv: | ||
| denv["MAD_SYSTEM_HIP_VERSION"] = "" | ||
| if "MAD_SYSTEM_GPU_PRODUCT_NAME" not in denv: | ||
| denv["MAD_SYSTEM_GPU_PRODUCT_NAME"] = "" |
Comment on lines
795
to
+802
| if "HOST_UBUNTU" in host_os: | ||
| print(self.console.sh("apt show rocm-libs -a", canFail=True)) | ||
| print(self.console.sh("timeout 10 apt show rocm-libs -a", canFail=True)) | ||
| elif "HOST_CENTOS" in host_os: | ||
| print(self.console.sh("yum info rocm-libs", canFail=True)) | ||
| print(self.console.sh("timeout 10 yum info rocm-libs", canFail=True)) | ||
| elif "HOST_SLES" in host_os: | ||
| print(self.console.sh("zypper info rocm-libs", canFail=True)) | ||
| print(self.console.sh("timeout 10 zypper info rocm-libs", canFail=True)) | ||
| elif "HOST_AZURE" in host_os: | ||
| print(self.console.sh("tdnf info rocm-libs", canFail=True)) | ||
| print(self.console.sh("timeout 10 tdnf info rocm-libs", canFail=True)) |
Comment on lines
+527
to
+531
| # Pre-load gpu_vendor from manifest context into additional_context so that | ||
| # _init_runtime_context() can skip local GPU detection on nodes (e.g. SLURM | ||
| # login nodes) where the GPU is on remote compute nodes, not the local host. | ||
| # The build phase records the target GPU vendor; re-use it here instead of | ||
| # trying to detect a GPU that does not exist on the login node. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On SLURM login nodes (no local GPU),
_execute_local()calls_init_runtime_context()which triggers GPU detection →RuntimeError("Unable to determine gpu vendor")→ exit 3.This happens before
skip_gpu_archor any model filtering runs, causing CI failures for Docker-based models (e.g.sglang_disagg_deepseek-r1) that should be dispatched to GPU compute nodes via SLURM but whose run-phasemadengineinvocation happens on the login node.Fix
The build phase already records the target GPU vendor in the manifest context (
"gpu_vendor": "AMD"). Pre-load it intoadditional_contextbefore_init_runtime_context()soinit_gpu_context()sees"gpu_vendor"in ctx and skips the local detection call. Also pre-populate GPUdocker_env_varswith placeholder values (-1/ empty) so the subsequent per-field detection calls (MAD_SYSTEM_NGPUS, etc.) are also skipped.Actual GPU values are set correctly inside the container where the GPU is actually present.