feat(bare-metal): add conda-based bare-metal execution backend [WIP]#153
Draft
coketaste wants to merge 3 commits into
Draft
feat(bare-metal): add conda-based bare-metal execution backend [WIP]#153coketaste wants to merge 3 commits into
coketaste wants to merge 3 commits into
Conversation
Add a non-Docker execution path that runs models directly on the host inside a conda/mamba environment. Target is inferred from a "bare_metal" key in additional_context (highest precedence over slurm/k8s). - conda_env: CondaEnvManager for env lifecycle (create/update/reuse) and conda_run_prefix; resolve_conda_env_name with config/model/derived fallback - bare_metal_runner: BareMetalRunner mirrors ContainerRunner's manifest/run interface, wraps scripts in `conda run`, handles GPU visibility env, pre/post scripts, and perf.csv reporting - run_reporting: shared Docker-independent performance extraction, status determination, and perf.csv/perf_super writing - build_orchestrator: _execute_bare_metal_build creates conda envs and writes a synthetic manifest (no Docker build) - run_orchestrator: route "bare_metal" target to BareMetalRunner - discover_models: carry conda_env/environment_file/python_version/setup_script model fields with path normalization Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Add a dummy_bare_metal model to the dummy fixture plus a hermetic e2e test exercising the conda execution backend end-to-end. Since CI has no real conda, a checked-in fake_conda.sh shim execs the wrapped command directly; the test points bare_metal.conda_bin at its absolute path (required because the runner cd's into the model working dir before invoking conda). The test asserts the run routes to the bare-metal path (deployment_type= bare_metal, launcher=conda) and that performance is extracted from the log. Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
…hash reuse invalidation
Extend the conda-based bare-metal backend so a model's dependency setup can
pull ROCm userspace (and torch) pip wheels from TheRock's per-arch index,
install a plain requirements.txt, and invalidate env reuse when dependency
file content changes.
- conda_env.py: add resolve_rocm_index_url ("auto" -> gfx-arch index),
_install_rocm_wheels, requirements_file support, and content-hash stamp
reuse-invalidation. create_or_update pipeline is now create/reuse env ->
[rocm.enabled] rocm+torch -> [requirements_file] pip install -> setup_script,
each step optional and additive. ROCm wheels stay skip-on-reuse.
- CondaEnvManager gains a gpu_arch param (falls back to bare_metal.gpu_arch)
to resolve rocm.index_url=auto.
- build_orchestrator.py / bare_metal_runner.py: pass gpu_arch from
docker_env_vars[MAD_SYSTEM_GPU_ARCHITECTURE]; add rocm/requirements_file to
the manifest built_models entry.
- tests: cover index resolution, rocm wheel install (default/torch/custom),
fresh-install vs skip-on-reuse, requirements install + missing-file error,
and dep-hash reuse invalidation; isolate stamp writes to tmp.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
coketaste
requested review from
Cemberk,
Rohan138,
gargrahul and
leconcio
as code owners
July 9, 2026 14:18
coketaste
changed the title
coketaste
marked this pull request as draft
July 9, 2026 14:19
There was a problem hiding this comment.
Pull request overview
This PR introduces a new “bare-metal” execution backend that runs model scripts directly on the host inside a conda/mamba environment, as an alternative to the existing Docker/K8s/SLURM paths.
Changes:
- Adds a conda environment manager and a bare-metal runner, plus routing in build/run orchestrators.
- Factors out Docker-independent run reporting (performance extraction, status determination, perf.csv/perf_super writing).
- Adds a dummy bare-metal fixture and unit/e2e coverage (including a fake conda shim for CI).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_bare_metal.py | Unit tests for conda env lifecycle, vendor file resolution, ROCm wheel install, dep-hash reuse invalidation, and reporting helpers |
| tests/fixtures/dummy/scripts/dummy_bare_metal/run.sh | Dummy bare-metal model script emitting a performance line |
| tests/fixtures/dummy/scripts/dummy_bare_metal/fake_conda.sh | Fake conda shim used to exercise bare-metal routing in CI without real conda |
| tests/fixtures/dummy/models.json | Adds dummy_bare_metal model fixture with conda fields |
| tests/e2e/test_run_workflows.py | Adds an e2e test validating bare-metal routing and perf extraction |
| src/madengine/utils/discover_models.py | Adds bare-metal model fields and path normalization for new dependency/setup files |
| src/madengine/orchestration/run_orchestrator.py | Routes bare_metal deployments to the new BareMetalRunner path |
| src/madengine/orchestration/build_orchestrator.py | Adds bare-metal “build” phase that creates/updates conda envs and writes a synthetic manifest |
| src/madengine/execution/run_reporting.py | New shared reporting helpers for perf/status + perf.csv/perf_super writing |
| src/madengine/execution/conda_env.py | New conda/mamba/micromamba env manager with dependency hashing and ROCm wheel install support |
| src/madengine/execution/bare_metal_runner.py | New bare-metal runner that wraps script execution in conda run and emits perf records |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ) | ||
|
|
||
| try: | ||
| script_path, working_dir, interpreter = self._resolve_script(model_info) |
Comment on lines
+511
to
+516
| write_perf_records( | ||
| run_details, | ||
| model_info, | ||
| self.perf_csv_path, | ||
| run_results.get("status", "FAILURE"), | ||
| ) |
| "docker_image": "", | ||
| "dockerfile": "N/A (bare-metal mode)", | ||
| "build_status": "SUCCESS", | ||
| "build_time": 0, |
| cache_dir = os.path.join( | ||
| os.path.expanduser("~"), ".cache", "madengine", "envhash" | ||
| ) | ||
| return os.path.join(cache_dir, env_name) |
Comment on lines
+240
to
+242
| if os.path.isfile(configured) or shutil.which(configured): | ||
| self._conda_bin = configured | ||
| return self._conda_bin |
|
|
||
|
|
||
| def bootstrap_micromamba() -> str: | ||
| """Download a pinned micromamba static binary when no conda/mamba exists. |
Comment on lines
+1408
to
+1411
| from madengine.execution.conda_env import ( | ||
| CondaEnvManager, | ||
| resolve_conda_env_name, | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a non-Docker execution path that runs models directly on the host inside a conda/mamba environment, as an alternative to the existing Docker/K8s/SLURM targets.
"bare_metal"key inadditional_context(highest precedence overslurm/k8s).CondaEnvManager(newconda_env.py) handles env lifecycle: create/update/reuse,conda_run_prefix, and env name resolution (config/model/derived fallback).BareMetalRunner(newbare_metal_runner.py) mirrorsContainerRunner's manifest/run interface: wraps scripts inconda run, handles GPU visibility env vars, pre/post scripts, and perf.csv reporting.run_reporting.pyfactors out Docker-independent performance extraction, status determination, and perf.csv/perf_super writing (shared by both container and bare-metal paths).BuildOrchestratorgains_execute_bare_metal_build, which creates conda envs and writes a synthetic manifest (no Docker build involved).RunOrchestratorroutes thebare_metaltarget toBareMetalRunner.discover_models.pycarries new model fields:conda_env,environment_file,python_version,setup_script, with path normalization.rocm.index_url=autoresolves to the gfx-arch index via a newgpu_archparam onCondaEnvManager).requirements_file.dummy_bare_metalfixture model and a hermetic e2e test. Since CI has no real conda, a checked-infake_conda.shshim execs the wrapped command directly; the test asserts the run routes through the bare-metal path (deployment_type=bare_metal,launcher=conda) and extracts performance from the log.Test plan
tests/unit/test_bare_metal.py— conda env manager, ROCm index resolution/wheel install, requirements install, dep-hash reuse invalidationtests/e2e/test_run_workflows.py— bare-metal end-to-end run viadummy_bare_metalfixturetests/integration/test_console_integration.py,tests/unit/test_slurm_multi.py— related coveragepytestfull suite