feat(eval): LocalRuntime serves fresh in-process envs; placement resolves from what is known#484
Conversation
…me serves sources Live Environment objects become a placement, not data: the new LocalRuntime accepts an env or a mapping of env name -> env/factory and serves it in this process over the same control channel as every placement. Rows join by env name; an instance is a shared substrate (daemons refcounted across acquisitions, one bound channel per rollout so concurrent task lifecycles never collide), a factory is fresh per acquisition. The child-process provider keeps its exact semantics under its honest name, SubprocessRuntime. Task rows are now pure data all the way: the _source private attr is gone, template factories no longer capture their defining file, and Task/Taskset run defaults collapse to runtime= else the HUD tunnel by env name. _local splits into _bind_channel (channel over a started env) + the single-use start/serve/stop form.
Mapping values are live Environments only — shared is the single semantic of the data form, no type-dispatch on values. The fresh case moves to an explicit build= callable that receives the placed row, which zero-arg factories could not express (per-row construction like one env per task dir).
…olves from what is known LocalRuntime(source, env=, ready_timeout=) serves a fresh env per rollout in this process, from any pointer to it: a .py path (throwaway module import per acquisition), a live module-level Environment (its declaring file, located by identity in sys.modules, is the recipe - the instance itself is never served), or a (task) -> Environment constructor. ready_timeout bounds env.start(). runtime= stays Runtime-family typed; there is no shared local form - shared substrates are Runtime(url). SubprocessRuntime keeps the child-process form for isolation. Task rows stay pure data: _source is gone and rows never carry placement. With no runtime, what is already known decides: a module-loaded taskset serves its source directory; a platform taskset runs on the platform; rows naming envs declared in imported modules serve each fresh from its file (ambiguity across live envs is a loud error, resolution is logged); anything else raises, naming the forms to pass - there is no silent fallback to the platform.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4006aabb3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from hud.environment import load_environment | ||
|
|
||
| def _load(task: Task) -> _Environment: | ||
| loaded = load_environment(path, name=pinned or task.env) |
There was a problem hiding this comment.
When LocalRuntime("env.py") is used with an env that imports local helper modules holding mutable state, this acquisition only re-imports the target file via load_environment; helper modules imported under their normal names remain cached in sys.modules and are reused by later rollouts in the same process. That breaks the fresh-per-rollout contract and can contaminate grouped/concurrent evaluations with counters, caches, RNG state, or singleton resources that used to be isolated by the subprocess runtime.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Accurate observation, addressed as a documented contract rather than a code change (c4f8465): the freshness boundary is the env's own source — re-imported per rollout — while modules it imports follow normal Python import caching and are shared process-wide; SubprocessRuntime remains the whole-process isolation option. Purging sys.modules per rollout can't distinguish local helpers from site-packages and breaks isinstance/identity across reloads, so it would trade a documented boundary for reload hazards. Docstring and runtime.mdx now state the boundary explicitly.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
…ntime local docs _local's docstring claimed AgentTool sub-rollouts enter it; nothing in hud/agents does — its consumers are LocalRuntime (per acquisition, with the fresh env it built) and test harnesses. Chat's missing-runtime error now names the same placement forms as the taskset resolution error. The CLI --runtime local row says what the mechanism is: a child process per rollout (SubprocessRuntime).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e46233bcb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def _resolve_placement(self) -> Provider | HUDRuntime: | ||
| if self.origin and self.origin.startswith("module:"): | ||
| source = Path(self.origin[len("module:") :]) | ||
| return LocalRuntime(source if source.is_dir() else source.parent) |
There was a problem hiding this comment.
When a taskset is loaded from a specific module such as Taskset.from_module("env_a.py"), replacing that file with its parent directory makes LocalRuntime call load_environment over every *.py in the folder. If a sibling file declares another Environment with the same env.name (or matching attribute), the default run now fails with “multiple Environments” even though the selected taskset came only from env_a.py; this regresses single-file authoring layouts that keep variants side by side. Use the actual module file when it can define the placed env, or defer to the declared-env resolution instead of broadening unconditionally.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Real — fixed in c4f8465 together with the tasks-only-module finding (same root cause): the origin rung serves the exact origin path, never its parent directory, so a same-named sibling variant is never dragged in. Regression test: test_single_file_taskset_never_drags_in_a_same_named_sibling.
…runs on failed startup Addresses the three real bot findings on #484: - The module: origin rung probed nothing and broadened a file to its parent directory. Both halves bit: a tasks-only module importing its envs from elsewhere failed instead of falling through to the live-env resolution, and a single-file taskset could hit 'multiple Environments' from a same-named sibling variant. The rung now claims the rows only when the origin source actually declares their env names, and serves the exact origin path. - _local awaited env.start() outside its try/finally, so a failed or timed-out initialize hook leaked already-started daemons; stop() now runs on startup failure (shutdown hooks are best-effort per hook). - The in-process freshness boundary is documented where it lives: the env's own source is re-imported per rollout, while modules it imports follow normal import caching and are shared process-wide - SubprocessRuntime is the whole-process isolation option. (Purging sys.modules per rollout would trade a documented boundary for reload hazards.)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c4f8465429
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if any(value is env for value in list(module_vars.values())): | ||
| return Path(module_file) |
There was a problem hiding this comment.
When an env is re-exported by a package, e.g. pkg/__init__.py does from .env import env and the caller passes LocalRuntime(pkg.env), this identity scan visits the package module before pkg.env in sys.modules and returns __init__.py. The fresh import then executes __init__.py as a standalone file, so relative imports fail (or an absolute re-export can reuse the cached env) instead of rebuilding the actual declaring env module; package-authored envs therefore cannot use the new live-env pointer reliably.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Real — fixed in 17f77c9: the live-env pointer now validates candidate files instead of taking the first identity match. Each candidate is loaded fresh — a declarer yields a new instance, a re-exporter yields the same live one or fails to import standalone — with init.py holders tried last. Package re-exports (pkg/init.py doing from .env import env) resolve to the declaring submodule; envs that no file can rebuild (e.g. declared via relative imports that don't survive standalone import) fail loudly at construction pointing at the constructor form. Regression test: test_package_reexported_env_pointer_uses_the_declaring_submodule.
| value.name | ||
| for module in iter_modules(source) | ||
| for value in vars(module).values() | ||
| if isinstance(value, Environment) |
There was a problem hiding this comment.
For a task-only file that imports or re-exports the env object, such as from sums_envmod import env, add, this treats the imported env in tasks.py globals as if tasks.py declared that environment. _resolve_placement then chooses LocalRuntime(tasks.py) instead of falling through to the actual env module, so the throwaway import of tasks.py reuses the already-cached imported env module and leaks module/env state across rollouts (or fails if that external env import is not available from the task file's directory).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Real — fixed in 17f77c9: the origin probe now counts only envs the source itself declares, using the freshness discriminator (a fresh execution yields new instances for declared envs and the already-live instance for imported ones). A tasks file re-exporting its env falls through to the env's real module, so every rollout rebuilds it fresh. Regression test: test_reexporting_tasks_module_does_not_claim_the_env (asserts per-rollout freshness via module state, group=2).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1323cdc79
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return { | ||
| value.name | ||
| for module in iter_modules(source) | ||
| for value in vars(module).values() | ||
| if isinstance(value, Environment) |
There was a problem hiding this comment.
When a task-only module imports the env object as well as templates (from sums_envmod import env, add), this scan still treats that imported Environment as declared by tasks.py, so _resolve_placement() stops at the origin and uses LocalRuntime(tasks.py). Each acquisition then reimports only tasks.py while sums_envmod stays cached from Taskset.from_module, so env/module state leaks across group/concurrent rollouts instead of being rebuilt from the real env file. Fresh evidence: the current regression imports only add, while line 83 still accepts any imported Environment in module globals.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same root cause as the earlier re-export finding — fixed together in 17f77c9 via the freshness discriminator in _declared_names; see test_reexporting_tasks_module_does_not_claim_the_env.
| from hud.environment import load_environment | ||
|
|
||
| def _load(task: Task) -> _Environment: | ||
| loaded = load_environment(path, name=pinned or task.env) |
There was a problem hiding this comment.
For LocalRuntime('subdir/env.py'), load_environment() only has subdir on sys.path during the initial import; after the fresh env is returned, hooks and templates run under _local() with that path removed. Env code that performs a sibling import at execution time (for example import helper inside a template) now fails with ModuleNotFoundError even though the old child-process runtime and the new SubprocessRuntime run with the source directory as the working/importable directory.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Real — fixed in 17f77c9: the source directory now stays on sys.path for the whole acquisition (insert-and-remove one entry per acquisition, balanced under concurrency), so templates can lazily import sibling modules at run time as they could under the child-process runtime. Regression test: test_template_can_lazily_import_a_sibling_module (group=2, max_concurrent=2).
The three discovery helpers had sprawled across two modules with two separate sys.modules scans; they now share one iterator (_live_envs) next to their consumers in runtime.py, and taskset.py imports them instead of defining its own. LocalRuntime's docstring lists the three source forms as bullets and drops a reference to a function that does not exist.
…ent; source dir stays importable Addresses the second round of bot findings on #484. One discriminator fixes the two re-export findings: a fresh execution of a source yields new instances for envs it declares and the already-live instance for envs it merely imports. The origin probe counts only newly-created instances, so a tasks file re-exporting its env falls through to the env's real module and every rollout is rebuilt fresh; the live-env pointer validates candidate files the same way, skipping package __init__ re-exports (tried last) and anything whose standalone import fails or returns the live instance. An empty taskset schedules nothing, so it resolves no placement (it previously raised). And the source directory stays on sys.path for the whole acquisition, not just the initial import, so a template can lazily import a sibling module at run time as it could under the child-process runtime (insert-and-remove one entry per acquisition, balanced under concurrency).
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 17f77c9. Configure here.
…or extracted rows
|
Docs PR opened: #485 Rewrote titles and descriptions on five v6 pages so each is specific, unique, and within SEO length targets. |
1 participant
What
Placement gets one coherent local story, and
Taskrows become pure data all the way.LocalRuntime(source, *, env=None, ready_timeout=120.0)Serves a fresh env per rollout, in this process, from any pointer to it:
The live-env form never serves the instance itself — its declaring module is located by identity in
sys.modulesand the file re-imported per acquisition, so every rollout is still fresh.ready_timeoutnow boundsenv.start()(initialize hooks). There is no shared local form: shared substrates remainRuntime(url). The previous child-process mechanism keeps its exact semantics asSubprocessRuntime, for envs that should not share the orchestrator's process (the CLI uses it forhud eval --runtime local).Rows never carry placement
Task._sourceis removed; template factories no longer capture their defining file. With noruntime=, what is already known decides:.pysource (from_module/from_file) serves that source's directoryfrom_api) runs on the platformfix_add().run(agent)works in the same project that defines the env (resolution is logged; a name declared by two live envs is a loud error naming both files)Guard rails
runcall) fails naming the actual mistake and the__main__guardunknown tasklaterruntime=LocalRuntime(env))Breaking changes
task.run(agent)against a deployed env now raises instead of silently tunneling — the error namesruntime=HUDRuntime()LocalRuntime(path)keeps its signature and fresh-per-rollout contract but serves in-process (no child); useSubprocessRuntime(path)for the previous process isolation_source; the resolution ladder above covers the same cases explicitlyTests
12 end-to-end tests in
hud/eval/tests/test_local_runtime.pydrive the real rollout engine over the wire: all three pointer forms (with freshness proofs via module state), every rung of the resolution ladder, the ambiguity/teaching errors, and the three guard rails. 669 pass; lint and pyright clean.Note
High Risk
This changes core eval placement defaults and breaks
task.run(agent)withoutruntime=for deployed/unknown envs; in-processLocalRuntimealso shares the event loop and import cache with rollouts, which can affect isolation and concurrency behavior.Overview
Local vs subprocess placement —
LocalRuntimeis redefined to load a fresh env per rollout in the orchestrator process (from a source path, a live module-levelEnvironment, or a(task) -> Environmentconstructor), withready_timeoutonenv.start()and stricter errors for bad imports or post-import drift. The previous “spawnhud.environment.serverin a child” behavior is preserved asSubprocessRuntime;hud eval --runtime localandhud taskspawn paths now use that class.Tasks are pure data —
Task._sourceand template-time source capture are removed. Whenruntime=is omitted,Taskset._resolve_placement()picks placement from taskset origin (local module vsapi:), or from envs already declared in imported modules; otherwise it raises with suggested runtimes (no silentHUDRuntime()fallback for arbitrary rows).Exports & tests —
SubprocessRuntimeis exported fromhud/hud.eval; docs describe the new semantics. Rollout/chat tests that need process isolation switch toSubprocessRuntime; newtest_local_runtime.pycovers in-process placement and the resolution ladder end-to-end.Reviewed by Cursor Bugbot for commit b8be4a2. Bugbot is set up for automated code reviews on this repo. Configure here.