Skip to content

Fix BNGsim bridge silent fallbacks: strict routing error (#109) + Perl-faithful list-arg grammar (#110)#111

Merged
jrfaeder merged 7 commits into
RuleWorld:mainfrom
wshlavacek:fix/bngsim-bridge-routing-issues
Jul 20, 2026
Merged

Fix BNGsim bridge silent fallbacks: strict routing error (#109) + Perl-faithful list-arg grammar (#110)#111
jrfaeder merged 7 commits into
RuleWorld:mainfrom
wshlavacek:fix/bngsim-bridge-routing-issues

Conversation

@wshlavacek

@wshlavacek wshlavacek commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Two related fixes to the optional BNGsim bridge, both addressing cases where a model silently ran on the legacy BNG2.pl subprocess when the caller expected BNGsim. One hardens the routing contract; the other removes the most common trigger.

Closes #109
Closes #110

#109simulator='bngsim' must error, not silently fall back

Under simulator='bngsim', the routing bridge silently routed to the legacy BNG2.pl subprocess whenever the Python parser (modelapi.bngmodel) couldn't inspect a model's actions — e.g. an unrecognized simulate argument that bngmodel rejects but BNG2.pl tolerates. That violates the simulator='bngsim' contract ("use BNGsim, or error"): the caller got legacy output with no exception and no warning. A parity sweep over 895 models hit this on 9 models.

_classify_bngl_actions_for_bngsim() returned ROUTE_SUBPROCESS for the un-inspectable case unconditionally, with no reference to simulator.

Fix (bionetgen/core/tools/bngsim_bridge.py), at the classify_bngsim_route call site where simulator is in scope:

  • simulator='bngsim' + un-inspectable actions → ROUTE_ERROR, propagating the underlying parse reason so the caller learns why it couldn't route to BNGsim. Both consumers (main.py, runner.run) already raise BNGSimError(route.reason) on ROUTE_ERROR.
  • simulator='auto' keeps the subprocess fallback but emits a one-time WARNING per file.
  • simulator='subprocess' is unchanged (returns before action inspection).

To surface the reason, the routing parse now returns (actions, reason) through the memoized loader (_load_bngl_routing_actions; _load_bngl_actions_for_routing kept as a thin list-only wrapper for its other callers). The unavailable-BNGsim case already returned ROUTE_ERROR and is untouched.

#110 — action grammar: accept e-notation and trailing commas in list args

ActionList.define_parser (bionetgen/core/utils/utils.py) was stricter than BNG2.pl/Perl for list-valued arguments. arg_type_list matched elements with pp.Word(pp.nums + ".") (digits and . only) inside a plain pp.delimitedList, so it rejected:

  • scientific notation, e.g. par_scan_vals=>[2.3e-10,5.1e-10]
  • a trailing comma, e.g. par_scan_vals=>[1,2,3,]

Both are valid Perl that BNG2.pl parses and runs. When a model used either (commonly par_scan_vals on parameter_scan), bngmodel raised BNGParseError — which is exactly what triggered the #109 silent-fallback class. Real models hit this: RuleHub's Mitra2019/15-igf1r fits (e-notation and trailing commas) and Salazar-Cavazos2019 CHO_EGFR best-fit (e-notation).

Fix: broaden arg_type_list to use arg_type_expr (which already spans e/E and +/-), allow an empty list, and tolerate one optional trailing comma:

arg_type_list = (
    "[" + pp.Optional(pp.delimitedList(quote_word ^ arg_type_expr)) + pp.Optional(",") + "]"
)

Still rejects genuinely-malformed lists — double commas ([1,,2]) and unclosed lists ([1,2,).

#109 follow-up — strict mode also errors on unrecognized actions

#109's first fix closed the parse-failure hole (actions can't be inspected → None). But classify_bngsim_route returned the _classify_bngl_actions_for_bngsim result verbatim, so under an explicit simulator='bngsim' request a BNGL with an action the router doesn't recognize still silently ran on the legacy stack (its ROUTE_SUBPROCESS was never promoted to ROUTE_ERROR).

BNGsim can run such a model via the BNG2.pl backend hook — the router just can't confirm an unknown action is safe to delegate — so strict mode now surfaces ROUTE_ERROR with the reason instead of degrading silently. PLA and BNGsim-unsupported methods are genuine BNGsim incapabilities, so they stay on the subprocess route in every mode; auto keeps the legacy fallback for unrecognized actions too. simulator is threaded into _classify_bngl_actions_for_bngsim (default 'auto', back-compatible).

#109 follow-up — unknown simulation methods error instead of silent legacy

The router treated every BNGsim-unsupported method the same (ROUTE_SUBPROCESS), so a malformed/unknown method — a typo like oed, or quadratic — silently ran on the legacy stack in both auto and bngsim. BNG2.pl's own simulate() validates the method against its $METHODS hash (cvode/ssa/pla/psa/nf, plus the ode alias) and rejects anything else ("Simulation method '...' is not a valid option"), so this is faithful, not stricter than BNG2.pl.

Now split (_unsupported_method_route):

  • A valid BNG method this BNGsim build can't run — pla, or nf without an NFsim build — stays on the subprocess route (genuine/build incapability), in both modes.
  • A method outside the valid set errors in both modes, surfacing it.

_KNOWN_BNGL_METHODS = {ode, cvode, ssa, psa, pla, nf, rm} is taken verbatim from BNG2.pl's $METHODS keys (+ ode alias, + rm RuleMonkey extension), so valid methods like the cvode alias keep their existing legacy routing rather than newly erroring.

Note the deliberate asymmetry with actions: BNG2.pl supports many actions outside the router's curated list (setVolume, setOption, readSBML, quit, …), so auto keeps the legacy fallback for unrecognized actions (only bngsim errors). Methods are a closed, BNG2.pl-validated set; actions are open.

#109 follow-up — rm/RuleMonkey awareness; build incapabilities error under strict mode

method=>"rm" routes to BNGsim's RuleMonkey (via the rmnf rewrite + backend hook) only when RuleMonkey is installed — but the router marked rm supported unconditionally (_method_supported_by_bngsim_for_routing never checked RuleMonkey, unlike nf/NFsim), and classify_bngsim_route took no RuleMonkey arg. So an rm model on a BNGsim without RuleMonkey still classified as ROUTE_BNGL_BNGSIM, then run_bngl_with_bngsim (which hardcodes simulator="auto") silently fell back to legacy — invisibly even under simulator='bngsim'.

Now the classifier is RuleMonkey-aware (bngsim_has_rulemonkey threaded through), so the decision is made up front where strict mode is enforced, not in the simulator-blind runner. nf-without-NFsim and rm-without-RuleMonkey are treated uniformly as build incapabilities (_BNGSIM_OPTIONAL_COMPONENT_METHODS): simulator='bngsim' errors (surfacing "install NFsim/RuleMonkey"), auto falls back. This matches the existing direct BNG-XML nf behavior. pla stays subprocess in both modes (a categorical incapability, not a missing component); the cvode alias keeps its legacy route.

Why together

#110 makes these models parseable; #109 ensures that when something still isn't inspectable, bngsim errors instead of silently degrading. Shipping them together closes the silent-fallback gap end to end.

Tests

…spect actions

Under simulator='bngsim', the routing bridge silently fell back to the
legacy BNG2.pl subprocess engine whenever the Python parser
(modelapi.bngmodel) could not inspect a BNGL's actions — e.g. a typo'd or
unknown simulate argument (atoll, abstol, par_scan_vals) that bngmodel
rejects but BNG2.pl tolerates. This violated the simulator='bngsim'
contract ("use BNGsim, or error"): the caller got legacy output with no
exception and no warning. A parity sweep over 895 models hit this on 9
models.

_classify_bngl_actions_for_bngsim() returned ROUTE_SUBPROCESS for the
un-inspectable case unconditionally, with no reference to simulator.

Fix, at the classify_bngsim_route call site where simulator is in scope:
- simulator='bngsim' + un-inspectable actions -> ROUTE_ERROR, propagating
  the underlying parse reason so the caller learns why it couldn't route
  to BNGsim. Both consumers (main.py, runner.run) already raise
  BNGSimError(route.reason) on ROUTE_ERROR.
- simulator='auto' keeps the subprocess fallback but emits a one-time
  WARNING per file (deduped across the ~4 routing queries per run).
- simulator='subprocess' is unchanged (returns before action inspection).

To surface the reason, the routing parse now returns (actions, reason):
_parse_bngl_actions_for_routing and the memoized loader (renamed
_load_bngl_routing_actions; _load_bngl_actions_for_routing kept as a thin
list-only wrapper for its other callers) carry the parse message. The
unavailable-BNGsim case already returned ROUTE_ERROR and is untouched.

Adds tests covering the bngsim error, the auto one-time-warning fallback,
the subprocess no-inspect path, reason propagation through the cache, and
an end-to-end repro of the atoll-typo model from the issue.
…n args

ActionList.define_parser's list-valued argument grammar was stricter than
BNG2.pl/Perl: arg_type_list matched elements with `pp.Word(pp.nums + ".")`
(digits and '.' only) inside a plain `pp.delimitedList`, so it rejected

  - scientific notation, e.g. par_scan_vals=>[2.3e-10,5.1e-10]
  - a trailing comma, e.g. par_scan_vals=>[1,2,3,]

Both are valid Perl that BNG2.pl parses and runs. When a model used either
(commonly par_scan_vals on parameter_scan), modelapi.bngmodel raised
BNGParseError, so under simulator='bngsim' the bridge couldn't inspect the
actions and silently fell back to the legacy subprocess (the RuleWorld#109 class) —
the model never ran on bngsim. Real models hit this: RuleHub's
Mitra2019/15-igf1r fits and Salazar-Cavazos2019 CHO_EGFR best-fit.

Broaden arg_type_list to use arg_type_expr (which already spans e/E and
+/-), allow an empty list, and tolerate one optional trailing comma:

  arg_type_list = "[" + Optional(delimitedList(quote_word ^ arg_type_expr))
                      + Optional(",") + "]"

Still rejects genuinely-malformed lists (double commas `[1,,2]`, unclosed
`[1,2,`). Adds parametrized accept/reject tests over the issue's matrix and
the affected real-model forms.
…legacy

Follow-up to the RuleWorld#109 contract. classify_bngsim_route returned the
_classify_bngl_actions_for_bngsim result verbatim, so under an explicit
simulator='bngsim' request a BNGL with an action the router doesn't
recognize still silently ran on the legacy BNG2.pl stack (ROUTE_SUBPROCESS
was never promoted to ROUTE_ERROR). RuleWorld#109 only closed the parse-failure
(actions==None) hole; this closes the unrecognized-action hole.

BNGsim can run such a model via the BNG2.pl backend hook — the router just
can't confirm an unknown action is safe to delegate — so strict mode now
surfaces ROUTE_ERROR with the reason instead of degrading silently. PLA and
BNGsim-unsupported methods are genuine BNGsim incapabilities, so they stay
on the subprocess route in every mode (auto and bngsim alike); auto keeps
the legacy fallback for unrecognized actions too.

Threads `simulator` into _classify_bngl_actions_for_bngsim (default 'auto',
back-compatible) and promotes only the unrecognized-action branch. Adds
tests: bngsim unknown-action -> error, auto unknown-action -> subprocess,
and bngsim PLA / unsupported-method -> subprocess.
The router lumped "method BNGsim can't run" together as ROUTE_SUBPROCESS,
so a malformed/unknown method (a typo like 'oed', or 'quadratic') silently
ran on the legacy stack in auto AND bngsim modes. BNG2.pl's simulate()
validates the method against its $METHODS hash (cvode/ssa/pla/psa/nf, plus
the 'ode' alias) and itself rejects anything else with "Simulation method
'...' is not a valid option" — so routing an unknown method to legacy just
defers to a less clear BNG2.pl error.

Split the unsupported-method handling (_unsupported_method_route):
- A method in the valid BNG universe that this BNGsim can't run — pla, or nf
  without an NFsim build — stays on the subprocess route (genuine/build
  incapability), in both modes.
- A method outside the valid set is malformed and now errors in both auto
  and bngsim, surfacing it instead of silently degrading.

_KNOWN_BNGL_METHODS = {ode, cvode, ssa, psa, pla, nf, rm} is taken verbatim
from BNG2.pl's $METHODS keys (+ ode alias, + rm RuleMonkey extension), so
valid methods like the cvode alias keep their existing legacy routing rather
than newly erroring.

Unrecognized *actions* are deliberately NOT treated this way: BNG2.pl
supports many actions outside the router's curated list (setVolume,
setOption, readSBML, quit, ...), so auto keeps the legacy fallback for them
(only bngsim errors, per the prior commit). Methods are a closed,
BNG2.pl-validated set; actions are open.

Adds parametrized tests: unknown method -> error in both modes; cvode alias
and nf-without-nfsim -> subprocess (regression guards, no spurious errors).
method=>"rm" routes to BNGsim's RuleMonkey (via the rm->nf rewrite + backend
hook) only when RuleMonkey is installed. But the router marked rm supported
UNCONDITIONALLY — _method_supported_by_bngsim_for_routing never checked
RuleMonkey, unlike nf/NFsim — and classify_bngsim_route took no rulemonkey
arg. So an rm model on a BNGsim without RuleMonkey still classified as
ROUTE_BNGL_BNGSIM, then run_bngl_with_bngsim (which hardcodes
simulator="auto") silently fell back to legacy — invisibly even under
simulator='bngsim'. Same silent-downgrade class as RuleWorld#109.

Make the classifier RuleMonkey-aware (thread bngsim_has_rulemonkey through
classify_bngsim_route -> _classify_bngl_actions_for_bngsim ->
_method_supported_by_bngsim_for_routing) so the decision is made up front
where strict mode is enforced, not in the simulator-blind runner.

Treat nf-without-NFsim and rm-without-RuleMonkey uniformly as *build*
incapabilities (_BNGSIM_OPTIONAL_COMPONENT_METHODS): BNGsim could run them
with the component installed, so simulator='bngsim' errors (surfacing
"install NFsim/RuleMonkey") while auto falls back to legacy. This matches
the existing direct BNG-XML nf behavior (classify already errored there
under bngsim).

NOTE: this revises the BNGL nf-without-NFsim case shipped in the previous
commit (was subprocess in both modes) so nf and rm are consistent and the
strict-bngsim silent gap is closed for both. pla stays subprocess in both
modes (a categorical incapability, not a missing component); the cvode alias
keeps its legacy route. Updates tests accordingly and adds rm coverage.
test_atomic_supported_bngl_methods_use_bngsim forced bngsim_available=True
but let bngsim_has_nfsim/bngsim_has_rulemonkey default to the ambient module
values, which are False in any environment without a feature-complete BNGsim
(all CI runners). After "rm" routing was made RuleMonkey-aware, the rm case
correctly routes to subprocess there, so the test's hard-coded bngl-bngsim
expectation failed on all 15 matrix jobs.

Force bngsim_has_nfsim=True and bngsim_has_rulemonkey=True in the test so the
"supported methods route to BNGsim" assertion is environment-independent,
matching how it already forces bngsim_available. Verified by rerunning the
routing + method-defaults suites with both component flags monkeypatched to
False (the CI condition): 93 passed.
@jrfaeder
jrfaeder merged commit d40470c into RuleWorld:main Jul 20, 2026
17 checks passed
jrfaeder pushed a commit that referenced this pull request Jul 20, 2026
…#111)

* Fix: use temporary directory universally for BNGVisualize execution

Resolves a TODO inside `bionetgen/core/tools/visualize.py` where execution was happening directly in the target directory (or current directory) instead of uniformly executing in a safe TemporaryDirectory context.

* Removes the `if self.output is None` execution branch split.
* Uses `with TemporaryDirectory() as out` for all visualization generation, ensuring `BNGCLI` isolates intermediate outputs safely.
* Extracts the finalized artifacts via `VisResult._dump_files()` seamlessly to the user's intended output folder, restoring the initial directory state cleanly on success or failure.

Co-authored-by: akutuva21 <44119804+akutuva21@users.noreply.github.com>

* chore: PR #111 remove forbidden artifacts and run black

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
wshlavacek added a commit to lanl/bngsim that referenced this pull request Jul 22, 2026
Bumps parity_checks/requirements-pybionetgen.txt from 5109a46 to 43b09a5.
The old pin carried RuleWorld/PyBioNetGen#109: under simulator='bngsim', a
model whose actions PyBioNetGen's Python parser could not inspect was routed
SILENTLY to legacy BNG2.pl instead of raising, so an explicit bngsim request
could return legacy output with no error. RuleWorld/PyBioNetGen#111 fixed it
(strict bngsim -> ROUTE_ERROR carrying the parse reason; auto keeps the
fallback plus a one-time warning) and made the list-arg grammar Perl-faithful
(#110). 43b09a5 is main at that fix plus the merges after it, CI green.

Verified live on issue #109's own repro (a simulate carrying a typo'd `atoll`,
which bngmodel() rejects and BNG2.pl tolerates): simulator='bngsim' -> error,
simulator='auto' -> subprocess. Note this could never have moved the suite's
numbers -- since GH #175 the sweep drives bngsim in-process via
run_bngsim_job, not through bionetgen.run(simulator='bngsim'), so the bridge's
routing pass is provenance, not the execution path.

Golden regenerated on the new pin: 895/895 jobs ok, _meta.bionetgen_commit
5109a46e58ec -> 43b09a534640. 885 of 895 checksums reproduce byte-for-byte
across a bngsim 0.9.59 -> 0.11.35 gap, which makes the 10 that moved legible:
6 are exactly CURATED_SIX (Lang_2024, Kocieniewski_2012, Barua_2007,
Blinov_2006, Barua_2013__PATCHED, fceri_fyn), re-sourced to house-curated
bug-fixed bodies after the golden was last generated -- those records had gone
stale. Of the rest, ml_q_learning is ODE drift at max_rel 1.0e-06 and the three
Lin2019 models are single seed=1 stochastic trajectories, chaotic across a
version change by contract D2. fceri_fyn also leaves the D1 .cdat fallback
(8 -> 7 records) since its corrected body has observables.

Also fixes two things that blocked the documented verification path:

* bootstrap_parity_env.py built its venv on uv's default interpreter, so the
  ABI-tagged bngsim wheel failed at the LAST step, after the whole PyBioNetGen
  build. The venv now defaults to the version of the interpreter running the
  script (the one --build-bngsim targets), --python overrides. Re-running the
  bootstrap is idempotent again (uv venv --allow-existing); current uv aborts
  on an existing venv rather than reusing it.

* ship_wheel.py forced MACOSX_DEPLOYMENT_TARGET=10.15 unconditionally. That is
  correct on the x86_64 build box but on arm64 yields a macosx_10_15_arm64 tag,
  and since Apple Silicon starts at 11.0 no installer ever generates a
  macosx_10_*_arm64 compatibility tag -- such a wheel installs nowhere,
  including on the machine that built it. Now chosen per build architecture
  (10.15 on x86_64, 11.0 on arm64), leaving x86_64 output unchanged.

Checks: 278 passed / 0 skipped in the bootstrapped parity env, where the
bridge/engine tests (test_bngsim_golden_engine, test_multi_segment_replay,
test_state_setup_prefix) actually execute instead of skipping for want of
BNG2.pl; 251 passed in the dev venv.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants