Disallow further step() calls after episode completion - #91
Open
pangwangshu wants to merge 1 commit into
Open
Conversation
Prevents an agent from continuing to act (and inflating its score) after isCompleted is set, by ignoring further step() calls and returning the cached terminal state instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #31.
Problem
step()recomputesisCompletedon every call (success, failure via negative score, or step-limit reached), but never actually stops processing new actions once it's set. An agent that doesn't check the returned flag can keep callingstep()after the episode is "done," and those actions still reach the simulator and can change the score — exactly the score-gaming risk described in the issue, especially for forced-choice tasks.Fix
ScienceWorldEnv.step()now short-circuits onceself.isCompletedisTrue: it skipsself.server.step(...)entirely and returns the cached terminal observation/infos withreward=0, instead of letting the action through. The flag is reset toFalseinload()andreset()so a new episode isn't blocked by a stale value from the previous one.This follows the same pattern already used a few lines above for the step-limit and negative-score checks (see the existing
# New: Handle this in the API rather than the agentcomment) — completion handling lives in this Python wrapper, not the Scala simulator, so no simulator/JAR changes are needed here.Scope
The issue thread also touched on two related-but-separate things that this PR intentionally does not address, to keep it focused:
isCompletedgets set rather than what happens after.Testing
test_step_after_completion_is_ignored, which drives an episode to its step limit, confirms a furtherstep()call is ignored (same observation/infos repeated,reward=0,donestillTrue), and confirmsreset()clears the flag so a new episode works normally.pytestsuite passes (8/8).flake8 --max-line-length 120andpre-commit run --all-fileson the changed files are clean.CHANGELOG.mdentry added, matching Complete type hinting across scienceworld package; add opt-in mypy tox env #89/Fix reversed anode/cathode polarity on battery/generator #90 which didn't add one either.🤖 Generated with Claude Code