parse: prefer top-level class over nested def in extract_function_name - #61
Open
prann-ab wants to merge 1 commit into
Open
parse: prefer top-level class over nested def in extract_function_name#61prann-ab wants to merge 1 commit into
prann-ab wants to merge 1 commit into
Conversation
`extract_function_name` searches for `def ...(` first and only falls back to
`class ...(`. A function_header for a class-based step carries the class line
*and* its first method:
class Maxwell:
""" The base class for evolution of Maxwell's equations. """
def __init__(self, n_grid, x_out):
so the `def` pattern matches first and the name comes back as `__init__`.
`get_function_from_code` then returns only that method, and the class -- plus
every other method on it -- is dropped from the code accumulated into later
sub-steps' prompts and test scripts. Any later sub-step that instantiates the
class fails with NameError regardless of what the model generated.
Two related problems in the same function:
* the class pattern `\bclass\s+(\w+)\s*\(` requires a base-class paren, so
`class Maxwell:` would not match even if it were tried first;
* both patterns are unanchored, so they can match inside the header's
docstring -- several headers document a parameter as `env: class Block`,
which yields the "function name" `env`.
Anchoring to a top-level definition line fixes all three: the first line at
column 0 opening a `class` or `def` is by construction the declared signature.
Measured over the 341 published sub-step headers: 11 change, all of them
genuine class headers that previously resolved to `__init__` (13.6, 30.1,
30.2, 30.3, 46.1, 46.2, 62.1, 68.1, 68.2, 68.3, 68.4). The other 330 are
identical.
Fraction of each skip-step's reference file that survives extraction:
13.6 1855/2098 (88%, a bare __init__) -> 2016/2098 (96%, class Maxwell)
62.1 151/978 (15%, a bare __init__) -> 493/978 (50%, class EnlargedBlock)
76.3 987/1020 (97%) -> 987/1020 (97%, unchanged)
NOTE this changes benchmark scores: sub-steps downstream of a class-based step
were previously unwinnable. SciCode numbers produced before and after this
commit are not comparable.
Refs scicode-bench#49, scicode-bench#59.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the class-extraction defect reported in #59 and #49.
The bug
extract_function_nametriesdef ...(beforeclass ...(. Afunction_headerfor a class-based step contains the class line and its first method:so the
defpattern wins and the name comes back as__init__.get_function_from_codethen returns only that method — the class and all its other methods are dropped from the code accumulated into later sub-steps' prompts and test scripts. Any later sub-step that instantiates the class raisesNameErrorno matter what the model generated.Two related problems in the same function:
\bclass\s+(\w+)\s*\(requires a base-class paren, soclass Maxwell:would not match even if it were tried first;re.search, so they match inside the header's docstring. Several headers document a parameter asenv: class Block, which yields the "function name"env. I hit this with a first attempt that only reordered the two patterns — worth knowing if anyone tries the one-line version.Anchoring to a top-level definition line fixes all three: the first line at column 0 opening a
classordefis by construction the signature the header declares.Effect
Ran the patched
extract_function_nameover all 341 published sub-step headers:All 11 are genuine class headers that previously resolved to
__init__: 13.6, 30.1, 30.2, 30.3, 46.1, 46.2, 62.1, 68.1, 68.2, 68.3, 68.4. No false positives.Fraction of each skip-step's
eval/data/reference file that survives extraction:__init__class Maxwell__init__class EnlargedBlockCaveats
This changes scores. Sub-steps downstream of a class-based step were previously unwinnable, so SciCode numbers produced before and after this commit are not comparable. That seemed worth stating loudly rather than burying.
62.1 is only half fixed. Its reference file defines two classes (
BlockandEnlargedBlock) and single-symbol extraction can only return one. Sinceeval/data/*.txtis curated reference code rather than model output, injecting those three files verbatim instead of extracting a symbol would fix it completely — but that is a design call for the harness, so I have left it out of this PR and will raise it on #59 instead.No test in
tests/exercises either function, so nothing there needed updating.