Skip to content

parse: prefer top-level class over nested def in extract_function_name - #61

Open
prann-ab wants to merge 1 commit into
scicode-bench:mainfrom
prann-ab:fix/class-header-extraction
Open

parse: prefer top-level class over nested def in extract_function_name#61
prann-ab wants to merge 1 commit into
scicode-bench:mainfrom
prann-ab:fix/class-header-extraction

Conversation

@prann-ab

@prann-ab prann-ab commented Aug 7, 2026

Copy link
Copy Markdown

Fixes the class-extraction defect reported in #59 and #49.

The bug

extract_function_name tries def ...( before class ...(. A function_header for a class-based step contains the class line and its first method:

class Maxwell:
    """ The base class for evolution of Maxwell's equations. """
    def __init__(self, n_grid, x_out):

so the def pattern wins and the name comes back as __init__. get_function_from_code then returns only that method — the class and all its other methods are dropped from the code accumulated into later sub-steps' prompts and test scripts. Any later sub-step that instantiates the class raises NameError no matter what the model generated.

Two related problems in the same function:

  • the class pattern \bclass\s+(\w+)\s*\( requires a base-class paren, so class Maxwell: would not match even if it were tried first;
  • both patterns are unanchored re.search, so they match inside the header's docstring. Several headers document a parameter as env: class Block, which yields the "function name" env. I hit this with a first attempt that only reordered the two patterns — worth knowing if anyone tries the one-line version.

Anchoring to a top-level definition line fixes all three: the first line at column 0 opening a class or def is by construction the signature the header declares.

Effect

Ran the patched extract_function_name over all 341 published sub-step headers:

count
unchanged 330
changed 11

All 11 are genuine class headers that previously resolved to __init__: 13.6, 30.1, 30.2, 30.3, 46.1, 46.2, 62.1, 68.1, 68.2, 68.3, 68.4. No false positives.

Fraction of each skip-step's eval/data/ reference file that survives extraction:

step before after
13.6 1855/2098 (88%) — a bare __init__ 2016/2098 (96%) — class Maxwell
62.1 151/978 (15%) — a bare __init__ 493/978 (50%) — class EnlargedBlock
76.3 987/1020 (97%) 987/1020 (97%), unchanged

Caveats

This changes scores. Sub-steps downstream of a class-based step were previously unwinnable, so SciCode numbers produced before and after this commit are not comparable. That seemed worth stating loudly rather than burying.

62.1 is only half fixed. Its reference file defines two classes (Block and EnlargedBlock) and single-symbol extraction can only return one. Since eval/data/*.txt is curated reference code rather than model output, injecting those three files verbatim instead of extracting a symbol would fix it completely — but that is a design call for the harness, so I have left it out of this PR and will raise it on #59 instead.

No test in tests/ exercises either function, so nothing there needed updating.

`extract_function_name` searches for `def ...(` first and only falls back to
`class ...(`. A function_header for a class-based step carries the class line
*and* its first method:

    class Maxwell:
        """ The base class for evolution of Maxwell's equations. """
        def __init__(self, n_grid, x_out):

so the `def` pattern matches first and the name comes back as `__init__`.
`get_function_from_code` then returns only that method, and the class -- plus
every other method on it -- is dropped from the code accumulated into later
sub-steps' prompts and test scripts. Any later sub-step that instantiates the
class fails with NameError regardless of what the model generated.

Two related problems in the same function:

  * the class pattern `\bclass\s+(\w+)\s*\(` requires a base-class paren, so
    `class Maxwell:` would not match even if it were tried first;
  * both patterns are unanchored, so they can match inside the header's
    docstring -- several headers document a parameter as `env: class Block`,
    which yields the "function name" `env`.

Anchoring to a top-level definition line fixes all three: the first line at
column 0 opening a `class` or `def` is by construction the declared signature.

Measured over the 341 published sub-step headers: 11 change, all of them
genuine class headers that previously resolved to `__init__` (13.6, 30.1,
30.2, 30.3, 46.1, 46.2, 62.1, 68.1, 68.2, 68.3, 68.4). The other 330 are
identical.

Fraction of each skip-step's reference file that survives extraction:

    13.6   1855/2098 (88%, a bare __init__)  ->  2016/2098 (96%, class Maxwell)
    62.1    151/978  (15%, a bare __init__)  ->   493/978  (50%, class EnlargedBlock)
    76.3    987/1020 (97%)                   ->   987/1020 (97%, unchanged)

NOTE this changes benchmark scores: sub-steps downstream of a class-based step
were previously unwinnable. SciCode numbers produced before and after this
commit are not comparable.

Refs scicode-bench#49, scicode-bench#59.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant