Skip to content

Fix asdiv prompt truncating multi-character gold answers#1298

Open
vineethsaivs wants to merge 1 commit into
huggingface:mainfrom
vineethsaivs:fix/asdiv-choices-list
Open

Fix asdiv prompt truncating multi-character gold answers#1298
vineethsaivs wants to merge 1 commit into
huggingface:mainfrom
vineethsaivs:fix/asdiv-choices-list

Conversation

@vineethsaivs

Copy link
Copy Markdown

Problem

asdiv_prompt sets choices to a bare string instead of a list:

def asdiv_prompt(line, task_name: str = None):
    return Doc(
        task_name=task_name,
        query=f"{line['body']}\nQuestion:{line['question']}\nAnswer:",
        choices=line["answer"].split(" (")[0],   # bare string, e.g. "35"
        gold_index=[0],
    )

Doc.choices is a list[str], and Doc.get_golds() indexes it with the gold index:

for gold_ix in as_list(self.gold_index):
    golds.extend(as_list(self.choices[gold_ix]))

When choices is the string "35", choices[0] indexes the string and yields "3", so the gold answer is silently truncated to its first character. ASDiv answers have the form "<value> (<unit>)" (which is why the .split(" (")[0] is there), so the extracted value is usually multi-character:

answer "35 (apples)" -> gold ["3"]   (should be ["35"])
answer "128"         -> gold ["1"]   (should be ["128"])
answer "3.5 (m)"     -> gold ["3"]   (should be ["3.5"])

get_golds() feeds the reference into the task's exact_match metric, so a model that correctly outputs 35 is scored against gold 3 and marked wrong. Every sibling generative task wraps the value in a list (unscramble, numeracy, arithmetic, ...); asdiv is the only one missing the brackets.

Fix

Wrap the extracted answer in a list, matching the sibling tasks and the Doc.choices: list[str] contract:

choices=[line["answer"].split(" (")[0]],

Test

Adds tests/unit/tasks/test_asdiv.py, which calls asdiv_prompt on a multi-character answer and asserts doc.choices == ["35"] and doc.get_golds() == ["35"]. It fails before (get_golds() returns ["3"]) and passes after.

asdiv_prompt set choices to a bare string (line["answer"].split(" (")[0])
rather than a list. Doc.choices is a list[str] and Doc.get_golds() indexes
it with the gold index, so a bare string was indexed by character: gold
"35" became "3", "128" became "1". ASDiv answers are typically
multi-character (multi-digit numbers, decimals), so exact_match scored
correct model outputs as wrong. Every sibling generative task wraps the
value in a list; asdiv was the only one that did not.

Wrap the extracted answer in a list so the full gold answer is preserved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant