fix(normalizations): exclude -1 padding from token-norm length + guard mismatch (#1170)

[WatchTree-19]

problem

LogProbTokenNorm in src/lighteval/metrics/normalizations.py crashes with IndexError for belebele CFFormulation (#1170), and there's a related correctness bug @Rijgersberg flagged in the thread. two issues:

1. padding is counted in the token length. continuations are right-padded with -1 (pad_sequence(..., padding_value=-1) in transformers_model.py), and those -1 pads end up in choices_tokens. len(choices_tokens[ix]) then counts padding, so the per-choice length used to normalize the log-prob is inflated - e.g. the reporter's [236743, 236812, -1] is treated as length 3 instead of 2. this silently skews token-normalized scores whenever continuations differ in length, not only in the crashing case.

2. opaque IndexError on a length mismatch. the comprehension indexes choices_tokens[ix] over range(len(choices_logprob)) with no length check - unlike the LogProbCharNorm case just above, which validates and raises a clear ValueError. when the backend returns fewer output_tokens than choices (reporter: 3 token-lists for 4 choices/logprobs), you get an IndexError with no context.

fix

• exclude -1 padding from the per-choice token count via a small _num_continuation_tokens helper.
• validate len(choices_tokens) == len(choices_logprob) and raise a clear ValueError, matching the existing LogProbCharNorm guard.

tests

• test_token_norm_excludes_padding - fails on main (the -1 inflates the length and skews the value), passes here.
• test_token_norm_length_mismatch_raises - was IndexError, now a clear ValueError.
• existing test_token_norm / test_empty_input are unaffected.

remaining root cause (not fixed here, flagged for maintainers)

the reason belebele CFFormulation hits the mismatch is upstream: the model response's output_tokens comes back with fewer entries than logprobs/choices (reporter: 3 vs 4). that lives in the loglikelihood backend (_loglikelihood_tokens, where continuations are padded across both the length and the choice dimension) and needs a repro on the actual model to fix correctly. this PR makes the token count correct and turns the crash into an actionable error; happy to follow up on the backend alignment with a pointer if that's useful.

[ErenAta16]

Checked the padding-exclusion logic and the new length guard against the actual pad_sequence(..., padding_value=-1) usage in transformers_model.py — _num_continuation_tokens correctly counts only non-negative tokens, and the real or len(tokens) or 1 fallback chain avoids a divide-by-zero for the degenerate all-padding/empty case without changing behavior for any normal input. The two new tests reproduce both parts of #1170 (the padding-inflation case and the crash case) directly.

While comparing this to the sibling match arms in normalize_log_probs, I noticed the same missing-length-validation pattern this PR fixes for LogProbTokenNorm also exists in two other arms that aren't touched here:

• LogProbCharNorm(ignore_first_space=False) doesn't crash on a mismatch, but silently truncates: [choices_logprob[ix] / len(choice) for ix, choice in enumerate(choices_text)] iterates over choices_text, so if it's shorter than choices_logprob you just get a shorter result list back with no error, e.g. 3 logprobs + 2 texts silently returns 2 normalized values instead of 3.
• LogProbPMINorm() has the same shape as the old LogProbTokenNorm code: choices_logprob[ix] - unconditioned_logprob[ix] for ix in range(len(choices_logprob)) raises a bare IndexError if unconditioned_logprob is shorter.

Confirmed both concretely by replicating the exact comprehensions. Not asking to fix those here since they're outside the scope of #1170, but might be worth a quick follow-up PR applying the same guard-and-raise pattern to keep all four match arms consistent, since LogProbCharNorm(ignore_first_space=True) already has the same check this PR adds for LogProbTokenNorm.

This PR itself looks correct and ready.