Fix max_tokens sent as a list by the litellm judge backend

[AbhilakshSinghReen]

Fixes #1296

What

JudgeLM.__call_litellm builds max_tokens with a trailing comma, making it a 1-tuple:

# src/lighteval/metrics/utils/llm_as_judge.py:341
kwargs["max_tokens"] = (max_new_tokens,)

litellm serializes that to the JSON array [64] instead of the integer 64, so the request body sent to the provider is malformed.

Why it matters

Spec-compliant OpenAI-compatible servers reject the request with 400 Bad Request. JudgeLM then retries 3x, gives up, and returns the string

"ERROR: Failed to get response from the API."

as the judge's response — which is passed to process_judge_response() and scored as if it were a real judgment. The result is silently corrupted judge scores rather than a loud failure.

Evidence

Verified against a real vLLM server. Same server, same model, same prompt; the only variable is the tuple:

POST /v1/chat/completions  400 Bad Request   <- judge, attempt 1
POST /v1/chat/completions  400 Bad Request   <- judge, attempt 2   (3x retry)
POST /v1/chat/completions  400 Bad Request   <- judge, attempt 3
POST /v1/chat/completions  200 OK            <- control: max_tokens=64 as an int

With this patch the judge call returns 200 OK and produces a real judgment.

Full repro (including the captured wire payload showing "max_tokens": [512]) is in the linked issue.

Scope

• Only the litellm backend is affected. vllm, transformers, openai/tgi and inference-providers all pass self.max_tokens through directly.
• Only fires when max_tokens is explicitly set on the judge (it defaults to None, which skips the branch).
• Present on main and in the released 0.13.0 on PyPI.

Note

Happy to add a regression test if you'd like one — the natural place is a unit test that stubs litellm.completion and asserts the max_tokens kwarg is an int, since litellm isn't in the base test deps.