Skip to content

Fix unreachable response_format default in JudgeLM#1303

Open
oraziooztas wants to merge 1 commit into
huggingface:mainfrom
oraziooztas:fix/judge-response-format-default
Open

Fix unreachable response_format default in JudgeLM#1303
oraziooztas wants to merge 1 commit into
huggingface:mainfrom
oraziooztas:fix/judge-response-format-default

Conversation

@oraziooztas

Copy link
Copy Markdown

What does this PR do?

Fixes a conditional in JudgeLM.__init__ that never applies the intended default:

self.response_format = response_format if not None else DEFAULT_FORMAT

The condition evaluates the constant expression not None (always True), so:

  • DEFAULT_FORMAT = {"type": "text"} (line 46) is unreachable dead code;
  • when response_format is not provided, None is stored and forwarded as-is to the backend request kwargs (litellm kwargs and the OpenAI/TGI chat.completions calls), instead of the documented text-format default.

The fix tests the variable, as intended:

self.response_format = response_format if response_format is not None else DEFAULT_FORMAT

Tests

Added tests/unit/metrics/test_llm_as_judge_response_format.py:

  • test_response_format_defaults_when_not_provided — fails on main (response_format stays None), passes with this fix;
  • test_explicit_response_format_is_preserved — guards against regressing the explicit-format path.

No network, no external dependencies (pure __init__ behavior). ruff check and ruff format --check pass on both changed files.

🤖 Generated with Claude Code

https://claude.ai/code/session_0158ZooCCJEy6xtme9c8iksT

The conditional tested the constant expression `not None` (always True),
so DEFAULT_FORMAT was dead code and response_format=None was forwarded
as-is to the backend request kwargs instead of the documented default.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0158ZooCCJEy6xtme9c8iksT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant