fix: honor output_format="tokens" in LLMDetector (#65)

[mobinert]

Summary

LLMDetector validated output_format="tokens" but every entry point returned the raw span dicts from _predict, so callers asking for tokens got spans back. And because span filtering is skipped for token output, passing min_confidence alongside it silently did nothing.

I went with Option A (actually implement token output) instead of making the detector spans-only. The facade defaults to output_format="tokens", so HallucinationDetector(method="llm").predict(...) would otherwise break by default, and this keeps the LLM detector in line with the other detectors.

What changed:

• Added _convert_to_tokens, which maps predicted character spans onto whitespace tokens ({token, pred, prob}), the same way RAGFactCheckerDetector does it. A token is flagged when its char range overlaps a predicted span. prob uses the span confidence when it's present (only set with include_reasoning=True), otherwise a fixed constant.
• Routed predict / predict_prompt / predict_prompt_batch through it before the confidence filter.
• Added tests in tests/test_llm_detector_pytest.py covering all three entry points. The spans path is untouched.

Related issue

Closes #65.

Type of change

• Bug fix
• Feature
• Documentation
• Tests
• Refactor or maintenance

Testing

• ruff format --check lettucedetect/ lettucedetect_api/ tests/
• ruff check lettucedetect/ lettucedetect_api/ tests/ --extend-exclude lettucedetect/integrations/
• python -m pytest
• Other:

Checklist

• I kept the PR focused on one change.
• I added or updated tests/docs when needed.
• I checked that no secrets, API keys, or credentials are included.

Rights & sign-off (required)

• I certify that I have the right to submit this code and that it may be
  distributed under the repository's MIT license
  (see CONTRIBUTING).

[mobinert]

Updated the description to tick the contribution-rights box from the PR template, which is what the earlier rights-check failure was about. The edit queued a fresh run but it's waiting on approval since this is coming from my fork. Could a maintainer kick off the re-run when you get a sec? Tests and lint are already green. Thanks!