Integrate LiteLLM: multi-model chat, per-user budgets, and automatic fallback

[androemeda]

Summary

Integrates LiteLLM as an OpenAI-compatible proxy sitting between the backend and model providers. This replaces the single shared Azure/OpenAI client with per-user, budget-aware routing, and adds three capabilities: user-controlled model switching (OpenAI vs. local), per-user spend budgets, and silent fallback to a local model when a user's budget is exhausted.

Architecture

• LiteLLM proxy (litellm service in docker-compose.yml, config in litellm_config.yaml) sits in front of two model aliases - openai-model (GPT-4o) and local-model (Ollama) - and is the only thing that talks to model providers directly.
• Per-user virtual keys: each Auth0 user gets a LiteLLM "virtual key" on first login (litellm_service.py: get_or_create_virtual_key), cached in Supabase (users.litellm_virtual_key) so it's provisioned once and reused thereafter. Every chat/analysis request authenticates to LiteLLM with the caller's own key instead of a single shared credential - this is what makes per-user spend tracking and budgets possible.
• Budgets: new users are provisioned with a DEFAULT_USER_BUDGET / DEFAULT_BUDGET_DURATION (from backend/.env). GET /users/me/budget exposes spend/remaining budget, surfaced in the sidebar (real progress bar, replacing the previous hardcoded placeholder).
• Fallback: LiteLLM's own config-level fallbacks: doesn't catch budget-exhaustion (it's rejected at LiteLLM's auth gate before model routing ever runs), so fallback is handled in application code (llms_service.py: _create_chat_completion) - on a budget-shaped error, it retries the same request against local-model, switching credentials to LITELLM_MASTER_KEY since the user's own key is budget-blocked across every model, not just the one they requested.
• Model selector: ChatBox.tsx lets the user explicitly pick OpenAI or Local; the response reports which model actually served it (model_used, read from LiteLLM's own response, not inferred), shown as a subtle "Responded via Local LLM" indicator - covering both explicit selection and silent fallback.
• Observability: Langfuse tracing moved from a per-client OpenAI wrapper to LiteLLM's native success_callback.

Request flow

ChatBox.tsx → POST /query (validated via ChatRequest) → llms_controller.get_llm_response (resolves the caller's virtual key) → llms_service.get_response_with_tools → _create_chat_completion (tries the requested model, falls back to local on budget exhaustion) → LiteLLM proxy → provider.

Known limitations (not blocking)

• provision_user has no recovery from LiteLLM's "user already exists" error if a user's Supabase row is ever lost after LiteLLM-side provisioning succeeds.
• Login has no fallback if LiteLLM itself is unreachable during virtual-key provisioning.
• Fallback calls (using the master key) are attributed to a generic LiteLLM identity, not the real user - no financial impact (local calls are free) but an observability gap.

[Copilot]

Pull request overview

This PR introduces a LiteLLM proxy layer to enable per-user model routing and spend/budget enforcement, replacing the prior single shared LLM client approach. It adds UI support for selecting between hosted (OpenAI) and local models, exposes per-user budget information, and implements application-level fallback to a local model when a user’s LiteLLM budget is exhausted.

Changes:

• Add LiteLLM proxy configuration + docker-compose service, and backend LiteLLM client plumbing for per-user virtual keys.
• Add /users/me/budget and a real budget progress bar in the sidebar.
• Add a model selector in chat and return/display model_used (including local fallback cases).

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
litellm.env.example	Example env vars for LiteLLM proxy, providers, Langfuse, and DB persistence.
litellm_config.yaml	LiteLLM model aliases, Langfuse callback, and proxy database/master key settings.
docker-compose.yml	Adds a LiteLLM proxy service container.
.gitignore	Ignores local qdrant storage and litellm.env.
backend/.env.example	Documents backend LiteLLM configuration defaults.
backend/app/core/config.py	Adds LiteLLM-related settings (URL, master key, default budget, duration).
backend/app/core/litellm.py	Adds an AsyncOpenAI client configured to talk to LiteLLM with per-user keys.
backend/app/core/singleton.py	Refactors the OpenAI singleton into an “embeddings/document” client.
backend/app/services/litellm_service.py	Provisions LiteLLM users/virtual keys and fetches per-user budget info.
backend/app/services/users_service.py	Ensures users are provisioned in LiteLLM during /users/me sync.
backend/app/controllers/users_controller.py	Adds controller for fetching current user budget details.
backend/app/routes/users_router.py	Adds GET /users/me/budget and types /users/me response.
backend/app/schemas/users_schema.py	Adds response models for user sync and budget API.
backend/app/schemas/llms_schema.py	Adds typed request/response models for chat queries and model_used.
backend/app/services/llms_service.py	Routes chat/tooling calls through LiteLLM and implements budget-based local fallback.
backend/app/controllers/llms_controller.py	Resolves per-user virtual keys and returns model_used for chat.
backend/app/routes/llms_router.py	Updates /query to accept ChatRequest, adds model param to analysis endpoints.
backend/app/services/knowledge_base_service.py	Switches embeddings/doc extraction to the refactored embeddings client.
frontend/components/ChatBox.tsx	Adds model selector and displays “Responded via Local LLM” when applicable.
frontend/components/app-sidebar.tsx	Fetches and displays real per-user budget usage/remaining.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jcschaff]

please revisit observability for local llm calls, approved.

[androemeda]

please revisit observability for local llm calls, approved.

Sure, thanks!