Add LiteRT trait and LiteRTLanguageModel backend#175
Open
john-rocky wants to merge 1 commit into
Open
Conversation
Runs .litertlm models (e.g. Gemma 4) fully on-device via Google's LiteRT-LM runtime, with Metal GPU acceleration on iOS and macOS. - LiteRT package trait, gating a swift-litert-lm dependency to iOS/macOS; default builds are unaffected - LiteRTLanguageModel: respond/streamResponse, image inputs for models with a vision tower, structured generation via schema-in-prompt + JSON extraction, and prompt-driven tool calling for respond (with the ToolExecutionDecision delegate flow) - Env-gated tests (LITERT_TEST_MODEL), README section and provider table updates
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #174: a
LiteRTpackage trait and aLiteRTLanguageModelbackend that runs.litertlmmodels (Gemma 4 E2B/E4B, or any.litertlmon the Hub) fully on-device via Google's LiteRT-LM runtime, with Metal GPU acceleration on iOS and macOS.What's included
Package.swift: aLiteRTtrait; thejohn-rocky/swift-litert-lmdependency is gated by.when(platforms: [.iOS, .macOS], traits: ["LiteRT"]), so default builds and other platforms are unaffected.LiteRTLanguageModel(behind#if LiteRT), following the existing backend patterns:.litertlm, or a local file. Engine bring-up is lazy (first request orprewarm), matching howMLXLanguageModeldefers loading..instructions+ prior turns asinitialMessages+ trigger message), so multi-turn sessions and multiple sessions per model instance behave correctly.respond/streamResponsefor text; image segments (data, file URL, or fetched remote URL) for models with a vision tower.includeSchemaInPrompt).respond: prompt-driven tool descriptions, tool-call parsing, and the standardToolExecutionDecisiondelegate flow, with the result fed back for a final answer (capped at 4 rounds).GenerationOptionsmapping:temperature,.greedy,.topK,.nucleus→ LiteRT-LM's sampler config.LITERT_TEST_MODEL=/path/to/model.litertlm), same spirit as the MLX suite — the weights are multi-GB, so they don't run in CI.Verification
swift build --traits LiteRT— builds clean (no warnings) on macOS arm64.swift build(default traits) — unaffected; the new file is fully#if LiteRT-gated and the dependency isn't resolved into the build graph without the trait.swift format lint— clean on the new files.Known limitations (documented in the README section)
respond-only for now (same starting point as MLX, cf. MLX: support tool calling duringstreamResponse(currentlyrespond()-only) #164).maximumResponseTokensisn't mapped (LiteRT-LM sets the token budget at engine bring-up, not per request).Transcript.Segmenthas no audio case yet, so this backend is text + image.