Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion Package.resolved

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ let package = Package(
.trait(name: "CoreML"),
.trait(name: "MLX"),
.trait(name: "Llama"),
.trait(name: "LiteRT"),
.trait(name: "AsyncHTTPClient"),
.default(enabledTraits: []),
],
Expand All @@ -43,6 +44,7 @@ let package = Package(
.package(url: "https://github.com/mattt/llama.swift", .upToNextMajor(from: "2.7484.0")),
.package(url: "https://github.com/mattt/PartialJSONDecoder", from: "1.0.0"),
.package(url: "https://github.com/ml-explore/mlx-swift-lm", from: "3.0.0"),
.package(url: "https://github.com/john-rocky/swift-litert-lm", from: "0.1.1"),
.package(url: "https://github.com/swiftlang/swift-syntax", from: "602.0.0"),
.package(url: "https://github.com/swift-server/async-http-client.git", from: "1.24.0"),
],
Expand Down Expand Up @@ -94,6 +96,11 @@ let package = Package(
package: "llama.swift",
condition: .when(traits: ["Llama"])
),
.product(
name: "LiteRTFoundation",
package: "swift-litert-lm",
condition: .when(platforms: [.iOS, .macOS], traits: ["LiteRT"])
),
.product(
name: "AsyncHTTPClient",
package: "async-http-client",
Expand Down
38 changes: 37 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ session.toolExecutionDelegate = ToolExecutionObserver()
- [x] [Core ML](https://developer.apple.com/documentation/coreml) models
- [x] [MLX](https://github.com/ml-explore/mlx-swift) models
- [x] [llama.cpp](https://github.com/ggml-org/llama.cpp) (GGUF models)
- [x] [LiteRT-LM](https://github.com/google-ai-edge/litert-lm) (`.litertlm` models, via [swift-litert-lm](https://github.com/john-rocky/swift-litert-lm))
- [x] Ollama [HTTP API](https://github.com/ollama/ollama/blob/main/docs/api.md)
- [x] Anthropic [Messages API](https://docs.claude.com/en/api/messages)
- [x] Google [Gemini API](https://ai.google.dev/api/generate-content)
Expand Down Expand Up @@ -117,6 +118,8 @@ This results in smaller binary sizes and faster build times.
(depends on `ml-explore/mlx-swift-lm`)
- `Llama`: Enables llama.cpp support
(requires `mattt/llama.swift`)
- `LiteRT`: Enables LiteRT-LM support for Gemma 4 and other `.litertlm` models
(requires `john-rocky/swift-litert-lm`; iOS and macOS only)

By default, no traits are enabled.
To enable specific traits, specify them in your package's dependencies:
Expand Down Expand Up @@ -366,13 +369,14 @@ Image support varies by provider:
| Core ML | — |
| MLX | model-dependent |
| llama.cpp | — |
| LiteRT-LM | model-dependent |
| Ollama | model-dependent |
| OpenAI | yes |
| Open Responses | yes |
| Anthropic | yes |
| Google Gemini | yes |

For MLX and Ollama,
For MLX, LiteRT-LM, and Ollama,
use a vision-capable model
(for example, a VLM or `-vl` variant).

Expand Down Expand Up @@ -610,6 +614,38 @@ let response = try await session.respond(
)
```

### LiteRT-LM

Runs `.litertlm` models (for example, Gemma 4) fully on-device via Google's
[LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime with Metal GPU
acceleration (requires `LiteRT` trait; iOS and macOS only):

```swift
let model = LiteRTLanguageModel(model: .gemma4_E2B)

let session = LanguageModelSession(model: model)
let response = try await session.respond(to: "What is the capital of France?")
```

The `.litertlm` file is downloaded from Hugging Face on first use and cached
under Application Support. You can also load any Hugging Face repo hosting a
`.litertlm`, or a local file:

```swift
// Any Hugging Face repo
let model = LiteRTLanguageModel(
huggingFaceRepo: "litert-community/gemma-4-E4B-it-litert-lm",
fileName: "gemma-4-E4B-it.litertlm")

// Local file (for example, a fine-tuned model)
let model = LiteRTLanguageModel(modelFileURL: url)
```

Image inputs are supported for models that ship a vision tower (pass
`modalities: .textImage`). Structured generation is prompt-driven — the JSON
schema is included in the prompt and the response is parsed. Tool calling is
supported for `respond` (not yet for streaming).

### Ollama

Run models locally via Ollama's
Expand Down
Loading