diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 2d5890ee..596a0529 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -1,203 +1,7 @@ -# Claude Instructions for Cotabby +# Claude Code Instructions -Cotabby is a community-driven open-source macOS menu bar app that provides -on-device inline autocomplete in any text field. It watches Accessibility focus, -monitors global input, generates a local continuation through Apple Intelligence -or llama.cpp, renders ghost text near the caret, and inserts accepted text when -the user presses `Tab`. +@../AGENTS.md -This is a production app with real users and external contributors. Treat every -change as shipping to end users, not as an exercise. - -## How To Work In This Repo - -- Read the relevant subsystem before editing. Cotabby is stateful, permission-heavy, - and tied to macOS Accessibility behavior, so guessing usually creates regressions. -- Talk through architecture before coding when ownership, lifecycle, or async - cancellation is unclear. -- Diagnose failures step by step before touching code. Many bugs come from stale - focus snapshots, AX timing, permission state, runtime lifecycle, or cancellation. -- Keep changes narrow. Prefer pure helpers in `Support/` before changing services, - coordinators, or SwiftUI views. -- Protect the user's worktree. Do not revert unrelated dirty files. - -## Project Map - -- `Cotabby/App/`: app lifecycle, dependency construction, and coordinators. -- `Cotabby/UI/`: SwiftUI/AppKit presentation such as settings, onboarding, overlays, - and menu-facing state. -- `Cotabby/Services/`: side-effectful boundaries: Accessibility, input monitoring, - screenshots/OCR, llama runtime, permissions, downloads, updates, and insertion. -- `Cotabby/Models/`: shared value types, settings snapshots, state machines, and - protocol contracts. -- `Cotabby/Support/`: pure rules and low-level helper logic that should be easy to - test. -- `CotabbyTests/`: focused tests for prompt rendering, request building, availability, - runtime behavior, and pure state transitions. - -## Key Subsystems - -- App ownership starts in `CotabbyAppEnvironment` and `AppDelegate`. These construct - and retain the long-lived services. SwiftUI views should observe this graph, - not recreate service objects. -- The suggestion state machine lives in `SuggestionCoordinator` plus its extension - files: lifecycle, input, prediction, and acceptance. Keep pure rules out of the - coordinator when they can live in `Support/`. -- Focus comes from `FocusTracker`, `FocusSnapshotResolver`, `AXTextGeometryResolver`, - and `AXHelper`. Treat AX data as eventually consistent and app-specific. -- Visual context flows through `VisualContextCoordinator`, - `ScreenshotContextGenerator`, `ScreenTextExtractor`, and `WindowScreenshotService`. - OCR text is cleaned by the pure `OCRTextHygiene`; there is no model summarization step. -- Runtime generation flows through `SuggestionEngineRouter`, - `FoundationModelSuggestionEngine`, `LlamaSuggestionEngine`, - `LlamaRuntimeManager`, and the serialized `LlamaRuntimeCore` actor. The OSS - (llama.cpp) path drives base models via `BaseCompletionPromptRenderer`; Apple - Foundation Models stays instruct via `FoundationModelPromptRenderer`. - -## Comments - -- Comments should explain why, not what. Explain which invariant a design - protects or which macOS/Swift pitfall it avoids. -- Prefer file-level and type-level `///` comments for new important files/types. -- Add targeted inline comments for tricky lifecycle, `@MainActor`, `Task`, - cancellation, Accessibility/Core Foundation bridging, unsafe pointer work, and - llama.cpp runtime state. -- Avoid comments that merely restate the next line of code. - -## Contributing Workflow - -- External contributors open PRs against `main`. Greptile reviews automatically. -- `Cotabby.xcodeproj` is generated from `project.yml` by XcodeGen and committed to the - repo. `project.yml` is the source of truth. Sources under `Cotabby/` and `CotabbyTests/` - are auto-discovered by folder, so a new file (including a new test) needs no project edit. - Only structural changes (targets, build settings, package dependencies, scheme) require - editing `project.yml` followed by `xcodegen generate`. The `XcodeGen` CI workflow fails the - PR if the committed project drifts from `project.yml`. -- Run SwiftLint before pushing: `swiftlint lint --quiet`. The project config is - in `.swiftlint.yml` (line length 140/200, trailing commas disallowed). -- Wiki lives at https://github.com/FuJacob/Cotabby/wiki for contributor onboarding. - -## GitHub Automation Rules - -- **No Co-Authored-By lines.** Never add `Co-Authored-By` trailers to commits. -- **PRs must use the repo template.** When creating a pull request, read - `.github/PULL_REQUEST_TEMPLATE.md` and fill in every section (Summary, - Validation, Linked issues, Risk / rollout notes). Do not invent your own - format or use a generic body. -- **Issues must use the repo templates.** When opening an issue, read the - matching template in `.github/ISSUE_TEMPLATE/` (bug_report.md or - feature_request.md) and fill in every field. Do not invent your own format. - -## Swift And macOS Expectations - -- UI, AppKit, SwiftUI, and most Accessibility interactions belong on the main actor. -- CPU-heavy work, OCR, screenshots, and llama.cpp generation must not block the UI. -- Keep cancellation and stale-result handling explicit. The focused field can change - while async work is still running. -- Use protocol contracts in `SuggestionSubsystemContracts.swift` when the coordinator - only needs a behavior-shaped dependency. -- Do not show dev-only diagnostics as normal user settings unless the feature is - intentionally productized. - -## Debugging & Logs - -Cotabby ships a structured logging system built for AI-assisted debugging. During development -the app is launched with `-cotabby-debug`, which enables on-disk JSONL sinks in addition to -the always-on Console.app stream. - -**Verbosity floor.** Every handler's level comes from `CotabbyDebugOptions.minimumLogLevel`. The -default is `.info`, which lets swift-log skip per-keystroke `.debug`/`.trace` calls before they -allocate, so they cost nothing in release and do not distort energy measurements. `-cotabby-debug` -raises the floor to `.trace`. `COTABBY_LOG_LEVEL=` overrides it explicitly, -e.g. to get Console `.debug` output without the heavier file/screenshot artifacts. A -`Logging initialized` line at startup records the active `debug_mode`, `min_log_level`, and sink paths. - -**Log file locations** (only populated when `-cotabby-debug` is set): - -- `~/Library/Logs/Cotabby/cotabby.jsonl` — main event stream. One JSON object per line. All - metadata (request IDs, engine names, token counts, latencies, error reasons) is flattened - as top-level fields so it can be filtered with `jq`. -- `~/Library/Logs/Cotabby/llm-io.jsonl` — full LLM prompts and completions, one record per - generation. Shares `request_id` with the main log so a single suggestion can be joined - across files. -- **Dev-identity builds log to `~/Library/Logs/Cotabby Dev/` instead** (same file names). The - `Cotabby Dev` scheme — the daily-driver way to run from Xcode — ships a separate app identity, - and its logs land in the identity's own directory. When debugging a dev-built app, read that - directory first; an apparently silent `~/Library/Logs/Cotabby/` does not mean the flag is off. -- `~/Desktop/cotabby-ax-dump.txt` — most recent Chrome AX tree snapshot. Overwritten on each - Chrome focus change (debounced by focused-element identity). -- Rotated previous logs: `*.jsonl.1` (one-step rotation when a file exceeds 10 MB). - -**Correlation IDs.** Every prediction gets a `request_id` like `req_a3f9k2lq`, stamped on -every log line that touches that request (coordinator state transitions, router selection, -engine generation, LLM I/O capture). To pull the complete history of one suggestion: - -```bash -jq 'select(.request_id == "req_a3f9k2lq")' ~/Library/Logs/Cotabby/cotabby.jsonl -jq 'select(.request_id == "req_a3f9k2lq")' ~/Library/Logs/Cotabby/llm-io.jsonl -``` - -**Useful `jq` recipes:** - -```bash -# Recent errors across the app -jq 'select(.level == "error")' ~/Library/Logs/Cotabby/cotabby.jsonl - -# Llama generations slower than 500 ms -jq 'select(.engine == "llama" and .latency_ms > 500)' ~/Library/Logs/Cotabby/llm-io.jsonl - -# Coordinator state transitions -jq 'select(.category == "suggestion" and .stage != null)' ~/Library/Logs/Cotabby/cotabby.jsonl - -# Runtime model load/decode events -jq 'select(.category == "runtime")' ~/Library/Logs/Cotabby/cotabby.jsonl -``` - -**Symptom → category map** (jump straight to the right filter): - -- Ghost text didn't appear → `suggestion` + `focus` -- Wrong text inserted → look up the request in `llm-io.jsonl`, then walk `suggestion` for - acceptance -- Model won't load / decode fails → `runtime` + `models` -- Permission dialog loop → `app` (permission state transitions) -- Chrome-specific weirdness → start with `~/Desktop/cotabby-ax-dump.txt`, then `focus` -- Wrong backend chosen → `suggestion` router selection log (`engine`, `fallback_engine`) - -**Console.app fallback** (when `-cotabby-debug` wasn't set or the user hasn't relaunched yet): - -```bash -log show --predicate 'subsystem == "com.cotabby.app"' --last 10m -log stream --predicate 'subsystem == "com.cotabby.app"' --level debug -``` - -Note the default verbosity floor is `.info`, so Cotabby's `.debug`/`.trace` lines are not emitted -unless the app was launched with `-cotabby-debug` or `COTABBY_LOG_LEVEL=debug`. The `--level debug` -flag above controls what `log` *displays*, not what Cotabby *emits*. The `.info`, warning, and error -lines (including model-load config and permission transitions) always stream. - -**Rule of thumb.** When a user reports a bug, first `tail` / `jq` the relevant file with the -symptom → category map. Do not ask the user to re-explain symptoms before checking the logs. -If `-cotabby-debug` clearly isn't on (no JSONL files exist), use the `log show` fallback -first; only ask for a relaunch with the flag if the OSLog stream is genuinely insufficient. - -## Validation - -Prefer the narrowest useful validation first, then broaden when the change touches -shared behavior: - -```bash -xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build \ - -derivedDataPath build/DerivedData -xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build-for-testing \ - -derivedDataPath build/DerivedData -``` - -Always pass `-derivedDataPath build/DerivedData` so output lands in the -repo-scoped `build/` (already gitignored) instead of accumulating under -`~/Library/Developer/Xcode/DerivedData/Cotabby-*`, where every build leaves a -fresh multi-GB module cache and SwiftPM checkout that nothing trims. When a -task is done and the build artifacts are no longer needed, `rm -rf -build/DerivedData` before reporting completion. - -Run targeted tests when possible. If app-hosted tests fail because of local signing -or Team ID mismatch, report the exact failure and still run `build-for-testing`. +`AGENTS.md` is the canonical project instruction file. Keep shared architecture, workflow, safety, +debugging, validation, and documentation rules there so Claude Code and other coding agents receive +the same guidance without duplicated copies drifting apart. diff --git a/AGENTS.md b/AGENTS.md index 781d76f6..0069419d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,266 +1,156 @@ -# Cotabby Codex Instructions - -## Project Identity - -Cotabby is a macOS menu bar app for on-device inline autocomplete. The core loop is: - -1. Track the currently focused editable field through Accessibility. -2. Monitor global keyboard input without stealing focus. -3. Decide whether the field, permissions, settings, and runtime are eligible. -4. Build an autocomplete request from the focused text context and optional visual context. -5. Generate locally through Apple Intelligence or llama.cpp. -6. Normalize the model output into a short continuation. -7. Render ghost text near the caret. -8. Insert accepted chunks when the user presses `Tab` while keeping the remaining tail alive. - -Privacy and local-first behavior matter. Do not introduce hosted API dependencies unless the user -explicitly asks for that direction. +# Cotabby Coding-Agent Instructions + +## Source of Truth + +- Code and tests are the final authority for shipping behavior. +- Read `ARCHITECTURE.md` before changing lifecycle, Accessibility, input, generation, context, + insertion, or presentation. It is the tracked ten-minute system map. +- `.internal/` may contain private, gitignored study notes in a local checkout. Use them for deeper + navigation when present, but verify every claim against code and never make tracked documentation + depend on them. +- When a tracked document conflicts with code, follow the code and update the stale document in the + same change. + +## Product Contract + +Cotabby is a macOS menu bar agent for inline autocomplete in other applications. The core loop is: + +1. Resolve a supported focused field through Accessibility. +2. Observe global keyboard input without taking focus. +3. Gate work using permissions, field capability, settings, and runtime state. +4. Build a bounded immutable request from caret text and optional context. +5. Generate through Apple Intelligence, in-process llama.cpp, or a configured OpenAI-compatible + endpoint. +6. Normalize the result into a safe short continuation. +7. Render inline ghost text or a mirror card near the caret. +8. Reconcile typing and insert accepted chunks through configurable shortcuts. + +Cotabby is local-first, not unconditionally offline. Apple Intelligence and llama.cpp run on the +Mac. The endpoint backend may send the bounded request to loopback, LAN, or public HTTPS when the +user explicitly selects it. Do not add or broaden hosted transmission without explicit user scope, +clear disclosure, and the existing privacy boundaries. + +Secure and unsupported fields fail closed for generation, presentation, acceptance, and inline +commands. Do not claim they are never acquired: bounded AX context and optional visual work can +currently occur before the secure capability block. Treat that as documented privacy debt until the +acquisition boundary itself changes. ## Learning-First Collaboration -- Explain both the "what" and the "why" for architecture and code changes. -- Assume the user is actively learning Swift, AppKit, Accessibility APIs, llama.cpp integration, - async/await, actor isolation, and macOS app architecture. -- Teach at the file, type, and subsystem level, not just the line level. -- Call out tradeoffs when there are multiple valid approaches. -- Prefer clean boundaries over quick coupling, especially across `App`, `UI`, `Services`, `Models`, - and `Support`. - -When creating or editing a file, explain: - -- what the file is responsible for -- why the file exists as its own boundary -- which objects own it or collaborate with it -- how data flows into and out of it - -When adding a `struct`, `class`, `enum`, actor, or protocol, explain: - -- what responsibility it owns -- what other objects it collaborates with -- why it should exist as its own type instead of being folded into another file -- how long it lives and who owns it - -## Repository Map - -- `Cotabby/App/`: app entrypoint, composition root, lifecycle wiring, and coordinators. -- `Cotabby/UI/`: SwiftUI/AppKit presentation: settings, onboarding, menu views, overlays, and - visual affordances. -- `Cotabby/Services/`: side-effectful boundaries: Accessibility, input monitoring, text insertion, - screenshots/OCR, visual context, llama runtime, permissions, downloads, updates, and launch - services. -- `Cotabby/Models/`: shared value types, settings snapshots, states, domain models, and protocol - contracts. -- `Cotabby/Support/`: pure helper logic, prompt rendering, availability rules, normalization, - reconciliation, geometry helpers, and low-level bridging utilities. -- `CotabbyTests/`: unit and microbench tests. Prefer testing pure `Support/` and `Models/` logic - when possible. -- `CotabbyInference`: the llama.cpp wrapper, consumed as a SwiftPM package - (`github.com/FuJacob/cotabbyinference`, pinned to `main`) rather than vendored in-tree. - -## App Ownership - -Start here when you need to understand lifecycle: +- Explain both what changed and why, including ownership, lifetime, collaborators, and data flow. +- Assume the user is learning Swift, AppKit, Accessibility, async/await, actor isolation, llama.cpp, + and macOS app architecture. +- Teach at the file, type, and subsystem level. Call out tradeoffs when several designs are valid. +- Prefer clean boundaries over quick coupling across `App`, `UI`, `Services`, `Models`, and + `Support`. +- Comments should explain an invariant, ownership rule, macOS quirk, or failure mode. Do not add + comments that merely narrate the next line. + +## Repository and Ownership Map + +- `Cotabby/App/`: composition root, lifecycle wiring, and coordinators. +- `Cotabby/UI/`: SwiftUI views for menus, settings, onboarding, overlays, and inline features. +- `Cotabby/Services/`: side effects and OS/runtime boundaries for focus, input, insertion, visual + capture, inference, model management, permissions, power, presentation, spelling, and updates. +- `Cotabby/Models/`: domain values, settings, states, and protocol contracts. +- `Cotabby/Support/`: deterministic policy, prompting, normalization, reconciliation, geometry, + sanitization, logging, and low-level bridging helpers. +- `CotabbyTests/`: unit and microbench tests. Prefer testing pure Models and Support behavior. +- `CotabbyInference`: external SwiftPM llama.cpp wrapper pinned to its `main` branch. +- `project.yml`: XcodeGen source of truth. `Cotabby.xcodeproj` is generated and committed. + +For app ownership, read in order: 1. `Cotabby/App/Core/CotabbyApp.swift` 2. `Cotabby/App/Core/AppDelegate.swift` 3. `Cotabby/App/Core/CotabbyAppEnvironment.swift` -`CotabbyAppEnvironment` builds the long-lived dependency graph once. `AppDelegate` starts, stops, -and wires cross-subsystem subscriptions. SwiftUI views should observe objects from that graph -rather than creating services directly. - -This ownership rule prevents duplicate Accessibility observers, duplicate input monitors, runtime -reload races, and mismatched settings state. - -## Suggestion Pipeline - -Read the coordinator in this order: - -1. `Cotabby/App/Coordinators/SuggestionCoordinator.swift` -2. `Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift` -3. `Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift` -4. `Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift` -5. `Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift` - -The coordinator owns orchestration and user-facing state. It should not absorb every rule. Prefer: - -- `SuggestionRequestFactory` for pure request construction -- `SuggestionAvailabilityEvaluator` for pure gating decisions -- `SuggestionSessionReconciler` for acceptance and active-tail reconciliation -- `SuggestionTextNormalizer` for backend-independent output cleanup -- `SuggestionWorkController` for generation task identity/cancellation -- `SuggestionInteractionState` for active suggestion session storage - -This split matters because autocomplete is a state machine. Pure rules are easier to test and reason -about than coordinator mutations. - -## Focus And Accessibility - -Focus and geometry live in: - -- `FocusTracker`: observes focus/value/selection changes and publishes snapshots. -- `FocusSnapshotResolver`: reduces raw AX elements into Cotabby-supported focus snapshots. -- `AXTextGeometryResolver`: resolves caret and input geometry. -- `AXHelper`: low-level Accessibility/Core Foundation helper calls. -- `FocusModels`: pure focus values, identities, capabilities, and debug inspection data. - -Accessibility data is eventually consistent and app-specific. Browser editors, Electron apps, -native AppKit fields, and secure fields expose different AX shapes. Preserve stale-result guards, -`focusChangeSequence`, and capability checks unless the change explicitly replaces them. - -## Visual Context And OCR - -Visual context currently flows through: - -- `VisualContextCoordinator`: field-scoped visual-context session lifecycle. -- `ScreenshotContextGenerator`: screenshot -> OCR -> `OCRTextHygiene` cleanup -> bounded excerpt. -- `WindowScreenshotService`: captures the relevant window or region. -- `ScreenTextExtractor`: Vision OCR extraction, carrying per-line recognition confidence. -- `OCRTextHygiene`: pure cleanup of raw OCR (drops low-confidence lines and chrome noise). There is - no model summarization step; a base model conditions fine on cleaned raw context. -- `VisualContextModels`: configuration, status, and excerpt values. - -Do not put raw screenshots, unbounded OCR dumps, or noisy AX tree text directly into prompts. -Normalize, bound, and mark unavailable states explicitly. Screen Recording permission is separate -from Accessibility and Input Monitoring. - -## Runtime And Prompting - -Runtime generation is split by responsibility: - -- `SuggestionEngineRouter`: selects Apple Intelligence vs Open Source. -- `FoundationModelSuggestionEngine`: Apple on-device generation path. -- `LlamaSuggestionEngine`: request-to-prompt, llama result handling, and cache reset handoff. -- `LlamaRuntimeManager`: UI-facing runtime state, model selection, warmup, and lifecycle control. -- `LlamaRuntimeCore`: serialized actor around mutable llama.cpp pointers, prompt tokenization, - KV-cache reuse, sampling, an optional deterministic constrained decoder - (`runConstrainedDecode`, gated behind the default-off `cotabbyConstrainedDecoderEnabled`), and - shutdown. -- `BaseCompletionPromptRenderer`: prompt construction for the Open Source path. The llama models are - now *base* (non-instruct) GGUFs, so this renders a pure text continuation: no instruction preamble, - custom rules and context fold into a short conditioning preface (a base model conditions on - description, it does not obey commands), sections are character-budgeted via `PromptSectionBudget`, - and the caret prefix comes last. `FoundationModelPromptRenderer` stays instruct-shaped because - Apple's Foundation Models path gives us a first-class instructions channel. - -Keep llama.cpp pointer work serialized inside `LlamaRuntimeCore`. The manager should publish state; -the core should own native correctness. - -## UI And Overlays - -- `OverlayController` owns the ghost-text panel lifecycle and positioning. -- `SuggestionOverlayPresenter` decides whether a suggestion should be shown or hidden. -- `ActivationIndicatorController` owns the optional caret/field-edge indicator. -- `FocusDebugOverlayController` is for developer visibility and should stay gated behind debug - options, not normal user settings. -- Settings panes (under `Cotabby/UI/Settings/Panes/`) and onboarding views should remain - presentation-focused. Push behavior into services, models, or support helpers. - -## Swift And Concurrency Rules - -- Use `@MainActor` for UI, AppKit, SwiftUI state, most Accessibility access, and published models. -- Use actors or explicit serialization for mutable native/runtime state. -- Do not block the main actor with OCR, screenshots, model loading, or generation. -- Make cancellation and stale-result checks explicit around async work. The user can keep typing, - switch apps, focus another field, or accept a partial suggestion while work is still running. -- Prefer narrow protocols from `SuggestionSubsystemContracts.swift` when the coordinator only needs - behavior, not a concrete service. -- Treat Core Foundation and AX bridging as unsafe boundaries. Add comments that explain ownership, - casting, and failure handling. - -## Teaching Comment Standard - -- Add real teaching comments, not labels. -- Prefer file-level and type-level `///` comments that explain purpose, ownership, and design. -- Add targeted inline comments for tricky lifecycle behavior, concurrency, cancellation, AX timing, - Core Foundation bridging, native pointer state, and macOS quirks. -- Comments should explain why the code is written this way, which invariant it protects, or which - pitfall it avoids. -- Avoid useless comments that merely restate the code. -- If Swift syntax is likely to be unfamiliar, annotate it briefly the first time it appears in a new - concept-heavy area. Examples: `@Published`, `@ObservedObject`, `@StateObject`, `@MainActor`, - `Task`, async/await, actor isolation, closures, convenience initializers, `AXUIElement`, - `CFTypeRef`, and `unsafeBitCast`. +`CotabbyAppEnvironment` constructs and retains one app-lifetime dependency graph. `AppDelegate` +starts and stops side effects and owns process-lifecycle subscriptions. Views observe shared objects; +they do not create process-wide services. Construction does not imply startup, production services +do not start in the XCTest host, and shutdown stops new work before native teardown. + +For the suggestion state machine, read: + +1. `SuggestionCoordinator.swift` +2. `SuggestionCoordinator+Lifecycle.swift` +3. `SuggestionCoordinator+Input.swift` +4. `SuggestionCoordinator+Prediction.swift` +5. `SuggestionCoordinator+Acceptance.swift` + +Keep orchestration in the coordinator and pure rules in focused owners such as +`SuggestionAvailabilityEvaluator`, `SuggestionRequestFactory`, `SuggestionSessionReconciler`, and +`SuggestionTextNormalizer`. `SuggestionWorkController` owns work identity and cancellation; +`SuggestionInteractionState` owns the active suggestion session. + +## Reliability and Concurrency Rules + +- Treat AX state as eventually consistent and application-specific. Polling is authoritative. +- Revalidate work ID, focus/content signatures, settings continuity, and session state after every + await that can outlive them. AX element identity alone is not a stale-result guard. +- A generation request is immutable once asynchronous work starts. Engines never reach back into + live editor state. +- Steady input observation is listen-only. Consume a key only after the owning feature succeeds; + stale or rejected interception must fail open to the host application. +- Keep overlay text equal to the active session tail before acceptance. Host publication after + insertion is reconciled, never assumed. +- Use Unicode key events for short ordinary insertion, the IME-safe paste path during active + composition, and preserve clipboard contents unless newer user clipboard activity wins. +- Bound and sanitize every context source before request assembly. Never put raw screenshots, + unbounded OCR, or noisy AX dumps into prompts. +- Use `@MainActor` for UI, AppKit, published state, and most AX access. OCR, capture, downloads, + model loading, endpoint calls, and generation must not block it. +- Use actors or explicit serialization for mutable non-UI state. `LlamaRuntimeCore` is not a Swift + actor: its native pointers, cache/decode state, and shutdown lifecycle are protected by explicit + locks and a condition. Do not weaken that serialization. +- Context reset must prevent cache data from crossing interaction boundaries. Switching away from + the Open Source engine must release native runtime resources. +- Endpoint credentials belong in Keychain. Preserve rejection of insecure public HTTP. +- Cancellation is expected lifecycle behavior, not automatically a backend failure. ## Change Strategy -Prefer this order when changing behavior: - -1. Pure rules in `Support/` -2. Domain models and contracts in `Models/` -3. Service boundary behavior in `Services/` -4. Coordinator orchestration in `App/` -5. SwiftUI/AppKit presentation in `UI/` - -This order reduces regression risk because deterministic code changes before stateful orchestration. -It also creates better tests. - -## Debugging & Logs - -Cotabby has a structured logging system built for AI-assisted debugging. During development the app -is launched with `-cotabby-debug`, which enables on-disk JSONL sinks in addition to the always-on -Console.app stream. - -**Log file locations** (only populated when `-cotabby-debug` is set): - -- `~/Library/Logs/Cotabby/cotabby.jsonl` — main event stream. One JSON object per line, with all - metadata flattened as top-level fields so it can be filtered with `jq`. -- `~/Library/Logs/Cotabby/llm-io.jsonl` — full LLM prompts and completions, one record per - generation. Shares `request_id` with the main log so a single suggestion can be joined across - files. -- `~/Desktop/cotabby-ax-dump.txt` — most recent Chrome AX tree snapshot. Overwritten on each - Chrome focus change (debounced by focused-element identity). -- Rotated previous logs: `*.jsonl.1` (one-step rotation when a file exceeds 10 MB). +Prefer this dependency direction when changing behavior: -**Correlation IDs.** Every prediction gets a `request_id` like `req_a3f9k2lq`, stamped on every log -line touching that request (coordinator state transitions, router selection, engine generation, LLM -I/O capture). Pull a complete history of one suggestion: +1. Pure policy in `Support/` +2. Domain values and contracts in `Models/` +3. Side-effectful boundaries in `Services/` +4. Orchestration in `App/` +5. Presentation in `UI/` -```bash -jq 'select(.request_id == "req_a3f9k2lq")' ~/Library/Logs/Cotabby/cotabby.jsonl -jq 'select(.request_id == "req_a3f9k2lq")' ~/Library/Logs/Cotabby/llm-io.jsonl -``` - -**Useful `jq` recipes:** - -```bash -# Recent errors across the app -jq 'select(.level == "error")' ~/Library/Logs/Cotabby/cotabby.jsonl +This is not a requirement to touch every layer. It prevents deterministic rules from being buried in +stateful coordinators or views. Prefer narrow protocols from `SuggestionSubsystemContracts.swift` +when a coordinator needs behavior rather than a concrete service. -# Llama generations slower than 500 ms -jq 'select(.engine == "llama" and .latency_ms > 500)' ~/Library/Logs/Cotabby/llm-io.jsonl +Treat Core Foundation, AX bridging, and native pointers as unsafe boundaries. Explain ownership, +casting, lifetime, cancellation, and failure handling where they are not obvious. -# Coordinator state transitions -jq 'select(.category == "suggestion" and .stage != null)' ~/Library/Logs/Cotabby/cotabby.jsonl +## Debugging and Logs -# Runtime model load/decode events -jq 'select(.category == "runtime")' ~/Library/Logs/Cotabby/cotabby.jsonl -``` - -**Symptom → category map:** +When a bug is reported, inspect existing diagnostics before asking the user to reproduce or +re-explain it. `-cotabby-debug` enables privacy-sensitive JSONL and capture artifacts; the default +log floor is `.info`, while debug mode raises it to `.trace`. `COTABBY_LOG_LEVEL` overrides the floor. -- Ghost text didn't appear → `suggestion` + `focus` -- Wrong text inserted → look up the request in `llm-io.jsonl`, then walk `suggestion` for - acceptance -- Model won't load / decode fails → `runtime` + `models` -- Permission dialog loop → `app` (permission state transitions) -- Chrome-specific weirdness → start with `~/Desktop/cotabby-ax-dump.txt`, then `focus` -- Wrong backend chosen → `suggestion` router selection log (`engine`, `fallback_engine`) +- Production: `~/Library/Logs/Cotabby/cotabby.jsonl` and `llm-io.jsonl` +- Dev identity: `~/Library/Logs/Cotabby Dev/cotabby.jsonl` and `llm-io.jsonl` +- Chrome AX snapshot: `~/Desktop/cotabby-ax-dump.txt` +- Visual debug captures: `~/Desktop/cotabby-debug-screenshots/` -**Console.app fallback** (when `-cotabby-debug` wasn't set): +Use `request_id` to join suggestion and LLM-I/O events. Start with `focus` for field/geometry issues, +`suggestion` for state/acceptance, `runtime` and `models` for backend failures, and `app` for +permissions/lifecycle. If JSONL files do not exist, use unified logging: ```bash log show --predicate 'subsystem == "com.cotabby.app"' --last 10m log stream --predicate 'subsystem == "com.cotabby.app"' --level debug ``` -**Rule of thumb.** When a user reports a bug, first `tail` / `jq` the relevant file with the -symptom → category map. Do not ask the user to re-explain symptoms before checking the logs. +The `log` display level cannot recover `.debug` or `.trace` events that Cotabby's emission floor +already skipped. ## Validation -Use the narrowest meaningful validation first, then broaden if the change touches shared behavior. -Common commands: +Use the narrowest meaningful test first, then broaden for shared behavior: ```bash xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build \ @@ -269,20 +159,28 @@ xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=mac -derivedDataPath build/DerivedData ``` -Always pass `-derivedDataPath build/DerivedData` so the output lands in the repo-scoped `build/` -directory (already gitignored) instead of accumulating under -`~/Library/Developer/Xcode/DerivedData/Cotabby-*`, where every build leaves a fresh multi-GB module -cache and SwiftPM checkout that nothing trims. When a task is done and the artifacts are no longer -needed, `rm -rf build/DerivedData` before reporting completion. - -Run targeted tests for changed pure logic when available. If `xcodebuild test` fails locally because -of app-hosted test bundle signing or Team ID mismatch, report the exact failure and still provide the -successful build/build-for-testing result. - -## Git And Worktree Safety - -- The worktree may already contain user edits. Never revert unrelated changes. -- Before editing, inspect `git status -sb` and the relevant files. -- Keep commits scoped. Do not silently include unrelated dirty files. -- Avoid destructive commands such as `git reset --hard` or `git checkout --` unless the user - explicitly asks for that operation. +- Run focused tests for changed pure logic and `swiftlint lint --quiet` for Swift changes. +- If app-hosted tests fail because of signing or Team ID mismatch, report the exact failure and the + successful build/build-for-testing result. +- Remove `build/DerivedData` after the artifacts are no longer needed. +- Sources are recursively discovered. Edit `project.yml` and run `xcodegen generate` only for + structural target, build-setting, package, or scheme changes. CI regenerates the project and fails + on drift. + +## Documentation Maintenance + +- Update `ARCHITECTURE.md` when the ten-minute mental model, ownership, data flow, privacy boundary, + backend set, or reliability invariant changes. +- Keep links resolvable and name concrete owners. Do not document planned behavior as shipping. +- Update setup, release, permission, and contributor docs when their contracts change. +- If private `.internal/` guides exist, update the affected guide for local study, but remember those + files are gitignored and cannot carry a public PR's explanation. + +## Git and GitHub Safety + +- Inspect `git status -sb` and relevant diffs before editing. Preserve unrelated worktree changes. +- Stage only intended paths and keep commits scoped. Never use destructive reset/checkout commands + unless the user explicitly requests them. +- Do not add `Co-Authored-By` trailers. +- Pull requests must use `.github/PULL_REQUEST_TEMPLATE.md` and report actual validation. +- Issues must use the matching template under `.github/ISSUE_TEMPLATE/`. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 715b831e..dff6703a 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,101 +1,393 @@ # Cotabby Architecture -This document is the maintainer map for Cotabby. Read this before making changes to the suggestion -pipeline, Accessibility integration, or runtime lifecycle. - -If you are new to Swift or macOS APIs, treat this file as the system-level map and follow the -linked source files in order rather than hunting through the project tree at random. - -## System Shape - -Cotabby is a macOS menu bar app with one long-lived dependency graph and one main product loop: - -1. Resolve the currently focused editable field through Accessibility. -2. Watch keyboard input globally. -3. Decide whether a suggestion should be requested. -4. Ask the local llama runtime for a continuation. -5. Render ghost text near the caret. -6. Reconcile live typing against the active suggestion session. -7. Insert accepted text back into the host app when the user presses `Tab`. - -The key design rule is separation by responsibility: - -- `Cotabby/App/`: lifecycle owners and composition root. -- `Cotabby/UI/`: SwiftUI presentation only. -- `Cotabby/Services/`: side effects, async work, and OS/runtime boundaries. -- `Cotabby/Models/`: shared value types and contracts. -- `Cotabby/Support/`: pure rules and low-level bridging helpers. - -## Lifecycle Ownership - -Start with these files in order: - -1. `Cotabby/App/Core/CotabbyApp.swift` -2. `Cotabby/App/Core/AppDelegate.swift` -3. `Cotabby/App/Core/CotabbyAppEnvironment.swift` - -`CotabbyAppEnvironment` builds the long-lived object graph once. `AppDelegate` owns app lifecycle and cross-subsystem subscriptions. SwiftUI views observe those objects; they do not create them. - -This is similar to a React app with a root provider tree plus a small top-level controller that wires subscriptions and startup behavior. - -## Suggestion Pipeline - -The suggestion subsystem is centered on `SuggestionCoordinator`, but it is no longer intended to be read as one giant file. +This is the ten-minute maintainer map for Cotabby. It explains the product loop, ownership +boundaries, reliability rules, and the best files to read before changing behavior. It is intentionally +a roadmap rather than an encyclopedia. + +## What Cotabby Is + +Cotabby is a macOS menu bar agent that provides inline autocomplete in other applications: + +1. Find the focused editable field through macOS Accessibility. +2. Observe global keyboard input without taking focus. +3. Gate work using permissions, field capability, settings, and runtime state. +4. Build a bounded request from caret text and optional context. +5. Generate through Apple Intelligence, an in-process llama.cpp model, or a user-configured + OpenAI-compatible endpoint. +6. Normalize the result into a safe short continuation. +7. Render inline ghost text or a mirror card near the caret. +8. Reconcile typing against the active suggestion and insert accepted chunks through configurable + shortcuts. + +The product is local-first, not unconditionally offline. Apple Intelligence and the bundled +open-source path run on the Mac. An endpoint can be loopback, on the local network, or a public HTTPS +service; when selected, the bounded request is sent to that server. + +## Architectural Constraints + +These rules explain most of the structure: + +- There is one app-lifetime dependency graph. Views never create process-wide services. +- Accessibility state is eventually consistent and app-specific. Every async result can be stale. +- Generation, presentation, and insertion fail closed for secure and unsupported fields. Early + acquisition has a current secure-field caveat described under privacy below. +- User text and optional context are bounded before generation. +- On-device work stays local unless the user explicitly selects an endpoint engine. +- Global input is observed in a fail-open way; Cotabby consumes only events it successfully handles. +- MainActor owns UI, published state, AppKit, and most AX access. OCR, downloads, and generation do + not block it. +- Mutable native llama state is explicitly serialized and released before process teardown. +- Pure policy belongs outside coordinators so the state machine remains testable. + +## Repository Map + +- [Cotabby/App](Cotabby/App): application entry point, composition root, lifecycle, and coordinators. +- [Cotabby/UI](Cotabby/UI): SwiftUI and AppKit-facing presentation for settings, onboarding, menus, + previews, and user surfaces. +- [Cotabby/Services](Cotabby/Services): side-effectful boundaries for AX, event taps, insertion, + capture/OCR, generation, downloads, permissions, updates, and AppKit panels. +- [Cotabby/Models](Cotabby/Models): shared values, settings, states, configuration, and protocol + contracts. +- [Cotabby/Support](Cotabby/Support): deterministic rules, prompt rendering, normalization, + reconciliation, layout, and low-level bridging helpers. +- [CotabbyTests](CotabbyTests): unit tests and microbenchmarks, with emphasis on pure Support and + Models behavior. +- CotabbyInference: the llama.cpp Swift wrapper consumed from an external SwiftPM package pinned to + its main branch; native code is not vendored here. + +Folder names describe the dominant responsibility, not the UI framework. Cotabby/UI contains +SwiftUI views, while AppKit panel/window controllers live mostly under Cotabby/Services/UI or app +coordinators because they own process-level presentation behavior. + +## End-to-End Data Flow + +~~~text +CGEvent and focus poll + -> FocusTracker + -> FocusSnapshotResolver + AXTextGeometryResolver + -> FocusSnapshot + -> SuggestionCoordinator + -> availability and native correction + -> debounce + current work identity + -> SuggestionRequestFactory + -> bounded AX / surface / clipboard / visual / user context + -> SuggestionEngineRouter + -> Apple | in-process llama | OpenAI-compatible endpoint + -> SuggestionTextNormalizer + seam guards + -> SuggestionInteractionState + -> SuggestionOverlayPresenter -> OverlayController + -> typing reconciliation or configured acceptance + -> SuggestionInserter + -> host AX publication check and next prediction +~~~ + +The request is immutable after generation begins. Engines do not reach back into live AX state. +Before a partial, final, or insertion applies, the coordinator verifies current work identity, focus, +content signatures, settings continuity, and session state. + +## Lifecycle and Ownership + +Read these first: + +1. [CotabbyApp.swift](Cotabby/App/Core/CotabbyApp.swift) +2. [AppDelegate.swift](Cotabby/App/Core/AppDelegate.swift) +3. [CotabbyAppEnvironment.swift](Cotabby/App/Core/CotabbyAppEnvironment.swift) + +| Owner | Responsibility | Lifetime | +| --- | --- | --- | +| CotabbyApp | SwiftUI scenes, MenuBarExtra, AppDelegate bridge | Process | +| CotabbyAppEnvironment | Construct the shared object graph and retain graph-internal subscriptions | Process | +| AppDelegate | Start/stop services and retain lifecycle-driven subscriptions | Process | +| Coordinators | Orchestrate one product surface across services | App or window | +| Services | Own one side effect, OS boundary, or mutable subsystem | Injected | +| Views | Render shared state and send narrow user intents | SwiftUI/AppKit surface | + +AppDelegate starts the selected runtime, focus polling, input monitoring, updates, suggestion +coordination, inline commands, and onboarding after launch. It stops new work before tearing down +global taps, polling, and native resources at termination. Production service startup is skipped in +the XCTest host. + +CotabbyAppEnvironment also owns important subscriptions: focus cadence, global-toggle tap binding, +power-profile application, engine/model selection, and endpoint connection invalidation. AppDelegate +owns permission reactions, engine runtime start/stop, overlays, model-directory refresh, and +process-lifecycle behavior. Both retain subscriptions because both own different relationships. + +[SuggestionSettingsModel.swift](Cotabby/Models/SuggestionSettingsModel.swift) is the published source +of app behavior. [SuggestionSettingsStore.swift](Cotabby/Support/SuggestionSettingsStore.swift) +persists non-secret preferences in UserDefaults. Endpoint credentials live in Keychain. + +## Suggestion State Machine Read the coordinator in this order: -1. `Cotabby/App/Coordinators/SuggestionCoordinator.swift` -2. `Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift` -3. `Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift` -4. `Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift` -5. `Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift` - -The coordinator owns: - -- published UI/debug state -- top-level orchestration -- debounce/generation task ownership through `SuggestionWorkController` -- active suggestion session ownership through `SuggestionInteractionState` -- overlay/insertion/logging decisions - -The coordinator should not own pure decision rules or low-level OS logic. Those live elsewhere: - -- `Cotabby/Support/SuggestionRequestFactory.swift`: pure request building -- `Cotabby/Support/SuggestionSessionReconciler.swift`: pure session and acceptance rules -- `Cotabby/Support/SuggestionAvailabilityEvaluator.swift`: pure gating logic -- `Cotabby/Services/Visual/VisualContextCoordinator.swift`: screenshot/OCR lifecycle; OCR text is cleaned by the pure `OCRTextHygiene` filters (no model-summarization step) -- `Cotabby/Services/Runtime/LlamaSuggestionEngine.swift`: prompt/result normalization over the runtime - -## Focus And Accessibility - -Focus detection is a small pipeline of its own: - -1. `FocusTracker` polls on a timer. -2. `FocusSnapshotResolver` walks the AX tree and validates field capability. -3. `AXTextGeometryResolver` computes caret and text geometry. -4. `AXHelper` contains the low-level Core Foundation / Accessibility bridging. - -If the issue is “Cotabby does not recognize this field” or “the ghost text is in the wrong place,” start in those files before touching the coordinator. - -## Runtime And Models - -The local model runtime is intentionally split: - -- `LlamaRuntimeManager`: published bootstrap state and user-facing control flow -- `LlamaRuntimeCore`: serialized low-level runtime work -- `LlamaSuggestionEngine`: suggestion-specific normalization and error mapping - -That split matters because runtime lifecycle concerns change at a different rate than prompt strategy or output cleanup. - -The constrained decoder, beam search, and fill-in-middle prompting ship behind default-off developer flags. +1. [SuggestionCoordinator.swift](Cotabby/App/Coordinators/SuggestionCoordinator.swift) +2. [SuggestionCoordinator+Lifecycle.swift](Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift) +3. [SuggestionCoordinator+Input.swift](Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift) +4. [SuggestionCoordinator+Prediction.swift](Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift) +5. [SuggestionCoordinator+Acceptance.swift](Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift) + +The coordinator owns orchestration and user-facing suggestion state. It delegates rules and mutable +sub-state to smaller boundaries: + +- [SuggestionAvailabilityEvaluator.swift](Cotabby/Support/SuggestionAvailabilityEvaluator.swift): + pure permission, settings, focus, and runtime gates. +- [SuggestionRequestFactory.swift](Cotabby/Support/SuggestionRequestFactory.swift): pure bounded + request construction and prompt preview. +- [SuggestionWorkController.swift](Cotabby/Services/Suggestion/SuggestionWorkController.swift): + debounce/generation tasks and monotonically increasing work IDs. +- [SuggestionInteractionState.swift](Cotabby/Services/Suggestion/SuggestionInteractionState.swift): + active session, materialized context, consumed prefix, and known post-insertion AX lag. +- [SuggestionSessionReconciler.swift](Cotabby/Support/SuggestionSessionReconciler.swift): type-through, + acceptance, and live-host reconciliation. +- [SuggestionTextNormalizer.swift](Cotabby/Support/SuggestionTextNormalizer.swift): backend-independent + cleanup, echo removal, whitespace policy, trailing-text deduplication, and unsafe-output rejection. + +A native correction path runs before model generation. NSSpellChecker and bundled SymSpell indexes +can suppress completion while a likely typo is forming, offer a green atomic replacement, or apply +an opt-in automatic fix after Space. + +Engines can stream cumulative partials. The coordinator coalesces UI work, accepts only monotonic +extensions, and allows a displayed partial to become an active accept-ready session. The final result +remains authoritative and can replace or suppress provisional text. + +Normal sessions support exact type-through, word or phrase acceptance, full-tail acceptance, CJK-aware +segmentation, punctuation handling, optional trailing space, and speculative generation after final +acceptance. Corrections commit atomically rather than exposing partial acceptance. + +## Focus and Accessibility + +Read: + +1. [FocusTracker.swift](Cotabby/Services/Focus/FocusTracker.swift) +2. [FocusSnapshotResolver.swift](Cotabby/Services/Focus/FocusSnapshotResolver.swift) +3. [FocusModels.swift](Cotabby/Models/FocusModels.swift) +4. [AXTextGeometryResolver.swift](Cotabby/Services/Focus/AXTextGeometryResolver.swift) +5. [AXHelper.swift](Cotabby/Support/AXHelper.swift) + +FocusTracker uses timer polling as the authoritative source because AX notifications are inconsistent +across AppKit, browsers, Electron, and custom editors. Activity resets the cadence; idle unchanged +state backs it off. Input and acceptance paths may request an explicit fresh capture, but do not trust +event payloads as complete field state. + +FocusSnapshotResolver finds a usable editable candidate, blocks secure/unsupported surfaces, bounds +text on both sides of the caret, resolves the focused process, and publishes stable domain values. +Chromium/Electron require accessibility priming, cursor hit-test recovery, and out-of-process iframe +handling. All fallbacks are revalidated and yield to a valid system-focused element. + +AXTextGeometryResolver tries direct range bounds, browser text-marker bounds, a nearby measured +character, child static-text runs, and field-frame estimation. Geometry is labeled exact, derived, +estimated, or layoutEstimated; presentation uses that quality rather than pretending every rectangle +is equally trustworthy. + +AX is synchronous cross-process IPC on MainActor. Deep walks are gated, cached, bounded, and +throttled. Calendar has a narrow capture guard because enumerating its transient editor can dismiss +the editor. Async consumers carry focus/content signatures instead of relying on AX element identity +alone. + +## Global Input and Insertion + +[InputMonitor.swift](Cotabby/Services/Input/InputMonitor.swift) owns three event-tap responsibilities: + +- A steady listen-only observer for typing, deletion, navigation, and pointer activity. +- A conditional consuming tap while a suggestion or inline-command capture needs interception. +- A separate consuming tap for the configured global-toggle shortcut. + +The conditional tap consumes a matching key only after the owning coordinator succeeds. Stale taps, +missing overlays, rejected sessions, and revoked permission fail open so the host receives the key. +Word/phrase and full-tail acceptance have independent configurable key/modifier bindings. + +[InputSuppressionController.swift](Cotabby/Services/Input/InputSuppressionController.swift) marks and +counts Cotabby-generated events so insertion does not re-enter the typing pipeline. + +[SuggestionInserter.swift](Cotabby/Services/Suggestion/SuggestionInserter.swift) normally posts short +Unicode key events without touching the clipboard. Active IME composition uses a clipboard paste +commit. A default-off policy can also paste long or multiline chunks. Paste tries the target app's +Accessibility Paste menu item before synthetic Command-V and restores every pasteboard representation +unless newer user clipboard activity wins. + +Replacement sessions delete their validated literal run before inserting correction, emoji, or macro +text. Posting events is not treated as proof of success; the later AX snapshot is reconciled. + +## Engines and Prompting + +[SuggestionEngineRouter.swift](Cotabby/Services/Runtime/SuggestionEngineRouter.swift) selects one of: + +- [FoundationModelSuggestionEngine.swift](Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift) + for Apple Intelligence. It uses the framework's instructions channel, streams cumulative partials, + and keeps a one-use compatible prewarmed session. Unsupported language/locale can fall back to llama. +- [LlamaSuggestionEngine.swift](Cotabby/Services/Runtime/LlamaSuggestionEngine.swift) for an in-process + GGUF base model through CotabbyInference. [LlamaRuntimeManager.swift](Cotabby/Services/Runtime/LlamaRuntimeManager.swift) + publishes state; [LlamaRuntimeCore.swift](Cotabby/Services/Runtime/LlamaRuntimeCore.swift) owns native + pointers, tokenization, KV-cache reuse, prefill, sampling, abort, and shutdown. +- [OpenAICompatibleSuggestionEngine.swift](Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift) + for completion/chat APIs and SSE streams. The default is loopback Ollama at + http://127.0.0.1:11434/v1; LAN and public HTTPS endpoints are supported, while insecure public HTTP + is rejected. + +LlamaRuntimeCore is a nonisolated lock/condition-protected native boundary, not a Swift actor. Its +autocomplete lock serializes cache/decode state, and its lifecycle condition prevents shutdown from +racing active native work. Heavy work runs away from MainActor. + +[BaseCompletionPromptRenderer.swift](Cotabby/Support/BaseCompletionPromptRenderer.swift) renders a +base-model text continuation with optional budgeted context and the caret prefix last. It does not +wrap a base GGUF in an instruction conversation. [FoundationModelPromptRenderer.swift](Cotabby/Support/FoundationModelPromptRenderer.swift) +keeps Apple's instruction-shaped prompt separate. + +Prewarm is opportunistic and goes only to the selected backend. Context reset reaches every backend. +The local runtime is loaded only for the Open Source engine and is released when switching to Apple +or endpoint mode so mapped weights and Metal buffers do not stay resident unnecessarily. + +## Context, Privacy, and Permissions + +[PermissionManager.swift](Cotabby/Services/Permission/PermissionManager.swift) tracks: + +- Accessibility: required for focus, text, capability, geometry, and insertion validation. +- Input Monitoring: required for global keyboard observation and acceptance interception. +- Screen Recording: optional, used only for screenshot-derived context. + +Context sources are independently enabled and bounded: recent AX prefix/trailing text, surface +metadata, user rules/extended context, relevant clipboard content, visual OCR, language, and settings. +[PromptContextSanitizer.swift](Cotabby/Support/PromptContextSanitizer.swift) sanitizes optional text, +and prompt renderers apply per-section budgets. + +[ClipboardContextProvider.swift](Cotabby/Services/Utilities/ClipboardContextProvider.swift) reads a +fresh bounded value at request time rather than recording clipboard history. Relevance and distillation +policies drop unrelated or excessive content. + +Visual context is one field-scoped session: + +~~~text +VisualContextCoordinator + -> WindowScreenshotService (ScreenCaptureKit crop) + -> ScreenTextExtractor (Vision OCR + line confidence) + -> OCRTextHygiene + -> bounded sanitized excerpt +~~~ + +There is no model summarization step and raw screenshots do not enter prompts. Missing Screen Recording +produces an explicit unavailable state without disabling text-only autocomplete. Debug screenshot/OCR +artifacts exist only under the explicit debug launch mode. + +Secure fields cannot schedule generation, show a suggestion, accept text, or open an inline command, +so their context cannot flow into an engine request. The acquisition boundary is currently broader +than that guarantee: FocusSnapshotResolver still creates a bounded context for a secure field before +marking its capability blocked, and visual-capture eligibility deliberately ignores capability so +screenshot/OCR can warm for that field. The excerpt cannot be consumed by prediction, but explicit +debug mode can persist visual captures. Treat this as privacy debt; current architecture guarantees +no secure-field generation, not no secure-field acquisition. + +Endpoint credentials are stored in Keychain. A remote endpoint receives the same bounded request that +would otherwise be used locally, so its privacy scope must remain visible in settings and +documentation. + +## Presentation and Sibling Features + +[SuggestionOverlayPresenter.swift](Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift) +decides presentation actions. [OverlayController.swift](Cotabby/Services/UI/OverlayController.swift) +owns a reusable borderless non-activating NSPanel and SwiftUI-hosted content. + +Automatic presentation uses inline ghost text for exact/derived caret geometry and a mirror card for +estimated/layout-estimated geometry, mid-line editing, or explicit user preference. The overlay can +match host font/color, render corrections distinctly, respect right-to-left and multiline layout, +show an acceptance hint, and advance a partial tail without waiting for noisy AX geometry. + +[ActivationIndicatorController.swift](Cotabby/Services/UI/ActivationIndicatorController.swift) owns +the optional field/caret indicator. [FocusDebugOverlayController.swift](Cotabby/Services/UI/FocusDebugOverlayController.swift) +is developer-only and gated by -cotabby-debug. + +[InlineCommandCoordinator.swift](Cotabby/App/Coordinators/InlineCommandCoordinator.swift) arbitrates +the single input-capture slot between: + +- [EmojiPickerController.swift](Cotabby/App/Coordinators/EmojiPickerController.swift): colon query, + lazy catalog/matcher, non-activating picker, recency/frequency ranking, and literal-run replacement. +- [MacroController.swift](Cotabby/App/Coordinators/MacroController.swift): slash query and deterministic + date, random, unit, currency, and arithmetic evaluation through MacroEngine. + +Both use pure trigger state machines, stay pinned to one supported focus sequence, and cancel on +focus change or incompatible input. They do not call a language model. + +[SettingsCoordinator.swift](Cotabby/App/Coordinators/SettingsCoordinator.swift) and +[WelcomeCoordinator.swift](Cotabby/App/Coordinators/WelcomeCoordinator.swift) own app-lifetime AppKit +windows hosting SwiftUI content. Settings and onboarding observe the shared graph. Hiding the menu bar +icon retains a recovery path to Settings. + +## Concurrency and Reliability Rules + +- Revalidate after every await that can outlive focus, content, settings, or work identity. +- Treat cancellation as expected lifecycle, not automatically as a backend failure. +- Keep AX and AppKit access MainActor-isolated, but bound synchronous AX work aggressively. +- Use actors or explicit serialization for mutable non-UI state; do not move native pointers across + an ownership boundary casually. +- Never use only AX element identity as a stale-result guard. +- Keep overlay text and active-session remaining text equal before acceptance. +- Preserve the narrow known-insertion sentinel; general stale AX tolerance hides real divergence. +- Stop new generation and input work before releasing runtime state at termination. ## Safe Change Order -If you need to change behavior, prefer this order: - -1. Pure logic in `Support/` -2. Service boundary behavior in `Services/` -3. Coordinator orchestration in `App/` -4. SwiftUI presentation in `UI/` - -That order minimizes regression risk because the most deterministic code changes first and the most stateful code changes last. +When behavior changes, prefer: + +1. Pure policy and helpers in Support. +2. Domain values, state, settings, and contracts in Models. +3. Side-effectful boundaries in Services. +4. Orchestration in App. +5. Presentation in UI. + +This is a dependency direction, not a demand to touch every layer. A pure rule should not be added to +SuggestionCoordinator just because that is where its symptom becomes visible. + +## Debugging and Validation + +Development schemes launch with -cotabby-debug. That enables local privacy-sensitive diagnostics in +addition to unified logging: + +- ~/Library/Logs/Cotabby/cotabby.jsonl: structured event stream. +- ~/Library/Logs/Cotabby/llm-io.jsonl: full prompt/completion records. +- ~/Desktop/cotabby-ax-dump.txt: most recent Chrome focus AX tree. +- ~/Desktop/cotabby-debug-screenshots/: retained visual-context capture/OCR pairs. + +Every prediction carries a request_id through coordinator, router, engine, and LLM-I/O records. Start +with category focus for field/geometry failures, suggestion for state/acceptance failures, runtime for +model failures, and app for permissions/lifecycle. + +[project.yml](project.yml) is the Xcode project source of truth. XcodeGen produces the committed +[Cotabby.xcodeproj](Cotabby.xcodeproj); CI regenerates it and fails when the checked-in project differs. +Cotabby and Cotabby Dev build the same sources under distinct bundle/product identities so development +does not overwrite the production app's TCC grants. Swift default actor isolation is MainActor. + +Use the narrowest relevant tests first, then broaden. The standard build boundary is: + +~~~bash +xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build \ + -derivedDataPath build/DerivedData + +xcodebuild -project Cotabby.xcodeproj -scheme Cotabby -destination 'platform=macOS' build-for-testing \ + -derivedDataPath build/DerivedData +~~~ + +CI independently checks project generation, compilation, tests, and SwiftLint. Pure state machines, +policies, prompt utilities, normalization, and layout logic should receive focused unit coverage. + +## Problem-to-File Map + +| Symptom or change | Start here | +| --- | --- | +| App startup, duplicate service, shutdown | CotabbyAppEnvironment, AppDelegate | +| Field not detected or wrong app policy | FocusTracker, FocusSnapshotResolver | +| Ghost at wrong location | AXTextGeometryResolver, CompletionRenderModePolicy, OverlayController | +| Suggestion never starts | SuggestionAvailabilityEvaluator, SuggestionCoordinator+Prediction | +| Old suggestion appears | SuggestionWorkController, focus/content signatures | +| Wrong prompt or leaked optional context | SuggestionRequestFactory, prompt renderer, sanitizer | +| Backend selection or memory issue | SuggestionEngineRouter, AppDelegate, LlamaRuntimeManager | +| Native decode/cache/shutdown issue | LlamaRuntimeCore, CotabbyInference boundary | +| Partial/final output mismatch | streaming policy, SuggestionTextNormalizer, seam guards | +| Acceptance key passes through or is stolen | InputMonitor, acceptance validation | +| Wrong or repeated inserted text | SuggestionInserter, InputSuppressionController, reconciler | +| Clipboard context is irrelevant | ClipboardRelevanceFilter, ClipboardContentDistiller | +| Screenshot context is stale/noisy | VisualContextCoordinator, OCRTextHygiene | +| Emoji or macro conflicts with suggestions | InlineCommandCoordinator, feature trigger machine | +| Permission loop or lost grant | PermissionManager, PermissionGuidanceController, app identity | +| Settings/onboarding window issue | SettingsCoordinator, WelcomeCoordinator | + +When a change crosses several rows, keep ownership at these boundaries rather than teaching one +coordinator to perform every step itself.