Orb Assistant

A conversation enhancement agent that listens to live 2-person conversations and provides rare, high-value spoken interjections at exactly the right moments.

✨ What It Does

Orb sits quietly in the background of your conversations—therapy sessions, tutoring, negotiations, brainstorms—and speaks only when it has something genuinely valuable to add:

• Clarifies confusion when speakers are talking past each other
• Points out patterns across the conversation
• Offers gentle reframes when discussions get stuck
• Catches missed cues that might be important

The UI shows only the AI's contributions. No transcript. No distraction. Just a calm orb that turns green when it has something worth saying.

🎯 Two Modes

Manual Mode (Recommended to start)

• Orb turns green when it has something valuable
• Tap to hear the interjection
• Stay in control of when AI speaks

Auto Mode

• AI speaks at natural pauses when confident
• More seamless but requires trust
• Use after you've tuned the thresholds

🏗️ Architecture

┌─────────────┐     WebSocket      ┌─────────────────────────────────┐
│   iOS App   │◄──────────────────►│         Node.js Server          │
│             │   audio + events   │                                 │
│  • Orb UI   │                    │  • Deepgram STT (diarized)     │
│  • TTS      │                    │  • Fast/Slow reasoning loops   │
│  • History  │                    │  • Gatekeeper (rare = valuable)│
└─────────────┘                    │  • Policy packs per mode       │
                                   │  • Cross-session memory        │
                                   └─────────────────────────────────┘

🚀 Quick Start

Prerequisites

• Node.js 20+
• Docker (for PostgreSQL)
• Xcode 15+ (for iOS)
• API Keys: OpenRouter, Deepgram

Server

# Quick setup (recommended)
./scripts/setup.sh
# Then edit server/.env with your API keys

# Or manual setup:
cd server
cp env.example .env
# Add your API keys to .env

pnpm install

# Start database
cd ..
docker-compose up -d postgres
cd server

pnpm db:migrate
pnpm dev

Deploy to Production

# Install Fly CLI: brew install flyctl
fly auth login

# Set your secrets
fly secrets set DATABASE_URL="postgresql://..."
fly secrets set OPENROUTER_API_KEY="YOUR_OPENROUTER_API_KEY_HERE"
fly secrets set DEEPGRAM_API_KEY="YOUR_DEEPGRAM_API_KEY_HERE"

# Deploy
./scripts/deploy.sh

iOS

1. Open ios/OrbAssistant/OrbAssistant.xcodeproj in Xcode
2. Update Config.swift with your server's IP
3. Build and run on a physical iPhone

📖 Documentation

• Architecture - System design, diagrams, data flow
• Prompts - LLM prompts and schemas
• Runbook - Development setup guide
• Agent Notes - Implementation notes and philosophy

🎨 Design Philosophy

The magic is in the restraint. The best interjection is often no interjection.

• Timing over content - A mediocre insight at the perfect moment beats brilliance at the wrong time
• Trust through silence - Every time we don't speak when we could, we earn trust
• No filler, ever - If nothing valuable, say nothing
• Feel like a friend - Not a tool, not a feature, a thoughtful presence

🔐 Privacy

• Audio is processed in real-time and never stored
• Only AI interjections are saved (locally on device, summaries on server)
• No conversation transcripts sent to client
• User controls what's persisted

📄 License

MIT

---

Built to help people have better conversations.