Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 33 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
# Required API Keys
ANTHROPIC_API_KEY=your-anthropic-api-key-here
OPENROUTER_API_KEY=your-openrouter-api-key-here
FISH_API_KEY=your-fish-audio-api-key-here

# Optional: OpenRouter settings
# OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
# OPENROUTER_FAST_MODEL=anthropic/claude-3.5-haiku
# OPENROUTER_RESEARCH_MODEL=anthropic/claude-3.5-sonnet
# OPENROUTER_VISION_MODEL=anthropic/claude-3.5-sonnet
# OPENROUTER_SITE_URL=
# OPENROUTER_APP_NAME=JARVIS

# Optional: Fish Audio voice model (defaults to JARVIS MCU voice)
# FISH_VOICE_ID=612b878b113047d9a770c069c8b4fdfe

Expand All @@ -12,13 +20,37 @@ FISH_API_KEY=your-fish-audio-api-key-here
# If not set, JARVIS reads ALL calendars from Apple Calendar
# CALENDAR_ACCOUNTS=you@gmail.com,work@company.com

# Linux/Windows Calendar (CalDAV)
# Provide either explicit calendar collection URLs, or a base URL where calendars can be discovered.
# CALDAV_CALENDAR_URLS=https://caldav.example.com/user/calendars/personal/,https://caldav.example.com/user/calendars/work/
# CALDAV_URL=https://caldav.example.com/user/calendars/
# CALDAV_USERNAME=you@example.com
# CALDAV_PASSWORD=your-app-password

# Linux/Windows Mail (IMAP, read-only)
# IMAP_HOST=imap.gmail.com
# IMAP_PORT=993
# IMAP_SSL=true
# IMAP_USERNAME=you@example.com
# IMAP_PASSWORD=your-app-password

# Optional: Whether to skip Claude CLI permission prompts on subprocess calls.
# Defaults to "true" because JARVIS is voice-driven and cannot respond to
# interactive permission prompts (they would silently hang the subprocess).
# Set to "false" to require explicit permission prompts — only useful when
# running JARVIS interactively in a Terminal you can see and respond to.
# JARVIS_SKIP_PERMISSIONS=true

# Security (recommended)
# The server will auto-generate this on first run and write it into .env.
# JARVIS_AUTH_TOKEN=paste-from-generated-value
# JARVIS_ALLOWED_ORIGINS=http://localhost:5173,https://localhost:5173
# JARVIS_PROJECTS_DIR=~/JarvisProjects
# JARVIS_DEV_MODE=false

# Linux/Windows Notes (filesystem-backed)
# NOTES_DIR=/home/you/jarvis-notes

# Optional: Override the auto-detected weather location.
# By default, JARVIS uses your public-IP location (via ipwho.is) and reports
# Fahrenheit. Set these to pin a specific location:
Expand Down
35 changes: 35 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: CI

on:
pull_request:
push:
branches: ["main"]

jobs:
backend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-dev.txt
- name: Run tests
run: pytest -q

frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "18"
cache: "npm"
cache-dependency-path: frontend/package-lock.json
- name: Install
run: npm -C frontend ci
- name: Build
run: npm -C frontend run build
17 changes: 8 additions & 9 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# JARVIS — Voice AI Assistant

## Overview
JARVIS (Just A Rather Very Intelligent System) is a voice-first AI assistant for macOS. It runs locally on your machine, connecting to your Apple Calendar, Mail, Notes, and can spawn Claude Code sessions for development tasks.
JARVIS (Just A Rather Very Intelligent System) is a voice-first AI assistant for Linux. It runs locally on your machine, connecting to your calendar (CalDAV), mail (IMAP, read-only), notes (filesystem), and can spawn Claude Code sessions for development tasks.

## Quick Start
When a user clones this repo and starts Claude Code, help them:
1. Copy .env.example to .env
2. Get an Anthropic API key from console.anthropic.com
2. Get an OpenRouter API key from openrouter.ai
3. Get a Fish Audio API key from fish.audio
4. Install Python dependencies: pip install -r requirements.txt
5. Install frontend dependencies: cd frontend && npm install
Expand All @@ -20,25 +20,25 @@ When a user clones this repo and starts Claude Code, help them:
- **Backend**: FastAPI + Python (server.py, ~2300 lines)
- **Frontend**: Vite + TypeScript + Three.js (audio-reactive orb)
- **Communication**: WebSocket (JSON messages + binary audio)
- **AI**: Claude Haiku for fast responses, Claude Opus for research
- **AI**: LLM via OpenRouter (fast + deep models)
- **TTS**: Fish Audio with JARVIS voice model
- **System**: AppleScript for Calendar, Mail, Notes, Terminal integration
- **System**: CalDAV (calendar) + IMAP (mail, read-only) + filesystem notes

## Key Files
- `server.py` — Main server, WebSocket handler, LLM integration, action system
- `frontend/src/orb.ts` — Three.js particle orb visualization
- `frontend/src/voice.ts` — Web Speech API + audio playback
- `frontend/src/main.ts` — Frontend state machine
- `memory.py` — SQLite memory system with FTS5 search
- `calendar_access.py` — Apple Calendar integration via AppleScript
- `mail_access.py` — Apple Mail integration (READ-ONLY)
- `notes_access.py` — Apple Notes integration
- `calendar_access.py` — Calendar integration (CalDAV)
- `mail_access.py` — Mail integration (IMAP, read-only)
- `notes_access.py` — Notes integration (filesystem)
- `actions.py` — System actions (Terminal, Chrome, Claude Code)
- `browser.py` — Playwright web automation
- `work_mode.py` — Persistent Claude Code sessions

## Environment Variables
- `ANTHROPIC_API_KEY` (required) — Claude API access
- `OPENROUTER_API_KEY` (required) — LLM API access via OpenRouter
- `FISH_API_KEY` (required) — Fish Audio TTS
- `FISH_VOICE_ID` (optional) — Voice model ID
- `USER_NAME` (optional) — Your name for JARVIS to use
Expand All @@ -48,6 +48,5 @@ When a user clones this repo and starts Claude Code, help them:
- JARVIS personality: British butler, dry wit, economy of language
- Max 1-2 sentences per voice response
- Action tags: [ACTION:BUILD], [ACTION:BROWSE], [ACTION:RESEARCH], etc.
- AppleScript for all macOS integrations (no OAuth needed)
- Read-only for Mail (safety by design)
- SQLite for all local data storage
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Thanks for your interest in contributing! Here's how to get involved.

- **Bug fixes** — if something's broken, fix it
- **New integrations** — Spotify, Slack, Notion, etc.
- **Windows/Linux support** — the AppleScript integrations are macOS-only, cross-platform alternatives welcome
- **Desktop support** — improve KDE/GNOME compatibility (screenshots, active windows, app launching)
- **Better error handling** — things fail silently in places
- **Voice improvements** — alternative TTS providers, better speech recognition
- **New actions** — extend what JARVIS can do
Expand All @@ -32,7 +32,7 @@ Yes, `server.py` is a 2400-line monolith. It works. If you want to refactor part
## What NOT to Do

- Don't add telemetry or analytics
- Don't send data to external services beyond the existing API calls (Anthropic, Fish Audio)
- Don't send data to external services beyond the existing API calls (OpenRouter, Fish Audio)
- Don't add features that modify or delete user data in connected services (Mail, Calendar, Notes)
- Don't break the existing voice loop

Expand Down
56 changes: 40 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@

**Just A Rather Very Intelligent System.**

A voice-first AI assistant that runs on your Mac. Talk to it, and it talks back -- with a British accent, dry wit, and an audio-reactive particle orb straight out of the MCU.
A voice-first AI assistant that runs on Linux. Talk to it, and it talks back -- with a British accent, dry wit, and an audio-reactive particle orb straight out of the MCU.

JARVIS connects to your Apple Calendar, Mail, and Notes. It can browse the web, spawn Claude Code sessions to build entire projects, and plan your day -- all through natural voice conversation.
This project was originally built for macOS and has been converted to a Linux-first version. Calendar, mail, and notes are now powered by CalDAV, IMAP, and a local notes folder.

JARVIS can browse the web, spawn Claude Code sessions to build entire projects, and plan your day -- all through natural voice conversation.

> "Will do, sir."

Expand All @@ -29,11 +31,11 @@ JARVIS connects to your Apple Calendar, Mail, and Notes. It can browse the web,

## Requirements

- **macOS** (uses AppleScript for Calendar, Mail, Notes integration)
- **Linux** (CalDAV + IMAP + filesystem notes)
- **Python 3.11+**
- **Node.js 18+**
- **Google Chrome** (required for Web Speech API)
- **Anthropic API key** -- powers the AI brain ([get one here](https://console.anthropic.com/))
- **OpenRouter API key** -- powers the LLM brain ([get one here](https://openrouter.ai/keys))
- **Fish Audio API key** -- powers the voice ([get one here](https://fish.audio/))
- **Claude Code CLI** -- for spawning dev tasks ([install here](https://docs.anthropic.com/en/docs/claude-code))

Expand Down Expand Up @@ -72,22 +74,45 @@ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -node
# 6. Start the backend (Terminal 1)
python server.py

# On first run, the server generates JARVIS_AUTH_TOKEN and writes it into .env.
# Paste that token into the Settings panel (Access Token) so the UI can connect.

# 7. Start the frontend (Terminal 2)
cd frontend && npm run dev

# 8. Open Chrome
open http://localhost:5173
xdg-open http://localhost:5173 # Linux (or just paste into your browser)
```

Click the page once to enable audio, then speak. JARVIS will respond.

## Running as a Linux service (systemd)

An example `systemd --user` service file is included here:

- `docs/systemd/jarvis.service`

Typical setup:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt

mkdir -p ~/.config/systemd/user
cp docs/systemd/jarvis.service ~/.config/systemd/user/jarvis.service
systemctl --user daemon-reload
systemctl --user enable --now jarvis
journalctl --user -u jarvis -f
```

## Configuration

Edit your `.env` file:

```env
# Required
ANTHROPIC_API_KEY=your-anthropic-api-key-here
OPENROUTER_API_KEY=your-openrouter-api-key-here
FISH_API_KEY=your-fish-audio-api-key-here

# Optional -- your name (JARVIS will address you personally)
Expand All @@ -101,15 +126,15 @@ CALENDAR_ACCOUNTS=you@gmail.com,work@company.com
## Architecture

```
Microphone -> Web Speech API -> WebSocket -> FastAPI -> Claude (Haiku) -> Fish Audio TTS -> WebSocket -> Speaker
Microphone -> Web Speech API -> WebSocket -> FastAPI -> LLM (via OpenRouter) -> Fish Audio TTS -> WebSocket -> Speaker
|
v
Claude Code Tasks
(spawns real dev work)
|
v
AppleScript Bridge
(Calendar, Mail, Notes, Terminal)
System Integrations
(CalDAV/IMAP/notes folder)
```

| Layer | Technology |
Expand All @@ -120,7 +145,7 @@ Microphone -> Web Speech API -> WebSocket -> FastAPI -> Claude (Haiku) -> Fish A
| AI (fast) | Claude Haiku -- low-latency voice responses |
| AI (deep) | Claude Opus -- research and complex tasks |
| TTS | Fish Audio with JARVIS voice model |
| System | AppleScript for all macOS integrations |
| System | CalDAV (calendar) + IMAP (mail) + filesystem notes |

## How the Voice Loop Works

Expand All @@ -144,9 +169,9 @@ Microphone -> Web Speech API -> WebSocket -> FastAPI -> Claude (Haiku) -> Fish A
| `frontend/src/voice.ts` | Web Speech API + audio playback |
| `frontend/src/main.ts` | Frontend state machine |
| `memory.py` | SQLite memory system with FTS5 full-text search |
| `calendar_access.py` | Apple Calendar integration via AppleScript |
| `mail_access.py` | Apple Mail integration (read-only) |
| `notes_access.py` | Apple Notes integration |
| `calendar_access.py` | Calendar integration (CalDAV) |
| `mail_access.py` | Mail integration (IMAP, read-only) |
| `notes_access.py` | Notes integration (filesystem) |
| `actions.py` | System actions (Terminal, Chrome, Claude Code) |
| `browser.py` | Playwright web automation |
| `work_mode.py` | Persistent Claude Code sessions |
Expand All @@ -167,13 +192,12 @@ JARVIS uses action tags to trigger real system actions:
JARVIS remembers things you tell it using SQLite with FTS5 full-text search. Preferences, decisions, and facts persist across sessions.

### Calendar & Mail
All macOS integrations use AppleScript -- no OAuth flows, no token management. Just native system access. Mail is intentionally read-only for safety.
Calendar uses CalDAV and mail uses IMAP. Mail is intentionally read-only for safety.

## Contributing

Contributions are welcome. Some areas that could use work:

- **Linux/Windows support** -- replace AppleScript with cross-platform alternatives
- **Alternative TTS engines** -- add ElevenLabs, OpenAI TTS, or local models
- **Alternative LLMs** -- add OpenAI, Gemini, or local model support
- **Mobile client** -- a companion app for voice interaction on the go
Expand All @@ -189,7 +213,7 @@ Free for personal, non-commercial use. Commercial use requires a license — vis

Built by [Ethan](https://ethanplus.ai).

Powered by [Anthropic Claude](https://anthropic.com) and [Fish Audio](https://fish.audio).
Powered by [OpenRouter](https://openrouter.ai) (LLM routing) and [Fish Audio](https://fish.audio).

Inspired by the AI that started it all -- Tony Stark's JARVIS.

Expand Down
Loading
Loading