JARVIS

Just A Rather Very Intelligent System.

A voice-first AI assistant that runs on Linux. Talk to it, and it talks back -- with a British accent, dry wit, and an audio-reactive particle orb straight out of the MCU.

This project was originally built for macOS and has been converted to a Linux-first version. Calendar, mail, and notes are now powered by CalDAV, IMAP, and a local notes folder.

JARVIS can browse the web, spawn Claude Code sessions to build entire projects, and plan your day -- all through natural voice conversation.

"Will do, sir."

---

What It Does

• Voice conversation -- speak naturally, get spoken responses with a JARVIS voice
• Builds software -- say "build me a landing page" and watch Claude Code do the work
• Reads your calendar -- "What's on my schedule today?"
• Reads your email -- "Any unread messages?" (read-only, by design)
• Browses the web -- "Search for the best restaurants in Austin"
• Manages tasks -- "Remind me to call the client tomorrow"
• Takes notes -- "Save that as a note"
• Remembers things -- "I prefer React over Vue" (it remembers next time)
• Plans your day -- combines calendar, tasks, and priorities into a plan
• Sees your screen -- knows what apps are open for context-aware responses
• Audio-reactive orb -- a Three.js particle visualization that pulses with JARVIS's voice

Requirements

• Linux (CalDAV + IMAP + filesystem notes)
• Python 3.11+
• Node.js 18+
• Google Chrome (required for Web Speech API)
• OpenRouter API key -- powers the LLM brain (get one here)
• Fish Audio API key -- powers the voice (get one here)
• Claude Code CLI -- for spawning dev tasks (install here)

Quick Start (with Claude Code)

The fastest way to get running:

git clone https://github.com/yourusername/jarvis.git
cd jarvis
claude

Claude Code will read the project's CLAUDE.md and walk you through setup step by step -- API keys, dependencies, SSL certs, everything.

Manual Setup

# 1. Clone the repo
git clone https://github.com/yourusername/jarvis.git
cd jarvis

# 2. Set up environment
cp .env.example .env
# Edit .env with your API keys (see below)

# 3. Install Python dependencies
pip install -r requirements.txt

# 4. Install frontend dependencies
cd frontend && npm install && cd ..

# 5. Generate SSL certificates (needed for secure WebSocket)
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes -subj '/CN=localhost'

# 6. Start the backend (Terminal 1)
python server.py

# On first run, the server generates JARVIS_AUTH_TOKEN and writes it into .env.
# Paste that token into the Settings panel (Access Token) so the UI can connect.

# 7. Start the frontend (Terminal 2)
cd frontend && npm run dev

# 8. Open Chrome
xdg-open http://localhost:5173  # Linux (or just paste into your browser)

Click the page once to enable audio, then speak. JARVIS will respond.

Running as a Linux service (systemd)

An example systemd --user service file is included here:

• docs/systemd/jarvis.service

Typical setup:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt

mkdir -p ~/.config/systemd/user
cp docs/systemd/jarvis.service ~/.config/systemd/user/jarvis.service
systemctl --user daemon-reload
systemctl --user enable --now jarvis
journalctl --user -u jarvis -f

Configuration

Edit your .env file:

# Required
OPENROUTER_API_KEY=your-openrouter-api-key-here
FISH_API_KEY=your-fish-audio-api-key-here

# Optional -- your name (JARVIS will address you personally)
USER_NAME=Tony

# Optional -- specific calendar accounts (comma-separated)
# Leave empty to auto-discover all calendars
CALENDAR_ACCOUNTS=you@gmail.com,work@company.com

Architecture

Microphone -> Web Speech API -> WebSocket -> FastAPI -> LLM (via OpenRouter) -> Fish Audio TTS -> WebSocket -> Speaker
                                                |
                                                v
                                        Claude Code Tasks
                                        (spawns real dev work)
                                                |
                                                v
                                        System Integrations
                                        (CalDAV/IMAP/notes folder)

Layer	Technology
Backend	FastAPI + Python (server.py, ~2300 lines)
Frontend	Vite + TypeScript + Three.js
Communication	WebSocket (JSON messages + binary audio)
AI (fast)	Claude Haiku -- low-latency voice responses
AI (deep)	Claude Opus -- research and complex tasks
TTS	Fish Audio with JARVIS voice model
System	CalDAV (calendar) + IMAP (mail) + filesystem notes

How the Voice Loop Works

1. You speak into your microphone
2. Chrome's Web Speech API transcribes your speech in real-time
3. The transcript is sent to the server via WebSocket
4. JARVIS detects intent -- conversation, action, or build request
5. For actions: spawns a Claude Code subprocess or runs AppleScript
6. Generates a response via Claude Haiku (optimized for speed)
7. Fish Audio converts the response to speech with the JARVIS voice
8. Audio streams back to the browser via WebSocket
9. The Three.js orb deforms and pulses in response to the audio
10. Background tasks notify you proactively when they complete

Key Files

File	Purpose
server.py	Main server -- WebSocket handler, LLM, action system
frontend/src/orb.ts	Three.js particle orb visualization
frontend/src/voice.ts	Web Speech API + audio playback
frontend/src/main.ts	Frontend state machine
memory.py	SQLite memory system with FTS5 full-text search
calendar_access.py	Calendar integration (CalDAV)
mail_access.py	Mail integration (IMAP, read-only)
notes_access.py	Notes integration (filesystem)
actions.py	System actions (Terminal, Chrome, Claude Code)
browser.py	Playwright web automation
work_mode.py	Persistent Claude Code sessions
planner.py	Multi-step task planning with smart questions

Features in Detail

Action System

JARVIS uses action tags to trigger real system actions:

• [ACTION:BUILD] -- spawns Claude Code to build a project
• [ACTION:BROWSE] -- opens Chrome to a URL or search query
• [ACTION:RESEARCH] -- deep research with Claude Opus, outputs an HTML report
• [ACTION:PROMPT_PROJECT] -- connects to an existing project via Claude Code
• [ACTION:ADD_TASK] -- creates a tracked task with priority and due date
• [ACTION:REMEMBER] -- stores a fact for future context

Memory System

JARVIS remembers things you tell it using SQLite with FTS5 full-text search. Preferences, decisions, and facts persist across sessions.

Calendar & Mail

Calendar uses CalDAV and mail uses IMAP. Mail is intentionally read-only for safety.

Contributing

Contributions are welcome. Some areas that could use work:

• Alternative TTS engines -- add ElevenLabs, OpenAI TTS, or local models
• Alternative LLMs -- add OpenAI, Gemini, or local model support
• Mobile client -- a companion app for voice interaction on the go
• Plugin system -- make it easy to add new actions and integrations

Please open an issue before submitting large PRs so we can discuss the approach.

License

Free for personal, non-commercial use. Commercial use requires a license — visit ethanplus.ai for inquiries. See LICENSE for details.

Credits

Built by Ethan.

Powered by OpenRouter (LLM routing) and Fish Audio.

Inspired by the AI that started it all -- Tony Stark's JARVIS.

Disclaimer: This is an independent fan project and is not affiliated with, endorsed by, or connected to Marvel Entertainment, The Walt Disney Company, or any related entities. The JARVIS name and character are property of Marvel Entertainment.