N E O

Your Intelligent Desktop AI Assistant

Agentic AI brain · Computer-Use Vision · Voice-First · Glassmorphism UI

---

✨ What Makes NEO Different

Aria isn't a chatbot — it's a computer-use AI agent that can see your screen, understand what's on it, and take actions to accomplish goals. Think Claude Computer Use, but local-first and deeply integrated with your desktop.

"Open WhatsApp and send hello to Shreyas"
   → Opens WhatsApp via Windows Search
   → Finds Shreyas in contacts using OCR
   → Types and sends the message
   → Verifies the message was sent

---

🚀 Features

Category	Capabilities
👁️ Computer-Use Vision	5-layer perception: screenshot + OCR + UI Automation + pre-trained layouts + learning memory
🤖 Agentic AI	Multi-step planning, autonomous tool use, self-reflection, session memory
🧠 Multi-Model	GPT-4o, Claude 3.5/3.7, Gemini Pro, Local Llama 3.2 (Ollama)
🖥️ Desktop Automation	Click, type, screenshot, app launch — Windows/macOS/Linux
🎙️ Voice Control	Wake word detection, Faster-Whisper offline STT, Edge-TTS
🔍 Web Search	Real-time search with inline source cards
📧 Email	Gmail compose, send, preview with confirmation flow
📅 Calendar	Google Calendar reading, event creation, morning briefing
📝 Notion	Search, read, create Notion pages
🎵 Music Player	Local file playback, playlist management, voice-controlled
🗂️ System Control	Volume, WiFi, Bluetooth, clipboard, screenshots, power
💧 Water Reminders	Hydration tracker with smart interval prompts
💾 Smart Memory	ChromaDB RAG — semantic search across conversations
🎨 6 Premium Themes	Light & dark mode across all themes

---

👁️ Vision Agent — Computer-Use AI

Aria's Vision Agent operates like an autonomous computer user. It sees the screen, reasons about what to do, and takes precise actions.

5-Layer Perception Pipeline

Every step, the AI receives 5 layers of ground truth before deciding:

Step N:
  1. 📸 Screenshot (100px grid overlay with high-contrast labels)
  2. 📝 OCR Text Dump (Tesseract extracts ALL visible text)
  3. 🔍 Live UI Elements (Windows Accessibility API via pywinauto)
  4. 🗺️ Pre-trained Layout (known positions for 10+ apps)
  5. 📚 Past Lessons (what worked/failed in similar goals)
       ↓
  LLM Decides → Executes → Compares Screenshots → Learns

5-Tier Click System

Priority	Action	How It Works	Accuracy
1	click_ui	Windows Accessibility API	100%
2	click_element	Pre-trained layout positions	~90%
3	click_text	Tesseract OCR text finding	~85%
4	click	LLM reads grid coordinates	~60%

Self-Improving Memory

Aria learns from every task and gets smarter over time:

Run 1: No lessons → 12 steps, some failures → saves lesson
Run 2: Recalls lesson → avoids past mistakes → 7 steps
Run 5: Rich memory → near-optimal → 4 steps

Pre-Trained App Layouts

Aria knows the UI layout of 10+ common applications out of the box:

WhatsApp · Telegram · Chrome · Edge · Spotify · VS Code · Teams · Discord · File Explorer · Notepad

Add your own apps by editing backend/data/ui_layouts.json

---

🤖 Agentic Loop

Aria has a full agentic reasoning engine for complex multi-step tasks:

User Request
    ↓
PlannerBrain (decomposes goal into steps)
    ↓
AgentLoop (executes tools, observes results)
    ↓
ReflectionEngine (checks success, retries on failure)
    ↓
Heartbeat Monitor (ensures agent stays alive)
    ↓
Final Response

Available Agent Tools:

• web_search — real-time internet search
• shell — run terminal commands
• file_read / file_write — file system access
• screenshot — capture and analyze screen
• browser — open and interact with URLs
• email_send — compose and send emails
• calendar_read — check events
• notion_search — search knowledge base

---

🚀 Quick Start

Prerequisites

• Python 3.10+ (3.11 recommended)
• Node.js 18+
• Tesseract OCR — Download
• Ollama (optional, for local LLM) — Download
• API Key — OpenAI, Anthropic, or Google

Installation

# Clone repository
git clone https://github.com/shreyass0007/Aria.git
cd ARIA

# Setup Python environment
python -m venv .venv
.venv\Scripts\activate        # Windows
# source .venv/bin/activate   # macOS/Linux
pip install -r requirements.txt

# Optional: install advanced vision libraries
pip install pywinauto easyocr

# Setup Electron frontend
cd electron
npm install

# Launch Aria
npm start

Environment Setup

Create .env in the project root:

# Required — at least one AI provider
OPEN_AI_API_KEY=sk-your-key

# Optional providers
ANTHROPIC_API_KEY=sk-ant-your-key
GOOGLE_API_KEY=your-key

# Integrations
NOTION_API_KEY=your-key
OPENWEATHER_API_KEY=your-key

---

🗣️ Voice Commands

"Hey Aria, what's on my calendar today?"
"Open WhatsApp and send hello to Shreyas"
"Open VS Code"
"Set volume to 50%"
"Play my playlist"
"Search the web for latest AI news"
"Send an email to john@example.com about the meeting"
"What's the weather like?"
"Take a screenshot and describe what you see"
"Enable Deep Work mode"
"Remind me to drink water every 30 minutes"

---

🎨 Themes

6 premium themes with full light/dark mode support:

Theme	Accent	Vibe
Violet Dream	#6366f1 Purple	Elegant, default
Ocean Breeze	#0ea5e9 Blue	Calm, professional
Sunset Glow	#f97316 Orange	Warm, creative
Forest Mist	#10b981 Green	Natural, relaxing
Azure	#3b82f6 Blue-white	Clean, corporate
Mono	Black & White	Minimalist

---

🏗️ Architecture

┌──────────────────────────────────────────────────────┐
│            Electron Frontend (Renderer)               │
│  Glassmorphism UI · 6 Themes · Music Player · Chat   │
└────────────────────┬─────────────────────────────────┘
                     │ HTTP/REST + WebSocket
┌────────────────────▼─────────────────────────────────┐
│            FastAPI Backend                            │
│   25+ endpoints · APScheduler · WebSocket streaming  │
└────────────────────┬─────────────────────────────────┘
                     │
┌────────────────────▼─────────────────────────────────┐
│            Aria Core Engine                           │
│                                                       │
│  ┌─────────────┐ ┌─────────────┐ ┌───────────────┐  │
│  │ AgentLoop   │ │ PlannerBrain│ │ ReflectionEng │  │
│  │ (tools,     │ │ (decompose  │ │ (verify,      │  │
│  │  reasoning) │ │  goals)     │ │  retry)       │  │
│  └─────────────┘ └─────────────┘ └───────────────┘  │
│                                                       │
│  ┌─────────────────────────────────────────────────┐ │
│  │ VisionAgent (Computer-Use)                      │ │
│  │  Screenshot → OCR → UIA → Layout → Memory →    │ │
│  │  LLM Decides → Execute → Verify → Learn        │ │
│  └─────────────────────────────────────────────────┘ │
│                                                       │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐  │
│  │ ChromaDB │ │ Calendar │ │ Desktop  │ │ Email  │  │
│  │   RAG    │ │ Manager  │ │ Executor │ │Manager │  │
│  └──────────┘ └──────────┘ └──────────┘ └────────┘  │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐  │
│  │ TTS/STT  │ │  Notion  │ │  Music   │ │ Vision │  │
│  │ Manager  │ │ Manager  │ │ Library  │ │Memory  │  │
│  └──────────┘ └──────────┘ └──────────┘ └────────┘  │
└──────────────────────────────────────────────────────┘

---

📁 Project Structure

ARIA/
├── aria/                    # Core Python backend
│   ├── agent/               # Agentic AI engine
│   │   ├── loop.py          # AgentLoop — autonomous reasoning
│   │   ├── vision_agent.py  # Computer-use vision agent
│   │   ├── vision_memory.py # Self-improving learning memory
│   │   ├── ui_automation.py # pywinauto UI element detection
│   │   ├── reflection.py    # Self-reflection engine
│   │   └── tools/           # Agent tools (search, shell, etc.)
│   ├── brains/              # PlannerBrain, LearningManager
│   ├── executor/            # Desktop automation adapters
│   ├── handlers/            # Intent handlers
│   └── vision/              # VisionManager (OCR, template matching)
├── backend/
│   ├── data/
│   │   ├── ui_layouts.json  # Pre-trained UI element positions
│   │   └── vision_memory.json # Learned patterns (auto-generated)
│   └── routers/             # FastAPI route handlers
├── electron/                # Electron frontend
│   ├── main.js              # Main process
│   └── renderer/            # UI (HTML/CSS/JS)
└── .env                     # API keys and configuration

---

📡 API Endpoints

Method	Endpoint	Description
GET	/health	Health check
GET	/greeting	Time-based greeting
GET	/briefing	Morning briefing (calendar + weather)
POST	/message	Process message (agentic)
GET	/models/available	List AI models
POST	/models/set	Switch active model
GET	/conversations	List conversations
POST	/voice/start	Start voice input
POST	/tts/speak	Text-to-speech
GET	/notifications	Notification stream
POST	/email/send	Send email
GET	/system/analyze-screen	Computer vision analysis
POST	/music/play	Play music

---

🔧 Configuration

AI Models

Switch via the UI dropdown or POST /models/set:

Provider	Models
OpenAI	gpt-4o, gpt-4o-mini, gpt-3.5-turbo
Anthropic	claude-3-5-sonnet, claude-3-7-sonnet, claude-haiku
Google	gemini-pro, gemini-flash
Local	llama3.2 (via Ollama — no API key)

Customizing App Layouts

Edit backend/data/ui_layouts.json to add your own apps:

{
  "your_app": {
    "app_name": "Your App",
    "elements": {
      "send_button": { "rx": 0.95, "ry": 0.95, "description": "Send button" },
      "search_bar":  { "rx": 0.50, "ry": 0.04, "description": "Search field" }
    },
    "tips": ["Ctrl+F opens search"]
  }
}

Coordinates are relative (0.0–1.0), so they work at any screen resolution.

---

📚 Documentation

• User Manual
• Full Documentation
• Google Calendar Setup
• Notion Setup
• System Control Guide

---

🤝 Contributing

1. Fork the repository
2. Create feature branch (git checkout -b feature/AmazingFeature)
3. Commit changes (git commit -m 'Add AmazingFeature')
4. Push to branch (git push origin feature/AmazingFeature)
5. Open Pull Request

---

📝 License

MIT License — see LICENSE for details.

---

👤 Author

Shreyas — @shreyass0007

---

Made with ❤️ by Shreyas

⭐ Star this repo if you find it helpful!