See. Understand. Remember.
An AI-powered multimodal assistant that combines computer vision, conversational AI, memory retrieval, and voice interaction to help users understand the world around them.
ONAI is a personal AI vision assistant designed to interact with the physical world through a camera.
Traditional AI image-analysis applications generally follow a simple pattern:
Image → AI → Response
ONAI aims to go further:
Camera
↓
Current Visual Context
↓
Conversation
↓
Memory Retrieval
↓
Adaptive Prompt Construction
↓
Multimodal AI
↓
Context-Aware Response
↓
Memory
↓
Future Interactions
Instead of treating every image as an isolated request, ONAI is designed to maintain an understanding of the user's ongoing interaction.
For example, a user could point the camera at a phone, ask what it is, move the camera toward a book, and then ask a follow-up question. ONAI should be able to understand that the visual context has changed rather than blindly relying on an older frame.
The long-term goal is to create an assistant that can see, reason, remember, and interact naturally with its environment.
ONAI uses the device camera as its primary interface with the physical environment.
The camera system allows the user to:
- Capture visual information.
- Send frames to the AI backend.
- Analyze objects and scenes.
- Ask questions about what is currently visible.
- Continue conversations while the visual environment changes.
The system is designed around fresh visual context rather than relying permanently on previously captured images.
ONAI uses Google's Gemini multimodal models to process both:
- Natural-language prompts
- Visual information
This allows questions such as:
"What am I looking at?"
"What's written on this page?"
"What is this object used for?"
"Can you explain what I'm seeing?"
"Is there anything unusual about this?"
The AI response is generated using the combination of the user's request and the available visual context.
ONAI isn't designed to behave like a one-shot image classifier.
The user can maintain a conversation with the assistant.
Example:
User:
"What is this?"
ONAI:
"That's a mechanical keyboard."
User:
"How does it work?"
ONAI:
"It uses individual switches beneath each key..."
User:
"Would it be good for programming?"
ONAI:
"Yes. Mechanical keyboards are popular for programming because..."
The conversation provides additional context that can be used when constructing subsequent AI requests.
One of ONAI's most important features is its vision memory system.
A normal vision assistant might forget everything after generating a response.
ONAI instead stores structured information about previous interactions.
A simplified memory entry can contain:
{
prompt,
frame,
response,
timestamp
}
This allows ONAI to retrieve previous interactions when they are relevant to the current conversation.
Current Interaction
↓
Generate Response
↓
Create Memory
↓
Store Interaction
↓
Future User Request
↓
Search Previous Memories
↓
Retrieve Relevant Context
↓
Add Context to Prompt
↓
Generate Response
This provides the foundation for a more persistent AI assistant.
Simply storing thousands of previous interactions isn't enough.
ONAI needs to determine which memories are relevant.
The memory system therefore separates:
Stores previous interactions and their associated visual/contextual information.
Searches previous interactions for information that could help answer the current request.
Relevant memories are then incorporated into the prompt sent to the AI model.
This creates a pipeline similar to:
User Request
↓
Memory Search
↓
Relevant Memories
↓
Current Camera Frame
↓
Conversation History
↓
Adaptive Prompt
↓
Gemini
↓
Response
ONAI doesn't rely on one static prompt for every interaction.
The prompt can be dynamically constructed based on available context.
Potential context includes:
- Current user question
- Current camera frame
- Previous conversation
- Relevant memories
- Previous visual context
- Current interaction state
Conceptually:
Base Instructions
+
Current Question
+
Current Visual Context
+
Conversation Context
+
Retrieved Memories
↓
Adaptive Prompt
↓
Gemini
This allows the AI to receive only the context that is relevant to the current interaction.
One of the challenges with vision assistants is stale visual context.
For example:
1. Camera sees a phone.
2. ONAI identifies the phone.
3. User moves the camera.
4. Camera now sees a book.
5. User asks about the book.
A poorly designed system may continue talking about the phone because an old frame remains in memory.
ONAI addresses this by distinguishing between:
What the camera is seeing right now.
What ONAI saw during previous interactions.
This distinction allows memory to provide useful historical information without overriding the current visual state.
ONAI includes a visual scan guide to help users focus the camera on a particular region.
Conceptually:
┌─────────────────────────────┐
│ │
│ CAMERA VIEW │
│ │
│ ┌───────────┐ │
│ │ │ │
│ │ SCAN │ │
│ │ AREA │ │
│ │ │ │
│ └───────────┘ │
│ │
│ │
└─────────────────────────────┘
The guide is an interface feature rather than a restriction on the camera itself.
The camera can still provide the full image to the backend while the interface helps the user visually target the object or area they want ONAI to understand.
ONAI is being designed for interaction that doesn't require constant typing.
Voice functionality allows users to interact with the assistant through spoken commands.
A typical interaction can become:
User speaks
↓
Voice Input
↓
Text / Command
↓
Vision + Memory
↓
AI Response
↓
Text-to-Speech
↓
User hears response
This makes ONAI more suitable for situations where the user's hands are occupied.
ONAI also explores a more natural hands-free interaction model.
Instead of requiring the user to:
Open camera
→ Take image
→ Type question
→ Wait
→ Repeat
the long-term interaction model is closer to:
Look
↓
Ask
↓
ONAI understands
↓
Responds
↓
Continue
This is particularly important for the long-term goal of making ONAI feel like an assistant rather than an image-analysis tool.
ONAI is divided into several major components.
┌─────────────────────┐
│ USER │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ FRONTEND │
│ │
│ React + TypeScript │
│ Camera Interface │
│ Chat Interface │
│ Voice Interface │
└──────────┬──────────┘
│
│ HTTP
▼
┌─────────────────────┐
│ FASTAPI │
│ BACKEND │
│ │
│ API Routes │
│ Request Handling │
│ AI Integration │
└──────────┬──────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
┌─────────────────┐ ┌──────────────────┐
│ GEMINI AI │ │ MEMORY SYSTEM │
│ │ │ │
│ Vision │ │ Storage │
│ Reasoning │ │ Retrieval │
│ Generation │ │ Context │
└────────┬────────┘ └────────┬─────────┘
│ │
└────────────┬────────────┘
▼
┌─────────────────────┐
│ CONTEXT-AWARE │
│ RESPONSE │
└─────────────────────┘
The frontend is built with React and TypeScript.
Major components include:
CameraPanel
ChatPanel
Navbar
Responsible for:
- Camera access
- Capturing frames
- Displaying the live camera
- Scan guide UI
- Connecting camera state with the vision system
Responsible for:
- User messages
- AI responses
- Conversation display
- Loading states
- Error states
- Interaction with the chat system
The application maintains shared vision state so that components can access the current visual frame without tightly coupling the camera component to the rest of the application.
The backend uses FastAPI.
The backend provides API endpoints for communication between the frontend and the AI system.
A simplified request flow:
Frontend
│
│ POST /analyze
▼
FastAPI
│
├── Validate request
│
├── Process image
│
├── Build AI context
│
├── Retrieve memory
│
└── Call Gemini
│
▼
AI Response
│
▼
FastAPI
│
▼
Frontend
The backend currently exposes core endpoints such as:
Basic API/root endpoint.
Health-check endpoint used to determine whether the backend is running.
Main vision-analysis endpoint.
Conceptually:
POST /analyze
Input:
image
prompt
Processing:
image → Gemini Vision
Output:
AI-generated analysis
| Technology | Purpose |
|---|---|
| React | UI framework |
| TypeScript | Type safety |
| Vite | Development/build tooling |
| Tailwind CSS | Styling |
| React Router | Application routing |
| Axios | HTTP communication |
| React Webcam | Camera integration |
| Lucide React | UI icons |
| Technology | Purpose |
|---|---|
| Python | Backend language |
| FastAPI | API framework |
| Google GenAI SDK | Gemini integration |
| Gemini | Multimodal AI |
| Technology | Purpose |
|---|---|
| React Native | Mobile UI |
| Expo | Mobile development platform |
| TypeScript | Type safety |
The project is organized around separate frontend, backend, and mobile applications.
ONAI/
│
├── frontend/
│ │
│ ├── src/
│ │ │
│ │ ├── components/
│ │ │ ├── CameraPanel/
│ │ │ ├── ChatPanel/
│ │ │ └── Navbar/
│ │ │
│ │ ├── contexts/
│ │ │ └── VisionContext/
│ │ │
│ │ ├── hooks/
│ │ │ ├── useChat/
│ │ │ └── useMemory/
│ │ │
│ │ ├── services/
│ │ │ └── analyzeImage/
│ │ │
│ │ └── pages/
│ │ └── Home/
│ │
│ └── package.json
│
├── backend/
│ │
│ ├── main.py
│ ├── services/
│ │ └── gemini_service.py
│ ├── utils/
│ └── requirements.txt
│
├── mobile/
│ │
│ ├── app/
│ ├── components/
│ ├── assets/
│ └── package.json
│
├── .gitignore
├── README.md
└── ...
The project structure may change as development continues.
Before running ONAI, make sure you have:
- Python 3.10+
- Node.js
- npm
- Git
- A Google Gemini API key
- A webcam/camera for vision functionality
For mobile development:
- Expo
- Android Studio and/or an Android device
- Or an iOS development environment
git clone https://github.com/YOUR_USERNAME/ONAI.git
cd ONAICreate a Python virtual environment:
python -m venv venvWindows:
venv\Scripts\activatemacOS/Linux:
source venv/bin/activateInstall dependencies:
pip install -r requirements.txtCreate a .env file:
GEMINI_API_KEY=your_api_key_hereStart the backend:
uvicorn main:app --reloadThe API should then be available at:
http://localhost:8000
FastAPI documentation is available at:
http://localhost:8000/docs
Navigate to the frontend:
cd frontendInstall dependencies:
npm installStart the development server:
npm run devThe frontend should be available at:
http://localhost:5173
Navigate to the mobile project:
cd mobileInstall dependencies:
npm installStart Expo:
npx expo startYou can then run the application using:
- Android emulator
- iOS simulator
- Physical Android device
- Physical iPhone
API keys should never be committed to the repository.
Example:
GEMINI_API_KEY=your_gemini_api_keyMake sure your environment file is ignored:
.env
.env.local
A production deployment should use a secure secret-management system rather than exposing API credentials to the frontend.
ONAI is being developed incrementally using development sprints.
Each sprint focuses on a specific capability rather than attempting to build the entire system at once.
The development process has included:
Core Vision
↓
Camera Integration
↓
Conversational Interface
↓
Vision Context
↓
Scan Interface
↓
Memory
↓
Memory Retrieval
↓
Adaptive Prompts
↓
Context Awareness
↓
Voice / Hands-Free Interaction
↓
Mobile Application
↓
Future Multimodal Capabilities
This approach makes it possible to test each major subsystem independently before combining them.
- Initial project architecture
- React frontend
- FastAPI backend
- Gemini integration
- Camera integration
- Image analysis
- Conversational chat
- Vision context
- Camera scan guide
- Vision memory
- Memory search
- Memory retrieval
- Adaptive prompt builder
- Context awareness
- Voice interaction
- Hands-free interaction
- React Native mobile application
- Mobile camera integration
- Mobile voice interaction
- Cross-platform synchronization
- Improved memory architecture
- Improved real-time performance
- Real-time object tracking
- AI-generated visual annotations
- Live object identification
- OCR improvements
- Scene understanding
- Persistent long-term memory
- User personalization
- Offline/edge AI experimentation
- Wearable-device integration
- Smart-glasses compatibility
- Continuous environmental understanding
Imagine the user is sitting at a desk.
ONAI sees:
Laptop
Notebook
Phone
Coffee cup
"What objects are on my desk?"
Gemini analyzes the current frame.
The response might be:
"I can see a laptop, a notebook, a phone,
and a coffee cup on your desk."
The user moves the camera toward the notebook:
"What is this used for?"
ONAI should prioritize the current notebook frame, while still using previous conversation context where appropriate.
This distinction between current perception and historical memory is fundamental to the design of ONAI.
The project explores an important question:
What happens when an AI assistant can continuously perceive and remember the environment around its user?
Most AI assistants primarily interact through text or isolated images.
ONAI experiments with a different model:
Language
+
Vision
+
Memory
+
Voice
+
Context
=
Personal Multimodal Assistant
The goal isn't simply to make another chatbot.
The goal is to build an AI system that can develop an understanding of an ongoing interaction with the physical world.
The long-term vision for ONAI is a personal AI assistant that can accompany the user throughout their day.
A future version could potentially:
See an environment
↓
Understand objects
↓
Understand the user's request
↓
Remember relevant history
↓
Reason about the situation
↓
Respond through voice
↓
Continue observing
This could eventually enable applications such as:
- Accessibility assistance
- Educational assistance
- Object and scene explanation
- Reading and document assistance
- Navigation assistance
- Personal productivity
- Hands-free computing
- Real-world AI tutoring
- Wearable AI interfaces
Current Stage: Active Development
Latest Major Milestone: Sprint 12 — Smarter Vision Memory, Memory Search & Retrieval, Adaptive Prompt Builder & Context Awareness
Current Focus: Mobile application development and expanding ONAI into a cross-platform multimodal assistant.
ONAI is currently a personal development project, but feedback, ideas, and technical discussion are welcome.
To contribute:
git clone https://github.com/YOUR_USERNAME/ONAI.gitCreate a feature branch:
git checkout -b feature/your-featureMake your changes, test them, commit them, and open a pull request.
ONAI is currently under active development.
License information will be added as the project reaches a stable release.
Omar Nashiru-Deen
Software Engineering Student AI / Software Engineering Enthusiast
ONAI is an ongoing exploration into multimodal AI, computer vision, conversational systems, memory architectures, and human-computer interaction.
ONAI
See. Understand. Remember.