🔗 Live Web App:
https://flutter-ml-web.web.app
Companion Robot is an AI-powered interactive robotic system designed to operate entirely offline using edge-based intelligence. Compact enough to fit on a desk, the system integrates speech recognition, computer vision, and distance sensing through a Raspberry Pi without relying on cloud infrastructure.
Voice commands are captured via an onboard microphone and processed locally using Large Language Models (LLMs). Visual perception is provided by a camera module, while spatial awareness and collision avoidance are achieved using ultrasonic distance measurement. All computation occurs on-device, ensuring low latency, enhanced privacy, and uninterrupted operation.
The robot provides real-time feedback through an integrated display and audio output. This project demonstrates how affordable hardware combined with efficient AI pipelines can enable autonomous, privacy-preserving robotic assistants.
- Enable fully offline AI interaction
- Achieve real-time perception and response
- Ensure safe autonomous navigation
- Provide web-based control via Flutter
- Maintain modular and scalable architecture
Companion Robot follows a layered architecture, ensuring clean separation of hardware interaction, processing logic, and intelligence.
- Microphone: Captures speech commands.
- Camera Module: Supplies real-time visual data.
- Ultrasonic Sensor: Computes distance using echo timing.
Implemented in Python with Flask, this layer:
- Converts speech to text
- Normalizes sensor signals
- Routes commands between UI and AI
- Maintains real-time API communication
- Runs locally (offline) using LM Studio
- Performs:
- Natural language understanding
- Scene and object analysis
- Contextual decision-making
- Emotion classification
- Speaker: AI-generated speech output
- Display: System state and responses
- Motors: Physical movement execution
- Flutter Web UI: Control, logs, and visualization
| Component | Description |
|---|---|
| Raspberry Pi 4 Model B | Central processing unit |
| Camera Module | Visual perception |
| Ultrasonic Sensor (HC-SR04) | Obstacle detection |
| Microphone | Voice input |
| Speaker | Audio output |
| 3.5″ Touch Display | Visual feedback |
| Motor Driver + Motors | Locomotion |
| Regulated Power Supply | Stable power delivery |
- Frontend: Flutter (Web/Desktop)
- Backend: Python, Flask, Socket.IO
- AI Engine: LM Studio (Local LLMs)
- Vision: OpenCV
- Speech: Web Speech API / PyAudio
- Hardware Control: GPIO
- OS: Raspberry Pi OS
The Flask server acts as the central coordination unit.
Fail-safe logic ensures immediate stop or slowdown when obstacles are detected.
- Emotion mapping from AI responses
- Fullscreen emotion playback
- Duplicate-emotion prevention
- Automatic return to idle state
This enhances human-robot interaction quality.
- Manual control dashboard
- AI chat interface
- Auto Mode hotkey
- Live logs & diagnostics
- Network configuration UI
- Desktop full-screen support
💡 Design Principle:
The interface is inspired by mission-command dashboards, emphasizing safety, clarity, and immediate operator awareness without reliance on keyboard input.
Purpose:
This flow ensures real-time, offline intelligence with minimal latency and maximum safety.
Auto Mode enables autonomous behavior using local AI reasoning and sensor fusion.
Capabilities:
- Continuous environment scanning
- Voice-initiated task execution
- Obstacle-aware motion control
- Emotion-synchronized responses
Safety Behavior:
- Ultrasonic distance monitoring
- Automatic slow-down in congested areas
- Emergency stop override at all times
Auto Mode always yields priority to manual or emergency commands.
| Metric | Description |
|---|---|
| CPU Load | Current Raspberry Pi processing usage |
| Temperature | Live system thermal state |
| Distance (cm) | Ultrasonic obstacle measurement |
| Network Status | API connectivity health |
| Motor State | Active / Idle / Emergency Stop |
Telemetry updates continuously while commands are executed, ensuring operator awareness and system transparency.
The Companion Robot uses an emotion-mapped response system to enhance interaction.
Emotion Triggers:
- AI response classification
- System states (idle, thinking, alert)
- Voice interaction outcomes
UI Behavior:
- Fullscreen emotion playback
- Duplicate emotion prevention
- Automatic fallback to idle state
Examples:
- 😊 Happy → Joke or positive response
- 🤔 Thinking → Scanning or processing
- 🚨 Alert → Obstacle detected
This system improves clarity, relatability, and human-robot communication.
- Ultrasonic obstacle avoidance
- Speed reduction near objects
- Emergency stop mechanism
- Manual override priority
- Non-blocking asynchronous execution
 Manual Control Panel Directional and emergency controls |
 AI Chat Interface Offline command & response system |
 Auto Mode Active AI-driven autonomous behavior |
 Camera Preview Real-time vision input for AI |
 Emotion Display AI emotion-based visual feedback |
 Desktop Full-Screen View Optimized wide-screen layout |
- Capture user input
- Preprocess locally
- Interpret using LLM
- Apply safety constraints
- Execute action
- Provide feedback
- Log system state
- Educational robotics
- Offline AI assistants
- Human-robot interaction research
- Privacy-preserving AI systems
- Smart automation demos
- Follow-me mode
- Patrol / guard mode
- Face recognition (optional)
- Expanded emotion library
- Multi-language interaction
This project is developed for educational and experimental purposes.
Proper safety measures must be followed during physical deployment.
Warning
⚖️ LEGAL NOTICE & LICENSE: CC BY-NC 4.0
Copyright (c) 2026 Bibek Das. All rights reserved.
This repository is strictly for educational and study purposes. Under the Creative Commons Non-Commercial License, you may read and study this code, but you are legally prohibited from using this codebase for any commercial purposes, including startups, monetization, or revenue generation. Unauthorized commercial deployment is strictly tracked and enforced.
⭐ Companion Robot demonstrates the feasibility of intelligent, private, and autonomous edge-AI robotics using affordable hardware.