A streaming multimodal memory system for smart glasses, web capture, and everyday-life question answering.
Overview | Demo | System Design | Quick Start | Repository Layout | Related Works | Privacy
LightMem-Ego is an end-to-end egocentric memory system for everyday-life assistance. It connects a Rokid AI Glass Android app, a browser frontend, and an online backend service so users can stream first-person camera/audio context, build structured memory from daily experience, and ask questions about current or past moments.
The system organizes continuous visual-audio experience into three memory scopes:
- Current memory for ongoing scene understanding.
- Short-term memory for recent events, actions, and conversations.
- Long-term memory for consolidated episodes, routines, preferences, and semantic facts.
LightMem-Ego is designed for practical scenarios such as object finding, conversation recall, life summarization, routine discovery, and hands-free wearable assistance.
- Streaming egocentric capture: captures first-person visual frames and microphone audio from smart glasses or the browser.
- Timeline-aligned multimodal memory: aligns frames, audio chunks, transcripts, and metadata on a shared session timeline.
- Hierarchical memory organization: maintains current, short-term, and long-term memory for different temporal scopes.
- Memory-grounded question answering: retrieves timestamped multimodal evidence before generating answers.
- Glasses + web deployment: supports a Rokid AI Glass app for hands-free interaction and a browser frontend for desktop/mobile use.
- Modular backend: separates stream ingestion, session management, memory construction, retrieval, and QA workers.
Demo video: YouTube · Bilibili
LightMem-Ego is organized as three cooperating components:
-
AI Glass App Captures first-person camera frames and microphone audio, controls live sessions, submits voice questions, and displays memory-grounded answers on the glasses.
-
Backend Service Receives live streams, manages sessions, builds current/short-term/long-term memories, retrieves evidence, and returns answers.
-
Web Frontend Provides a browser interface for live capture, memory interaction, session review, and backend-powered QA.
Web frontend \
-> Backend API and workers -> Memory / retrieval / QA
Rokid AI Glass app /
At runtime, either the web frontend or the glasses app can open a live session with the backend, send visual/audio data, and receive memory-grounded answers.
src/
frontend/ # Vite + React web frontend
backend/ # FastAPI service, online workers, and memory-processing logic
ai_glass_app/ # Rokid AI Glass Android app
Component documentation:
The web frontend is a Vite + React app. It supports browser camera/microphone capture, session start/stop, live ingest controls, question submission, answer display, and evidence review.
The API base URL is configured with VITE_API_BASE_URL at build time, with a production fallback in online_web/src/api/lightmem_egoApi.js.
The backend is a FastAPI-based online server. It exposes stream and query APIs, manages live sessions, runs workers for preprocessing/ASR/memory updates, and serves memory-grounded answers.
The backend uses the src/em2mem/ runtime package for memory, LLM, and embedding components. Runtime sessions, logs, model weights, generated indexes, and private .env files are intentionally excluded.
The glasses app is an Android client for Rokid AI Glass. It starts and stops live capture, streams camera/audio data, records short voice questions, and renders answers on a glasses-friendly UI.
The backend endpoint is configured in:
src/ai_glass_app/app/src/main/java/cn/zjukg/lightmem/glass/lightmem_ego/LightMemEgoConfig.kt
Each component has its own setup and runtime requirements. Start with the README for the component you want to run.
cd src/frontend/online_web
npm install
npm run devFor production build:
npm run buildcd src/backend
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .
cp .env.example .env
scripts/start_api.shStart the default online worker set:
scripts/start_online_all_workers.shWindows:
cd src\ai_glass_app
.\gradlew.bat assembleDebugmacOS or Linux:
cd src/ai_glass_app
./gradlew assembleDebug
| Scenario | Example Query | Memory Scope |
|---|---|---|
| Object Finding | "Where did I leave my badge?" | Current / short-term memory |
| Conversation Recall | "What did the doctor tell me after checking the report?" | Short-term memory + transcript context |
| Life Summarization | "What did I do this afternoon?" | Short-term and long-term memory |
| Routine Discovery | "What do I usually do after arriving at the office?" | Long-term semantic memory |
| Wearable Assistance | "What am I looking at now?" | Current memory |
This repository belongs to ZJUNLP LightMem series, focusing on solving context bloat, excessive token consumption and low cache utilization for long-running LLM agents:
- LightMem — A lightweight and efficient memory management framework designed for Large Language Models and AI Agents
- LightMem2 — A modular framework for long-running agent memory and context management
- LightMem-Ego — A lightweight streaming multimodal memory system for everyday-life assistance
LightMem-Ego may process camera frames, microphone audio, transcripts, generated answers, and memory data depending on deployment configuration. Before deploying with real users, review endpoint configuration, data retention policy, access control, and user consent flow.
This repository is intended for research and demonstration. Production deployments should implement privacy-preserving capture, sensitive-content filtering, encrypted storage, access control, retention/deletion policies, and user-controlled memory editing.
See LICENSE.
Paper and citation information will be added when available.
LightMem-Ego builds on the broader line of work on memory-augmented agents, egocentric multimodal understanding, and wearable AI assistants. We thank all contributors and collaborators who helped develop the system.