ENEM_A — Emotional Nuance Engine

Real-time multimodal emotion detection that fuses text sentiment, voice tone, and facial expression into a unified emotional profile.

What it does

• Captures emotion from three input streams simultaneously
• Fuses modalities with weighted scoring into a dominant emotion + confidence
• Runs interactively (main.py) or as a background service that writes state to disk
• Pushes emotional context to an external AI gateway ("Clara") via signed WebSocket (gateway_client.py)
• Tracks emotional history in SQLite for pattern detection over time

Stack

Python · PyTorch · Hugging Face Transformers (DistilBERT) · SpeechBrain/Wav2Vec2 · OpenCV · FER · WebRTC VAD · Flask dashboard · Kokoro TTS · SQLite

Architecture

data_input_modules/     text, voice, facial expression capture
data_fusion_module/     multimodal weighted fusion
emotional_memory/       SQLite history tracking
prediction_module/      emotional shift prediction
response_module/        adaptive response generation
gateway_client.py       WebSocket push to external AI agent
wizard.py               device setup (mic/camera selection)

Quick start

run_wizard.bat          REM first-time mic/camera setup
RUN.bat                 REM interactive mode
run_service.bat         REM background service + gateway push

Service mode

When running as a service, ENEM writes ENEM_ACTIVE|<timestamp>|<emotion line> to a state file. The gateway client reads this and forwards emotional context to connected AI agents.

Model weights (not in repo)

Download or place locally under models/:

• models/distilbert_emotion/pytorch_model.bin
• models/voice_emotion/pytorch_model.bin

See ENEM_A-spec.txt for full system design, module requirements, and privacy model.