🧠 Sixth Sense — Architecture Guide

Hearing → Understanding → Feeling → Seeing

---

Project Structure

lib/
├── main.dart                        # App entry, dark theme, orientation lock
├── models/
│   └── sound_event.dart             # SoundEvent, SoundProfile, SoundRegistry, enums
├── screens/
│   ├── setup_screen.dart            # Onboarding: name + feedback mode picker
│   └── home_screen.dart             # Live listening screen, history feed
├── widgets/
│   ├── sound_visualizer.dart        # Animated canvas + overlay text
│   └── sound_history_tile.dart      # History list item
├── painters/
│   └── sound_painter.dart           # CustomPainter — 7 distinct visual styles
└── services/
    ├── audio_service.dart            # Mic capture + ML isolate manager
    └── haptic_service.dart           # Haptic vocabulary engine

---

Architecture at a Glance

Microphone (flutter_audio_capture)
        │ PCM float32 @ 16kHz
        ▼
  AudioService.start()
        │ sends AudioChunk via SendPort
        ▼
  ML Isolate (background thread)
        │ YAMNet .tflite inference
        │ returns InferenceResult
        ▼
  AudioService._handleResult()
        │ resolves SoundRegistry → SoundProfile
        │ emits SoundEvent on stream
        ▼
  HomeScreen._onSoundEvent()
        ├─► SoundVisualizer (CustomPainter, 60fps)
        └─► HapticService.play()

---

The 7 Visual Styles (SoundPainter)

Style	Sounds	What you see
sharpPulse	Siren, Alarm	Rotating angular red spikes, pulsing center
ripple	Rain	Expanding blue concentric circles from center
fracture	Glass breaking	Shards exploding outward, seeded randomness
softWave	Baby crying	Layered yellow sine waves, gentle rhythm
radialBurst	Knock, Car horn	Impact rings expanding from center
steadyGlow	Speech, Music	Morphing organic blob with radial glow
chaosSpike	Thunder	Frame-seeded lightning bolts, chaotic angles

All painters are driven by:

• phase — animation controller value (0→1 looping)
• amplitude — live RMS energy from PCM buffer

---

Haptic Vocabulary

Pattern	Sounds	Feel
singleTap	Speech, Unknown	One light tap
doubleTap	Knock, Car horn	Two medium taps
rapidPulse	Siren, Alarm	5× staccato bursts
continuousWave	Rain, Music	Soft rolling every 200ms
longBurst	Glass, Thunder	Heavy double + vibrate

---

Production Integration Checklist

Step 1 — Real Audio Capture

Replace the Timer in AudioService._startSimulatedCapture():

// pubspec.yaml: flutter_audio_capture: ^1.1.5

import 'package:flutter_audio_capture/flutter_audio_capture.dart';

final _recorder = FlutterAudioCapture();

await _recorder.start(
  (List<double> buffer) {
    _mlSendPort?.send(AudioChunk(buffer, 16000));
  },
  (Object error) => debugPrint('Audio error: $error'),
  sampleRate: 16000,
  bufferSize: 15360,   // 0.96s @ 16kHz = exact YAMNet window
);

Android permissions (AndroidManifest.xml):

<uses-permission android:name="android.permission.RECORD_AUDIO"/>

iOS permissions (Info.plist):

<key>NSMicrophoneUsageDescription</key>
<string>Sixth Sense listens to identify sounds around you.</string>

Step 2 — YAMNet Model

1. Download: https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1
2. Add to assets/yamnet.tflite
3. Download class map: assets/yamnet_class_map.csv
4. Uncomment in pubspec.yaml

Replace _runInference() in the ML isolate:

// pubspec.yaml: tflite_flutter: ^0.10.4

import 'package:tflite_flutter/tflite_flutter.dart';

// Call once at isolate start:
final interpreter = await Interpreter.fromAsset('assets/yamnet.tflite');
final labels = _loadLabels('assets/yamnet_class_map.csv');

// Per-frame inference:
InferenceResult _runInference(AudioChunk chunk) {
  // 1. Compute RMS amplitude
  final rms = sqrt(chunk.samples.map((s) => s*s).reduce((a,b)=>a+b) / chunk.samples.length);

  // 2. Normalize to [-1, 1] float32
  final input = [chunk.samples.map((s) => s.toDouble()).toList()];

  // 3. YAMNet outputs shape [1, 521]
  final output = List.filled(521, 0.0).reshape([1, 521]);

  interpreter.run(input, output);

  // 4. Top-1 label
  final scores = output[0] as List<double>;
  final topIdx = scores.indexOf(scores.reduce(max));

  return InferenceResult(
    labels[topIdx],
    scores[topIdx],
    (rms * 10).clamp(0.0, 1.0),
  );
}

Step 3 — Extend SoundRegistry

Add any YAMNet label (see yamnet_class_map.csv) to SoundRegistry._profiles:

'Dog': SoundProfile(
  label: 'Dog',
  displayName: 'Dog Barking',
  priority: SoundPriority.medium,
  primaryColor: Color(0xFFFF8C42),
  secondaryColor: Color(0xFFFFD700),
  visualStyle: VisualStyle.radialBurst,
  hapticPattern: HapticPattern.doubleTap,
  emoji: '🐕',
),

---

Tuning Tips

• Confidence threshold: adjust if (result.confidence < 0.55) in AudioService._handleResult()
• Animation speed: _controller.duration in SoundVisualizerState.didUpdateWidget()
• History length: if (_history.length > 30) in HomeScreen._onSoundEvent()
• Haptic intensity: scales with event.amplitude in HapticService._rapidPulse()