LipLink

Repository LRAI · App LipLink. The repository is named LRAI; the iPhone app it contains is called LipLink.

LipLink is an iPhone-first visual speech app. It tracks faces and mouth movement during live conversations, recognizes trained phrases and common single words, keeps notes per participant, and checks imported videos for face, mouth, and audio coverage.

Camera frames, uploaded video, speech transcription, notes, and phrase matching are all processed on the device. Firebase provides only sign-in and profile storage. An optional community-training setting shares numerical mouth-movement templates — never video, audio, or images.

Repository structure

LipLink/                   The iOS app — open this in Xcode
  LipLink.xcodeproj          Xcode project
  App/                       App entry point and authentication
  Features/                  Feature modules
    LipCommand/                On-device lip reading (geometry, DTW, lexicon, community training)
    LiveSession/               Live multi-person camera session + speech
    Notes/                     Conversation notes storage and history
    VideoUpload/               Imported-video coverage analysis
  UI/                        Screens, components, settings, privacy policy
  Models/                    Shared data models
  Support/                   Keychain and helpers
  Resources/                 Asset catalog + GoogleService-Info.example.plist
  AppInfo.plist              Info.plist
  LipLink.entitlements       App entitlements

training/                  Optional macOS training pipeline (Python) — not needed to build the app

README.md
LICENSE                    MIT (application source)
THIRD_PARTY_NOTICES.txt    Third-party model and dataset notices

What the app does

• Live lip reading of trained phrases and a built-in word lexicon, on-device.
• Multi-person sessions: introduces each face and attributes phrases per person.
• Optional live captions via Apple's on-device speech recognition.
• Conversation notes saved per participant, with export.
• Video coverage check for imported clips (face / mouth / audio presence).
• Optional community training (off by default) that pools numerical phrase templates across accounts to improve recognition for everyone.

Requirements

• macOS with Xcode 26 or newer
• iOS 26 or newer device or simulator
• A Firebase project with Authentication and Cloud Firestore
• An Apple development team for device signing

Getting started

Just want to try it? Clone, open LipLink/LipLink.xcodeproj, pick a simulator or your iPhone, and run. With no Firebase configuration the app launches straight into guest mode — every on-device feature (live lip reading, phrase training, notes, and video checks) works with no setup and no sign-in. Sign-in, cloud profiles, and community training stay hidden until you add a Firebase config below.

To enable the account features:

1. Clone the repository and open LipLink/LipLink.xcodeproj in Xcode.

2. Change the bundle identifier and select your Apple development team.

3. Create a Firebase project and register an iOS app.

4. Enable the sign-in methods you want (Email/Password, Google, Apple).

5. Download your GoogleService-Info.plist into LipLink/Resources/. GoogleService-Info.example.plist there shows the expected shape; the real file is intentionally git-ignored.

6. Copy your reversed Google client ID into the URL scheme in LipLink/AppInfo.plist.

7. Replace the example associated domains in LipLink/LipLink.entitlements with your own before enabling email-link sign-in.

8. Add Firestore security rules so community-training samples are uid-locked:

   match /communityTemplates/{phraseId}/samples/{userId} {
     allow read: if request.auth != null;
     allow write: if request.auth != null && request.auth.uid == userId;
   }
   

Camera features require a physical iPhone.

Training tools (optional)

The experimental training pipeline lives in training/ and is not needed to build or run the app. Datasets, virtual environments, checkpoints, logs, and downloaded model assets are git-ignored.

cd training
./scripts/setup_macos.sh
.venv/bin/python -m unittest discover -s tests -v

Privacy

Live media stays on the device. Account details are stored by Firebase. Community training is off by default and, when a signed-in user turns it on, uploads only numerical phrase templates (never video, audio, or images). Review the in-app privacy policy and your Firebase security rules before distributing a build.

Limitations

LipLink is an experimental prototype, not a general sentence-level lip reader. The visual recognizer uses trained templates and a word lexicon and works best one word or phrase at a time; accuracy depends on lighting, framing, and individual speech. The video importer reports coverage but does not yet produce a transcript. Do not rely on its output for medical, legal, safety-critical, or accessibility decisions without independent verification.

License

The application source is released under the MIT License (see LICENSE). Third-party model and dataset notices are in THIRD_PARTY_NOTICES.txt and the training pipeline documentation.