Skip to content

feat(cv): on-demand Python CV worker lifecycle + Windows setup#18

Merged
lukitasxue merged 5 commits into
mainfrom
feat/cv-electron-bridge
Jun 16, 2026
Merged

feat(cv): on-demand Python CV worker lifecycle + Windows setup#18
lukitasxue merged 5 commits into
mainfrom
feat/cv-electron-bridge

Conversation

@rxv801

@rxv801 rxv801 commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Summary

Connects the Electron app to the Python CV worker, and changes the worker so it only runs while something actually needs detection instead of staying up for the whole app session. Also adds the onboarding camera-check screens that use it, plus a Windows setup script.

What this PR does

On-demand CV worker lifecycle (python-bridge.ts)

  • The worker is reference-counted. It starts on the first request and stops a short grace period (2s) after the last release, so nothing runs while no session or check is active. The grace period keeps it from dying and respawning on a quick pause/resume, a page change, or React StrictMode's double-mount in dev.
  • If the worker crashes on its own it gets relaunched, capped at 5 rapid restarts so a broken setup can't spin forever. A worker that stays up long enough resets that counter.
  • If the venv is missing it logs a plain "run setup" message instead of a cryptic ENOENT. Detection just stays off and the rest of the app keeps working.
  • Removed the launch-time start from index.ts. The quit-time stop stays as a safety net.

Renderer to main wiring

  • ipc-handlers, preload, and vite-env expose window.taskmaster.cv.request / release.
  • useCvDetection asks for the worker before opening its WebSocket and releases it on cleanup.

Onboarding camera checks

  • New OnboardingFaceCheck and OnboardingPhoneCheck pages stream the camera to the worker through useCvDetection and show live feedback. Wired into OnboardingPage.

Windows setup

  • setup.ps1 is a PowerShell version of setup.sh (venv 3.11, pip deps, model downloads, npm install). The README points to both.

Testing

I ran the full app, not just the build. The worker started only once the renderer asked for it, MediaPipe gaze ran on a live camera frame over the WebSocket, and it shut down cleanly on quit. tsc -b && vite build and eslint . are both clean.

Notes

This branch already has the latest main merged in, including #16 (mini-timer). The two features don't touch each other and run side by side.

rxv801 added 2 commits June 15, 2026 17:18
Run the CV worker only while detection is wanted instead of for the whole
app session. The renderer requests/releases the worker via IPC; it is
reference-counted in the main process and started on the first request,
stopped a short grace period after the last release.

- python-bridge.ts: reference-counted request/release with a stop grace
  period, crash auto-restart (capped to avoid crash loops), and a loud
  check that fails clearly when the venv is missing
- index.ts: drop launch-time start; keep quit-time stop as a safety net
- ipc-handlers / preload / vite-env: expose window.taskmaster.cv.{request,release}
- useCvDetection: request the worker before connecting, release on cleanup
- onboarding: face/phone check pages wired to the worker via useCvDetection
- setup.ps1 + README: Windows PowerShell setup mirroring setup.sh
# Conflicts:
#	electron/src/main/index.ts
#	electron/src/main/ipc-handlers.ts
#	electron/src/preload/index.js
#	electron/src/renderer/vite-env.d.ts
@lukitasxue

Copy link
Copy Markdown
Collaborator

@copilot check for possible security issues

Copilot AI commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

@copilot check for possible security issues

Addressed in a108806.

I reviewed the CV worker path and added hardening in python/main.py:

  • reject oversized WebSocket frames (MAX_FRAME_BYTES, close with code 1009)
  • guard detector execution with exception handling so malformed input can’t crash-loop the worker

Validation run: Electron lint/build, Python compile check, secret scan, and CodeQL (0 alerts).

@lukitasxue

Copy link
Copy Markdown
Collaborator

Hey Saatvik, this PR looks good overall and I like the direction of the on-demand CV worker. I just noticed 2 small things I think we should fix before merging.

First, setup.ps1 still tells Windows users to run python cv\detection_loop.py at the end, but this PR now uses python/main.py through Uvicorn as the worker entry point. That instruction could confuse someone setting up the project. I’d either update it to something like:

cd python
.\.venv\Scripts\Activate.ps1
python -m uvicorn main:app --port 8765

or simply say that after running setup, Electron will start the CV worker automatically when detection is needed.

Second, in useCvDetection, we call window.taskmaster.cv.request() before opening the camera. If getUserMedia() fails because the camera is denied, missing, busy, or the saved camera ID is invalid, the catch block only logs the error. That means the Python worker request may stay active while the user remains on the screen, even though no frames are being sent. Cleanup will release it on unmount, but I think we should release it immediately if startup fails.

I’d fix it by tracking whether the CV worker was requested, then calling window.taskmaster.cv.release() inside the catch path when startup fails.

Apart from that, the architecture makes sense to me.

rxv801 added 2 commits June 16, 2026 19:08
…un hints

useCvDetection requested the CV worker before opening the camera, but if
getUserMedia failed (camera denied/missing/busy, invalid saved device id) the
catch only logged — leaving the worker requested until unmount even though no
frames were being sent. Release it immediately on startup failure, guarded so
the unmount cleanup doesn't double-release the reference count.

Also fix the post-setup hints in setup.ps1/setup.sh: they pointed at the old
cv/detection_loop.py entry point. The worker now runs as uvicorn main:app and
is started on demand by Electron, so the scripts now say that and give the
standalone uvicorn command for testing.
# Conflicts:
#	electron/src/main/index.ts
#	electron/src/main/ipc-handlers.ts

@lukitasxue lukitasxue left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good now!

@lukitasxue lukitasxue merged commit 1048f64 into main Jun 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants