focused_window_detector

Standalone Swift + CoreML detector for macOS windows. Single-pass RF-DETR (nano) over the live screen → live overlay window with bounding boxes.

Classes: focused_window, unfocused_window, cursor.

The model is strong at separating focused vs unfocused windows in real time. Known weak spots (cursor + Dock) are listed under Model improvements needed.

Model weights are hosted on Hugging Face: https://huggingface.co/Cianmcnally/focused-window-detector (fetched automatically into ./model/ on first run).

---

Contents

focused_window_detector.swift   # source (ScreenCaptureKit -> CoreML -> AppKit overlay)
build.sh                        # compile
download_model.sh               # fetch CoreML model from Hugging Face into ./model/
run.sh                          # fetch model + build + run
README.md                       # this file

---

Quick start

git clone https://github.com/cianmcnally/focused_window_detector
cd focused_window_detector
bash run.sh

run.sh downloads the model from Hugging Face (first run only), builds the binary, and launches. A live overlay window opens with detection boxes drawn on the screen. Quit with Cmd-Q or close the window.

Requirements

• macOS on Apple Silicon, Swift toolchain (Xcode command line tools).
• hf CLI for the model download: pip install huggingface_hub.
• Screen Recording permission for your terminal (System Settings → Privacy & Security → Screen Recording), else frames are black and nothing is detected.

---

Manual steps

bash download_model.sh   # fetch model -> ./model/
bash build.sh            # compile -> ./focused_window_detector

Then run with your own flags:

./focused_window_detector \
    --model ./model/rf-detr-nano-checkpoint_best_total-2-fp32.mlpackage \
    --width 384 --height 384 \
    --boxes-name var_2267 --logits-name var_2270 \
    --display-scale 2

---

Flags

--model PATH               CoreML .mlpackage
--width N --height N        model input size (this model: 384 x 384)
--fps N                     capture fps (default 30)
--threshold F               detection score threshold (default 0.5)
--display-scale F           overlay window scale
--no-cursor                 hide the cursor box
--no-display                run headless (metrics only, no window)
--logits-name S             logits output tensor name (this model: var_2270)
--boxes-name S              boxes output tensor name  (this model: var_2267)
--save-metrics-csv PATH     write per-frame timing to CSV

---

Model notes

• Input: 384 x 384 (--width 384 --height 384).
• Output tensors are named var_2267 (boxes, cxcywh normalised) and var_2270 (logits) — passed via --boxes-name / --logits-name in run.sh.
• If boxes render in the wrong places, swap those two flags — the name order is by convention, not read back from the model.

---

Model improvements needed

Window detection (focused / unfocused) is solid. The remaining weaknesses are around the cursor and one Dock confusion. These are training-data problems, not code problems — fix them by re-capturing / augmenting the dataset and retraining.

1. Better cursor variance recognition

The model only reliably catches a narrow range of cursor appearances. macOS cursors change size and shape constantly — arrow, I-beam, resize arrows, open/closed hand, loading spinner, text cursor, plus accessibility size scaling. The current training data does not cover this spread.

Fix: use the cursor separator to expand the cursor training data so it includes all cursor sizes and shapes. Isolating the cursor lets us synthesize / augment many cursor variants into frames, giving the model far broader coverage than hand-captured screens provide.

2. More precise cursor bounding boxes

Even when the cursor is detected, the bounding box is loose / imprecise — it doesn't tightly fit the actual cursor pixels. Downstream anything that relies on cursor position (click targeting, hover analysis) suffers from the slack.

Fix: tighter, pixel-accurate cursor annotations in the training data (again, the cursor separator gives clean cursor masks/boxes to train against).

3. Dock misclassified as an unfocused window on hover

When the cursor hovers over the bottom bar (the Dock), the model marks it as an unfocused_window. That is wrong — the Dock is not a window and should not be detected as one.

Fix: add training frames with the Dock hovered / magnified (and other system chrome) explicitly labelled as not a window, so the model stops firing unfocused_window on the Dock.

---

Summary of retraining work

Issue	Root cause	Fix
Misses many cursor shapes/sizes	Narrow cursor coverage in data	Cursor separator → augment all cursor variants
Loose cursor boxes	Imprecise cursor annotations	Tight pixel-accurate cursor labels
Dock read as unfocused_window on hover	Dock never labelled as non-window	Add hovered-Dock frames labelled not-a-window