Summary
Research epic capturing two future directions for accelerating KPM/NFT detection, motivated by what the pyramid perf series (#200–#208) revealed. Two independent tracks; either could be split into its own issue.
Motivation (what #200–#208 taught us)
So the biggest future wins are architectural, not in the pyramid.
Track A — GPU compute via wgpu / WebGPU
wgpu targets Vulkan/Metal/DX12/GL natively and WebGPU in the browser, with WGSL compute shaders.
Best-fit targets (NOT the pyramid):
- Descriptor matching — brute-force Hamming (XOR + popcount) over N×M descriptors is a textbook GPU win; best isolated experiment.
- FREAK extraction — parallel per keypoint (some irregular gather).
- Pyramid/DoG — only worth it inside a resident GPU pipeline, never standalone.
- Hough/RANSAC — poor GPU fit; keep on CPU.
Key rule: keep data GPU-resident across stages. Porting one stage means a per-frame upload/download round-trip that would dwarf the compute — for the small images here it would likely be slower (a sharper version of the SIMD finding). Only a contiguous resident chunk pays off.
Why it's compelling for this repo: WebGPU gives real GPU parallelism in the browser and sidesteps the wasm-threading friction (no SAB/COOP-COEP/nightly needed). For a web AR library this is arguably stronger than the native case.
Costs: bit-exactness breaks (GPU f32 won't match the -ffp-contract=off C++ baseline → a separate, tolerance-tested backend, not dual-mode-parity); async dispatch + buffer/layout management + WGSL shaders; a 4th backend to maintain (scalar/SIMD/rayon/gpu).
Track B — Replace DoG+FREAK with ORB (FAST + rBRIEF) from purecv
webarkit/purecv (already our CV dependency) provides ORB. ORB = oriented FAST corners + rotated BRIEF binary descriptors.
Why it could be a bigger win than any pyramid optimization:
- FAST is integer corner detection — it eliminates the entire f32 Gaussian scale-space pyramid (the thing we've been optimizing). ORB uses a cheap image pyramid + FAST per level instead of DoG.
- rBRIEF descriptors are binary (like FREAK) → the existing Hamming-matching path largely carries over.
- ORB is explicitly designed for real-time; reusing purecv reduces code we maintain.
Considerations / risks (this is accuracy-sensitive, not just perf):
- Match quality / robustness must be validated — ORB's scale handling is coarser than DoG scale-space; for planar NFT tracking we must confirm match rate and pose accuracy don't regress (measure on the
pinball fixture vs current DoG+FREAK).
- Reference-dataset format changes —
.fset3 stores FREAK descriptors at DoG keypoints; ORB needs rBRIEF-at-FAST descriptors → new reference format + marker dataset regeneration, breaking compat with existing assets and with jsartoolkitNFT's format.
- Breaks C++ parity entirely (different algorithm) — no
dual-mode comparison.
- Assess maturity/quality of purecv's ORB first.
Suggested next steps (spikes, measure first)
- Track A spike: prototype GPU brute-force Hamming matching in
wgpu, benchmark vs the current CPU matcher at realistic descriptor counts; measure including transfer. Decide native vs WebGPU-first.
- Track B spike: wire purecv ORB as an alternative detector behind a feature/flag; on the
pinball fixture compare (a) detection+match time and (b) match rate / pose error vs DoG+FREAK.
- Gate either on a measured win without an unacceptable accuracy regression.
References
Summary
Research epic capturing two future directions for accelerating KPM/NFT detection, motivated by what the pyramid perf series (#200–#208) revealed. Two independent tracks; either could be split into its own issue.
Motivation (what #200–#208 taught us)
wontfix) and rayon gave 1.5–2.4× on the kernel but only ~1.05–1.09× end-to-end onkpm_matching(perf(kpm): parallelize (rayon) the Gaussian pyramid filter passes #207/perf(kpm): parallelize Gaussian pyramid filter passes with rayon (#207) #208) — because the pyramid is just ~10–15% of detection time.SharedArrayBuffer+ COOP/COEP + nightlybuild-std).So the biggest future wins are architectural, not in the pyramid.
Track A — GPU compute via wgpu / WebGPU
wgputargets Vulkan/Metal/DX12/GL natively and WebGPU in the browser, with WGSL compute shaders.Best-fit targets (NOT the pyramid):
Key rule: keep data GPU-resident across stages. Porting one stage means a per-frame upload/download round-trip that would dwarf the compute — for the small images here it would likely be slower (a sharper version of the SIMD finding). Only a contiguous resident chunk pays off.
Why it's compelling for this repo: WebGPU gives real GPU parallelism in the browser and sidesteps the wasm-threading friction (no SAB/COOP-COEP/nightly needed). For a web AR library this is arguably stronger than the native case.
Costs: bit-exactness breaks (GPU f32 won't match the
-ffp-contract=offC++ baseline → a separate, tolerance-tested backend, notdual-mode-parity); async dispatch + buffer/layout management + WGSL shaders; a 4th backend to maintain (scalar/SIMD/rayon/gpu).Track B — Replace DoG+FREAK with ORB (FAST + rBRIEF) from
purecvwebarkit/purecv(already our CV dependency) provides ORB. ORB = oriented FAST corners + rotated BRIEF binary descriptors.Why it could be a bigger win than any pyramid optimization:
Considerations / risks (this is accuracy-sensitive, not just perf):
pinballfixture vs current DoG+FREAK)..fset3stores FREAK descriptors at DoG keypoints; ORB needs rBRIEF-at-FAST descriptors → new reference format + marker dataset regeneration, breaking compat with existing assets and with jsartoolkitNFT's format.dual-modecomparison.Suggested next steps (spikes, measure first)
wgpu, benchmark vs the current CPU matcher at realistic descriptor counts; measure including transfer. Decide native vs WebGPU-first.pinballfixture compare (a) detection+match time and (b) match rate / pose error vs DoG+FREAK.References
wontfix— memory-bound), perf(kpm): parallelize (rayon) the Gaussian pyramid filter passes #207/perf(kpm): parallelize Gaussian pyramid filter passes with rayon (#207) #208 (rayon, modest end-to-end), chore(kpm): decide fate of unused BoxFilterPyramid8u (wire in or remove) #203 (box-filter dead-code)kpm_matching; bottleneck is FREAK/matching/Hough