RFD 0011: EdgeTAM/sam3.cpp runtime refactor (U0–U4) + CoreML/ANE serving path (hybrid 6 fps shipping; all 4 stages exported, pure-CoreML 20 fps)

CMakeLists.txt

@@ -27,6 +27,42 @@ target_include_directories(sam3 PUBLIC
 target_link_libraries(sam3 PUBLIC ggml)
 target_compile_features(sam3 PUBLIC cxx_std_14)
 
+# RFD 0011 U0: per-frame "hold" timing instrumentation. OFF by default so the
+# library has strictly zero timing overhead in normal builds (the SAM3_TIME_*
+# macros expand to nothing and sam3_take_frame_timing() returns zeros). The
+# RFD-0011 EdgeTAM bench requires this ON. PUBLIC so the definition propagates
+# to consumers (e.g. sam3_edgetam_bench), keeping the bench's own copy of the
+# HoldFrameTiming layout in agreement with the library's.
+option(SAM3_TIMING "Compile per-frame hold-timing instrumentation into sam3" OFF)
+if(SAM3_TIMING)
+    target_compile_definitions(sam3 PUBLIC SAM3_TIMING)
+endif()
+
+# RFD 0011 (CoreML/ANE hybrid): run the EdgeTAM image encoder via CoreML on the
+# Apple Neural Engine (~12 ms) instead of the ggml/Metal RepViT path (~122 ms).
+# Apple-only; compiles the Objective-C++ bridge (coreml/edgetam_coreml.mm) and
+# links the CoreML + Foundation frameworks. OFF by default. Even with this ON,
+# the encoder path is gated at RUNTIME by the SAM3_COREML_ENCODER env var (+ the
+# model path in SAM3_COREML_MODEL), so a SAM3_COREML build still defaults to ggml.
+if(APPLE)
+    option(SAM3_COREML "Build the CoreML/ANE EdgeTAM encoder bridge" OFF)
+    if(SAM3_COREML)
+        # Compile the Objective-C++ bridge as CXX with -x objective-c++ rather than
+        # enable_language(OBJCXX). The latter (a) must sit after add_subdirectory(ggml)
+        # or it reclassifies ggml's Metal .m files as C++ and breaks them, and (b) under
+        # CMake 4.x, enabled in this conditional sub-scope, fails to emit a
+        # CMAKE_OBJCXX_COMPILE_OBJECT rule ("cmake may not be built correctly"). Treating
+        # the single .mm as CXX+objective-c++ sidesteps both and never touches ggml.
+        target_sources(sam3 PRIVATE coreml/edgetam_coreml.mm)
+        set_source_files_properties(coreml/edgetam_coreml.mm PROPERTIES
+            LANGUAGE CXX
+            COMPILE_FLAGS "-x objective-c++ -fobjc-arc")
+        target_include_directories(sam3 PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/coreml)
+        target_compile_definitions(sam3 PUBLIC SAM3_COREML)
+        target_link_libraries(sam3 PUBLIC "-framework CoreML" "-framework Foundation")
+    endif()
+endif()
+
 # Examples (only when top-level)
 option(SAM3_BUILD_EXAMPLES "Build example executables" ON)
 if(SAM3_BUILD_EXAMPLES AND CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)

coreml/README.md

@@ -0,0 +1,233 @@
+# EdgeTAM on CoreML / Apple Neural Engine (RFD 0011)
+
+The CoreML/ANE serving path for EdgeTAM — the route to real-time on Apple
+silicon, which the ggml/Metal runtime cannot reach (ggml is ~10× off Apple's
+kernels on the heavy stages). **Both bottleneck stages — the image encoder and
+the memory attention — are wired into the tracker and working.**
+
+## Measured on M4 Pro (verified, FP16, parity vs PyTorch)
+
+| Stage | ggml/Metal | best CoreML | speedup | best device | parity |
+|---|--:|--:|--:|---|--:|
+| Image encoder (RepViT+FPN) | 122 ms | **11 ms** | ~11× | **ANE** | cosine 0.992 |
+| Memory attention | ~105 ms | **17.8 ms** | ~6× | **GPU** | cosine 1.000 |
+
+**Heterogeneous placement matters:** the conv-heavy encoder is fastest on the
+ANE; the attention (matmul+softmax) is fastest on the GPU. The ANE is NOT
+universally faster.
+
+## Live hybrid — both legs (DONE)
+- **Encoder** (ANE): replaces the ggml RepViT+FPN in `edgetam_encode_image`. The
+  256-ch neck features (`neck(trunk(x))[0:3]`, matching ggml `neck_trk`) are
+  exported **channels-last `[1,H,W,D]`**, byte-identical to ggml `[D,W,H]`, so the
+  load is a `memcpy`.
+- **Memory attention** (GPU): replaces the ggml `sam3_build_mem_attn_graph` in
+  `sam3_propagate_single` at the fixed steady-state capacity (7 memory frames ×
+  512 + 16 obj-ptrs × 4 = **3648 tokens**). ggml's `[feature,token]` CPU buffers
+  are byte-identical to the model's `[token,1,feature]`, so `curr`/`memory`/
+  positions hand straight over and the output injects as the decoder's
+  `cond_spatial`. Early/transition frames (capacity ≠ 3648) fall back to ggml.
+
+**Measured (goldeneval, dancetrack, seed-once, tf@0.5):**
+
+| | fps | dt0004 | dt0006 | wrong |
+|---|--:|--:|--:|--:|
+| ggml | 2.66 | 0.976 | 0.939 | 0 / 1 |
+| + CoreML encoder | 3.92 | 0.939 | 0.939 | 0 / 1 |
+| **+ CoreML mem-attn (both)** | **6.11** | **0.939** | **0.939** | **0 / 1** |
+
+**~2.3× over ggml, accuracy held, wrong-person preserved.** The FP16 box-jitter
+(IoU ~0.88 vs ggml) does not move `tracked_fraction` — the tracker is robust to it.
+
+### Run it
+```
+cmake -S . -B build -DSAM3_TIMING=ON -DSAM3_COREML=ON
+cmake --build build --target sam3_edgetam_bench -j
+SAM3_COREML_ENCODER=1 SAM3_COREML_MODEL=<...>/edgetam_encoder_neck_nhwc.mlpackage \
+SAM3_COREML_MEMATTN=1 SAM3_COREML_MEMATTN_MODEL=<...>/edgetam_memory_attention.mlpackage \
+  ./build/examples/sam3_edgetam_bench --n-frames 40 --warmup 20
+```
+
+## What's here
+- `edgetam_coreml.{h,mm}` — Objective-C++ C-ABI bridge: `edgetam_coreml_encode`
+  (encoder) + `edgetam_coreml_memattn` (4-input mem-attn), load/compile a
+  `.mlpackage`, predict, widen FP16→FP32, compute-unit selectable. Behind the
+  `SAM3_COREML` CMake option (Apple-only, OFF by default; default builds
+  unaffected) + runtime env gates `SAM3_COREML_ENCODER` / `SAM3_COREML_MEMATTN`.
+
+## How the CoreML models are produced (tooling vendored under `coreml/`)
+`coreml/export/` produces the models, `coreml/bench/` measures them (see
+`coreml/bench/README.md`; set `SAM3_GE_DIR` to your eval dir for the EdgeTAM ckpt):
+- encoder: `neck(trunk(x))[0:3]` channels-last via the freeze trick
+  (`torch.jit.freeze` + `run_frozen_optimizations` before `ct.convert`).
+  → `edgetam_encoder_neck_nhwc.mlpackage`.
+- memory attention: `export/convert_memattn_coreml.py` (real-valued RoPE + constant-shape
+  reshapes + rank-≤5 k-rope). → `edgetam_memory_attention.mlpackage`.
+- mask decoder: `export/convert_maskdec_coreml_nhwc.py` (channels-last NHWC).
+  → `edgetam_mask_decoder_nhwc.mlpackage`.
+- memory-encode (stage 4): `export/convert_memenc_coreml.py` (memory_encoder +
+  spatial_perceiver) — constant-shape perceiver windowing + head-split + injected constant
+  PEs. **Exported, parity cosine 1.000000, ~5 ms on ANE/GPU.**
+  → `edgetam_memory_encode.mlpackage`. All 4 stages are now exported.
+
+## Pure-CoreML pipeline — MEASURED 19 fps (the real-time path)
+
+A standalone pure-CoreML chain (`audits/goldenclip-eval/pipeline_coreml.py`, no
+ggml) chains the three CoreML stages with real output→input handoffs and times
+the per-frame loop on this M4 Pro (100 frames):
+
+| stage | ms (3-stage run) | ms (4-stage run) |
+|---|--:|--:|
+| encoder (ANE) | 21.0 | 19.3 |
+| memory attention (GPU) | 23.8 | 17.7 |
+| decoder (ANE) | 7.3 | 6.5 |
+| memory-encode (ANE/GPU) | — | **5.0** |
+| **glue / handoff** | **0.17** | — |
+| **per-frame** | **52.3 → 19.1 fps** | **49.3 → 20.3 fps** |
+
+**The 4th stage now exports** (`export/convert_memenc_coreml.py`, parity cosine
+**1.000000**), so the full pipeline is **measured at 20.3 fps**, not estimated — and
+memory-encode is only ~5 ms (Egor's "unsung hero"), so the earlier ~15-16 fps estimate
+was pessimistic. All 4 stage models are now parity-verified.
+
+**This settles the 6-vs-15-20 question.** The cross-stage handoff is **0.17 ms**
+(negligible) — so the hybrid's 6.1 fps was **entirely** the ggml↔CoreML
+marshalling (84 MB decoder inputs, per-stage input copies), NOT chaining or
+compute. A pure-CoreML pipeline (everything on the CoreML side, no ggml
+round-trips) hits **19 fps for the 3 stages**, and the **full 4-stage chain is now
+measured at 20.3 fps** (`bench/pipeline_coreml_4stage.py`) — the 4th stage
+(memory-encode) exported once its perceiver head-split was given the same
+constant-shape patch the other stages used, and it runs only ~5 ms — matching the
+EdgeTAM-paper / PABannier "15–20 on M-series" figure. Per-stage times vary a bit run
+to run (ANE/GPU contention back-to-back), but the throughput is real.
+
+**Bottom line:** the **hybrid (6.1 fps)** is the validated, accuracy-held config shipping
+in this PR (CoreML stages bolted onto the proven ggml tracker). The **pure-CoreML pipeline
+(20.3 fps full 4-stage sequential / 23.7 fps threaded-3-stage)** is *measured*
+(`bench/pipeline_coreml*.py`). The **C++ runtime core** is now **built + measured**
+(`examples/sam3_coreml_pipeline.cpp`): no-GIL threading reaches **~15-20 fps sustained
+(~27 cold)** and is **placement-critical** — encoder ANE ∥ consumer all-GPU gives a 1.1-1.35×
+threaded speedup, but putting 3 stages on the ANE makes threading *regress to 0.62×*. It does
+**not** yet hold a stable 28-30; the GIL was not the dominant cap (per-frame ANE↔GPU
+switching + thermal + CPU-marshalling contention are). See `coreml/RUNTIME.md` for the
+findings + remaining levers (resident buffers, fewer unit transitions, host-side memory bank,
+cgo). **All 4 stage models are exported + parity-verified** (encoder 0.992, mem-attn 1.000,
+decoder 0.999, memory-encode 1.000).
+
+## U5 threaded pipeline — MEASURED (the real-time leg, native coremltools)
+
+`audits/goldenclip-eval/pipeline_coreml_threaded.py` runs the three CoreML stages as a
+3-thread / 2-bounded-queue pipeline (`Queue(maxsize=2)`), overlapping frames across
+compute units exactly as Egor does; `encoder_compute_sweep.py` is the per-unit encoder
+benchmark. Measured on this M4 Pro.
+
+**Encoder compute-unit sweep — Egor's CPUOnly surprise does NOT reproduce here:**
+
+| unit | p50 ms |
+|---|--:|
+| CPUOnly (BNNS/AMX) | 44.6 |
+| **ANE (CPUAndNeuralEngine)** | **16.7** |
+| GPU (CPUAndGPU) | 20.4 |
+| ALL | 23.9 |
+
+On M4 Pro the ANE wins by 2.7× over CPUOnly — the reverse of his M1 Pro (where CPUOnly
+won). M4's Neural Engine is far stronger, our encoder is RepViT (conv/ANE-friendly) not
+his ViT, and native coremltools compiles the whole graph to the ANE instead of ORT's
+partitioned fallback. **Encoder stays on the ANE — validated, not a gap.**
+
+**Threaded pipeline throughput (100 frames; sequential baseline 19.1 fps):**
+
+| decoder placement | fps | speedup |
+|---|--:|--:|
+| dec → ANE (enc+dec share the ANE) | 22.5 | 1.18× |
+| **dec → GPU (best)** | **23.7** | **1.24×** |
+| dec → CPU | 20.8 | 1.09× |
+
+**The pipeline works, but the lever is ~1.24×, not Egor's 1.9×.** Direct probe: two
+concurrent predicts on different units (enc ANE ∥ mem-attn GPU) overlap only **~1.15×**,
+far below the 2× that "physically separate silicon" implies. Why: `coremltools.predict()`
+releases the GIL *only during kernel execution*; the per-call CPU-side marshalling
+(MLMultiArray alloc, input/output copy, graph orchestration) holds it, and that CPU
+portion is a large fraction of each predict — so two predicts mostly serialize on the
+CPU/GIL. Co-resident models also contend (encoder 16.7 → 20–30 ms with a second model
+loaded). Egor's 1.9× came partly from splitting out a genuinely-CPU preprocess stage
+(pure cv2/numpy, clean CPU∥accelerator overlap) that our chain doesn't have, on ORT
+(whole-`run()` GIL release) + EfficientTAM + M1 Pro.
+
+**Production caveat:** this is the *Python-harness* number. The C++/Obj-C++ bridge path
+has no GIL, can keep `MLMultiArray`s resident, and can use GCD / async `MLModel`
+prediction — it should overlap better than Python threads, but that is **unmeasured**.
+The other unmeasured lever is the decoder's **84 MB/frame** input (the 32/64-ch
+`conv_s0`/`conv_s1` shrink described below) which would lighten the heaviest stage and
+rebalance the pipeline.
+
+---
+
+## Alignment with Egor Dmitriev's CoreML methodology (part-3)
+
+Egor's [part-3](https://egordmitriev.dev/blog/2026-05-18-optimizing-samurai-part-3)
+is the reference deployment for this exact model class on Apple silicon. He runs the
+SAM2/EfficientTAM ONNX modules through **ONNX Runtime's CoreML EP**; we run **native
+`coremltools` MLProgram models** behind a CoreML.framework bridge, bolted onto the ggml
+tracker. Different substrate, same physics. Where we land vs his steps:
+
+| Egor's part-3 step | here | match |
+|---|---|---|
+| **MLProgram** container (his #1 win: ORT op-coverage 9%→93%, 3.2×) | `coremltools` emits MLProgram by default | ✓ same target (free for us; he had to flip ORT off its legacy NeuralNetwork default) |
+| **Kill RoPE dynamic int-ops** that fragment the graph (Gather→Split/Squeeze, ScatterND→Split/Concat, found in Netron) | real-valued RoPE + constant-shape reshapes + `jit.freeze`, at the torch level | ✓ same fix, one layer earlier (we never emit ONNX) |
+| **Heterogeneous per-module compute units** | encoder→ANE, mem-attn→GPU, decoder→ANE | ✓ principle; winners differ (gap below) |
+| **FP16**, verify small accuracy cost | FP16 exports; box-jitter doesn't move `tracked_fraction` | ✓ (he: −0.012 mIoU; us: same shape) |
+| **Boundary-crossing cost dominates kernel speed** | quantified: 0.17 ms glue proves the hybrid's 6 fps was *all* ggml↔CoreML marshalling | ✓ same lesson, different boundary |
+| **3-stage threaded pipelining** (his 15.9→29.6 fps, ~1.9×) | **built + measured: 1.24×** (19.1→23.7 fps), see U5 section | ⚠ lever is far weaker on this stack |
+| runtime = **ORT CoreML EP over ONNX** | native `coremltools` + ggml hybrid | different by design |
+
+**Both gaps are now MEASURED (U5 build, see the section above) — and the result corrects
+my earlier optimistic estimate:**
+1. **Pipelining yields 1.24×, not 1.9×.** Threaded 3-stage pipeline = 23.7 fps (best,
+   dec→GPU) vs 19.1 sequential. Cross-unit overlap is only ~1.15× because
+   `coremltools.predict()` holds the GIL during its CPU-side marshalling. Egor's 1.9×
+   does **not** transfer to native coremltools at the same magnitude.
+2. **The CPUOnly encoder finding does not reproduce.** On M4 Pro the encoder is fastest on
+   the **ANE** (16.7 ms), 2.7× faster than CPUOnly (44.6 ms) — the opposite of his M1 Pro.
+   Our ANE placement was already optimal here.
+
+**Corrected FPS reconciliation:** 19.1 fps sequential → **23.7 fps pipelined (measured,
+3 stages)**. Egor's 28–30 came from a ~1.9× pipelining lever that this stack does not get.
+Closing the rest is *unmeasured* work, not a free pipelining step: a GIL-free C++ pipeline,
+the 32/64-ch decoder-input shrink, and the 4th-stage (memory-encode) export. My earlier
+"≈1.9× → 28–30" line was wrong on measurement; the real lever here is 1.24×.
+
+Two corrections from reading the post: part-3 is **plain FP16** (the attention-in-FP32
+mixed-precision was part-2 / TensorRT), and part-3 leans on **CPU/GPU, not the ANE** (he
+skips the ANE for memory_attention too) — the opposite of our encoder placement, which is
+why the CPUOnly encoder benchmark is worth running.
+
+## Decoder leg in the HYBRID — transfer-bound, NOT recommended there (2-leg is optimal)
+NB: the decoder being "transfer-bound" is a **hybrid-only** artifact (its 84 MB
+256-ch high-res inputs cross the ggml↔CoreML boundary each frame). In the
+*pure-CoreML* chain above the decoder is just 7 ms with ~0 handoff — so it's only
+a problem when bolted onto ggml.
+A third leg (CoreML mask decoder, `SAM3_COREML_DECODER`) is implemented and
+**functionally correct** (goldeneval dt0004 0.913, tracks, wrong-person 0), but it
+is a **net regression** and is OFF by default:
+
+| config | fps | dt0004 |
+|---|--:|--:|
+| **2 legs (encoder + mem-attn)** | **6.11** | **0.939** |
+| 3 legs (+ decoder) | 2.62–5.0 | 0.913 |
+
+Why: the decoder *compute* is fast on CoreML (80 ms ggml → ~11 ms, export parity
+0.999), but its **256-ch high-res inputs are ~84 MB/frame** (`feat_s0` 67 MB +
+`feat_s1` 17 MB). CoreML's `predict()` copies inputs to the compute device every
+call, and that transfer costs more than the ggml decoder — which keeps those
+features on-device. So moving the decoder out trades a cheap on-device op for an
+expensive host→device copy. The slight accuracy dip is extra FP16 compounding.
+
+**The fix (the real remaining work):** feed the decoder **32/64-ch** high-res
+features (apply the decoder's `conv_s0`/`conv_s1` 256→32/64 inside the *encoder*
+export) → ~13 MB/frame instead of 84 MB. Or fuse encoder+decoder into one CoreML
+graph so the high-res features never leave the device. Either makes the decoder a
+real win and, with memory-encode, lands the EdgeTAM-paper ~15 fps. The decoder
+export + bridge + integration here are the reusable building blocks for that.
+
+**Recommended config: 2 legs (encoder ANE + mem-attn GPU) — 6.11 fps, accuracy held.**