ios vnc

.devin/skills/ios-simulator-accessibility/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: ios-simulator-accessibility
+description: Reading the iOS Simulator accessibility tree and hit testing it — bridge delegate tokens, full-screen backdrops, app-scoped versus display-scoped results, and reaching web content that the tree cannot see. Use when working on get_tree, element picking or the inspector.
+triggers:
+  - user
+  - model
+---
+
+Reading and hit testing the simulator's accessibility tree, via `AXPTranslator`
+in `packages/accessibility-ios-sys`.
+
+The theme: failures here are **silent and look like absence**. A missed token
+returns an empty label rather than an error, so the symptom is "this element
+cannot be selected" rather than anything that points at the cause.
+
+## Hit testing needs the token applied twice
+
+`objectAtPoint:` returns a platform element whose *own* translation must also
+be given the bridge delegate token — it is not necessarily the translation you
+tokenized on the way in.
+
+Miss that and attribute reads do not fail, they silently return an empty label
+and a zero frame. `get_tree` already does this; see the matching step in
+`get_element_at_point`.
+
+## Backdrops swallow hit tests
+
+Every app has full-screen backdrops: the Application node plus one or more
+container groups. Hit testing empty space resolves to one, and highlighting it
+paints over the whole device. They are filtered out rather than drawn.
+
+## The tree and the hit test have different scopes
+
+The tree is app-scoped and the hit test is display-scoped, so the status bar
+appears in hit tests but never in `get_tree`.
+
+## Reaching web content
+
+`get_tree` walks the frontmost app and so cannot see anything in another
+process; a Safari page reports five elements of chrome. Hit testing *does*
+cross that boundary — `objectAtPoint:` resolves elements inside a `WKWebView`
+while `get_tree` returns only the host app's chrome, and there is no hierarchy
+to traverse from either end. This applies to Safari and to any embedded web
+view.
+
+So `?scan=true` marks everything the tree walk explained on a coverage grid and
+then probes the cells left over. On a real page that takes 5 elements at 4%
+coverage to 15 at 50%, using 114 probes in under half a second.
+
+Swept elements are tagged `point_grid` rather than `recursive`: they are point
+samples with no parent, no children and no document order, and should not be
+treated as equivalent to nodes the tree walk returned.
+
+Coverage is reported whether or not a scan runs, and is a useful signal on its
+own — a full web page reporting 4% describes the problem far better than an
+empty element list does.
+
+Background on the approach, and how idb solves the same problem, is in
+`docs/WEB_CONTENT_ACCESSIBILITY_PLAN.md`.

.devin/skills/ios-simulator-input/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: ios-simulator-input
+description: Injecting touch, keyboard and rotation into the iOS Simulator — the two coordinate spaces, USB HID keycodes, system edge gestures and device settings. Use when working on taps, swipes, scrolling, typing, orientation or anything that drives the simulator.
+triggers:
+  - user
+  - model
+---
+
+Everything about getting input *into* the simulator. The recurring hazard is
+that wrong input is usually accepted without complaint: the event is delivered,
+it just does the wrong thing or nothing at all.
+
+## Coordinate spaces
+
+There are **two** normalized spaces and they only coincide in portrait, which
+makes conflating them very easy and the bug invisible until you rotate.
+
+- **Raw framebuffer space** — what HID input uses. The framebuffer is always
+  portrait-native: rotating the device rotates the *content* inside a
+  fixed-size surface, so pointer coordinates must be un-rotated before
+  injection.
+- **Logical space** — what accessibility uses. iOS has already applied the
+  rotation: in landscape the app reports its own bounds as 874x402 rather than
+  402x874, so normalizing against them yields upright coordinates that need no
+  further rotation.
+
+Concretely, in the web UI a tap sends raw coordinates while a hit test sends
+display coordinates, and AX rects are drawn without rotation.
+
+Normalize AX rects with `(rect.origin - app_bounds.origin) / app_bounds.size`.
+`get_screen_bounds` is only populated after a tree has been read.
+
+## HID keycodes
+
+`IndigoHIDMessageForKeyboardArbitrary` takes **USB HID usage codes**, not
+HIToolbox virtual keycodes — measured, by sending values and reading back what
+appeared in a text field. `a`-`z` are `4`-`29`, Left Shift is `225`. idb's own
+comment claiming HIToolbox is wrong.
+
+The two code spaces overlap in range and disagree on nearly every value, so
+getting it wrong types different letters rather than failing.
+
+Modifiers are ordinary key events held around the target key, so shifted
+characters are Shift-down, key-down, key-up, Shift-up. There is no shift flag
+on the Indigo message.
+
+The character-to-key table lives in `accessibility-serve/src/keymap.rs` and is
+US-ASCII only; unmappable characters fail the whole string rather than typing
+a subtly wrong one.
+
+On Xcode 27 / CoreSimulator 1155.4+ an active `dtuhidd` silently disables
+legacy Indigo keyboard events; they are delivered correctly and produce no
+text. See `docs/IDB_LEARNINGS.md`.
+
+## System edge gestures
+
+Swipe-up-to-home does not work unless the touch is flagged with the screen
+edge it started from. That edge is the 7th argument to
+`IndigoHIDMessageForMouseNSEvent`; on arm64 it lands in x4 while the `NSSize`
+argument occupies d0/d1, so a wrong declaration can silently pass zero there
+and every gesture just becomes an ordinary drag.
+
+The same edge must be supplied for every event in the gesture, and the edge is
+in *raw* framebuffer space, so it rotates with the device.
+
+## Orientation
+
+The framebuffer never changes size, so orientation cannot be recovered from
+the video. It is tracked server-side and seeded at startup from the
+accessibility bounds aspect ratio, which is the only cheap signal — and it
+only distinguishes landscape from portrait, not left from right.
+
+Rotation itself is a GSEvent mach message to `PurpleWorkspacePort`, not an
+Indigo event. It needs Simulator.app running, because the runtime alone does
+not publish that port.
+
+## Device settings
+
+`simctl ui` only implements `appearance`, `increase_contrast` and
+`content_size`. The other options in Xcode's Devices window (reduce motion,
+colour filters, transparency, VoiceOver) have no simctl verb and need a helper
+binary spawned inside the simulator that drives the private libAccessibility
+setters.

.devin/skills/ios-simulator-internals/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: ios-simulator-internals
+description: CoreSimulator and SimulatorKit private API mechanics — remote proxies, blocks, framebuffer port discovery and IOSurface lifetime. Use when touching accessibility-ios-sys or debugging why a private call silently does nothing.
+triggers:
+  - user
+  - model
+---
+
+How to talk to the simulator's private frameworks from Rust, in
+`packages/accessibility-ios-sys`. Every note here cost real debugging time and
+none of it is obvious from the outside. The common thread is that these APIs
+tend to **fail silently** rather than error, so the symptom is usually "nothing
+happens" rather than a crash.
+
+Simulator work needs Xcode, not just Command Line Tools:
+
+```sh
+export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
+xcrun simctl list devices booted
+```
+
+`xcode-select -p` commonly points at `/Library/Developer/CommandLineTools`,
+which has no simulator frameworks, so `DEVELOPER_DIR` matters.
+
+## CoreSimulator hands back proxies
+
+IO ports and display descriptors are `ROCKRemoteProxy` objects that implement
+their interface through forwarding. `objc2::msg_send!` panics on them in debug
+builds because its verification looks the selector up with
+`class_getInstanceMethod`, which a forwarding proxy does not answer.
+
+Use the helpers in `macos/dynamic.rs`, always guarded by `responds_to`.
+
+## Blocks need a type signature
+
+ROCKit marshals block arguments across the proxy boundary by reading the
+block's ObjC type encoding, which requires `BLOCK_HAS_SIGNATURE`. `block2` does
+not emit that flag (there is a TODO in its `global.rs`), so passing an
+`RcBlock` aborts with "Block is missing signature field".
+
+`macos/blocks.c` creates the blocks with clang instead. See
+`macos/void_block.rs`.
+
+## Registering screen callbacks is load-bearing
+
+Registration is what makes SimulatorKit attach the display pipeline and
+populate `framebufferSurface`. Reading the property without registering does
+not reliably work.
+
+## Picking the right framebuffer
+
+Several ports share `com.apple.framebuffer.display` — the main screen plus
+secondary planes. Register on all of them, then pick the descriptor whose
+state reports `displayClass == 0`, falling back to the largest live surface.
+
+A booted iPhone exposes two descriptors, classes 0 and 1. Largest-area happens
+to pick correctly but is a heuristic standing in for a value the API actually
+reports, and it would choose wrong on tvOS, which renders on a non-zero class.
+
+## The framebuffer IOSurface is recycled in place
+
+Retaining the `CVPixelBuffer` does not help, because the surface mutates
+underneath it. The sink must finish with a frame, or copy it, before
+returning. The encoder's pixel transfer is what does that copy.
+
+## SimulatorKit moved in Xcode 27
+
+From `Developer/Library/PrivateFrameworks` to `Contents/SharedFrameworks`.
+Both are probed.
+
+## Verifying without a browser
+
+```sh
+cargo run -p accessibility-ios-sys --example framebuffer_probe
+```
+
+Fails loudly if no frames arrive, if no keyframe is produced, or if any access
+unit is not Annex-B framed.

.devin/skills/ios-simulator-video/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: ios-simulator-video
+description: Capturing, encoding, streaming and recording the iOS Simulator screen — bits per pixel, VideoToolbox rate control, pixel transfer, MP4 recording and browser-side WebCodecs. Use when working on stream quality, frame rate, the encoder or screen recording.
+triggers:
+  - user
+  - model
+---
+
+The capture and encode pipeline, and how to tell whether a change to it
+actually helped.
+
+**Measure before changing anything here.** Most of the notes below exist
+because something that sounded obviously right turned out to be wrong when
+measured.
+
+## Stream quality: read bits per pixel first
+
+`GET /api/stats` reports frames, fps, bitrate, mean frame size, keyframe
+requests, lag events and — the number that matters — **bits per pixel** against
+the *encoded* resolution.
+
+Rules of thumb for screen content:
+
+    < 0.05 bpp    heavy blocking, and the encoder starts dropping frames
+    0.10-0.20     what to aim for
+    > 0.30        wasted bandwidth
+
+## Starving the encoder costs frame rate, not just quality
+
+With low-latency rate control, VideoToolbox drops frames to stay inside its
+per-frame budget, which is `AverageBitRate / ExpectedFrameRate` regardless of
+the rate actually being achieved. Measured on a 1206x2622 device during
+scrolling:
+
+    6 Mbps    12.2 KB/frame   0.0315 bpp   33.9 fps
+    24 Mbps   32.2 KB/frame   0.0835 bpp   60.3 fps
+
+So "chunky, slow and janky" was a single root cause, not three.
+
+The fix is resolution, not bitrate. A phone framebuffer is roughly fifteen
+times the pixels the browser actually displays, so the long edge is capped at
+1280 by default (`--max-dimension`, or `--native-resolution` to disable) and
+the bitrate is derived from the encode resolution at ~0.15 bpp rather than
+being a fixed number. Same stimulus, same bandwidth:
+
+    native 3.16 MP @ 6 Mbps   4.99 Mbps   0.0297 bpp
+    588x1280 @ derived        4.95 Mbps   0.1338 bpp
+
+Raise `--max-dimension` if viewing in a large or retina window; the default
+trades sharpness for bits on the assumption of a normal-sized preview.
+
+`MaxKeyFrameInterval` counts frames, so on its own a "2 second" interval
+stretches to twenty when the device is idle at 5fps.
+`MaxKeyFrameIntervalDuration` is what actually bounds it in time; both are set.
+
+## The frame copy is a pixel transfer, not a memcpy
+
+Three things must happen between the framebuffer and the encoder, and
+`VTPixelTransferSession` does all three in one hardware pass into a pooled
+buffer: copy off the recycled live surface, convert BGRA to NV12, and
+downscale.
+
+This does not contradict the earlier finding that a plain Metal blit was
+*slower* than `memcpy` (0.517 ms against 0.377 ms on a cache-cold 12.1 MB
+surface). That measured a bare copy doing one job against a purpose-built
+transfer doing three; the transfer also removes the BGRA-to-NV12 conversion
+that `VTCompressionSession` would otherwise do internally.
+
+## Latency and quality are one choice, not two knobs
+
+`kVTCompressionPropertyKey_Quality` is **ignored** while
+`EnableLowLatencyRateControl` is set. Measured with it on, quality 1.0 gave
+0.0221 bpp and quality 0.4 gave 0.0223 — no effect whatsoever. Measured with
+it off, quality 0.3 gave 0.70 Mbps and quality 0.9 gave 11.82 Mbps, a
+sixteenfold range.
+
+So the two settings are mutually exclusive, and `Tuning` pairs them into a
+single choice rather than letting the useless combination be expressed:
+
+- `Interactive { bitrate }` — low-latency rate control and `MaxFrameDelayCount`
+  0, spending a bitrate derived from the encode resolution. Omitting
+  low-latency costs roughly 300ms of decoder buffering, so this is what any
+  live viewer wants.
+- `Recording { quality }` — no low-latency constraint, so the quality target is
+  honoured and bits go where the picture needs them. Latency is unbounded in
+  principle.
+
+## Recording
+
+`start_recording` runs a **second, independent encode** of the same frames
+rather than retuning the streaming encoder. That is what lets a recording use
+B-frames: the live path cannot, because WebRTC's payloader and the raw stream
+framing both assume the encoder emits frames in submission order. It also lets
+a recording have its own resolution and quality regardless of what the viewer
+is watching.
+
+`AVAssetWriter` does the encoding and the muxing. Feeding it pixel buffers
+through an `AVAssetWriterInputPixelBufferAdaptor`, rather than encoding
+ourselves and appending sample buffers, avoids having to order decode and
+presentation timestamps by hand — which is precisely what B-frames complicate.
+
+Verify a recording really is what it claims to be:
+
+```sh
+ffprobe -v error -select_streams v:0 -show_entries frame=pict_type \
+  -of csv=p=0 -read_intervals "%+#120" recording.mp4
+```
+
+Expect a mix of `I`, `P` and `B`. All `I` and `P` means frame reordering
+silently failed to take.
+
+Timestamps are wall-clock elapsed at nanosecond timescale, so the variable
+capture rate is recorded at the speed it actually happened. Finalizing blocks
+until the writer flushes: an MP4 has no playable index until then.
+
+## Browser-side decode
+
+Do not pass `hardwareAcceleration: "prefer-hardware"` to `VideoDecoder`.
+Despite the name it is treated as a requirement, and phone-shaped resolutions
+like 1206x2622 exceed what hardware decoders accept, making the configuration
+unsupported outright.
+
+The real WebCodecs member is `optimizeForLatency`, not `optimizeFor`.
+
+`configure()` reports success synchronously and only surfaces the failure
+through the async error callback, so check `isConfigSupported` first.