ios vnc - #3
Merged
Merged
Conversation
Replaces the need to shell out to `xcrun simctl io screenshot` for frame acquisition by driving SimulatorKit's display pipeline directly. Three things made this non-obvious: - CoreSimulator vends its IO ports and display descriptors as ROCKRemoteProxy objects that implement their interface via forwarding, so objc2's msg_send! rejects them in debug builds. Messages now go through raw objc_msgSend helpers guarded by respondsToSelector:. - ROCKit marshals block arguments across the proxy boundary by reading the block's ObjC type encoding, which requires BLOCK_HAS_SIGNATURE. block2 does not emit that flag yet, so the three screen callbacks are created by a small C shim where clang emits a conforming block. - SimulatorKit recycles its framebuffer IOSurface in place, so frames are deep-copied into a pooled buffer before going downstream. Also carries over the seed-based dirty check, the 200ms forced re-emit floor (needed so idle screens still paint for late joiners), and the re-wire retry that covers lazily created descriptors. Verified against a booted iPhone 17: 1206x2622, ~5fps idle, ~68fps animating. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Encodes on the capture queue, since CVPixelBuffer and CMSampleBuffer are not Send and only the compressed bytes need to travel. Emits either Annex-B (for WebRTC) or AVCC plus a separate avcC record (for browser VideoDecoder) from the same session. Configured for interactive latency: low-latency rate control, no frame reordering, MaxFrameDelayCount 0, Baseline profile for broad WebRTC codec negotiation. SPS/PPS are prepended only to IDRs rather than every frame. Verified against a booted iPhone 17: 211 access units over 5s at ~60fps, all Annex-B framed, ~4 Mbps, keyframes on the configured 2s interval. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Adds `accessibility-cli serve-sim`, which serves a live, interactive view of a booted simulator with accessibility element inspection. accessibility-core gains a platform-agnostic `video` module: a VideoCapture trait plus encoded-frame types. It is encoded-frame oriented rather than surface oriented because every platform that can do this has a hardware encoder next to its capture API, and passing raw surfaces across the boundary would force a copy. Only the iOS Simulator implements it; other platforms report it unsupported, matching how capture_screen already stubs. accessibility-serve is a new crate holding the server so that webrtc, axum and their transitive dependencies stay out of the library crates. Notes on the design: - Video is offered over WebRTC by default and as length-prefixed H.264 over a WebSocket for WebCodecs clients. One Annex-B encoder feeds both; the WebSocket path converts framing on the way out. - Input and accessibility each get a dedicated thread. The simulator's Objective-C objects are not Sync and their calls block, and an accessibility tree fetch can take hundreds of milliseconds, which pointer events must not queue behind. - Element picking uses the cached tree for hover feedback and confirms with a real objectAtPoint: hit test once the pointer settles. The cached tree mis-picks overlapping views; the simulator does not. - Accessibility frames arrive in macOS screen points wherever the Simulator window happens to sit, so rects are normalized against the app's own bounds before leaving the server. That also makes them scale-independent. The framebuffer wiring state moved into CaptureState so the idle timer can actually rebuild the pipeline; previously it only counted attempts. This also let the unsafe Send/Sync impls on SimFramebuffer go away, leaving the unsafety localized to the raw pointer wrapper. Verified against a booted iPhone 17 over both transports: video renders, taps and the home button drive the device, and the inspector highlights elements and reports a copyable CLI selector. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Records the things that cost real debugging time: CoreSimulator's proxied objects breaking msg_send!, blocks needing BLOCK_HAS_SIGNATURE, the load- bearing screen-callback registration, and the WebCodecs hardwareAcceleration trap. Also notes that the cli_macos tests need a TCC-permitted GUI session, so their failures are environmental rather than regressions. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Adds SimulatorHID::set_orientation. Orientation does not travel over Indigo
like touches do: it is a GSEvent delivered by mach message to the simulator's
PurpleWorkspacePort, the same path Simulator.app uses for Device > Rotate.
Requires Simulator.app to be running, since it publishes the port.
Also investigated replacing the per-frame framebuffer memcpy with a Metal
blit, since both surfaces are already IOSurface-backed. Benchmarked against a
real 1206x2622 surface with cache-cold sources:
CPU memcpy 0.377 ms/copy 33.5 GB/s
Metal blit 0.517 ms/copy 24.5 GB/s
The GPU path is 37% slower. Command buffer submission and the
waitUntilCompleted round trip cost more than the copy saves, because unified
memory already makes the CPU path fast. At 60fps the memcpy is ~23 ms/s, about
2% of one core, so it was never the bottleneck it looked like. Reverted the
Metal path and recorded the numbers so this is not retried blindly.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Scroll. iOS has no scroll wheel, so wheel deltas are synthesized into a touch drag. The awkward part is that a real finger runs out of screen, so the virtual contact point is lifted and re-planted in the middle whenever it nears an edge, and lifted entirely once the wheel goes quiet. The input worker moved to a channel with a timeout so it can notice that quiet period. Orientation. Rotation goes out as a GSEvent to PurpleWorkspacePort. The framebuffer never rotates — the simulator draws rotated content into a fixed portrait surface — so the browser rotates for display and un-rotates pointer coordinates. Orientation is tracked server-side and seeded at startup from the accessibility bounds aspect, so attaching to an already-rotated device no longer renders sideways. Device settings. Appearance, increase contrast and content size over simctl ui. Those are the only three simctl implements; the rest of Xcode's Devices window needs an in-simulator helper and is deliberately out of scope. System edge gestures. Swipe-up-to-home never worked: the edge argument to IndigoHIDMessageForMouseNSEvent was declared as a bool in the register the ABI uses for the edge, so every gesture was flagged as edgeless. Fixed the signature and plumbed the edge through, rotating it with the device. Fixed an accessibility coordinate bug this surfaced. AX frames are in logical space — iOS reports landscape bounds as 874x402 — so they were being rotated a second time for display and hit tests were being sent in the wrong space. Correct in portrait, wrong everywhere else. The two spaces are now documented where they are used. The UI is now a vertical ribbon beside the device rather than a toolbar underneath, with the inspector and settings as drawers. Also investigated replacing the per-frame memcpy with a Metal blit. It is 37% slower (0.517 ms vs 0.377 ms on a cache-cold 12.1 MB surface) because the command buffer round trip costs more than the copy saves on unified memory. Reverted, with the numbers recorded. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
…solution
The stream was blocky and stuttery even on loopback. Both symptoms had the
same cause.
Bits per pixel was 0.0297 — roughly a quarter of what screen content needs.
The important part is that starving the encoder does not just soften the
picture: with low-latency rate control VideoToolbox drops frames to stay
inside its per-frame budget, which is AverageBitRate / ExpectedFrameRate
regardless of the rate actually being achieved. Measured during scrolling:
6 Mbps 12.2 KB/frame 0.0315 bpp 33.9 fps
24 Mbps 32.2 KB/frame 0.0835 bpp 60.3 fps
So the blockiness and the jank were one problem, not two.
Raising the bitrate is the wrong fix. A phone framebuffer is about fifteen
times the pixels the browser ever displays — 3.16 MP encoded for a 0.21 MP
preview — and every one of those pixels costs bits. The long edge is now
capped at 1280 by default and the bitrate derived from the encode resolution
at ~0.15 bpp instead of being a fixed number, which also makes it correct for
a watch or an iPad rather than tuned for one phone. Same stimulus, same
bandwidth:
native 3.16 MP @ 6 Mbps 4.99 Mbps 0.0297 bpp 53.2 fps
588x1280 @ derived 4.95 Mbps 0.1338 bpp 49.2 fps
Also fixes the keyframe interval, which counted frames only and so stretched a
"2 second" interval to twenty whenever the device idled at 5fps, and stops the
WebCodecs client painting a downscaled frame onto a full-resolution canvas.
Adds GET /api/stats, since none of the above was diagnosable without it, and
ruled out two suspects along the way: no keyframe storms and no subscribers
falling behind.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Most of the screen was unselectable. Three separate causes. objectAtPoint: returns a platform element carrying its own translation, which is not necessarily the one tokenized on the way in. Without the bridge delegate token on that object, attribute reads do not fail — they quietly return an empty label and a zero-size frame. The picker then drew a zero-size highlight, which reads as "this element cannot be selected". get_tree already did this and had a comment marking it IMPORTANT; the hit-test path did not. Sampling 240 points across Settings: 62 returned degenerate elements before, 0 after, and status bar items became selectable for the first time. Hit testing empty space resolves to a full-screen backdrop — every app has an Application node and usually a full-bleed container group — and highlighting one painted a box over the entire device. Backdrops are now filtered from both the tree and the hit test, and empty space reports nothing rather than everything. The client's containment search also broke ties towards the outermost element, so for the very common Button-wrapping-Label-wrapping-text case it selected the wrapper instead of the control. Not fixed, because it is not fixable here: Safari web content lives in a separate WebContent process and is genuinely absent from the tree, which is what made the original Safari test page look so broken. Noted in the backlog. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Measured rather than assumed, and the earlier note in the backlog was wrong. Hit testing does reach web content: objectAtPoint: resolves links, buttons and text inside a WKWebView. It is get_tree that is blind, returning only the host app's own chrome. There is also no hierarchy to traverse — a hit-test result has no children at any depth, and its parent is a web-area group that reports zero children — so enumeration has to be sampled or come from elsewhere. Hit tests cost 3.2 ms median, which is what makes a bounds-guided sweep a realistic option at 0.2-0.5 s per page. The same applies to any embedded web view, since Safari's content area is a WKWebView like any other, so hybrid apps are drivable by coordinate and invisible to every tree-based tool. No implementation yet; the plan deliberately starts with a 30-minute experiment to find out whether SimulatorBridge sees web content at all, since that decides between two very differently sized pieces of work. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
…SimulatorBridge Read idb rather than benchmarking it, which turned out to matter. SimulatorBridge is dead. accessibilityElementsWithDisplayId: survives only in idb's private headers and is called from nowhere; git log -S shows it added and then removed by "Delete FBSimulatorBridge". All that remains is the output format, kept for downstream compatibility. Building it would have reimplemented something its own authors deleted. idb's current path is AXPTranslator over sendAccessibilityRequestAsync:completionQueue:completionHandler:, with exactly the two request kinds we already use, so it would inherit the same blindness. They solve web content by grid hit testing, and say so outright: "Remote elements are in separate processes and require grid-based hit-testing." That is first-party confirmation that no better API exists. Their implementation is more careful than a naive sweep in ways worth copying: a coverage grid filled during the ordinary traversal so probes skip explained regions, pid-based identification of remote content, frame dedup, and provenance recorded per element as recursive vs point_grid. Worth noting that the experiment I proposed last time would have produced a false negative: remoteContentOptions defaults to nil and is not plumbed through the proto or the Python CLI, so idb ui describe-all shows no web content even though idb can fetch it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Reading idb turned up a bug in our own code. IndigoHIDMessageForKeyboardArbitrary takes USB HID usage codes, not HIToolbox virtual keycodes. Sending 0, 11, 8 — our values for abc — into Safari's address bar types "he", which is exactly what those mean as USB HID usages. Keyboard input has never worked; it has been silently typing different letters. idb's own comment in FBSimulatorIndigoHID.swift claims HIToolbox and is wrong, while their Python layer uses USB HID correctly. Also worth having: modifiers are held key events rather than a flag, which is the fix for capitals and shifted symbols; Xcode 27 silently disables legacy Indigo keyboard entirely when dtuhidd is active, and idb has built a whole XPC transport in response; the main display is identified by displayClass 0 rather than by picking the largest surface; VTPixelTransferSession can replace our memcpy, the BGRA to NV12 conversion and the downscale in a single GPU pass; and rate control can target quality instead of a bitrate, which avoids the per-frame budget that was making the encoder drop frames. One place we are ahead: idb has no screen-edge parameter, so it cannot flag a touch as a system edge gesture. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Keyboard input had never worked. IndigoHIDMessageForKeyboardArbitrary takes USB HID usage codes, but the client sent HIToolbox virtual keycodes, so every keystroke produced a different letter — sending our codes for "abc" typed "he". There was also no modifier support, so capitals and shifted symbols were impossible and were dropped or silently lowercased. Text now goes to the server as text and is expanded there, so the US-ASCII table lives in one tested place instead of being duplicated in JavaScript. Shift is a held key event around the base usage, because the Indigo message has no shift flag. Unmappable characters fail the whole string rather than typing a subtly wrong one, and paste is wired up as the practical way to enter anything long. Verified on device: "Test@Example.com?" now types verbatim. It previously became "testexample.com". Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
The keymap in accessibility-serve owns the character-to-key policy and its own usage constants; a second unused copy in the sys crate was just another thing to keep in sync. The important fact — that these are USB HID usages and not HIToolbox keycodes — now lives on send_key_with_modifiers where it is needed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Web content lives in a separate process and has no traversable hierarchy, so a page shows up as nothing but Safari's own chrome. It is reachable by point though, which is the opening: mark every element the tree walk found on a coverage grid, then hit test the cells left over. Following idb's design, which solves the same problem the same way: skip cells already explained, dedup by frame because a coarse grid lands on a large element repeatedly, cap the probe count, and record how each element was found so a point sample is never mistaken for a node with real parentage and order. Coverage is reported whether or not a scan runs, and is a useful signal on its own — a full web page reporting 4% is a much better description of the problem than an empty element list. On a Safari page: 5 elements at 4.1% coverage becomes 15 at 49.5%, using 114 probes in 0.47s, and every element of the page is found. Opt-in via ?scan=true, and used by the inspector so hover works over web content. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
…by class Two changes to the capture path, plus one thing tried and rejected. VTPixelTransferSession now does the copy off the recycled live surface, the BGRA to NV12 conversion, and the downscale in a single hardware pass into a pooled buffer. That removes the row-wise memcpy and also removes the implicit conversion VTCompressionSession was doing internally. Measured on the same stimulus: 47-55 fps against 49 before, at an unchanged 0.13 bits per pixel and about 6% of a core. This does not contradict the earlier result that a Metal blit was slower than memcpy — that was a bare copy doing one job, against a transfer doing three. Main display selection now reads displayClass from the descriptor's state and takes class 0, falling back to largest live area. A booted iPhone exposes two framebuffer descriptors, classes 0 and 1, so the old heuristic was relying on the main display also happening to be the biggest. Constant-quality rate control was implemented and then removed: with EnableLowLatencyRateControl the Quality property is ignored outright, measured at 0.0221 bpp for quality 1.0 versus 0.0223 for quality 0.4. Shipping a knob that silently does nothing is the same mistake as --fps. Low latency matters more here than constant quality, so bitrate stays. Also fixes a self-deadlock the stats change introduced: two lock() calls in one struct expression, where the first guard outlives the second acquisition. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Constant quality was removed last commit because the knob did nothing. It did
nothing because EnableLowLatencyRateControl makes VideoToolbox ignore the
Quality property — not because quality control is unavailable. Turning the
low-latency constraint off makes it work:
low-latency on quality 0.4 -> 0.0223 bpp, quality 1.0 -> 0.0221 bpp
low-latency off quality 0.3 -> 0.70 Mbps, quality 0.9 -> 11.82 Mbps
So the two settings are one decision, and Tuning models it as one:
Interactive { bitrate } — low-latency rate control, no frame delay, a
bitrate derived from the encode resolution. Anything with a live viewer.
Recording { quality } — no low-latency constraint, so the quality target is
honoured and bits go where the picture needs them.
Pairing them this way makes the combination that silently does nothing
impossible to ask for, which was the actual hazard. Frames stay in decode
order in both tunings: B-frames would help recording quality, but WebRTC's
payloader and the raw stream framing both assume output order matches input
order, so that is left for when there is a real recording path to feed.
Interactive remains the default and is unchanged at 0.136 bpp and ~56fps.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
Recording runs a second, independent encode of the same frames rather than retuning the streaming encoder. That separation is what makes B-frames usable: the live path cannot reorder frames because WebRTC's payloader and the raw stream framing both assume the encoder emits them in submission order, but a file has no such constraint and reordering is where most of the quality per bit comes from. It also lets a recording keep its own resolution and quality regardless of what the viewer happens to be watching. AVAssetWriter does the encoding and the muxing, fed pixel buffers through an adaptor rather than sample buffers we encoded ourselves. That avoids ordering decode and presentation timestamps by hand, which is exactly the part B-frames complicate. Timestamps are wall-clock elapsed, so the simulator's very variable paint rate plays back at the speed it actually happened. Verified: 521 frames at 882x1920, High profile, with a genuine mix of frame types — IBBBPBBBPBBBPBBBPBBBPBPBPPPP... All I and P would have meant frame reordering silently failed to take. The live stream is unaffected while recording, 54.3fps against 51.7 idle, for about 10.5% of a core in total. Stopping twice is a clean 409 rather than a panic. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
AGENTS.md had grown to 299 lines, most of it private-framework detail that
only matters when working on one particular area. It was read in full for
every task regardless, and the parts that mattered were buried among the parts
that did not.
The specifics now live in .devin/skills, loaded when relevant:
ios-simulator-internals proxies, block signatures, framebuffer port
discovery, IOSurface lifetime
ios-simulator-input coordinate spaces, USB HID keycodes, edge
gestures, orientation, device settings
ios-simulator-video bits per pixel, rate control, pixel transfer,
recording, browser decode
ios-simulator-accessibility bridge tokens, backdrops, scope differences,
reaching web content
Each groups notes that share a failure mode, which is the useful axis here:
the internals ones fail silently at the proxy boundary, the input ones deliver
an event that does the wrong thing, the accessibility ones return emptiness
that reads as absence. Stating that theme up front is most of the value.
AGENTS.md keeps what every task needs — how to build, test and run — plus a
table pointing at the skills. Verified no content was lost by diffing the
distinctive tokens of the old file against the new set.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Devin Review