feat: drive physical iOS devices over the accessibility audit daemon - #93
Open
subdiox wants to merge 1 commit into
Open
feat: drive physical iOS devices over the accessibility audit daemon#93subdiox wants to merge 1 commit into
subdiox wants to merge 1 commit into
Conversation
Adds `sim-use ios-device` (devices / ui / tap), a third backend alongside the simulator and Android ones. A connected iPhone or iPad is reached through `com.apple.accessibility.axAuditDaemon.remoteserver` over usbmux lockdown, so no XCUITest runner is installed, nothing is code signed and no Developer Disk Image is mounted — the device only has to be unlocked. The channel exposes no element geometry, so this surface deliberately omits coordinate taps, swipes and gestures and interacts through accessibility actions instead; the outline is ordered by the accessibility reading order rather than by screen position. Notable details, all measured against iOS 27: - The daemon wants the DTX stream in plaintext once lockdown has started the service. `FBAMDServiceConnection` send/receive push bytes through the connection's SSL context, so the transport talks to the raw socket instead; encrypted frames make the device hang up on the first message. - `FBDeviceSet` receives AMDevice attachment through CFRunLoop sources on the main queue, so discovery pumps the main run loop. Without that a CLI only ever sees the restorable half of the device. - AMDevice does not publish the lockdown UDID before a session, so a device is addressable by ECID as well. - Sessions must release the daemon's inspector state on the way out. Skipping it leaves the device answering every element query with nothing, which is easy to misread as a locked screen. Requests are pipelined over one connection, which is worth about 1.4x; past that the daemon serialises, so `--fast` trades roughly a quarter of the elements for 40% off the wall clock. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
Contributor
|
Wow, this is quite extensive and amazing! Thank you so much for putting this together. I’ll take some time to go through it in detail. |
subdiox
marked this pull request as ready for review
August 12, 2026 02:58
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
sim-use drives simulators and Android devices, but not a physical iPhone or iPad. Every existing tool for real iOS devices — Appium, Maestro, agent-device, devicekit-ios — goes through an XCUITest runner, which means building a runner app, code signing it with the user's team and mounting a Developer Disk Image. Adopting that would have cost sim-use its "install it and go" property on exactly the platform where setup friction hurts most.
Change
Adds
sim-use ios-device(devices/ui/tap) as a third backend. A connected device is reached throughcom.apple.accessibility.axAuditDaemon.remoteserverover usbmux lockdown: nothing is installed on the device, nothing is code signed, and no Developer Disk Image is mounted. The device only has to be unlocked.Layering mirrors the existing backends:
Transport/(DTX framing and the lockdown service connection),AXAudit/(selector layer — FB*-free and unit-testable behind aDTXInvokingprotocol),Domain/(tree walk, outline),Verbs/.What this surface deliberately lacks
iOS serves no element geometry on this channel; Xcode's own Accessibility Inspector shows no Frame row for a physical device either. So there is no coordinate tap,
swipe,gestureormulti-touch— interaction goes through accessibility actions, and the outline follows the accessibility reading order rather than screen position. Element handles encode a live pointer and die with the connection, sotapresolves by--textwithin one session instead of reusing an@Nalias from a previousui.Notes for review
Three behaviours that are invisible in the diff and cost the most to find, all measured against iOS 27:
FBAMDServiceConnection's send/receive push bytes through the connection's SSL context, but the daemon wants the stream in plaintext once lockdown has started the service — encrypted frames make the device hang up on the first message. The transport therefore uses the raw socket. There is no.DVTSecureSocketProxyvariant of this service.FBDeviceSetreceives AMDevice attachment through CFRunLoop sources on the main queue, so discovery pumps that loop. Without it a CLI only ever sees the restorable half of the device andstartService:fails with "not AMDevice backed".Requests are pipelined over one connection (~1.4x). Past that the daemon serialises — raising the in-flight count or spreading load over several connections buys nothing — so
--fastis the remaining lever, trading about a quarter of the elements for 40% off the wall clock.Verification
make test— 993 tests pass, including 5 new ones pinning the wire envelope shape. That shape is the whole reasonAXAuditValueis a type rather than hand-built dictionaries: the reference Python client mis-nests the element token, and the daemon answers a malformed element by doing nothing and reporting success.uirenders the foreground app,tap --textflips a segmented control, and the follow-upuishows the new selection.make e2e-ios— 22 of 23 suites pass. Six suites failed on the first run and passed on a re-run after clearing a wedged daemon.RemoteContentRecoveryTestsfails deterministically for a reason unrelated to this change: it matches the document picker with--label-regex '选取文件|Choose File', and on a Japanese-locale simulator the picker renders asファイルを選択. Fixed separately.