Skip to content

feat: drive physical iOS devices over the accessibility audit daemon - #93

Open
subdiox wants to merge 1 commit into
lycorp-jp:mainfrom
subdiox:feat/physical-ios-device-backend
Open

feat: drive physical iOS devices over the accessibility audit daemon#93
subdiox wants to merge 1 commit into
lycorp-jp:mainfrom
subdiox:feat/physical-ios-device-backend

Conversation

@subdiox

@subdiox subdiox commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Motivation

sim-use drives simulators and Android devices, but not a physical iPhone or iPad. Every existing tool for real iOS devices — Appium, Maestro, agent-device, devicekit-ios — goes through an XCUITest runner, which means building a runner app, code signing it with the user's team and mounting a Developer Disk Image. Adopting that would have cost sim-use its "install it and go" property on exactly the platform where setup friction hurts most.

Change

Adds sim-use ios-device (devices / ui / tap) as a third backend. A connected device is reached through com.apple.accessibility.axAuditDaemon.remoteserver over usbmux lockdown: nothing is installed on the device, nothing is code signed, and no Developer Disk Image is mounted. The device only has to be unlocked.

$ sim-use ios-device ui
@4   Button  "Chats Button, Selected"
@5   Button  "Friends"
...
117 elements (316 nodes) in 6647 ms

$ sim-use ios-device tap --text "Friends"
✓ Activated Friends Button

Layering mirrors the existing backends: Transport/ (DTX framing and the lockdown service connection), AXAudit/ (selector layer — FB*-free and unit-testable behind a DTXInvoking protocol), Domain/ (tree walk, outline), Verbs/.

What this surface deliberately lacks

iOS serves no element geometry on this channel; Xcode's own Accessibility Inspector shows no Frame row for a physical device either. So there is no coordinate tap, swipe, gesture or multi-touch — interaction goes through accessibility actions, and the outline follows the accessibility reading order rather than screen position. Element handles encode a live pointer and die with the connection, so tap resolves by --text within one session instead of reusing an @N alias from a previous ui.

Notes for review

Three behaviours that are invisible in the diff and cost the most to find, all measured against iOS 27:

  • Plaintext DTX. FBAMDServiceConnection's send/receive push bytes through the connection's SSL context, but the daemon wants the stream in plaintext once lockdown has started the service — encrypted frames make the device hang up on the first message. The transport therefore uses the raw socket. There is no .DVTSecureSocketProxy variant of this service.
  • Main run loop. FBDeviceSet receives AMDevice attachment through CFRunLoop sources on the main queue, so discovery pumps that loop. Without it a CLI only ever sees the restorable half of the device and startService: fails with "not AMDevice backed".
  • Session teardown. A session has to release the daemon's inspector state on the way out. Skipping it leaves the device answering every element query with nothing, which reads exactly like a locked screen.

Requests are pipelined over one connection (~1.4x). Past that the daemon serialises — raising the in-flight count or spreading load over several connections buys nothing — so --fast is the remaining lever, trading about a quarter of the elements for 40% off the wall clock.

Verification

  • make test — 993 tests pass, including 5 new ones pinning the wire envelope shape. That shape is the whole reason AXAuditValue is a type rather than hand-built dictionaries: the reference Python client mis-nests the element token, and the daemon answers a malformed element by doing nothing and reporting success.
  • Verified on a physical iPhone (iOS 27): ui renders the foreground app, tap --text flips a segmented control, and the follow-up ui shows the new selection.
  • make e2e-ios — 22 of 23 suites pass. Six suites failed on the first run and passed on a re-run after clearing a wedged daemon. RemoteContentRecoveryTests fails deterministically for a reason unrelated to this change: it matches the document picker with --label-regex '选取文件|Choose File', and on a Japanese-locale simulator the picker renders as ファイルを選択. Fixed separately.

Adds `sim-use ios-device` (devices / ui / tap), a third backend alongside
the simulator and Android ones. A connected iPhone or iPad is reached
through `com.apple.accessibility.axAuditDaemon.remoteserver` over usbmux
lockdown, so no XCUITest runner is installed, nothing is code signed and
no Developer Disk Image is mounted — the device only has to be unlocked.

The channel exposes no element geometry, so this surface deliberately
omits coordinate taps, swipes and gestures and interacts through
accessibility actions instead; the outline is ordered by the
accessibility reading order rather than by screen position.

Notable details, all measured against iOS 27:

- The daemon wants the DTX stream in plaintext once lockdown has started
  the service. `FBAMDServiceConnection` send/receive push bytes through
  the connection's SSL context, so the transport talks to the raw socket
  instead; encrypted frames make the device hang up on the first message.
- `FBDeviceSet` receives AMDevice attachment through CFRunLoop sources on
  the main queue, so discovery pumps the main run loop. Without that a
  CLI only ever sees the restorable half of the device.
- AMDevice does not publish the lockdown UDID before a session, so a
  device is addressable by ECID as well.
- Sessions must release the daemon's inspector state on the way out.
  Skipping it leaves the device answering every element query with
  nothing, which is easy to misread as a locked screen.

Requests are pipelined over one connection, which is worth about 1.4x;
past that the daemon serialises, so `--fast` trades roughly a quarter of
the elements for 40% off the wall clock.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
@onevcat

onevcat commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Wow, this is quite extensive and amazing! Thank you so much for putting this together. I’ll take some time to go through it in detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants