fix: scale AX coordinates into HID space on display-downscaled devices - #90
Merged
onevcat merged 1 commit intoAug 6, 2026
Merged
Conversation
iPhone 12/13 mini and the 6/6s/7/8 Plus family lay out a larger point space than they display: the mini renders 375x812 points into a 1125x2436 buffer and downsamples it onto a 1080x2340 panel. The space HID normalizes against is pixels / scale, so NativePortraitSize reads 360x780 there while AX frames stay in 375x812 — and because the portrait mapping was the identity, every AX-derived coordinate reached HID 4.1% away from the origin. Measured on iPhone 12 mini / iOS 26.3.1: tapping a Settings row spanning y 648...700 opened the row below it, and a 48pt-tall button could only be hit by dividing its center by 1.041 first. Calibration now carries a UIPointScale — the metric ratio recovered from the AX-reported screen size, which rotation swaps but never changes — and uiToFramebuffer / framebufferToUI apply it in both directions. The gesture canvas, the stale-snapshot advisory and the quadtree's UI-space probes read the UI size through the same scale, so they agree with HID again. Devices that render at their panel resolution keep the identity transform, and an axis-skew guard rejects a UI size that is a window rather than the display (a resized iPad scene) instead of turning it into a bogus scale. Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
Contributor
|
Good catch! LGTM. Thank you! |
Contributor
|
I have some minor following fixes in #92, but it does not block this PR. |
onevcat
added a commit
that referenced
this pull request
Aug 6, 2026
The orientation self-calibration README still stated the #34-era premise (hit-test == HID framebuffer space) in its space table, Decision, Implementation notes, and Open items, and showed the quadtree probe wrapped through hidCGPoint. A prominent update note now records the metric split (PR #90/#92) with the live evidence, the affected statements are corrected in place, and the Open-items caveat that anticipated exactly this divergence is marked resolved. Also the last stale code comments: the OrientationCalibrator enum doc ("shares the HID coordinate space"), the EmptyShellTreeTests coverage- grid note, and the OrientationRecoveryTests fixture wording (1:1 fixture, behaviour unaffected) now use the axes/metric terms. Released CHANGELOG entries are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On iPhone 12/13 mini (and the 6/6s/7/8 Plus family) every AX-derived tap, swipe and point hit-test lands ~4% away from its target.
These models lay out a larger point space than they display. The mini renders 375x812 points into a 1125x2436 buffer and downsamples it onto a 1080x2340 panel, so
NativePortraitSize—pixels / scale, the spaceFBSimulatorIndigoHIDnormalizes against — reads 360x780 while AX frames stay in 375x812.DisplayOrientation's portrait mapping is the identity, so the UI point was handed to HID verbatim and arrived multiplied by 812/780 = 1.041.Measured live on iPhone 12 mini / iOS 26.3.1 (Settings.app,
sim-usev0.12.0):674.5 x 1.041 = 702. Same cause in the wild: a 48pt-tall button in a login flow could only be pressed by dividing its center by 1.041 by hand, and a
describe-ui-driven agent has no way to know that.It also degrades calibration itself — the AX screen size matches neither
nativenor its swap, soorderedCandidatescannot prune and the probe budget is spent re-deriving portrait.Fix
Calibration carries a
UIPointScale: the metric ratio between the UI point space and the framebuffer point space, recovered from the AX-reported screen size (rotation swaps the two dimensions but never changes the two numbers, so sorting them yields the portrait-major UI size before the orientation is known).uiToFramebuffer/framebufferToUIapply it in both directions, swapping the factors along with the dimensions in landscape.Everything that reads the UI screen size goes through the same scale, so they agree with HID again: the gesture preset canvas, the stale-snapshot advisory, and the quadtree's UI-space probes (
wrappedProbenow wraps in portrait too, since portrait is no longer necessarily identity).Devices that render at their panel resolution are unaffected — the scale is exactly identity and
hidCGPointshort-circuits as before. A UI size that is a window rather than the display (a resized iPad scene) is rejected by an axis-skew guard instead of becoming a bogus scale.Verification
Live, iPhone 12 mini / iOS 26.3.1 — same command, same screen:
#com.apple.settings.homeScreen(row y 648...700)Regression, iPhone 17 / iOS 26.5 (renders 1:1):
tap '#com.apple.settings.camera'opens カメラ, unchanged.Unit:
make test— 978 tests green, including a newDisplay-downscaled devicessuite (portrait dispatch pinned to the live measurement,uiSizein UI space, round-trip identity in all four orientations) and aUIPointScalesuite (identity on 1:1 panels, degenerate sizes rejected, recovered from either orientation, window-sized UI rejected).