fix: keep hit-test probes in the UI point metric on display-downscaled devices - #92
Merged
Conversation
iPhone 12/13 mini and the 6/6s/7/8 Plus family lay out a larger point space than they display: the mini renders 375x812 points into a 1125x2436 buffer and downsamples it onto a 1080x2340 panel. The space HID normalizes against is pixels / scale, so NativePortraitSize reads 360x780 there while AX frames stay in 375x812 — and because the portrait mapping was the identity, every AX-derived coordinate reached HID 4.1% away from the origin. Measured on iPhone 12 mini / iOS 26.3.1: tapping a Settings row spanning y 648...700 opened the row below it, and a 48pt-tall button could only be hit by dividing its center by 1.041 first. Calibration now carries a UIPointScale — the metric ratio recovered from the AX-reported screen size, which rotation swaps but never changes — and uiToFramebuffer / framebufferToUI apply it in both directions. The gesture canvas, the stale-snapshot advisory and the quadtree's UI-space probes read the UI size through the same scale, so they agree with HID again. Devices that render at their panel resolution keep the identity transform, and an axis-skew guard rejects a UI size that is a window rather than the display (a resized iPad scene) instead of turning it into a bogus scale. Signed-off-by: yuta.ooka <yuta.ooka@lycorp.co.jp>
The AX point hit-test shares HID's native-portrait axes but consumes points in the UI point metric, not the pixels/scale metric HID normalizes against. Verified live on iPhone 12 mini / iOS 26.4 (Settings.app): querying the un-scaled UI point returns the element at that point, while the pixels/scale-shrunk point lands ~4% off target. The two metrics coincide on devices that render 1:1, which is why the distinction was invisible until the UIPointScale work. Applying the HID scale to probe traffic shifted the calibrator's discriminator probes, the quadtree recovery probes, and the --point re-issued query ~4% off on downscaled panels: small discriminators (a 17x22 toolbar button) missed outright, wasting the probe budget and degrading collapsed-children recovery. Probe transforms now rotate on the UI-sized canvas via the new OrientationCalibration.probeCGPoint; only HID dispatch carries the metric scale. Portrait probes are the identity everywhere, keeping the single-probe --point fast path valid on downscaled panels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
type "" documents itself as a strict no-op, but execute() ran setup() (xcode-select check + private-framework load) before the empty-input guard, so the no-op still depended on a usable Xcode and paid a subprocess spawn. On loaded CI runners that subprocess intermittently failed, flaking the empty-input unit test with "Xcode is not available". Resolve the input and convert it to HID events first; only reach the preflight once there is something to dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
Review follow-ups on the probe-metric split: - The remote-content retry's sampling region (issue #64) still used the pixels/scale portrait bounds, leaving the right/bottom ~4% of a display-downscaled screen unprobed — exactly where a picker's bottom bar tends to live. The region now uses the hit-test canvas, with the UIPointScale recovered from the shell root's own display frame (bare shells keep the identity fallback). - Documented the deliberate gap in --point's bounds gate: UI-only edge bands skip the identity probe and take the tree-calibration fallback, since no scale is known pre-fetch and soleOrientation clamps to the native canvas anyway. - Removed a stale pre-experiment comment block left in PointQueryOrientationTests that contradicted the suite it introduced, and reworded the remaining framebuffer-metric-era comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
Remaining comments still described the pre-experiment model (hit-test == HID framebuffer space): the OrientationCalibration header routed hit-tests through hidPoint, DisplayOrientation called both consumers the same native framebuffer space, and LegacyAccessibilityBridge asked for "native-portrait bounds" feeding a "framebuffer-space" hit-test. All now state the invariant explicitly — HID and the hit-test share native-portrait AXES, but the hit-test keeps the UI point METRIC — so the scale does not get reintroduced into probe traffic later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
The orientation self-calibration README still stated the #34-era premise (hit-test == HID framebuffer space) in its space table, Decision, Implementation notes, and Open items, and showed the quadtree probe wrapped through hidCGPoint. A prominent update note now records the metric split (PR #90/#92) with the live evidence, the affected statements are corrected in place, and the Open-items caveat that anticipated exactly this divergence is marked resolved. Also the last stale code comments: the OrientationCalibrator enum doc ("shares the HID coordinate space"), the EmptyShellTreeTests coverage- grid note, and the OrientationRecoveryTests fixture wording (1:1 fixture, behaviour unaffected) now use the axes/metric terms. Released CHANGELOG entries are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
The Decision's probe step still said the discriminator center is inverse-transformed "into a framebuffer point", contradicting the axes/metric model the rest of the document now states; it and the measured-transform table intro use portrait-axis / hit-test canvas terms, with a note that the 1:1 measurement device makes the axis table metric-agnostic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #90 — the first commit here is #90. Review the two commits on top; once #90 merges, this PR's diff reduces to just them.
Problem
#90 correctly identified that HID dispatch normalizes against
pixels / scale(360x780 on the iPhone 12/13 mini) while AX frames live in the 375x812 UI point space, and introducedUIPointScaleto carry AX-derived taps across. But it applies that scale to alluiToFramebuffertraffic — including points fed to the AX point hit-test, and the hit-test turns out not to share HID's metric.Verified live on iPhone 12 mini / iOS 26.4 (Settings.app), using the v0.12.0 binary (no scale handling anywhere) as the control:
(187.5, 674.5)674.5 / 0.9606 = 702.3(317.5, 760)(330.7, 791.2)Meanwhile the HID side behaves exactly as #90 found (re-confirmed on 26.4): the un-scaled tap opens the next row, the scaled tap opens the right one.
So the two spaces share axes (native portrait, issue #34) but not metric: the hit-test consumes UI points, HID consumes
pixels / scale. On devices that render 1:1 the metrics coincide, which is why the distinction was invisible until now.Consequences of #90 as-is on downscaled panels (all in plain portrait):
wrappedProbenow wraps portrait too) query ~4% off systematically, degrading recovery on the screens that need it most.describe-ui --point's tree-calibration fallback re-issues the query through the scaled mapping and returns the element ~4% away from the one asked about.Fix
Split the two transforms on
OrientationCalibration:probeCGPoint— hit-test space: rotates on the UI-sized portrait canvas (NativePortraitSize.uiMetric(_:), which documents the finding), never applies the metric scale. Portrait is the identity everywhere — including downscaled panels — which is what keeps--point's single-probe fast path valid there.hidCGPoint— unchanged from fix: scale AX coordinates into HID space on display-downscaled devices #90: axes + metric scale. All of fix: scale AX coordinates into HID space on display-downscaled devices #90's HID-side behavior (tap/swipe dispatch, gesture canvas, stale-snapshot advisory,orderedCandidatespruning, scale recovery) is preserved.wrappedProbe, the calibrator's probe loop (including its discriminator-area gate), and the--pointfallback now ride the probe transform.Second commit —
type ""short-circuits before the framework preflight: the documented no-op previously still ran thexcode-selectcheck + private-framework load, so it depended on the host's Xcode state for no benefit. That subprocess is what intermittently flaked the empty-input unit test on #90's first CI run ("Xcode is not available", 10.5s — whilemainwas 10/10 green and the same suite passes locally; the rerun passed). Input is now resolved and converted first; the preflight only runs when there are events to dispatch.Verification
make test: 1290 green, including two red-first suites written against the live-verified semantics —OrientationCalibrator on display-downscaled devices(mock hit-test consumes UI-metric points; both tests fail on fix: scale AX coordinates into HID space on display-downscaled devices #90 as-is, pass here) andProbe vs HID space on display-downscaled devices(pins the two transforms apart numerically, plus thewrappedProbeportrait pass-through).--pointon the Dictate button and on a row-boundary point both return the element at the queried point;describe-ui --jsonadvisory staysnull(calibration converges without wasting probes);tap '#com.apple.settings.homeScreen'opens the right page (HID side intact);type ""no-ops cleanly. Control runs with v0.12.0 confirm the hit-test metric independently of this branch's code.uiMetricreturnsselfunder an identity scale, so every 1:1 path is byte-identical to fix: scale AX coordinates into HID space on display-downscaled devices #90.