Skip to content

fix: render value= in the iOS outline for TextArea, Slider, and Button - #87

Merged
onevcat merged 2 commits into
mainfrom
fix/textarea-outline-value
Aug 3, 2026
Merged

fix: render value= in the iOS outline for TextArea, Slider, and Button#87
onevcat merged 2 commits into
mainfrom
fix/textarea-outline-value

Conversation

@onevcat

@onevcat onevcat commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Problem

The describe-ui text outline whitelists which roles get a value="…" tag (OutlineFormatter.valueBearingTypes), and the list stopped at TextField / SecureTextField / Switch. A multi-line editor surfaces as role TextArea (UITextView / SwiftUI TextEditor — the shape of real chat message inputs), so text typed into one was invisible in the outline whenever the element carries an accessibility label:

@10  TextArea  "Message editor"  #text-area-field  (16,236 370x160)        ← typed text nowhere

while --json had it all along (entries[].value: "hello textarea"). Agents driving an app via the outline (as the bundled skill recommends) could not confirm a type, paste, or clear landed without hand-rolling JSON spelunking.

Role-vocabulary research

The whitelist additions are grounded in a live probe of the AX bridge rather than the XCUIElementType names — the bridge speaks macOS-style roles, and several familiar names simply never appear. New playground screen Value Roles (SwiftUI + UIKit sections) is the reusable probe; measured on the iOS 26.5 bridge:

Control Actual role AXValue
UITextView / TextEditor TextArea typed text
UISearchBar TextField (no SearchField role) typed text
UISlider Slider thumb position
UIPageControl Slider (no PageIndicator) page 2 of 3 == label
UIDatePicker (wheels) Slider ×N == label
UIProgressView GenericElement (no ProgressIndicator) 70%
UISwitch CheckBox (not Switch) 0 / 1
DatePicker (compact) Button Sep 9, 2001
UIStepper / SwiftUI Stepper 2 × Button none / current value

Fix

  • valueBearingTypes += TextArea (the bug), Slider (thumb position), Button (compact date pickers / stepper halves whose AXValue is the only readable state).
  • New guard: a value that merely echoes the label slot is suppressed — a label-less element already surfaces its value through the rawLabel fallback, so the tag would print the same string twice. Mirrors Android's AndroidClassifier.effectiveValue (text == label → nil). This also fixes the pre-existing TextField "query" value="query" double print, and keeps page controls / picker wheels (Slider with value == label) quiet.
  • Not added: GenericElement (would drag every status-bar item's value into the outline) and CheckBox (its 0/1 needs an unchecked-style tag design, not a cryptic value="0" — follow-up material).

Android needs no code change: multi-line EditText keeps className android.widget.EditText → classifies as TextField → already value-bearing.

After (same screen):

@10  TextArea  "Message editor"  #text-area-field  (16,236 370x160)  value="hello textarea"

Fixtures & tests

  • Playground iOS: new text-area screen (labeled TextEditor, the regression shape) and value-roles probe screen; Playground Android: labeled multi-line EditText on the text screen.
  • Unit: 5 new OutlineFormatterTests pinning labeled/unlabeled TextArea, Slider echo suppression, Button value, and the TextField double-print fix.
  • E2E: TypeTests "Typed text surfaces in the text outline for a labeled TextArea" (iOS) and AndroidTypeTests "typed text surfaces as value= on a labeled multiline field" (parity).

Verification

  • make build + make test: 1259 passed, 0 failed.
  • New iOS E2E test green against a booted iPhone 17 Pro (iOS 26.5); full TypeTests suite green (15 tests).
  • New Android E2E test green against emulator (API 36); AndroidTypeTests + AndroidKeyboardStateTests green.
  • Live spot-checks of the text-area and value-roles screens: TextArea/Slider/Button values render, page-control and date-wheel Sliders stay quiet, search-bar double print gone, status bar unchanged.

🤖 Generated with Claude Code

onevcat added 2 commits July 30, 2026 17:20
Text typed into a labeled multi-line editor (role TextArea — the shape
of real chat inputs) was invisible in the describe-ui text outline:
the label owned the label slot and the valueBearingTypes whitelist
skipped the value= tag, so the content existed only in --json.

The whitelist now includes TextArea, Slider, and Button, grounded in a
live probe of the AX bridge's role vocabulary (see the playground's new
Value Roles screen): UITextView -> TextArea, UISearchBar -> TextField,
UIPageControl -> Slider, UIProgressView -> GenericElement, UISwitch ->
CheckBox. GenericElement stays out so status-bar values don't flood the
outline.

A value that merely echoes the label slot is suppressed (same guard as
Android's AndroidClassifier.effectiveValue): a label-less element
already surfaces its value via the rawLabel fallback, so the tag would
double-print — this also fixes the pre-existing
`TextField "query" value="query"` duplication and keeps page controls /
picker wheels (Slider with value == label) quiet.

Android already classified multi-line EditText as TextField and
rendered the value; it gains a labeled multi-line playground fixture
and a parity E2E test. New iOS playground screens: text-area
(regression fixture) and value-roles (role-vocabulary probe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Signed-off-by: onevcat <onevcat@gmail.com>
@onevcat
onevcat merged commit 16e2cdf into main Aug 3, 2026
4 checks passed
@onevcat
onevcat deleted the fix/textarea-outline-value branch August 3, 2026 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant