fix: render value= in the iOS outline for TextArea, Slider, and Button - #87
Merged
Conversation
Text typed into a labeled multi-line editor (role TextArea — the shape of real chat inputs) was invisible in the describe-ui text outline: the label owned the label slot and the valueBearingTypes whitelist skipped the value= tag, so the content existed only in --json. The whitelist now includes TextArea, Slider, and Button, grounded in a live probe of the AX bridge's role vocabulary (see the playground's new Value Roles screen): UITextView -> TextArea, UISearchBar -> TextField, UIPageControl -> Slider, UIProgressView -> GenericElement, UISwitch -> CheckBox. GenericElement stays out so status-bar values don't flood the outline. A value that merely echoes the label slot is suppressed (same guard as Android's AndroidClassifier.effectiveValue): a label-less element already surfaces its value via the rawLabel fallback, so the tag would double-print — this also fixes the pre-existing `TextField "query" value="query"` duplication and keeps page controls / picker wheels (Slider with value == label) quiet. Android already classified multi-line EditText as TextField and rendered the value; it gains a labeled multi-line playground fixture and a parity E2E test. New iOS playground screens: text-area (regression fixture) and value-roles (role-vocabulary probe). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
…value # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
describe-uitext outline whitelists which roles get avalue="…"tag (OutlineFormatter.valueBearingTypes), and the list stopped atTextField/SecureTextField/Switch. A multi-line editor surfaces as roleTextArea(UITextView / SwiftUITextEditor— the shape of real chat message inputs), so text typed into one was invisible in the outline whenever the element carries an accessibility label:while
--jsonhad it all along (entries[].value: "hello textarea"). Agents driving an app via the outline (as the bundled skill recommends) could not confirm atype,paste, or clear landed without hand-rolling JSON spelunking.Role-vocabulary research
The whitelist additions are grounded in a live probe of the AX bridge rather than the
XCUIElementTypenames — the bridge speaks macOS-style roles, and several familiar names simply never appear. New playground screen Value Roles (SwiftUI + UIKit sections) is the reusable probe; measured on the iOS 26.5 bridge:SearchFieldrole)PageIndicator)page 2 of 3== labelProgressIndicator)70%Switch)0/1Sep 9, 2001Fix
valueBearingTypes+=TextArea(the bug),Slider(thumb position),Button(compact date pickers / stepper halves whose AXValue is the only readable state).rawLabelfallback, so the tag would print the same string twice. Mirrors Android'sAndroidClassifier.effectiveValue(text == label → nil). This also fixes the pre-existingTextField "query" value="query"double print, and keeps page controls / picker wheels (Slider with value == label) quiet.GenericElement(would drag every status-bar item's value into the outline) andCheckBox(its0/1needs anunchecked-style tag design, not a crypticvalue="0"— follow-up material).Android needs no code change: multi-line
EditTextkeeps classNameandroid.widget.EditText→ classifies asTextField→ already value-bearing.After (same screen):
Fixtures & tests
TextEditor, the regression shape) and value-roles probe screen; Playground Android: labeled multi-lineEditTexton the text screen.OutlineFormatterTestspinning labeled/unlabeled TextArea, Slider echo suppression, Button value, and the TextField double-print fix.TypeTests"Typed text surfaces in the text outline for a labeled TextArea" (iOS) andAndroidTypeTests"typed text surfaces as value= on a labeled multiline field" (parity).Verification
make build+make test: 1259 passed, 0 failed.TypeTestssuite green (15 tests).AndroidTypeTests+AndroidKeyboardStateTestsgreen.🤖 Generated with Claude Code