Add MyAccessibilityService to fill in some notification parsing gaps during startup/launch#1
Open
paulpv wants to merge 12 commits into
Open
Add MyAccessibilityService to fill in some notification parsing gaps during startup/launch#1paulpv wants to merge 12 commits into
paulpv wants to merge 12 commits into
Conversation
…sibility requirement Introduce MyAccessibilityService (with XML manifest entry and service config) to inspect the SystemUI notification shade and extract structured notification rows. Wire the new ACCESSIBILITY_SERVICE into the startup/permissions flow: extend Requirement to include ACCESSIBILITY_SERVICE with a missing(context) helper, show an Accessibility requirement card in the permissions UI, and make the notification listener idle/unbind until all requirements (including accessibility) are met. Also add a small string resource and .gitignore entry. The service is scoped to com.android.systemui events for privacy and includes debug/heuristic approaches for finding notification containers/rows.
…y notifications and log telemetry Detect notifications that NLS cannot parse (empty GROUP_SUMMARY), schedule a short pending lookup window, and if no sibling content arrives trigger MyAccessibilityService to open the shade, expand/scan rows, and read matching app rows. Introduce a PendingRowSearch state machine in MyAccessibilityService to expand collapsed rows, scroll for off-screen rows, and return matched ShadeRow data to MyNotificationListenerService for TTS. Add ObscuredNotification telemetry model and ObscuredNotificationLogger (pluggable sink) to record outcomes (FOUND / NOT_FOUND / ACCESSIBILITY_UNAVAILABLE). Wire ACCESSIBILITY_SERVICE into startup requirements, add explanatory strings and docs (parsers/README, Handling Obscured Notifications), and minor logging tweaks. This enables catch-up readback for notifications whose content is only present in the system shade (e.g. Google Chat) while preserving normal NLS behavior for live deliveries.
Move and split the large AccessibilityService scanning logic into focused classes/files: - Introduce NotificationShadeSnapshot (ShadeRow data model + snapshot utilities) - Add ShadeRowSearchQueue to serialize/find/expand/scroll searches - Add DebugShadeScan for a full top-to-bottom diagnostic scan - Add ShadeDelays for configurable timing (fast/slow debug modes) - Replace the previous monolithic MyAccessibilityService implementation with a smaller service wiring these components - Adjust MyNotificationListenerService to use the relocated ShadeRow type These changes modularize the shade traversal/row-extraction logic. This prepares the code for clearer responsibilities and easier testing/maintenance.
There was a problem hiding this comment.
Pull request overview
This PR introduces an AccessibilityService-based fallback path to read “obscured” notifications (e.g., empty GROUP_SUMMARY payloads) from the rendered notification shade during app startup/catch-up, and wires it into the app’s startup gating + UI so the service can be enabled when needed.
Changes:
- Add
MyAccessibilityService+ shade snapshot/search utilities to open the notification shade, expand rows, and extract structured notification text via the accessibility tree. - Add NLS→Accessibility bridging in
MyNotificationListenerServiceby scheduling a delayed lookup forParsedIgnorednotifications and logging outcomes viaObscuredNotificationLogger. - Update startup requirement evaluation + permission gate UI/resources/docs to include and explain the Accessibility permission/service.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| smartfoo/src/main/java/com/smartfoo/android/core/notification/FooNotification.kt | Silences debug logging in notification-listener enablement checks. |
| mobile/src/main/res/xml/my_accessibility_service.xml | Declares the AccessibilityService capabilities/events/packages. |
| mobile/src/main/res/values/strings.xml | Adds the user-facing Accessibility service description string. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/startup/StartupCoordinator.kt | Adds ACCESSIBILITY_SERVICE to hard requirements and centralizes missing-requirements evaluation. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/ShadeRowSearchQueue.kt | Implements a serialized queue/state machine for shade row search/expand/scroll. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/ShadeDelays.kt | Introduces configurable timing constants for the shade scanning state machine. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/NotificationShadeSnapshot.kt | Implements notification shade snapshotting + row extraction utilities. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/notification/parsers/README.md | Documents the two-path (NLS + Accessibility) notification parsing approach. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/notification/ObscuredNotification.kt | Adds the telemetry data model and resolution outcomes for obscured notifications. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/notification/ObscuredNotificationLogger.kt | Adds a swappable sink for logging obscured-notification telemetry (default: logcat). |
| mobile/src/main/java/llc/lookatwhataicando/notifai/MyNotificationListenerService.kt | Schedules/cancels delayed accessibility lookups for ParsedIgnored notifications; speaks shade-derived content. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/MyAccessibilityService.kt | Adds the AccessibilityService implementation that snapshots and searches shade rows. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/MainActivity.kt | Adds an Accessibility requirement card to the permissions gate UI. |
| mobile/src/main/java/llc/lookatwhataicando/notifai/DebugShadeScan.kt | Adds an optional debug-only full-shade scan mode. |
| mobile/src/main/AndroidManifest.xml | Registers MyAccessibilityService. |
| mobile/README.md | Updates architecture documentation to describe the dual delivery paths and the accessibility requirement scope. |
| docs/Handling Obscured Notifications.md | Adds a design/plan doc for obscured notification handling and telemetry. |
| .gitignore | Ignores a local scratch/ directory. |
Comments suppressed due to low confidence (1)
mobile/src/main/java/llc/lookatwhataicando/notifai/notification/parsers/README.md:102
- The constants table is now out of sync with the implementation: there is no
SHADE_OPEN_SETTLE_DELAY_MS/MAX_SCROLL_ATTEMPTSinMyAccessibilityServiceanymore (these are effectively inShadeDelaysandShadeRowSearchQueue). Updating the table to reflect the current sources of truth will prevent future debugging confusion.
## Constants
| Constant | Location | Value | Purpose |
|---|---|---|---|
| `PENDING_LOOKUP_DELAY_MS` | `MyNotificationListenerService` | 300 ms | Window for content-bearing child to cancel the accessibility lookup |
| `SHADE_OPEN_SETTLE_DELAY_MS` | `MyAccessibilityService` | 600 ms | Time for shade animation + first accessibility events before scanning |
| `MAX_SCROLL_ATTEMPTS` | `MyAccessibilityService` | 10 | Max `ACTION_SCROLL_FORWARD` calls before giving up on off-screen rows |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+52
to
+61
| private data class PendingRowSearch( | ||
| val appLabel: String, | ||
| val callback: (List<ShadeRow>?) -> Unit, | ||
| var shadeOpenedByUs: Boolean, | ||
| var readyToScan: Boolean, | ||
| val rowsToExpand: MutableList<AccessibilityNodeInfo> = mutableListOf(), | ||
| var scrollAttemptsLeft: Int = MAX_SCROLL_ATTEMPTS, | ||
| var rowExpanded: Boolean = false, | ||
| var settling: Boolean = false, | ||
| ) |
Introduce docs and README explaining "obscured" notifications (empty GROUP_SUMMARY whose content is only in the shade). Add ObscuredNotification data model and detection/readout flow: schedule a short pending lookup for ParsedIgnored posts, and if no sibling arrives call MyAccessibilityService to open the shade, expand/scan rows, and read matching app rows. Wire in handler timing, explanatory comments, and a README for notification parsers; include telemetry concepts for logging FOUND / NOT_FOUND / ACCESSIBILITY_UNAVAILABLE outcomes so packages that require special handling can be identified (e.g. stale-obscured Google Chat).
Introduce a new NotificationParseResult.ParsedEmpty and return it when a notification yields no readable parts. Keep ParsedIgnored for cases where the text-to-speech manager is unavailable so we don't schedule a deferred accessibility lookup when TTS can't run. Update MyNotificationListenerService to schedule the pending accessibility lookup only for ParsedEmpty results. Also tidy up the parsing logic order (move the TTS null check) and add a TODO about the current control flow.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ility Agent-Logs-Url: https://github.com/LookAtWhatAiCanDo/Shaide/sessions/0d937efc-98a6-4edd-a33c-57a280f1908d Co-authored-by: paulpv <1393897+paulpv@users.noreply.github.com>
…ches for row expand/collapse Switches expand/collapse logic to rely on AccessibilityNodeInfo.actionList (ACTION_EXPAND / ACTION_COLLAPSE) rather than string-matching chevron contentDescription or searching for direct child buttons. This is locale-agnostic, avoids accidentally descending into child expandable rows, and simplifies the codepaths by removing the now-unused EXPAND/COLLAPSE constants and findDirectRowButton helper.
…, tighten expand logic, shorten delays, and improve logging Centralize and simplify the shade-search/scan logic by consistently using NotificationShadeSnapshot helpers and the concrete ShadeRow type, replacing ad-hoc live-node lists with fresh lookups. Introduced a single selfAdvance state-machine path (driven by timers and events) instead of separate timer/event flows, removed stale rowsToExpand buffering, and proactively re-advances after settle delays so searches don't stall. Make expand detection more robust by requiring ACTION_EXPAND present and ACTION_COLLAPSE absent (avoids acting on transitional/ambiguous nodes). Adjust debug flags/log messages (prefixed markers like #ACCESSIBILITY/#OBSCURED) and shorten FAST delay values for quicker test runs. Minor logging/text tweaks and small parser log message clarification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.