Application Lists (with Android adb) by thomasm789 · Pull Request #68 · unfoldedcircle/integration-androidtv

thomasm789 · 2025-04-29T16:02:32Z

Summary
Introduces python-adb, handles ADB authentication, and adds support for dynamic app selection based on installed apps (via if enabled ADB) and/or pre-defined apps list.

Features

Added app configurator UI allowing users to enable/disable and update the names of apps where friendly names could not be found
Supports both static app data maintained in Apps.py and ADB-discovered apps (if enabled)
For ADB packages, if friendly names already exist in Apps.py they will be pulled, if external metadata is enabled then an attempt to pull friendly app name will be made as well. Otherwise the user has the ability to edit the name in the UI
Friendly names are auto-filled using IdMappings, and fetched via get_app_metadata() if external metadata is enabled
De-duplication of apps if the same packages already exist in Apps.py and are pulled via ADB. Apps.py will take priority

Application Source Selection

Per-device app selections saved to appslist_.json
tv.py and select_source() updated to read from saved app lists
Falls back to static metadata when no saved list exists
Only explicitly enabled apps (*_enabled == true) are saved to reduce config noise

ADB Integration

Introduced python-adb for managing ADB connections
Handles ADB authentication and retrieval of installed packages
Authentication for ADB currently follows successful androidremotetv2 pincode process
Automatically cleans up ADB keys and appslist_*.json on device reset/remove

General Improvements

Improved logging for debugging app loading and metadata failures
General cleanup and refactoring of setup flow and app selection logic

…lso setting friendly names

…ys and appslist.json file

…orm dynamic

zehnm

Thanks a lot for this addition, can't wait to use it!

Unfortunately I can't configure a new device and use ADB, there is at least one required module missing by adb-shell:

DEBUG:setup_flow:ADB is enabled, proceeding with ADB setup
ERROR:ucapi.api:Exception in setup handler, aborting setup! Exception: No module named 'async_timeout'

I've tried pip3 install -r requirements.txt --force-reinstall, but that didn't install it.

zehnm · 2025-05-07T16:52:32Z

+def _get_config_root() -> Path:
+    config_home = Path(os.environ.get("UC_CONFIG_HOME", "./config"))
+    config_home.mkdir(parents=True, exist_ok=True)
+    return config_home
+
+
+def _get_data_root() -> Path:
+    data_home = Path(os.environ.get("UC_DATA_HOME", "./data"))
+    data_home.mkdir(parents=True, exist_ok=True)
+    return data_home


These functions are now used in other files and shouldn't be "pseudo-private" anymore. Please remove the leading underscore.
TBH I don't really know the Python customs for "private things", that's all a bit weird to me :-)

zehnm · 2025-05-07T16:57:27Z

+            adb_cert_path = Path(os.environ.get("UC_CONFIG_HOME", "./config")) / "certs" / f"adb_{choice}"
+            adb_cert_path_pub = Path(os.environ.get("UC_CONFIG_HOME", "./config")) / "certs" / f"adb_{choice}.pub"
+            if adb_cert_path.exists():
+                adb_cert_path.unlink()
+            if adb_cert_path_pub.exists():
+                adb_cert_path_pub.unlink()
+            appslist_path = Path(os.environ.get("UC_CONFIG_HOME", "./config")) / f"appslist_{choice}.json"


Use get_config_root to retrieve the configuration directory. The UC_CONFIG_HOME env var with default path shouldn't be copy pasted to several source files. See also src/adb_tv.py

thomasm789 · 2025-05-07T17:09:21Z

Thanks a lot for this addition, can't wait to use it!

Unfortunately I can't configure a new device and use ADB, there is at least one required module missing by adb-shell:
DEBUG:setup_flow:ADB is enabled, proceeding with ADB setup
ERROR:ucapi.api:Exception in setup handler, aborting setup! Exception: No module named 'async_timeout'
I've tried pip3 install -r requirements.txt --force-reinstall, but that didn't install it.

Can you try installing adb-shell in your virtual env and let me know what package you get?

"pip list" shows the following for me.

adb_shell 0.4.4

"pip show adb_shell"
Name: adb_shell
Version: 0.4.4
Summary: A Python implementation of ADB with shell and FileSync functionality.
Home-page: https://github.com/JeffLIrion/adb_shell

thomasm789 · 2025-05-07T17:11:33Z

Thanks a lot for this addition, can't wait to use it!
Unfortunately I can't configure a new device and use ADB, there is at least one required module missing by adb-shell:
DEBUG:setup_flow:ADB is enabled, proceeding with ADB setup
ERROR:ucapi.api:Exception in setup handler, aborting setup! Exception: No module named 'async_timeout'
I've tried pip3 install -r requirements.txt --force-reinstall, but that didn't install it.
Can you try installing adb-shell in your virtual env and let me know what package you get?

"pip list" shows the following for me.

adb_shell 0.4.4

"pip show adb_shell" Name: adb_shell Version: 0.4.4 Summary: A Python implementation of ADB with shell and FileSync functionality. Home-page: https://github.com/JeffLIrion/adb_shell

Ah I made a mistake in requirements.txt... I used a dash instead of an underscore.....

zehnm · 2025-05-07T19:36:24Z

Missing entry in requirements.txt: async_timeout~=5.0.1 Then it works!

Is there a way to reconfigure the app-list? I've only seen it in the initial setup when adding a new device.
Is there an option to always show all retrieved apps by ADB and automatically refresh them? I've selected 2 apps and now it seems that I'm stuck with only those 2 :-) Maybe I've missed that in the setup screen and I haven't looked at the setup code yet.

I have to check if the "keep-alive" option for the setup-flow is working in the core & web-configurator. Then we simply have to send the "setup-is-still-alive" message while the app configuration page is shown. Otherwise the setup-screen automatically times out after 60s and the whole setup fails. For configuring a lengthy app-list, that's not user friendly.
I'll try to verify it this week.

zehnm · 2025-05-07T19:45:03Z

While checking the logs, there seems some URL encoding missing for retrieving the metadata after fetching the ADB app list. This might be related to the previous external metadata addition, but I haven't notices these errors before.

WARNING:external_metadata:Google Play metadata fetch failed for Prime Video: URL can't contain control characters. '/store/apps/details?id=Prime Video&hl=en&gl=us' (found at least ' ')
WARNING:external_metadata:Google Play metadata fetch failed for HBO Max: URL can't contain control characters. '/store/apps/details?id=HBO Max&hl=en&gl=us' (found at least ' ')
WARNING:external_metadata:Google Play metadata fetch failed for Apple TV: URL can't contain control characters. '/store/apps/details?id=Apple TV&hl=en&gl=us' (found at least ' ')
WARNING:external_metadata:Google Play metadata fetch failed for Steam Link: URL can't contain control characters. '/store/apps/details?id=Steam Link&hl=en&gl=us' (found at least ' ')

There are also a lot of "App not found(404)" errors for well-known apps like:

WARNING:external_metadata:Google Play metadata fetch failed for Kodi: App not found(404).
WARNING:external_metadata:Google Play metadata fetch failed for Spotify: App not found(404).
WARNING:external_metadata:Google Play metadata fetch failed for Disney+: App not found(404).
WARNING:external_metadata:Google Play metadata fetch failed for Netflix: App not found(404).

Time to look into the code...

thomasm789 · 2025-05-07T20:09:15Z

While checking the logs, there seems some URL encoding missing for retrieving the metadata after fetching the ADB app list. This might be related to the previous external metadata addition, but I haven't notices these errors before.


WARNING:external_metadata:Google Play metadata fetch failed for Prime Video: URL can't contain control characters. '/store/apps/details?id=Prime Video&hl=en&gl=us' (found at least ' ')

WARNING:external_metadata:Google Play metadata fetch failed for HBO Max: URL can't contain control characters. '/store/apps/details?id=HBO Max&hl=en&gl=us' (found at least ' ')

WARNING:external_metadata:Google Play metadata fetch failed for Apple TV: URL can't contain control characters. '/store/apps/details?id=Apple TV&hl=en&gl=us' (found at least ' ')

WARNING:external_metadata:Google Play metadata fetch failed for Steam Link: URL can't contain control characters. '/store/apps/details?id=Steam Link&hl=en&gl=us' (found at least ' ')

There are also a lot of "App not found(404)" errors for well-known apps like:


WARNING:external_metadata:Google Play metadata fetch failed for Kodi: App not found(404).

WARNING:external_metadata:Google Play metadata fetch failed for Spotify: App not found(404).

WARNING:external_metadata:Google Play metadata fetch failed for Disney+: App not found(404).

WARNING:external_metadata:Google Play metadata fetch failed for Netflix: App not found(404).

Time to look into the code...

Weird. It looks like the friendly name is being used instead of the package name. Although during testing it was working so maybe I changed something that caused this unexpected behaviour.

Some 404s are expected as not all apps will be found.

Will need to review.

thomasm789 · 2025-05-07T20:11:47Z

Missing entry in requirements.txt: async_timeout~=5.0.1 Then it works!

Is there a way to reconfigure the app-list? I've only seen it in the initial setup when adding a new device.

Is there an option to always show all retrieved apps by ADB and automatically refresh them? I've selected 2 apps and now it seems that I'm stuck with only those 2 :-) Maybe I've missed that in the setup screen and I haven't looked at the setup code yet.

I have to check if the "keep-alive" option for the setup-flow is working in the core & web-configurator. Then we simply have to send the "setup-is-still-alive" message while the app configuration page is shown. Otherwise the setup-screen automatically times out after 60s and the whole setup fails. For configuring a lengthy app-list, that's not user friendly.

I'll try to verify it this week.

I didn't touch the reconfiguration step as it was getting very messy to maintain. I wanted to split the views up from the logic so that things could be DRY otherwise it would have got a bit messy.

zehnm · 2025-05-13T13:38:25Z

I have to check if the "keep-alive" option for the setup-flow is working in the core & web-configurator.

That's something that cannot be changed quickly and requires support from the web-configurator and remote-ui.

keep-alive option is only possible for actions running in the integration driver, for example if the discovery or setup process takes a long time.
user input requests from the integration driver have a fixed timeout of 3 min, and there's an overall setup-flow timeout of 5 min.

We have to live with that at the moment. First we need to enhance the web-configurator to send additional events, that a user input screen is still shown.

@thomasm789 would you like me to sync your branch with main? It diverged with the latest additions.

thomasm789 added 17 commits April 23, 2025 22:25

updates to external_metadata.py for async and adding httpx dependancy

9ba1b32

fixes bugs in 64 ands 63

a558e10

update default metadata

15df3fa

further tweaks

b20e500

further tweaks

74e8fd4

adb poc

4709da4

update

27ec593

merge from main

1e2c497

update adb attempt

a7f53d2

Merge remote-tracking branch 'origin/main' into android-adb

fe13f34

updates

3ef5ba7

update

17c2690

lint

95ecaaf

isort

ced371e

update setupflow to allow for application list selection and if adb a…

9573f72

…lso setting friendly names

testing

ab015f7

final commit - adding in apps configurator + adb + cleaning up adb ke…

4ead16b

…ys and appslist.json file

thomasm789 marked this pull request as ready for review April 30, 2025 21:38

thomasm789 added 8 commits April 30, 2025 22:54

using idmapping if friendly names are already known + fixes

fdefbfb

fixes

ca21a78

linting

b0ae053

adding in requirements.txt

2b621da

clean up

3739dd7

clean up

a799181

replacing cache_root with config _get_data_root

e35a2f8

linting

f57eb1e

thomasm789 mentioned this pull request May 1, 2025

Send text using commands via simple cmds #69

Closed

thomasm789 added 3 commits May 2, 2025 13:47

clean up app list logic - remove duplicates and sort alpha and make f…

9d2525a

…orm dynamic

clean up app list logic

0405df2

clean up app list logic

00de323

zehnm self-requested a review May 5, 2025 16:40

thomasm789 mentioned this pull request May 6, 2025

Centralised Media Metadata Handling & General Cleanup #70

Closed

zehnm requested changes May 7, 2025

View reviewed changes

thomasm789 closed this May 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Application Lists (with Android adb)#68

Application Lists (with Android adb)#68
thomasm789 wants to merge 28 commits into
unfoldedcircle:mainfrom
thomasm789:android-adb

thomasm789 commented Apr 29, 2025 •

edited

Loading

Uh oh!

zehnm left a comment

Uh oh!

zehnm May 7, 2025

Uh oh!

zehnm May 7, 2025

Uh oh!

thomasm789 commented May 7, 2025 •

edited

Loading

Uh oh!

thomasm789 commented May 7, 2025

Uh oh!

zehnm commented May 7, 2025

Uh oh!

zehnm commented May 7, 2025

Uh oh!

thomasm789 commented May 7, 2025 •

edited

Loading

Uh oh!

thomasm789 commented May 7, 2025

Uh oh!

zehnm commented May 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

thomasm789 commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zehnm left a comment

Choose a reason for hiding this comment

Uh oh!

zehnm May 7, 2025

Choose a reason for hiding this comment

Uh oh!

zehnm May 7, 2025

Choose a reason for hiding this comment

Uh oh!

thomasm789 commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomasm789 commented May 7, 2025

Uh oh!

zehnm commented May 7, 2025

Uh oh!

zehnm commented May 7, 2025

Uh oh!

thomasm789 commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomasm789 commented May 7, 2025

Uh oh!

zehnm commented May 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

thomasm789 commented Apr 29, 2025 •

edited

Loading

thomasm789 commented May 7, 2025 •

edited

Loading

thomasm789 commented May 7, 2025 •

edited

Loading