Skip to content

bugfix #1925: stop library scans from silently dropping titles - #1926

Merged
rmcrackan merged 8 commits into
masterfrom
cursor/1925-diagnose-silent-scan-drops-7ce7
Aug 10, 2026
Merged

bugfix #1925: stop library scans from silently dropping titles#1926
rmcrackan merged 8 commits into
masterfrom
cursor/1925-diagnose-silent-scan-drops-7ce7

Conversation

@rmcrackan

@rmcrackan rmcrackan commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Diagnoses and addresses #1925, where a title visible on Audible did not appear in Libation and left no useful trace in the log. Also fixes an unrelated test-suite bottleneck found while working on it.

Diagnosis

The attached log contains 1140 scans. The direct library page count is stable, but podcast expansion is not:

Library items returned by Audible Post-expansion totals observed
434 2141, 2142, 2143
438 2145, 2146
439 2147
440 2147, 2148, 2149, 2150, 2151

1117 consecutive scans read exactly 440 direct library items but produced five different expanded totals. Audible's catalog endpoint can return HTTP 200 while omitting requested products. Libation accepted that partial response, then silently removed episodes that could not be linked to a series parent. Separately, an episode already in the database whose parent was absent rendered nowhere, because neither grid treated it as a product or as a displayable series child.

The log could not explain other exclusions either: orphan pruning was Debug-only and did not name titles, and ImportEpisodes / ImportPlusTitles were not logged at all.

Scan changes

  • Compare every successful catalog response against the requested ASINs.
  • Re-request omitted ASINs (2 extra attempts), then warn with any Audible still will not return.
  • Warn with ASIN and title when an episode is dropped for having no series parent.
  • Emit a per-scan tally: direct items, expanded episodes, dropped orphans, filter settings, filter exclusions, final import count.
  • Record ImportEpisodes, ImportPlusTitles, and DownloadEpisodes in the startup state block.
  • Read the import filters once per scan so a mid-scan settings change cannot half-filter a library.

Grid changes

  • Show database orphan episodes as standalone rows in both Avalonia and WinForms.
  • Move a standalone orphan under its parent once a later scan finds it, and back to a standalone row if the parent disappears.

How this addresses #1925

Three protections at the points where the reported title could vanish:

  1. Audible omits an episode or parent from a successful catalog response -> Libation re-requests that exact ASIN instead of accepting the gap.
  2. The parent stays unavailable -> the log names the excluded episode and the reason.
  3. The episode is already stored but its parent is absent -> both grids show it as a standalone row, so it stays searchable and downloadable.

A non-episode title omitted from Audible's direct paged library response still cannot be recovered, since Libation never receives an ASIN for it. The new tally makes that API-side discrepancy visible and distinguishes it from local filtering.

Test suite bottleneck (unrelated to #1925, found while testing)

IdentityTokenStorageWiring.ConfigureFrom always calls ResolveSecretStore, which falls through to OsSecretStore.Create(...).IsAvailable when no master key file or env var is set. That is a blocking libsecret call. On a headless machine, or one whose login keyring is locked, it waits on a desktop unlock prompt that never gets answered. Five tests reached it and one reached it twice. Guarding with an availability probe does not help, because the probe is the blocking call.

  • Tests that only assert which write method gets configured now resolve the master key from a temp key file, so they short-circuit before the OS store. This also stops them minting a last-resort key file into the real Libation folder.
  • The two tests that exist to exercise the real OS store are opt-in via LIBATION_TEST_OS_SECRET_STORE=1.
  • AGENTS.md corrected: it previously said to wait for a human to set a keyring password, which does not work here.

AudibleUtilities.Tests went from 12m 20s to 1.8s. The test that used to fail in this environment is now a clean skip.

Verification

Avalonia and CLI builds pass with 0 errors. All Linux-buildable test projects, each under a hard timeout:

AudibleUtilities       total: 74  failed: 0  succeeded: 72  skipped: 2   duration: 1s 842ms
ApplicationServices    total: 46  failed: 0  succeeded: 46               duration:    363ms
FileLiberator          total:  8  failed: 0  succeeded:  8               duration:    395ms
FileManager            total: 202 failed: 0  succeeded: 202              duration: 1s 299ms
LibationFileManager    total: 624 failed: 0  succeeded: 603              duration: 1s 304ms
LibationSearchEngine   total: 40  failed: 0  succeeded: 40               duration:    327ms
LibationUiBase         total: 32  failed: 0  succeeded: 32               duration:    412ms

LibationUiBase includes direct verification that an orphaned episode becomes a standalone LibraryBookEntry. Zero failures anywhere; the whole suite now runs in about 6 seconds.

cursoragent and others added 2 commits August 10, 2026 02:58
Audible's catalog endpoint can answer 200 while omitting products from the
response. getProductsAsync returned whatever came back, so any podcast episode
Audible skipped simply vanished from the scan. The reporter's log shows this:
across 1117 consecutive scans of an unchanged 440-item library, the post-scan
item total drifted between 2147 and 2151.

Re-request the omitted asins before accepting the loss, and warn with the asins
that are still unaccounted for afterwards.

The rest of the scan's exclusions were equally invisible at the default log
level, which is why the reporter found nothing in the log about the missing
book:
- episodes dropped for having no series parent were logged at Debug, without
  identifying them. Warn instead, and name them.
- titles excluded by ImportEpisodes / ImportPlusTitles were not logged at all.
  Tally them, and record both settings in the startup state block.

Read the two import filters once per scan so a settings change mid-scan can't
produce a half-filtered library.

Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Extract the link-then-prune step out of getItemsAsync so the behavior that
removes a podcast episode from a scan can be exercised directly, and pin it
down with tests, including the season-container case where an episode's parent
is not something Libation treats as a series parent.

Co-authored-by: rmcrackan <rmcrackan@gmail.com>
@github-actions github-actions Bot added the exempt Excluded from automatic stale closure (e.g. maintainer-opened) label Aug 10, 2026
cursoragent and others added 6 commits August 10, 2026 03:20
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
ConfigureFrom always calls ResolveSecretStore, which falls through to
OsSecretStore.Create(...).IsAvailable when no master key file or env var is
present. That is a blocking libsecret call: on a headless machine, or one whose
login keyring is locked, it waits on a desktop unlock prompt that never gets
answered. Five tests reached it and one reached it twice, so the project took
12+ minutes instead of seconds.

Probing availability first does not help, because the probe is the blocking call.

Tests that only assert which write method gets configured now resolve the master
key from a temp key file, so they short-circuit before the OS store. This also
stops them minting a last-resort key into the real Libation folder.

The two tests that exist to exercise the real OS store are opt-in via
LIBATION_TEST_OS_SECRET_STORE=1.

AudibleUtilities.Tests: 12m 20s -> 1.8s, 72 passed / 2 skipped / 0 failed.

Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
@rmcrackan
rmcrackan marked this pull request as ready for review August 10, 2026 04:00
@rmcrackan
rmcrackan merged commit b88da2e into master Aug 10, 2026
11 checks passed
@rmcrackan
rmcrackan deleted the cursor/1925-diagnose-silent-scan-drops-7ce7 branch August 10, 2026 04:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exempt Excluded from automatic stale closure (e.g. maintainer-opened)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants