Skip to content

Fix blocked wiki syncs and add offline knowledge library - #32

Merged
TrapEmAll merged 1 commit into
mainfrom
feature/offline-knowledge-archive
Aug 2, 2026
Merged

Fix blocked wiki syncs and add offline knowledge library#32
TrapEmAll merged 1 commit into
mainfrom
feature/offline-knowledge-archive

Conversation

@TrapEmAll

Copy link
Copy Markdown
Owner

Summary

  • Detect Cloudflare browser-verification responses from ConsoleMods and XenonLibrary and show an actionable error.
  • Preserve successful cache timestamps, retrieval metadata, stale-cache state, and known URLs across failed refreshes.
  • Stop retrying every page after a host blocks automated access, while continuing with available cached documents and other sources.
  • Add browser-saved HTML/folder/ZIP import for ConsoleMods, XenonLibrary, and Free60.
  • Build a self-contained dark offline knowledge library with local search, source attribution, timestamps, license notes, and no remote scripts or trackers.
  • Parse saved ConsoleMods TitleID and Multi-ID pages through the structured metadata adapters and enrich only unknown local fields.
  • Treat Unknown game and Unknown title as replaceable placeholders without overwriting known metadata.
  • Add GUI and CLI controls, migration 11, tests, architecture notes, source guidance, and README documentation.

Root Cause

Both ConsoleMods and XenonLibrary currently serve Cloudflare browser verification to automated clients. Their normal pages can remain available in a browser while MediaWiki API, sitemap, article, and raw endpoints return HTTP 403. The previous client surfaced a generic request failure and had no workflow for safely bringing browser-accessible pages into the local knowledge database.

This change does not bypass access controls. It makes the failure explicit and provides cache-first and user-driven offline paths.

Migration Notes

Schema migration 11 adds:

  • offline_archive_runs
  • offline_archive_documents
  • offline_page_import_runs

The migration is additive and runs automatically through the existing schema manager. Existing source documents and raw caches remain compatible.

Source And Licensing Notes

  • Offline pages retain source URLs, source names, timestamps, and stated license information.
  • Rendered pages contain readable text only and do not copy remote scripts, styles, trackers, or images.
  • ConsoleMods and XenonLibrary content stays source-attributed; XenonLibrary's CC BY-NC-SA terms remain visible.
  • Imported HTML is bounded, never executed, and accepts canonical URLs only for the selected source.
  • XboxUnity transport remains explicitly HTTP-only and is unchanged.

Validation

  • python tests.py: 108 passed
  • python integration_tests.py: 31 passed
  • python -m ruff check .: passed
  • focused mypy check for the knowledge modules: passed
  • compile and version consistency checks: passed
  • CLI help and desktop import smoke checks: passed

@TrapEmAll
TrapEmAll marked this pull request as ready for review August 2, 2026 10:19
@TrapEmAll
TrapEmAll merged commit 7375400 into main Aug 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant