Fix blocked wiki syncs and add offline knowledge library

ARCHITECTURE.md

@@ -38,6 +38,8 @@ results to Tk's main loop.
 - `profile_manager.py` owns Content-tree discovery, STFS ownership inspection,
   profile/save indexing, verified snapshots, and conflict-safe restore.
 - `knowledge_sync.py` exposes complete source-import workflows to CLI and GUI.
+- `offline_knowledge.py` safely imports browser-saved pages and renders the
+  self-contained local reading library.
 - `collection_intelligence.py` coordinates snapshots, exact MediaID matching,
   health, preservation matching, repair previews, and offline exports.
 - `console_sync.py` owns durable transfer jobs, resumable FTP, remote
@@ -72,7 +74,7 @@ Main schema groups:
 - Library: `titleids`, `title_updates`, `covers`, `download_history`
 - Knowledge: sources, documents, revisions, entities, names, identifiers,
   facts, citations, relationships, import runs, conflicts, source priorities,
-  conflict decisions, and scheduled sync state
+  conflict decisions, scheduled sync state, and offline archive/import runs
 - Backups: targets, scans, inventory, and operations
 - Profiles: scan runs, profiles, saves, snapshots, snapshot files, GPD
   inventories, achievements, comparisons, Xenia migration runs, and auditable
@@ -91,12 +93,13 @@ data and be covered by tests.
 ```text
 source discovery
   -> rate-limited fetch
-  -> raw cache
+  -> raw cache or validated browser-saved page
   -> source adapter
   -> normalized records
   -> citations and conflicts
   -> preferred fact selection
   -> fill unknown library metadata
+  -> script-free offline HTML archive
 ```
 
 Raw documents and retrieval metadata remain attached to their source. A parser

CHANGELOG.md

@@ -7,6 +7,13 @@ Notable changes to UnityScraper are documented here. The project follows
 
 ### Added
 
+- Additive schema migration 11 for offline knowledge archive builds, rendered
+  document state, and browser-saved wiki import history.
+- A self-contained offline Knowledge library with local search, source and
+  license attribution, stale-cache labels, and script-free article pages.
+- Safe `.html`, `.htm`, folder, and ZIP imports for pages saved from
+  ConsoleMods, XenonLibrary, or Free60 in a normal browser.
+
 - Additive schema migration 10 for metadata snapshot runs, library intelligence,
   preservation reports, correction packages, hardware records, and package extractions.
 - Visual Studio 2010-inspired dark desktop theme shared by the modern and legacy
@@ -107,6 +114,16 @@ Notable changes to UnityScraper are documented here. The project follows
 - Tagged-release packaging with SHA-256 checksums.
 - Contributor, security, community, and architecture documentation.
 
+### Fixed
+
+- ConsoleMods and XenonLibrary Cloudflare responses now produce actionable
+  browser-verification diagnostics instead of opaque HTTP failures.
+- Wiki refreshes retain known seed and cache URLs, preserve the original fetch
+  timestamp when falling back offline, and no longer discard successful pages
+  because another page failed.
+- Knowledge enrichment now recognizes `Unknown game` and `Unknown title` as
+  replaceable placeholders when a source provides a real title.
+
 ### Changed
 
 - Version advanced to `1.2.0-beta.1`.

DOCS_INDEX.md

@@ -13,8 +13,8 @@
   XEX identity, Title Update compatibility, preservation, and repair previews
 - [Console Sync](CONSOLE_SYNC.md) - persistent transfers, resume, snapshots,
   comparison, and verification
-- [Knowledge Sources](KNOWLEDGE_SOURCES.md) - imports, provenance, caching, and
-  source licensing
+- [Knowledge Sources](KNOWLEDGE_SOURCES.md) - imports, provenance, caching,
+  browser-verification recovery, offline viewing, and source licensing
 - [Advanced Features](ADVANCED_FEATURES.md) - rate limits, resume, diagnostics,
   portable mode, API, and conversion
 - [REST API](API.md) - authentication, endpoints, configuration, and safety

KNOWLEDGE_SOURCES.md

@@ -33,7 +33,8 @@ python main.py --sync-wikis
 
 ConsoleMods and XenonLibrary pages are discovered through MediaWiki's paginated
 all-pages API, with XML sitemap discovery as an additional path. Free60 uses
-its XML sitemap. Seed pages are used only when discovery is unavailable.
+its XML sitemap. Seed pages and URLs from successful earlier cache entries are
+always retained, so a temporary discovery failure does not hide known pages.
 
 Every fetched article is cached locally and imported as a searchable knowledge
 entity with its source URL, revision snapshot, summary, and visible article
@@ -42,6 +43,45 @@ receive a freshness warning.
 
 Use `--wiki-limit N` to restrict each source during testing or a first sync.
 
+## Browser Verification And Offline Use
+
+ConsoleMods and XenonLibrary currently may place Cloudflare browser
+verification in front of wiki and API requests. A normal browser can work while
+the same URL returns HTTP 403 to UnityScraper. The application deliberately
+does not imitate a browser session, solve challenges, or bypass source access
+controls.
+
+When this happens UnityScraper:
+
+- reports that browser verification blocked the refresh;
+- uses a prior cached page when one exists without changing its original fetch
+  timestamp;
+- records the stale-cache state and refresh error with the source document;
+- continues importing other available sources;
+- rebuilds the offline library from every usable cached page.
+
+In the desktop **Knowledge** workspace, choose a source and use **Import Saved
+Wiki Pages**. You can select individual `.html`/`.htm` files, a folder, or a ZIP
+containing saved pages. For command-line use:
+
+```powershell
+python main.py --import-saved-wiki "C:\Saved Wikis" --saved-wiki-source consolemods-wiki
+python main.py --import-saved-wiki "C:\Saved Wikis\xenon.zip" --saved-wiki-source xenonlibrary
+python main.py --build-offline-knowledge
+```
+
+The importer accepts at most 5,000 pages, 10 MB per page, and 250 MB total per
+operation. It never executes imported HTML. Canonical URLs are accepted only
+when they match the selected source. A saved ConsoleMods TitleID or Multi-ID
+list also runs through the structured game metadata parser and still enriches
+only unknown local names or publishers.
+
+The generated `offline_knowledge/index.html` is a private, self-contained,
+dark-theme reading library. It contains readable article text, cache status,
+source links, timestamps, and license attribution. Remote scripts, styles,
+trackers, and images are not copied into rendered pages. Raw source snapshots
+remain in the application cache for provenance and future reprocessing.
+
 ## Preservation DAT Import
 
 Download DATs directly from their source and import them locally:
@@ -71,6 +111,9 @@ The database migration adds these normalized tables:
 - `entity_relationships`
 - `knowledge_import_runs`
 - `knowledge_conflicts`
+- `offline_archive_runs`
+- `offline_archive_documents`
+- `offline_page_import_runs`
 
 Facts are source-attributed claims. If two sources disagree, both claims can
 exist and the disagreement is recorded in `knowledge_conflicts`.
@@ -110,6 +153,7 @@ The **Knowledge** page includes:
 - details with source names and citation URLs;
 - source license, document count, fact count, and latest import status;
 - ConsoleMods ID sync and whole-wiki sync;
+- saved-page import, offline-library rebuild, and local-browser access;
 - Redump and No-Intro file import;
 - per-property source priorities, where lower numbers are preferred for
   display;
@@ -120,7 +164,8 @@ The **Knowledge** page includes:
 ## Remaining Boundaries
 
 - Source availability, access controls, and licenses can change. Failed syncs
-  are isolated per source and previously cached pages remain available.
+  are isolated per source and previously cached or manually imported pages
+  remain available.
 - Redump and No-Intro DATs are not bundled. Users obtain them from the source.
 - Wiki content is reference material, not automatically trusted repair advice.
 - Commercial game images, firmware, keys, and leaked SDK files are never

PROJECT_STATUS.md

@@ -16,6 +16,8 @@ backup-management, and source-attributed knowledge application.
 - Searchable ConsoleMods, XenonLibrary, and Free60 wiki article ingestion using
   MediaWiki API or sitemap discovery, local caching, attribution, and per-source
   failure isolation.
+- Cloudflare-aware diagnostics, stale-cache recovery, browser-saved page
+  import, and a self-contained offline Knowledge reading library.
 - User-selected Redump and No-Intro XML DAT import for release identity, serial,
   size, status, and checksum metadata.
 - Desktop knowledge browser with source status, licenses, citations, import
@@ -65,6 +67,8 @@ backup-management, and source-attributed knowledge application.
   duplicate actions, plus selectable restore controls.
 - Additive schema migration 10 for metadata snapshots, library audits,
   preservation reports, correction exports, hardware records, and package extraction.
+- Additive schema migration 11 for offline archive builds, rendered document
+  state, and saved-page import history.
 - Visual Studio 2010-inspired shared desktop theme, classic menus, scoped API
   tokens, out-of-process plugin execution, and bounded community language packs.
 - Read-only consecutive STFS extraction, direct Xenia launch controls, and
@@ -78,8 +82,8 @@ backup-management, and source-attributed knowledge application.
 
 - Offline tests cover scraper configuration, database behavior, ConsoleMods parsing,
   DAT parsing, wiki parsing, knowledge search/provenance, downloads, queueing,
-  STFS/XBE inspection, safe archives, backup scanning, atomic copies, and an
-  end-to-end local workflow.
+  blocked-source recovery, saved-page rendering, STFS/XBE inspection, safe
+  archives, backup scanning, atomic copies, and an end-to-end local workflow.
 - Network-backed source syncs remain dependent on each source's availability and
   access policy. Cached copies are used when available.
 - Windows, Linux, and macOS artifacts are generated by CI and releases

README.md

@@ -35,6 +35,8 @@ service exposes. UnityScraper does not silently substitute HTTPS URLs.
 - Imports TitleID, publisher, region, and Multi-ID references from ConsoleMods.
 - Caches and indexes Xbox 360 articles from ConsoleMods, XenonLibrary, and
   Free60.
+- Builds a searchable, self-contained offline library from cached or
+  browser-saved wiki pages.
 - Imports user-supplied Redump and No-Intro XML DAT files.
 - Stores entities, identifiers, facts, citations, revisions, import runs, and
   conflicts with provenance.
@@ -299,10 +301,25 @@ python main.py --sync-wikis
 # Limit a first test sync per source
 python main.py --sync-wikis --wiki-limit 25
 
+# Build or refresh the local offline reading library
+python main.py --build-offline-knowledge
+
+# Import a page, folder, or ZIP saved in a browser when a wiki blocks automation
+python main.py --import-saved-wiki "C:\Saved Wikis" --saved-wiki-source xenonlibrary
+
 # Import a user-downloaded preservation DAT
 python main.py --import-dat "D:\DATs\xbox360.dat" --dat-source redump
 ```
 
+ConsoleMods and XenonLibrary may require Cloudflare browser verification and
+return HTTP 403 to command-line clients. UnityScraper does not bypass that
+protection. It continues using the last successful cache and identifies the
+blocked source clearly. In the **Knowledge** workspace, use **Import Saved Wiki
+Pages** for pages saved through a normal browser, then **Open Offline Library**
+to browse the local copy. The offline renderer keeps article text, source
+links, timestamps, and license attribution while excluding remote scripts and
+trackers.
+
 ### Backup Manager
 
 ```powershell

app_paths.py

@@ -148,6 +148,7 @@ def xdg_path(variable: str, fallback: Path) -> Path:
 PROFILE_BACKUPS_DIR = DATA_DIR / "profile_backups"
 PLUGINS_DIR = DATA_DIR / "plugins"
 LANGUAGE_PACKS_DIR = DATA_DIR / "languages"
+OFFLINE_KNOWLEDGE_DIR = DATA_DIR / "offline_knowledge"
 
 DATABASE_PATH = DATA_DIR / "unityscraper.db"
 CONFIG_PATH = CONFIG_DIR / "config.json"
@@ -176,6 +177,7 @@ def ensure_app_dirs() -> None:
         PROFILE_BACKUPS_DIR,
         PLUGINS_DIR,
         LANGUAGE_PACKS_DIR,
+        OFFLINE_KNOWLEDGE_DIR,
     ):
         path.mkdir(parents=True, exist_ok=True)
 

database_migrations.py

@@ -8,7 +8,7 @@
 from pathlib import Path
 
 
-SCHEMA_VERSION = 10
+SCHEMA_VERSION = 11
 
 
 def _now() -> str:
@@ -71,6 +71,7 @@ def ensure_application_schema(connection: sqlite3.Connection) -> int:
         (8, "community roadmap workspaces", _migration_community_roadmap),
         (9, "hardening and plugin runtime", _migration_hardening),
         (10, "release readiness workspaces", _migration_release_readiness),
+        (11, "offline knowledge archive", _migration_offline_knowledge),
     )
     for version, name, migration in migrations:
         if version in applied:
@@ -805,3 +806,39 @@ def _migration_release_readiness(connection: sqlite3.Connection) -> None:
         );
         """
     )
+
+
+def _migration_offline_knowledge(connection: sqlite3.Connection) -> None:
+    """Track generated offline pages and user-provided wiki imports."""
+    connection.executescript(
+        """
+        CREATE TABLE IF NOT EXISTS offline_archive_runs (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            started_at TEXT NOT NULL,
+            finished_at TEXT,
+            status TEXT NOT NULL,
+            documents_written INTEGER NOT NULL DEFAULT 0,
+            index_path TEXT,
+            errors TEXT
+        );
+        CREATE TABLE IF NOT EXISTS offline_archive_documents (
+            document_id INTEGER PRIMARY KEY,
+            archive_path TEXT NOT NULL,
+            rendered_at TEXT NOT NULL,
+            content_sha256 TEXT NOT NULL,
+            stale INTEGER NOT NULL DEFAULT 0,
+            FOREIGN KEY(document_id) REFERENCES source_documents(id)
+        );
+        CREATE TABLE IF NOT EXISTS offline_page_import_runs (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            source_slug TEXT NOT NULL,
+            source_path TEXT NOT NULL,
+            started_at TEXT NOT NULL,
+            finished_at TEXT,
+            status TEXT NOT NULL,
+            files_seen INTEGER NOT NULL DEFAULT 0,
+            files_imported INTEGER NOT NULL DEFAULT 0,
+            errors TEXT
+        );
+        """
+    )

knowledge_base.py

@@ -17,7 +17,16 @@
 
 logger = logging.getLogger(__name__)
 
-UNKNOWN_VALUES = {"", "unknown", "unknown publisher", "n/a", "none", "null"}
+UNKNOWN_VALUES = {
+    "",
+    "unknown",
+    "unknown game",
+    "unknown title",
+    "unknown publisher",
+    "n/a",
+    "none",
+    "null",
+}
 
 
 def utc_now() -> str: