Harden community workflows and preservation tools

.github/workflows/release.yml

@@ -62,7 +62,7 @@ jobs:
         shell: pwsh
         run: |
           New-Item -ItemType Directory -Path package | Out-Null
-          Copy-Item dist\UnityScraper.exe, README.md, CHANGELOG.md, LICENSE, DOCS_INDEX.md, BACKUP_MANAGER.md, COLLECTION_INTELLIGENCE.md, CONSOLE_SYNC.md, KNOWLEDGE_SOURCES.md, LINUX.md, PLUGIN_API.md package\
+          Copy-Item dist\UnityScraper.exe, README.md, CHANGELOG.md, LICENSE, DOCS_INDEX.md, BACKUP_MANAGER.md, COLLECTION_INTELLIGENCE.md, COMMUNITY_HUB.md, CONSOLE_SYNC.md, KNOWLEDGE_SOURCES.md, LINUX.md, PLUGIN_API.md, PROFILE_INTELLIGENCE.md, PROFILES_AND_SAVES.md, SECURITY.md, THIRD_PARTY_NOTICES.md package\
           Compress-Archive -Path package\* -DestinationPath UnityScraper-Windows-x64.zip
           $hash = (Get-FileHash UnityScraper-Windows-x64.zip -Algorithm SHA256).Hash.ToLower()
           "$hash *UnityScraper-Windows-x64.zip" | Set-Content UnityScraper-Windows-x64.zip.sha256

API.md

@@ -45,6 +45,12 @@ when a token is configured.
 | `GET` | `/api/titleids` | List library TitleIDs |
 | `GET` | `/api/titleid/<TitleID>` | Get one library record |
 | `GET` | `/api/search?q=` | Search the library |
+| `GET` | `/api/community/search?q=` | Search all local domains; repeat `category` to filter |
+| `GET` | `/api/preservation/dedup/actions` | List duplicate actions from the latest or selected plan |
+| `POST` | `/api/preservation/dedup/preview` | Create a read-only duplicate preview for JSON `root` |
+| `POST` | `/api/preservation/dedup/<id>/apply` | Apply `quarantine` or `hardlink` mode |
+| `POST` | `/api/preservation/dedup/<id>/restore` | Revalidate and restore a quarantined original |
+| `GET` | `/api/plugins` | List managed plugins and checksum trust state |
 | `POST` | `/api/metadata/<TitleID>` | Collect metadata |
 | `POST` | `/api/download/<TitleID>` | Process downloads |
 | `GET` | `/api/statistics` | Library statistics |
@@ -56,6 +62,9 @@ when a token is configured.
 | `POST` | `/api/config` | Update allowlisted runtime settings |
 
 TitleID routes require exactly eight hexadecimal characters.
+Duplicate apply and restore endpoints change local files and therefore require
+an explicit action ID created by a prior preview. They retain a recovery copy
+until restoration and use the same path and checksum validation as the desktop.
 
 ## Configuration
 

CHANGELOG.md

@@ -7,6 +7,16 @@ Notable changes to UnityScraper are documented here. The project follows
 
 ### Added
 
+- Additive schema migration 9 for plugin collection audits and reversible
+  duplicate recovery records.
+- Runtime plugin loading with approved-checksum enforcement, bounded results,
+  per-plugin failure isolation, provenance, and unknown-only enrichment.
+- Background Community Hub operations, actionable unified-search results, and
+  selectable duplicate apply/restore controls.
+- Unified-search and preservation API/CLI operations.
+- Read-only FATX partition geometry and Xbox 360 USB-container reporting.
+- Bounded read-only STFS file-table inventory in package workspaces.
+
 - Community Hub with unified local search across games, source knowledge,
   profiles, saves, achievements, files, and structured records.
 - Additive schema migration 8 for structured knowledge, guided console plans,

COMMUNITY_HUB.md

@@ -27,9 +27,9 @@ or press `Ctrl+K` to focus unified search.
     archive export layouts plus checksum manifests.
 12. Multi-disc completeness audits based on scanned disc number and count.
 13. Duplicate previews using size grouping and SHA-256 verification.
-14. Recoverable duplicate actions that quarantine the original and can create
-    a verified hardlink only after a second hash check.
-15. Read-only FATX signature and mounted-storage audits.
+14. Selectable, recoverable duplicate actions that quarantine the original,
+    optionally create a verified hardlink, and restore from the interface.
+15. Read-only FATX partition geometry, Xbox 360 USB container, and mounted-storage audits.
 16. Original Xbox `default.xbe` discovery alongside Xbox 360 collections.
 17. Plugin discovery, checksums, permission display, enable/disable state, and
     bounded ZIP installation or update with rollback.
@@ -46,7 +46,8 @@ or press `Ctrl+K` to focus unified search.
 - Sync plans are previews until the user confirms queueing. Queued jobs still
   run through the normal transfer controls.
 - Duplicate cleanup never deletes the only retained copy. Quarantined files
-  remain under `.unityscraper-dedup-quarantine` for manual recovery.
+  remain under `.unityscraper-dedup-quarantine` and can be restored from the
+  Preservation tab after checksum and path validation.
 - FATX images are detected read-only. Raw-device and raw-image writes are not
   implemented.
 - Package workspaces and ownership changes are previews. CON/LIVE/PIRS rebuild,
@@ -65,6 +66,11 @@ disc audits, dedup plans, storage audits, original Xbox records, plugin state,
 recovery events, dashboard tests, and accessibility preferences. It is additive
 and preserves existing databases.
 
+Migration 9 adds plugin collection audits and duplicate-recovery records. Enabled
+plugins run during normal metadata collection only while their approved entrypoint
+checksum still matches. Long Community Hub operations run in a background worker;
+search results can be opened with Enter or a double-click.
+
 Imported ConsoleMods, XenonLibrary, Free60, Redump, and No-Intro information
 continues to retain source, revision, citation, licensing, and conflict data.
 Redump and No-Intro DAT files remain user-supplied; copyrighted game content is

PLUGIN_API.md

@@ -21,9 +21,17 @@ plugins/
 }
 ```
 
-The entrypoint exports a `MetadataCollectorPlugin` subclass. The caller must
-pass the plugin ID in `enabled_plugins` before code is loaded. Permissions
-are disclosure metadata, not an operating-system sandbox, so only enable
-plugins whose source and publisher you trust.
+The entrypoint exports a `MetadataCollectorPlugin` subclass. Desktop installs
+live in the managed application plugin directory and begin disabled. Enabling a
+plugin records its SHA-256 checksum; normal metadata collection loads it only
+while the manifest ID and approved checksum still match. Each result is limited
+to 2 MiB, cover/update counts are bounded, failures are isolated, and every run
+is audited in SQLite. Known title and publisher values are never replaced by a
+plugin fallback.
+
+Requested access is disclosure metadata, not an operating-system sandbox.
+Plugin code executes with the user's account permissions, so only enable source
+and publishers you trust. Editing an enabled entrypoint automatically prevents
+it from loading until it is reviewed and enabled again.
 
 Root-level legacy Python plugins load only with `allow_legacy=True`.

PROFILE_INTELLIGENCE.md

@@ -21,9 +21,9 @@ For game GPDs, the application displays achievement ID, title, gamerscore,
 locked/unlocked state, and a valid online unlock timestamp when present. It
 also records totals for unlocked achievements and earned/possible gamerscore.
 
-The parser never writes to the source file. It does not unlock achievements,
-alter sync records, extract images, edit account settings, or repair malformed
-databases.
+The parser never writes to the source file. It can validate and export a selected
+embedded PNG/JPEG image to a new file, but it does not unlock achievements,
+alter sync records, edit account settings, or repair malformed databases.
 
 UnityScraper currently reads standalone or extracted GPD files. It does not
 silently unpack or rewrite the profile's STFS container.

PROJECT_STATUS.md

@@ -61,6 +61,11 @@ backup-management, and source-attributed knowledge application.
   recoverable deduplication, storage audits, original Xbox discovery, plugin
   controls, recovery actions, dashboard probes, and accessibility preferences.
 - Additive schema migration 8 and local audit history for every new workspace.
+- Additive schema migration 9 for audited plugin collection and reversible
+  duplicate actions, plus selectable restore controls.
+- Background Community Hub jobs, actionable unified-search navigation, CLI/API
+  parity for search and preservation, FATX geometry reports, and bounded STFS
+  file-table inventory.
 - Windows, Linux, and unsigned Apple Silicon macOS CI packaging.
 
 ## Validation
@@ -93,7 +98,8 @@ backup-management, and source-attributed knowledge application.
   re-signing remain disabled until complete package verification and reliable
   cross-platform signing support are available.
 - FATX and raw-device access remains read-only. Duplicate actions retain a
-  quarantine copy, and console plans require explicit queue confirmation.
+  tracked quarantine copy with validated restoration, and console plans require
+  explicit queue confirmation.
 
 ## Future Work
 

README.md

@@ -120,7 +120,11 @@ checksums, platform notes, and safety guidance.
 - Adds profile dashboards, read-only package workspaces, ownership previews,
   save comparison, played-title history, and validated GPD image export.
 - Manages preferred artwork, multi-disc audits, recoverable duplicate cleanup,
-  read-only FATX inspection, original Xbox discovery, plugins, and recovery.
+  read-only FATX geometry inspection, original Xbox discovery, plugins, and recovery.
+- Runs long Community Hub operations outside the interface thread, opens unified
+  search results in their native workspace, and restores quarantined duplicates.
+- Inventories STFS file tables in read-only package workspaces without claiming
+  unsupported package rebuilding or signing.
 - Stores high-contrast, large-text, reduced-motion, and keyboard-hint settings.
 
 See [COMMUNITY_HUB.md](COMMUNITY_HUB.md) for all twenty capabilities and their
@@ -323,6 +327,17 @@ python main.py --match-file game.iso
 
 # Capture a read-only console inventory
 python main.py --ftp-host 192.168.1.50 --ftp-snapshot /Hdd1
+
+# Search every local knowledge domain
+python main.py --search-all "Hitman"
+
+# Inspect FATX geometry or an Xbox 360 USB container without writing it
+python main.py --audit-storage E:\drive.img
+
+# Preview duplicate recovery actions, then apply or restore one explicitly
+python main.py --dedup-preview D:\XboxArchive
+python main.py --dedup-apply 42 --dedup-mode quarantine
+python main.py --dedup-restore 42
 ```
 
 ## Optional REST API

api.py

@@ -13,7 +13,7 @@
 from flask import Flask, jsonify, request
 from flask_cors import CORS
 
-from app_paths import EXPORTS_DIR
+from app_paths import DATABASE_PATH, EXPORTS_DIR, PLUGINS_DIR
 from app_version import DISPLAY_VERSION
 
 if TYPE_CHECKING:
@@ -134,6 +134,81 @@ def search():
                 )
             )
 
+        @self.app.get("/api/community/search")
+        def community_search():
+            from unified_search import UnifiedSearchService
+
+            query = request.args.get("q", "")[:200]
+            categories = tuple(
+                value.strip() for value in request.args.getlist("category") if value.strip()
+            )
+            limit = request.args.get("limit", default=100, type=int)
+            if len(query.strip()) < 2:
+                return jsonify({"error": "q must contain at least two characters"}), 400
+            if limit is None or limit < 1 or limit > 500:
+                return jsonify({"error": "limit must be between 1 and 500"}), 400
+            return self._execute(lambda: self._search_response(
+                UnifiedSearchService(self._database_path()).search(
+                    query, categories=categories, limit=limit
+                )
+            ))
+
+        @self.app.get("/api/preservation/dedup/actions")
+        def dedup_actions():
+            from community_services import PreservationPlanningService
+
+            plan_id = request.args.get("plan_id", type=int)
+            return self._execute(lambda: {
+                "actions": PreservationPlanningService(self._database_path()).list_dedup_actions(
+                    plan_id
+                )
+            })
+
+        @self.app.post("/api/preservation/dedup/preview")
+        def dedup_preview():
+            from community_services import PreservationPlanningService
+
+            payload = request.get_json(silent=True)
+            if not isinstance(payload, dict) or not isinstance(payload.get("root"), str):
+                return jsonify({"error": "A root directory is required"}), 400
+            return self._execute(
+                lambda: PreservationPlanningService(self._database_path()).create_dedup_plan(
+                    payload["root"]
+                )
+            )
+
+        @self.app.post("/api/preservation/dedup/<int:action_id>/apply")
+        def dedup_apply(action_id: int):
+            from community_services import PreservationPlanningService
+
+            payload = request.get_json(silent=True) or {}
+            mode = payload.get("mode", "quarantine") if isinstance(payload, dict) else ""
+            if mode not in {"quarantine", "hardlink"}:
+                return jsonify({"error": "mode must be quarantine or hardlink"}), 400
+            return self._execute(
+                lambda: PreservationPlanningService(self._database_path()).apply_dedup_action(
+                    action_id, mode
+                )
+            )
+
+        @self.app.post("/api/preservation/dedup/<int:action_id>/restore")
+        def dedup_restore(action_id: int):
+            from community_services import PreservationPlanningService
+
+            return self._execute(
+                lambda: PreservationPlanningService(self._database_path()).restore_dedup_action(
+                    action_id
+                )
+            )
+
+        @self.app.get("/api/plugins")
+        def plugins():
+            from community_services import PluginControlService
+
+            return self._execute(lambda: {
+                "plugins": PluginControlService(self._database_path()).discover(PLUGINS_DIR)
+            })
+
         @self.app.post("/api/metadata/<titleid>")
         def collect_metadata(titleid: str):
             normalized = self._titleid_or_error(titleid)
@@ -277,6 +352,11 @@ def _require_scraper(self) -> "UnityScraper":
             raise RuntimeError("Scraper not initialized")
         return self.scraper
 
+    def _database_path(self) -> Path:
+        if self.scraper is None:
+            return DATABASE_PATH
+        return Path(getattr(self.scraper.db, "db_path", DATABASE_PATH))
+
     def _titleid_or_error(self, titleid: str):
         if self.scraper is None:
             return jsonify({"error": "Scraper not initialized"}), 400

app_paths.py

@@ -146,6 +146,7 @@ def xdg_path(variable: str, fallback: Path) -> Path:
 EXPORTS_DIR = _PATHS.exports
 DIAGNOSTICS_DIR = _PATHS.diagnostics
 PROFILE_BACKUPS_DIR = DATA_DIR / "profile_backups"
+PLUGINS_DIR = DATA_DIR / "plugins"
 
 DATABASE_PATH = DATA_DIR / "unityscraper.db"
 CONFIG_PATH = CONFIG_DIR / "config.json"
@@ -172,6 +173,7 @@ def ensure_app_dirs() -> None:
         EXPORTS_DIR,
         DIAGNOSTICS_DIR,
         PROFILE_BACKUPS_DIR,
+        PLUGINS_DIR,
     ):
         path.mkdir(parents=True, exist_ok=True)