feat: deliver unified Xbox 360 collection platform

.github/workflows/release.yml

@@ -7,6 +7,8 @@ on:
 
 permissions:
   contents: write
+  id-token: write
+  attestations: write
 
 jobs:
   validate:
@@ -30,6 +32,14 @@ jobs:
           python tests.py
           python integration_tests.py
 
+      - name: Generate software bill of materials
+        run: python scripts/generate_sbom.py
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: release-sbom
+          path: UnityScraper-SBOM.cdx.json
+
   windows:
     runs-on: windows-latest
     needs: validate
@@ -52,7 +62,7 @@ jobs:
         shell: pwsh
         run: |
           New-Item -ItemType Directory -Path package | Out-Null
-          Copy-Item dist\UnityScraper.exe, README.md, CHANGELOG.md, LICENSE package\
+          Copy-Item dist\UnityScraper.exe, README.md, CHANGELOG.md, LICENSE, DOCS_INDEX.md, BACKUP_MANAGER.md, COLLECTION_INTELLIGENCE.md, CONSOLE_SYNC.md, KNOWLEDGE_SOURCES.md, LINUX.md, PLUGIN_API.md package\
           Compress-Archive -Path package\* -DestinationPath UnityScraper-Windows-x64.zip
           $hash = (Get-FileHash UnityScraper-Windows-x64.zip -Algorithm SHA256).Hash.ToLower()
           "$hash *UnityScraper-Windows-x64.zip" | Set-Content UnityScraper-Windows-x64.zip.sha256
@@ -103,7 +113,7 @@ jobs:
 
   publish:
     runs-on: ubuntu-latest
-    needs: [windows, linux]
+    needs: [validate, windows, linux]
 
     steps:
       - uses: actions/download-artifact@v4
@@ -112,6 +122,11 @@ jobs:
           path: release
           merge-multiple: true
 
+      - name: Attest release artifacts
+        uses: actions/attest-build-provenance@v2
+        with:
+          subject-path: "release/*"
+
       - name: Publish GitHub release
         env:
           GH_TOKEN: ${{ github.token }}

.gitignore

@@ -44,6 +44,7 @@ build/
 dist/
 *.spec.bak
 *.zip
+UnityScraper-SBOM.cdx.json
 
 # IDE and operating system
 .vs/

ARCHITECTURE.md

@@ -19,6 +19,8 @@ for CLI and optional REST automation. SQLite is the durable local store.
 - `modern_gui.py` builds the dark navigation shell and core pages.
 - `knowledge_gui.py` renders knowledge search, imports, sources, and conflicts.
 - `backup_gui.py` renders inventory, package, FTP, and converter workflows.
+- `collection_gui.py` renders collection analysis, matching, reports, and
+  repair previews.
 - `setup_wizard.py` handles first-run storage setup.
 
 GUI operations that can block are dispatched to background threads and return
@@ -31,12 +33,18 @@ results to Tk's main loop.
 - `backup_service.py` coordinates scans, installs, exports, verification, FTP,
   and audit records.
 - `knowledge_sync.py` exposes complete source-import workflows to CLI and GUI.
+- `collection_intelligence.py` coordinates snapshots, exact MediaID matching,
+  health, preservation matching, repair previews, and offline exports.
+- `console_sync.py` owns durable transfer jobs, resumable FTP, remote
+  snapshots, and PC/console comparisons.
+- `database_migrations.py` applies additive schema versions and provides
+  consistent SQLite backup/restore helpers.
 
 ### Domain and Adapters
 
 - `main.py` contains the XboxUnity collector and shared configuration.
 - `resume.py` handles partial download state and verification.
-- `backup_manager.py` parses public STFS/XBE fields and performs safe
+- `backup_manager.py` parses public STFS/XBE/XEX fields and performs safe
   filesystem or FTP operations.
 - `knowledge_base.py` defines normalized knowledge records and resolution.
 - `consolemods_adapters.py`, `wiki_adapters.py`, and `dat_adapters.py` parse

BACKUP_MANAGER.md

@@ -57,13 +57,15 @@ structural health check, not a substitute for Redump or No-Intro verification.
 ## FTP console transfer
 
 The Console Transfer tab supports a user-configured FTP server such as Aurora.
-It uses one connection per operation and uploads to a temporary remote name
-before renaming the completed file. The default content root is:
+It persists resumable upload and download jobs, retains partial data, recovers
+interrupted jobs, limits bandwidth, verifies final sizes, and can capture a
+read-only console inventory. The default content root is:
 
 `/Hdd1/Content/0000000000000000`
 
 FTP passwords remain in memory and are deliberately omitted from SQLite.
 Traditional FTP is unencrypted, so use it only on a trusted local network.
+See [CONSOLE_SYNC.md](CONSOLE_SYNC.md) for queue and comparison behavior.
 
 ## ISO conversion
 

CHANGELOG.md

@@ -7,6 +7,20 @@ Notable changes to UnityScraper are documented here. The project follows
 
 ### Added
 
+- Versioned additive migrations for collection snapshots, preservation
+  matches, repair plans, console inventories, resumable jobs, overrides, and
+  recovery state.
+- XEX2 identity parsing, mounted-storage discovery, and immutable read-only
+  Aurora database import.
+- Exact MediaID Title Update comparison, collection health scoring, and
+  non-destructive repair-plan previews.
+- Redump/No-Intro file matching, offline HTML reports, manifests, and
+  provenance exports.
+- Persistent resumable FTP upload/download jobs, bandwidth limits, transfer
+  verification, and read-only console snapshots.
+- Database backups, plugin API v1, Italian and Portuguese translation
+  foundations, UI scaling, and keyboard navigation.
+- CycloneDX SBOM generation and GitHub build-provenance attestations.
 - Native Linux desktop support with XDG data, configuration, cache, and state
   directories.
 - Linux x86_64 release bundle with user-level installation, application-menu
@@ -21,6 +35,10 @@ Notable changes to UnityScraper are documented here. The project follows
 
 ### Changed
 
+- Version advanced to `1.0.0-beta.1`.
+- Download queues now use atomic writes and recover interrupted items.
+- Update checks select a platform artifact and require its SHA-256 sidecar
+  before staging it.
 - PyInstaller configuration and desktop entry point now support Windows and
   Linux from the same source tree.
 - Knowledge source snapshots use the platform cache directory.

COLLECTION_INTELLIGENCE.md

@@ -0,0 +1,51 @@
+# Collection Intelligence
+
+UnityScraper 1.0 beta inventories Xbox 360 collections, retains snapshots in
+SQLite, compares TitleIDs and MediaIDs with catalogued Title Updates, and
+produces preservation-oriented reports.
+
+## Sources and Identification
+
+The **Collections** workspace accepts content trees, extracted `Games`
+directories, mounted USB/archive folders, and user-selected Aurora SQLite
+databases. Aurora databases are opened in immutable read-only mode.
+
+Mounted-storage discovery checks Windows drive roots, Linux `/media`,
+`/run/media`, and `/mnt`, and macOS `/Volumes`. XEX2 parsing reads the public
+header fields for TitleID, MediaID, versions, module flags, and disc position.
+It does not decrypt or extract executable content.
+
+Title Update status is conservative:
+
+- `compatible`: TitleID and MediaID both match
+- `media-id-required`: updates exist but the collection MediaID is unknown
+- `incompatible`: the TitleID exists but no MediaID matches
+- `none`: no update is catalogued
+- `unknown`: the TitleID could not be identified
+
+## Preservation and Repair
+
+Files can be hashed with CRC32, MD5, SHA-1, and SHA-256 and matched against
+user-imported Redump or No-Intro DAT metadata. UnityScraper stores metadata,
+hashes, and matches, never game content.
+
+Exports include JSON manifests, offline HTML collection reports, and fact
+provenance with sources and citations. Metadata overrides are stored
+separately from imported facts.
+
+A repair plan is a preview in `repair_plans` and `repair_actions`. Creating
+one does not delete, replace, download, or transfer anything.
+
+## Command Line
+
+```powershell
+python main.py --analyze-collection D:\Xbox360 `
+  --collection-manifest collection.json `
+  --collection-html collection.html `
+  --create-repair-plan
+
+python main.py --aurora-db content.db --collection-manifest aurora.json
+python main.py --match-file game.iso
+python main.py --export-provenance provenance.json
+python main.py --backup-database unityscraper-backup.db
+```

CONSOLE_SYNC.md

@@ -0,0 +1,34 @@
+# Console Sync
+
+Console Sync provides a persistent FTP queue for consoles and dashboards
+whose FTP server the user explicitly configures.
+
+## Behavior
+
+- Upload and download jobs survive restarts in SQLite.
+- Interrupted transfers return as paused.
+- `.partial` files are retained for resume.
+- FTP `REST` is used when the server supports ranged transfer.
+- Uploads are published by renaming the completed partial file.
+- Final sizes are verified; downloads can also require a SHA-256.
+- Each job can have a bytes-per-second bandwidth limit.
+- Passwords remain in memory and are never stored.
+
+Some console FTP servers do not implement ranged uploads correctly. Those
+servers may reject resume; the job retains its state and reports the error.
+
+**Snapshot Console** recursively reads remote metadata without changing
+files. A snapshot can be compared with a local directory to find files only
+on the PC, only on the console, different-sized files, and matching files.
+Discovery has a default 100,000-entry safety limit.
+
+```powershell
+python main.py --ftp-host 192.168.1.50 --ftp-user xbox --ftp-snapshot /Hdd1
+
+python main.py --ftp-host 192.168.1.50 --ftp-user xbox `
+  --ftp-download /Hdd1/Content/file `
+  --ftp-local-path D:\Xbox360\file `
+  --ftp-bandwidth-limit 1048576
+```
+
+Standard FTP is unencrypted. Use it only on a trusted local network.

DOCS_INDEX.md

@@ -5,6 +5,10 @@
 - [README](README.md) - product overview, installation, core workflows, and CLI
 - [Backup Manager](BACKUP_MANAGER.md) - layouts, installation, exports, FTP,
   verification, and external conversion
+- [Collection Intelligence](COLLECTION_INTELLIGENCE.md) - storage discovery,
+  XEX identity, Title Update compatibility, preservation, and repair previews
+- [Console Sync](CONSOLE_SYNC.md) - persistent transfers, resume, snapshots,
+  comparison, and verification
 - [Knowledge Sources](KNOWLEDGE_SOURCES.md) - imports, provenance, caching, and
   source licensing
 - [Advanced Features](ADVANCED_FEATURES.md) - rate limits, resume, diagnostics,
@@ -19,6 +23,7 @@
 
 - [Architecture](ARCHITECTURE.md) - modules, layers, schemas, data flows, and
   packaging
+- [Plugin API v1](PLUGIN_API.md) - manifests, opt-in loading, and compatibility
 - [Contributing](CONTRIBUTING.md) - environment, tests, PR expectations, and
   adapter rules
 - [Security](SECURITY.md) - private reporting and operational boundaries

PLUGIN_API.md

@@ -0,0 +1,29 @@
+# Plugin API v1
+
+Metadata collectors use a manifest-based, opt-in API. Disabled plugin code is
+discovered but never imported.
+
+```text
+plugins/
+  example/
+    plugin.json
+    collector.py
+```
+
+```json
+{
+  "id": "org.example.collector",
+  "name": "Example Collector",
+  "version": "1.0.0",
+  "api_version": 1,
+  "entrypoint": "collector.py",
+  "permissions": ["network"]
+}
+```
+
+The entrypoint exports a `MetadataCollectorPlugin` subclass. The caller must
+pass the plugin ID in `enabled_plugins` before code is loaded. Permissions
+are disclosure metadata, not an operating-system sandbox, so only enable
+plugins whose source and publisher you trust.
+
+Root-level legacy Python plugins load only with `allow_legacy=True`.

PROJECT_STATUS.md

@@ -24,18 +24,23 @@ backup-management, and source-attributed knowledge application.
   dynamically loaded knowledge adapters.
 - Local inventory for Xbox content roots, USB drives, archive folders, Games on
   Demand, XBLA, DLC, title updates, and extracted Xbox/Xbox 360 games.
-- Public STFS and XBE header inspection for TitleID, MediaID, content type, disc,
-  and display metadata.
+- Public STFS, XBE, and XEX2 header inspection for TitleID, MediaID, versions,
+  content type, disc, and display metadata.
+- Mounted-storage discovery, read-only Aurora database import, exact MediaID
+  Title Update comparison, collection health scoring, and repair previews.
+- Preservation hash matching against imported DAT metadata, offline HTML
+  reports, manifests, provenance exports, and separate local overrides.
 - Safe bare-package and ZIP installation with path validation, `.partial`
   staging, SHA-256 verification, and atomic final placement.
 - Verified archive export with portable JSON manifests and per-file checksums.
-- Aurora-oriented FTP package upload with one connection per operation,
-  temporary remote names, and no stored passwords.
+- Persistent resumable FTP upload/download jobs, partial-file recovery,
+  bandwidth limits, verified final sizes, read-only console snapshots, and no
+  stored passwords.
 - Explicit external converter integration for user-owned ISO images.
 - Local-by-default REST API with token-required remote binding, restricted
   browser origins, validated settings, and current version reporting.
-- Cross-platform CI, Windows packaging checks, tagged release archives, and
-  SHA-256 release checksums.
+- Cross-platform CI, Windows packaging checks, tagged release archives,
+  SHA-256 checksums, CycloneDX SBOMs, and build-provenance attestations.
 - Linux x86_64 packaging, XDG storage, application-menu integration, source
   launch scripts, and release artifacts.
 - Repository contribution, security, architecture, API, and release
@@ -70,8 +75,7 @@ backup-management, and source-attributed knowledge application.
 
 ## Future Work
 
-- Parse additional XEX fields and link scanned file identifiers directly to
-  normalized knowledge entities.
 - Add field-specific source-priority controls and conflict resolution actions.
-- Add optional scheduled knowledge refreshes and offline HTML reports.
-- Add resumable FTP queues and optional Aurora database inventory.
+- Add optional scheduled knowledge refreshes.
+- Validate console resume behavior against a broader matrix of dashboard FTP
+  servers and add opt-in remote hash verification where servers expose it.

README.md

@@ -50,9 +50,20 @@ service exposes. UnityScraper does not silently substitute HTTPS URLs.
   `Content/0000000000000000` trees.
 - Exports selected backups with per-file SHA-256 values and a preservation
   manifest.
-- Uploads packages to a configured Aurora-style FTP server.
+- Queues resumable uploads and downloads to a configured Aurora-style FTP server.
+- Captures read-only console inventories and compares PC and console content.
 - Runs a user-selected external ISO converter without bundling converter code.
 
+### Collection Intelligence and Preservation
+
+- Discovers mounted console, USB, and archive storage.
+- Parses XEX2 identity fields and imports Aurora databases read-only.
+- Compares installed content with catalogued updates using exact MediaIDs.
+- Scores collection health and creates non-destructive repair-plan previews.
+- Matches local hashes against user-imported Redump and No-Intro DAT metadata.
+- Exports preservation manifests, offline HTML reports, and fact provenance.
+- Keeps local metadata overrides separate from source-attributed knowledge.
+
 ## Install
 
 ### Windows Release
@@ -120,6 +131,7 @@ Linux source setup:
 | Add Games | Import or enter TitleIDs |
 | Downloads | Review and manage download activity |
 | Backup Manager | Scan, install, verify, export, convert, and transfer owned content |
+| Collections | Identify storage, compare Title Updates, verify preservation data, and preview repairs |
 | Knowledge | Search sources, facts, citations, imports, and conflicts |
 | Archive Health | Find missing or inconsistent downloaded files |
 | Settings | Configure storage and scraper behavior |
@@ -212,6 +224,24 @@ python main.py --ftp-upload game.live --ftp-host 192.168.1.50
 See [BACKUP_MANAGER.md](BACKUP_MANAGER.md) for layouts, conflict behavior,
 manifests, FTP considerations, and external converter arguments.
 
+### Collections and Console Sync
+
+```powershell
+# Analyze a collection and create offline reports
+python main.py --analyze-collection D:\Xbox360 `
+  --collection-manifest collection.json `
+  --collection-html collection.html
+
+# Read an Aurora database without modifying it
+python main.py --aurora-db content.db --collection-manifest aurora.json
+
+# Match a local file against imported preservation DAT hashes
+python main.py --match-file game.iso
+
+# Capture a read-only console inventory
+python main.py --ftp-host 192.168.1.50 --ftp-snapshot /Hdd1
+```
+
 ## Optional REST API
 
 Start the localhost-only API:
@@ -271,6 +301,9 @@ SHA-256 file.
 - [Linux support](LINUX.md)
 - [Knowledge sources and licensing](KNOWLEDGE_SOURCES.md)
 - [Backup manager](BACKUP_MANAGER.md)
+- [Collection intelligence](COLLECTION_INTELLIGENCE.md)
+- [Console sync](CONSOLE_SYNC.md)
+- [Plugin API v1](PLUGIN_API.md)
 - [REST API](API.md)
 - [Project status](PROJECT_STATUS.md)
 - [Changelog](CHANGELOG.md)