Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:
shell: pwsh
run: |
New-Item -ItemType Directory -Path package | Out-Null
Copy-Item dist\UnityScraper.exe, README.md, CHANGELOG.md, LICENSE, DOCS_INDEX.md, BACKUP_MANAGER.md, COLLECTION_INTELLIGENCE.md, COMMUNITY_HUB.md, CONSOLE_SYNC.md, KNOWLEDGE_SOURCES.md, LINUX.md, PLUGIN_API.md, PROFILE_INTELLIGENCE.md, PROFILES_AND_SAVES.md, SECURITY.md, THIRD_PARTY_NOTICES.md package\
Copy-Item dist\UnityScraper.exe, README.md, CHANGELOG.md, LICENSE, DOCS_INDEX.md, BACKUP_MANAGER.md, COLLECTION_INTELLIGENCE.md, COMMUNITY_HUB.md, CONSOLE_SYNC.md, KNOWLEDGE_SOURCES.md, LINUX.md, PLUGIN_API.md, PROFILE_INTELLIGENCE.md, PROFILES_AND_SAVES.md, RELEASE_TOOLKIT.md, SECURITY.md, THIRD_PARTY_NOTICES.md package\
Compress-Archive -Path package\* -DestinationPath UnityScraper-Windows-x64.zip
$hash = (Get-FileHash UnityScraper-Windows-x64.zip -Algorithm SHA256).Hash.ToLower()
"$hash *UnityScraper-Windows-x64.zip" | Set-Content UnityScraper-Windows-x64.zip.sha256
Expand Down
13 changes: 13 additions & 0 deletions API.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ X-API-Key: <token>
The health endpoint does not require authentication. All other endpoints do
when a token is configured.

For separate automation clients, `UNITYSCRAPER_API_TOKENS` accepts a JSON object
mapping tokens to `read`, `write`, or `transfer` scopes. A legacy single token
has all scopes. Requests are limited per client to 120 per minute by default.

## Endpoints

| Method | Path | Purpose |
Expand All @@ -51,6 +55,13 @@ when a token is configured.
| `POST` | `/api/preservation/dedup/<id>/apply` | Apply `quarantine` or `hardlink` mode |
| `POST` | `/api/preservation/dedup/<id>/restore` | Revalidate and restore a quarantined original |
| `GET` | `/api/plugins` | List managed plugins and checksum trust state |
| `GET` | `/api/library/audit` | Find incomplete names, publishers, covers, updates, and MediaIDs |
| `POST` | `/api/metadata-snapshots/export` | Export a non-personal `.usmeta` snapshot |
| `POST` | `/api/metadata-snapshots/import` | Merge a validated `.usmeta` snapshot |
| `POST` | `/api/packages/extract` | Extract supported STFS files read-only |
| `POST` | `/api/reports/preservation` | Create a privacy-conscious HTML report |
| `GET` | `/api/hardware` | List local console hardware notes |
| `POST` | `/api/hardware` | Add a local console hardware record |
| `POST` | `/api/metadata/<TitleID>` | Collect metadata |
| `POST` | `/api/download/<TitleID>` | Process downloads |
| `GET` | `/api/statistics` | Library statistics |
Expand Down Expand Up @@ -106,3 +117,5 @@ Invoke-RestMethod `
- Browser CORS is restricted to localhost origins unless the API is embedded
programmatically with an explicit origin list.
- Responses disable caching and include basic content and frame protections.
- Tokens can be separated by read, write, and transfer scope, and each client
has a bounded request window.
14 changes: 14 additions & 0 deletions BACKUP_MANAGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,17 @@ and atomically renamed. Existing files are skipped by default.
**Import ZIP** applies the same checks to packages in a user-supplied ZIP.
Absolute paths, parent traversal, and archive symlinks are rejected.

## Read-only STFS extraction

The Community Hub package workspace can inventory and extract files stored in
consecutive STFS blocks. Extraction rejects unsafe paths, limits total output,
never replaces existing files, stages each output atomically, and creates a
manifest with source and extracted-file SHA-256 values. Fragmented files are
reported and skipped rather than reconstructed without verified block-chain and
hash-tree support.

The parser follows the public [Free60 STFS format reference](https://free60.org/System-Software/Formats/STFS/).

## Export and verification

**Export Selected** copies an inventoried title to a separate archive folder
Expand Down Expand Up @@ -103,6 +114,9 @@ python main.py --verify-backups E:\ --backup-report health.json

# Upload a package to a console FTP server
python main.py --ftp-upload game.live --ftp-host 192.168.1.50

# Read-only extraction of supported STFS files
python main.py --extract-stfs save.con --extract-destination extracted
```

## Inspiration and licensing
Expand Down
18 changes: 17 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,22 @@ Notable changes to UnityScraper are documented here. The project follows

### Added

- Additive schema migration 10 for metadata snapshot runs, library intelligence,
preservation reports, correction packages, hardware records, and package extractions.
- Visual Studio 2010-inspired dark desktop theme shared by the modern and legacy
interfaces, including compact tool chrome, classic menus, dense tabs, and blue focus states.
- Read-only extraction of consecutive STFS files with traversal protection,
bounded output, atomic publication, hashes, and an extraction manifest.
- Portable `.usmeta` metadata snapshots that exclude profile and filesystem data
and merge source-attributed facts without replacing better library metadata.
- Library attention audits, privacy-conscious HTML preservation reports,
community correction exports, and console hardware inventory records.
- Xenia/Xenia Canary discovery and direct argument-vector game launching.
- Scoped API tokens, fixed-window request limits, and API/CLI parity for the new toolkit.
- Bounded JSON community language packs with English fallback.
- Timeout-controlled child-process execution for enabled metadata plugins, with
POSIX CPU, memory, and output limits where the operating system supports them.

- Additive schema migration 9 for plugin collection audits and reversible
duplicate recovery records.
- Runtime plugin loading with approved-checksum enforcement, bounded results,
Expand Down Expand Up @@ -93,7 +109,7 @@ Notable changes to UnityScraper are documented here. The project follows

### Changed

- Version advanced to `1.1.0-beta.1`.
- Version advanced to `1.2.0-beta.1`.
- CI now runs the Python suite on Windows, Linux, and macOS.
- Cached XboxUnity titles now resolve immediately in library lists and details,
enrich matching rows page by page, and recover after interrupted refreshes.
Expand Down
14 changes: 13 additions & 1 deletion COMMUNITY_HUB.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The Community Hub brings the wider Xbox and preservation workflows into one
source-attributed, offline-capable workspace. Open it from the desktop sidebar
or press `Ctrl+K` to focus unified search.

## Twenty Integrated Capabilities
## Integrated Capabilities

1. Unified local search across games, identifiers, wiki knowledge, profiles,
saves, achievements, files, and structured reference records.
Expand Down Expand Up @@ -40,6 +40,13 @@ or press `Ctrl+K` to focus unified search.
20. Cross-platform accessibility and packaging: scalable text, high contrast,
reduced-motion preferences, keyboard hints, and Windows, Linux, and macOS
build paths.
21. Read-only extraction of supported consecutive STFS files with hashes and a
portable extraction manifest.
22. Non-personal `.usmeta` snapshot export/import for offline title and knowledge data.
23. Library attention audits and privacy-conscious preservation reports.
24. Reviewed correction-package exports and local hardware inventory records.
25. Direct Xenia/Xenia Canary discovery and game launch controls alongside the
existing snapshot-first save migration workflow.

## Safety Rules

Expand Down Expand Up @@ -71,6 +78,11 @@ plugins run during normal metadata collection only while their approved entrypoi
checksum still matches. Long Community Hub operations run in a background worker;
search results can be opened with Enter or a double-click.

Migration 10 records metadata snapshot operations, library audits, preservation
reports, correction exports, hardware notes, and package extractions. Enabled
plugins execute in a timeout-controlled child process; this improves crash and
resource isolation but is not an operating-system permission sandbox.

Imported ConsoleMods, XenonLibrary, Free60, Redump, and No-Intro information
continues to retain source, revision, citation, licensing, and conflict data.
Redump and No-Intro DAT files remain user-supplied; copyrighted game content is
Expand Down
2 changes: 2 additions & 0 deletions DOCS_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
and signing limitations
- [Community Hub](COMMUNITY_HUB.md) - unified search, profile/package tools,
guided console plans, preservation, plugins, recovery, and safety boundaries
- [Release Toolkit](RELEASE_TOOLKIT.md) - portable metadata snapshots, library
audits, preservation reports, corrections, and hardware inventory
- [Project Status](PROJECT_STATUS.md) - completed work, boundaries, and roadmap
- [Changelog](CHANGELOG.md) - release history

Expand Down
11 changes: 2 additions & 9 deletions GUI.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
from main import UnityScraper, Config
from i18n import init_translator, get_translator, t
from updater import VersionChecker
from ui_theme import apply_vs2010_theme
from queue_manager import DownloadQueue
except ImportError as e:
print(f"Error: Missing required module: {e}")
Expand Down Expand Up @@ -89,15 +90,7 @@ def set_window_icon(self):

def setup_styles(self):
"""Configure ttk styles"""
style = ttk.Style()
style.theme_use('clam')

# Configure colors
style.configure('TFrame', background='#f0f0f0')
style.configure('Title.TLabel', font=('Arial', 16, 'bold'), background='#f0f0f0')
style.configure('Subtitle.TLabel', font=('Arial', 10), background='#f0f0f0')
style.configure('Success.TLabel', foreground='green', background='#f0f0f0')
style.configure('Error.TLabel', foreground='red', background='#f0f0f0')
apply_vs2010_theme(self.root)

def create_widgets(self):
"""Create all GUI widgets"""
Expand Down
6 changes: 5 additions & 1 deletion PLUGIN_API.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,12 @@ to 2 MiB, cover/update counts are bounded, failures are isolated, and every run
is audited in SQLite. Known title and publisher values are never replaced by a
plugin fallback.

Enabled collectors run in a child process with a 30-second timeout and a bounded
result file. POSIX builds also request CPU, address-space, and output-file limits.
A timeout or worker crash does not terminate the desktop collection job.

Requested access is disclosure metadata, not an operating-system sandbox.
Plugin code executes with the user's account permissions, so only enable source
Plugin code still executes with the user's account permissions, so only enable source
and publishers you trust. Editing an enabled entrypoint automatically prevents
it from loading until it is reviewed and enabled again.

Expand Down
12 changes: 10 additions & 2 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ backup-management, and source-attributed knowledge application.
- Additive schema migration 8 and local audit history for every new workspace.
- Additive schema migration 9 for audited plugin collection and reversible
duplicate actions, plus selectable restore controls.
- Additive schema migration 10 for metadata snapshots, library audits,
preservation reports, correction exports, hardware records, and package extraction.
- Visual Studio 2010-inspired shared desktop theme, classic menus, scoped API
tokens, out-of-process plugin execution, and bounded community language packs.
- Read-only consecutive STFS extraction, direct Xenia launch controls, and
portable non-personal metadata distribution.
- Background Community Hub jobs, actionable unified-search navigation, CLI/API
parity for search and preservation, FATX geometry reports, and bounded STFS
file-table inventory.
Expand Down Expand Up @@ -107,5 +113,7 @@ backup-management, and source-attributed knowledge application.
of real dashboard FTP servers.
- Add notarization and universal binaries after macOS signing infrastructure is
available.
- Consider package mutation only after complete STFS extraction, rehashing,
signing, verification, and automatic recovery have independent test vectors.
- Complete fragmented STFS block-chain traversal and hash-tree verification
against independent real-package test vectors before considering mutation.
- Consider package mutation only after complete extraction, rehashing, signing,
verification, and automatic recovery have independent test vectors.
34 changes: 33 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ service exposes. UnityScraper does not silently substitute HTTPS URLs.
- Reads extracted XDBF/GPD achievement databases without modifying them.
- Compares save hashes and imported achievement state across two profiles.
- Previews Xenia save mappings and creates a verified snapshot before migration.
- Discovers Xenia or Xenia Canary beside a selected content folder and launches
user-selected games directly without constructing a shell command.

The profile/package model is informed by Dalavin, also known as DJ
SkunkieButt, and the GPLv3 X360 library and Le Fluffie source. See
Expand Down Expand Up @@ -125,7 +127,19 @@ checksums, platform notes, and safety guidance.
search results in their native workspace, and restores quarantined duplicates.
- Inventories STFS file tables in read-only package workspaces without claiming
unsupported package rebuilding or signing.
- Extracts supported consecutive STFS files read-only with path validation,
output limits, atomic files, hashes, and a manifest.
- Stores high-contrast, large-text, reduced-motion, and keyboard-hint settings.
- Exports and imports non-personal metadata snapshots, audits incomplete library
metadata, creates preservation reports and correction packages, and records
local hardware notes.

### Desktop Style

The desktop uses a Visual Studio 2010-inspired dark tool aesthetic: compact
square controls, charcoal chrome, blue selection and focus states, classic menu
commands, dense tables, and status bars. The primary and Advanced Tools windows
share the same theme, with high-contrast and large-text alternatives retained.

See [COMMUNITY_HUB.md](COMMUNITY_HUB.md) for all twenty capabilities and their
safety boundaries.
Expand Down Expand Up @@ -207,7 +221,7 @@ Linux source setup:
| External Tools | Run XeXTool and other user-supplied command-line utilities |
| Collections | Identify storage, compare Title Updates, verify preservation data, and preview repairs |
| Knowledge | Search sources, facts, citations, imports, and conflicts |
| Community Hub | Unified search, console plans, profiles, preservation, plugins, recovery, and compatibility |
| Community Hub | Unified search, console plans, profiles, preservation, plugins, recovery, compatibility, and release toolkit |
| Archive Health | Find missing or inconsistent downloaded files |
| Settings | Configure storage and scraper behavior |
| Help & About | Version, diagnostics, storage, and advanced tools |
Expand Down Expand Up @@ -338,6 +352,17 @@ python main.py --audit-storage E:\drive.img
python main.py --dedup-preview D:\XboxArchive
python main.py --dedup-apply 42 --dedup-mode quarantine
python main.py --dedup-restore 42

# Share or consume a non-personal offline metadata snapshot
python main.py --metadata-snapshot-export xbox360.usmeta
python main.py --metadata-snapshot-import xbox360.usmeta

# Audit the library and produce a privacy-conscious report
python main.py --library-audit
python main.py --preservation-report preservation.html

# Extract supported files without changing the STFS source package
python main.py --extract-stfs save.con --extract-destination extracted
```

## Optional REST API
Expand All @@ -356,9 +381,16 @@ python main.py --api-mode --api-host 0.0.0.0
```

Clients send the token as `Authorization: Bearer <token>` or `X-API-Key`.
Multiple scoped tokens can be provided through `UNITYSCRAPER_API_TOKENS` as a
JSON object whose values contain `read`, `write`, or `transfer`. Requests are
rate-limited per client.
Remote HTTP is not encrypted; place it behind a trusted local reverse proxy or
use it only on an isolated network. See [API.md](API.md).

See [RELEASE_TOOLKIT.md](RELEASE_TOOLKIT.md) for metadata snapshot privacy,
library intelligence, reports, corrections, hardware records, and STFS
extraction boundaries.

## Build and Test

Install development dependencies:
Expand Down
51 changes: 51 additions & 0 deletions RELEASE_TOOLKIT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Release Toolkit

The **Community Hub > Toolkit** page groups portable metadata, collection
attention, reporting, correction, and hardware workflows.

## Metadata snapshots

`.usmeta` files are compressed JSON snapshots containing the XboxUnity title
catalog and normalized knowledge entities, identifiers, facts, citations, source
names, and licenses. They explicitly exclude profiles, saves, download history,
credentials, local paths, console identifiers, and game content.

Imports validate the archive shape, schema, expanded size, TitleIDs, and fact
records. Existing source-attributed claims can coexist; the normal preference and
conflict rules choose display values.

## Library intelligence

The audit highlights unknown game names or publishers, available but unarchived
covers and updates, and updates whose MediaID compatibility still needs evidence.
It records only a summary in SQLite and does not automatically download content.

## Preservation reports

The HTML report summarizes library attention and knowledge-source provenance.
Personal profile identifiers and filesystem paths are deliberately excluded.

## Correction packages

Correction exports contain reviewed local metadata overrides. They are suitable
for manual community review and do not publish or upload anything automatically.

## Hardware records

Users can keep local notes for motherboard, DVD drive, NAND, dashboard, and
console type. Serial numbers and keys are not requested.

## Command line

```powershell
python main.py --metadata-snapshot-export xbox360.usmeta
python main.py --metadata-snapshot-import xbox360.usmeta
python main.py --library-audit
python main.py --preservation-report preservation.html
python main.py --corrections-export corrections.json
python main.py --extract-stfs save.con --extract-destination extracted
```

STFS extraction currently supports files stored in consecutive blocks. A
fragmented file is reported and skipped until block-chain traversal and hash-tree
verification have independent real-package test vectors.
3 changes: 3 additions & 0 deletions UnityScraper.spec
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ a = Analysis(
'structured_knowledge',
'unified_search',
'plugins',
'plugin_worker',
'roadmap_services',
'ui_theme',
'gpd_parser',
'profile_gui',
'profile_intelligence',
Expand Down
14 changes: 11 additions & 3 deletions VERSION
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"version": "1.1.0b1",
"version": "1.2.0b1",
"name": "Unified Xbox 360 Collection and Preservation Manager",
"changes": [
"Unified Xbox 360 knowledge browser",
Expand All @@ -20,8 +20,16 @@
"Unified Community Hub across games, knowledge, profiles, preservation, and recovery",
"Guided console sync plans and recoverable duplicate cleanup",
"Played-title history and validated embedded GPD image export",
"Windows, Linux, and unsigned Apple Silicon macOS build paths"
"Windows, Linux, and unsigned Apple Silicon macOS build paths",
"Visual Studio 2010-inspired desktop theme and classic tool chrome",
"Read-only consecutive STFS extraction with manifests",
"Portable source-attributed metadata snapshots",
"Library intelligence, preservation reports, and correction packages",
"Out-of-process metadata plugin execution with timeouts",
"Scoped and rate-limited local REST API",
"Xenia discovery and direct game launching",
"Community language-pack loading and hardware inventory"
],
"download_url": "https://github.com/TrapEmAll/UnityScraper/releases",
"release_date": "2026-07-23"
"release_date": "2026-08-01"
}
Loading
Loading