Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ for CLI and optional REST automation. SQLite is the durable local store.
- `modern_gui.py` builds the dark navigation shell and core pages.
- `knowledge_gui.py` renders knowledge search, imports, sources, and conflicts.
- `backup_gui.py` renders inventory, package, FTP, and converter workflows.
- `profile_gui.py` renders privacy-aware profile/save inventory and snapshots.
- `collection_gui.py` renders collection analysis, matching, reports, and
repair previews.
- `setup_wizard.py` handles first-run storage setup.
Expand All @@ -32,6 +33,8 @@ results to Tk's main loop.
- `knowledge_service.py` provides search, source status, imports, and details.
- `backup_service.py` coordinates scans, installs, exports, verification, FTP,
and audit records.
- `profile_manager.py` owns Content-tree discovery, STFS ownership inspection,
profile/save indexing, verified snapshots, and conflict-safe restore.
- `knowledge_sync.py` exposes complete source-import workflows to CLI and GUI.
- `collection_intelligence.py` coordinates snapshots, exact MediaID matching,
health, preservation matching, repair previews, and offline exports.
Expand Down Expand Up @@ -62,6 +65,8 @@ Main schema groups:
- Knowledge: sources, documents, revisions, entities, names, identifiers,
facts, citations, relationships, import runs, and conflicts
- Backups: targets, scans, inventory, and operations
- Profiles: scan runs, profiles, saves, snapshots, snapshot files, and
auditable operations

Schema initialization is idempotent. New migrations should preserve existing
data and be covered by tests.
Expand Down Expand Up @@ -100,6 +105,23 @@ user-selected file or ZIP
Game payloads are not stored in SQLite. Inventory records contain paths,
identifiers, sizes, statuses, and notes.

### Profile Snapshot

```text
user-selected Content tree
-> read-only profile and save discovery
-> STFS ownership/header inspection
-> local inventory and mismatch detection
-> selected profile or saves
-> .partial verified copies
-> atomic snapshot manifest
-> conflict-preserving restore
```

Profile identifiers are masked in the UI by default. Snapshot payloads remain
on disk under the local application data directory; SQLite stores paths,
hashes, identifiers, and operation history.

## Storage

Normal Windows data:
Expand Down Expand Up @@ -134,6 +156,8 @@ works in source and PyInstaller one-file builds.
expanded size.
- Package and export copies use temporary files and verification.
- External converters run only through explicit user configuration.
- Profile scans never modify source content, and restores never overwrite a
different existing file.

See [SECURITY.md](SECURITY.md) for reporting and operational guidance.

Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ Notable changes to UnityScraper are documented here. The project follows

### Added

- Profiles & Saves workspace with read-only Content-tree discovery, masked
profile inventory, STFS ownership metadata, save search, duplicate and
mismatch reporting, verified snapshots, manifests, and conflict-safe restore.
- Additive schema migration 6 for profile scans, profiles, saves, snapshots,
snapshot files, and auditable operations.
- Dalavin / DJ SkunkieButt attribution and preserved GPLv3 provenance for the
X360 library and Le Fluffie technical reference.
- Bundled Windows XeXTool 6.3 integration with automatic discovery, xorloser
attribution, provenance, checksum, and preserved third-party license.
- External Tools workspace with XeXTool inspection presets, custom CLI
Expand Down
2 changes: 2 additions & 0 deletions DOCS_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
- [README](README.md) - product overview, installation, core workflows, and CLI
- [Backup Manager](BACKUP_MANAGER.md) - layouts, installation, exports, FTP,
verification, and external conversion
- [Profiles and Saves](PROFILES_AND_SAVES.md) - profile inventory, save
snapshots, privacy, restore behavior, and Le Fluffie attribution
- [Collection Intelligence](COLLECTION_INTELLIGENCE.md) - storage discovery,
XEX identity, Title Update compatibility, preservation, and repair previews
- [Console Sync](CONSOLE_SYNC.md) - persistent transfers, resume, snapshots,
Expand Down
110 changes: 110 additions & 0 deletions PROFILES_AND_SAVES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Profiles and Saves

The **Profiles & Saves** workspace inventories and protects Xbox 360 profile
data from an extracted `Content` directory. It is designed around read-only
discovery and snapshot-before-change workflows.

## Supported Sources

Choose any of these:

- a `Content` directory
- a folder containing `Content`
- an extracted `Hdd1` folder containing `Hdd1/Content`
- a modern FAT32 Xbox 360 USB drive whose content is directly visible

For FATX-formatted disks or older Xbox 360 USB system drives, mount or extract
the content using a trusted filesystem tool first. UnityScraper does not write
raw disks or mount FATX volumes.

## Inventory

UnityScraper recognizes non-public 16-character profile directories and scans:

```text
Content/
<ProfileID>/
FFFE07D1/00010000/ profile package
<TitleID>/00000001/ saved games
```

For supported STFS packages it records:

- package signature type (`CON`, `LIVE`, or `PIRS`)
- content type
- TitleID and cached game name
- save-game ID
- embedded profile ID
- creator console ID
- device ID
- package display name
- size, modified time, and SHA-256

Package fields are compared with their containing profile and TitleID folders.
Mismatches are reported and never rewritten automatically. Identical hashes are
reported as duplicates.

The UI masks profile identifiers by default. Select **Reveal profile
identifiers** only when needed. Profile and save data stays local and is never
sent to UnityScraper metadata sources.

## Snapshots

**Back Up Profile** copies every file in the selected profile directory.
**Back Up Selected Saves** copies only the selected save packages.

Each snapshot contains:

- the original relative directory layout
- SHA-256 for every file
- source metadata
- an atomic JSON manifest
- database operation history

Files are copied through a temporary `.partial` file, verified, and then
published atomically. Snapshots are stored in the application data directory
under `profile_backups`.

## Restore

Select a complete snapshot and choose **Restore to Folder**. Every snapshot
file is verified before it is copied.

UnityScraper never overwrites a different existing file. Matching files are
skipped; conflicts are restored alongside the existing file with a
`.restored-N` suffix. This lets the user compare both copies before deciding
which one belongs on a console.

Restore currently targets an ordinary folder. Use Backup Manager or your
preferred filesystem tool to transfer verified output to the console or USB
device.

## Current Safety Boundary

This release deliberately does not:

- edit achievements, GPD records, gamertags, or account blocks
- change profile, console, or device ownership fields
- rehash or resign modified CON packages
- authenticate to Xbox Live or Microsoft accounts
- store CPU keys, account credentials, or signing material
- write raw FATX disks

Those operations can make a profile or save unusable when implemented
incorrectly. Future editing and migration support should only ship with
complete package verification, automatic pre-change snapshots, and
well-tested cross-platform signing support.

## Le Fluffie Attribution

The profile/STFS field model is informed by Dalavin, also known as
**DJ SkunkieButt** and **DJ Shepherd**, through the GPLv3 X360 library and Le
Fluffie source archived at
[mtolly/X360](https://github.com/mtolly/X360).

UnityScraper uses a new Python implementation suited to its existing
cross-platform architecture. It does not bundle Le Fluffie's executable,
updater, embedded key resources, account-modification code, or artwork. The
archived GPL text is preserved with the application and the corresponding
source remains linked from About and the third-party notices.

10 changes: 10 additions & 0 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ backup-management, and source-attributed knowledge application.
bandwidth limits, verified final sizes, read-only console snapshots, and no
stored passwords.
- Explicit external converter integration for user-owned ISO images.
- Read-only profile and save discovery with public STFS ownership metadata,
masked identifiers, cached game-name resolution, duplicate and mismatch
reporting, verified snapshots, manifests, and conflict-safe restore.
- Credited GPLv3 technical lineage from Dalavin / DJ SkunkieButt's X360
library and Le Fluffie without bundling its updater, keys, or executable.
- Local-by-default REST API with token-required remote binding, restricted
browser origins, validated settings, and current version reporting.
- Cross-platform CI, Windows packaging checks, tagged release archives,
Expand Down Expand Up @@ -72,10 +77,15 @@ backup-management, and source-attributed knowledge application.
memory and omitted from database records.
- Remote API access requires a token but the built-in server does not provide
TLS. Remote deployments need a trusted network or TLS reverse proxy.
- Profile editing, achievement modification, ownership rewriting, and CON
re-signing remain disabled until complete package verification and reliable
cross-platform signing support are available.

## Future Work

- Add field-specific source-priority controls and conflict resolution actions.
- Add optional scheduled knowledge refreshes.
- Validate console resume behavior against a broader matrix of dashboard FTP
servers and add opt-in remote hash verification where servers expose it.
- Add read-only GPD/achievement views, Xenia save mapping, and audited
profile-migration previews before considering package mutation.
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,24 @@ service exposes. UnityScraper does not silently substitute HTTPS URLs.
- Captures read-only console inventories and compares PC and console content.
- Runs a user-selected external ISO converter without bundling converter code.

### Profiles and Save Data

- Discovers profile folders and save packages in extracted Xbox 360 `Content`
trees.
- Reads public STFS ownership, TitleID, package, and display metadata.
- Associates saves with cached game names and flags profile/TitleID mismatches.
- Detects duplicate save packages by SHA-256.
- Masks profile identifiers by default.
- Creates complete-profile or selected-save snapshots with manifests and
verified atomic copies.
- Restores snapshots without overwriting different existing files.
- Exports portable JSON preservation manifests.

The profile/package model is informed by Dalavin, also known as DJ
SkunkieButt, and the GPLv3 X360 library and Le Fluffie source. See
[PROFILES_AND_SAVES.md](PROFILES_AND_SAVES.md) and
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

### External Tools

- Runs Xbox command-line utilities inside the desktop interface.
Expand Down Expand Up @@ -149,6 +167,7 @@ Linux source setup:
| Add Games | Search cached game names, select TitleIDs, or import lists |
| Downloads | Review and manage download activity |
| Backup Manager | Scan, install, verify, export, convert, and transfer owned content |
| Profiles & Saves | Inventory profiles, back up saves, and restore snapshots |
| External Tools | Run XeXTool and other user-supplied command-line utilities |
| Collections | Identify storage, compare Title Updates, verify preservation data, and preview repairs |
| Knowledge | Search sources, facts, citations, imports, and conflicts |
Expand Down
7 changes: 7 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ filesystem contents in a report.
- ZIP imports reject traversal paths, symlinks, excessive entries, and
unreasonable expanded sizes.
- Package copies use temporary files, verification, and atomic publication.
- Profile and save identifiers are masked in the GUI by default.
- Profile scans are read-only. Snapshot restores preserve different existing
files and write the restored copy alongside them.
- Profiles, saves, gamertags, XUIDs, console IDs, and device IDs are not sent
to metadata sources.
- External converters run only when explicitly configured by the user.
- UnityScraper does not provide game images, firmware, keys, SDK files, or
copy-protection bypass tools.
- UnityScraper does not request Xbox Live credentials or store CPU keys,
account keys, or profile-signing material.
17 changes: 17 additions & 0 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,20 @@ preserved at
UnityScraper's GUI integration is independent code. Users may select a
different lawfully obtained XeXTool build or another command-line utility.

## X360 Library and Le Fluffie

- **Creator:** Dalavin, also known as DJ SkunkieButt and DJ Shepherd
- **Purpose:** Historical Xbox 360 STFS, SVOD, FATX, GPD, profile, and account
research and tooling
- **Archived source:** [mtolly/X360](https://github.com/mtolly/X360)
- **License:** GNU General Public License version 3

UnityScraper's profile and save implementation is new Python code informed by
the public package/profile model and field layout documented in the X360
library and Le Fluffie source. The original GPL text is preserved at
`assets/references/lefluffie/X360-GPL-3.0.txt`.

UnityScraper does not include Le Fluffie's executable, updater, embedded key
resources, account-modification code, or artwork. The application credits
Dalavin prominently and links to the archived corresponding source.

2 changes: 2 additions & 0 deletions UnityScraper.spec
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ a = Analysis(
'knowledge_service',
'knowledge_sync',
'plugins',
'profile_gui',
'profile_manager',
'updater',
'wiki_adapters',
],
Expand Down
2 changes: 2 additions & 0 deletions app_paths.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ def xdg_path(variable: str, fallback: Path) -> Path:
CACHE_DIR = _PATHS.cache
EXPORTS_DIR = _PATHS.exports
DIAGNOSTICS_DIR = _PATHS.diagnostics
PROFILE_BACKUPS_DIR = DATA_DIR / "profile_backups"

DATABASE_PATH = DATA_DIR / "unityscraper.db"
CONFIG_PATH = CONFIG_DIR / "config.json"
Expand All @@ -170,6 +171,7 @@ def ensure_app_dirs() -> None:
CACHE_DIR,
EXPORTS_DIR,
DIAGNOSTICS_DIR,
PROFILE_BACKUPS_DIR,
):
path.mkdir(parents=True, exist_ok=True)

Expand Down
Loading
Loading