Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
os: [ubuntu-latest, windows-latest, macos-latest]
python: ["3.10", "3.12"]

steps:
Expand Down Expand Up @@ -57,6 +57,10 @@ jobs:
sh -n packaging/linux/install.sh
sh -n packaging/linux/uninstall.sh

- name: Validate macOS shell script
if: runner.os == 'macOS'
run: sh -n build_macos.sh

windows-build:
name: Windows executable
runs-on: windows-latest
Expand Down Expand Up @@ -136,3 +140,35 @@ jobs:
dist/UnityScraper-Linux-x86_64.tar.gz
dist/UnityScraper-Linux-x86_64.tar.gz.sha256
if-no-files-found: error

macos-build:
name: macOS application
runs-on: macos-14
needs: test

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip

- name: Install build dependencies
run: python -m pip install -r requirements.txt pyinstaller

- name: Build macOS application
run: ./build_macos.sh

- name: Validate application bundle
run: |
test -x dist/UnityScraper.app/Contents/MacOS/UnityScraper
plutil -lint dist/UnityScraper.app/Contents/Info.plist

- uses: actions/upload-artifact@v4
with:
name: UnityScraper-macOS-arm64
path: |
dist/UnityScraper-macOS-arm64.tar.gz
dist/UnityScraper-macOS-arm64.tar.gz.sha256
if-no-files-found: error
23 changes: 22 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,30 @@ jobs:
dist/UnityScraper-Linux-x86_64.tar.gz
dist/UnityScraper-Linux-x86_64.tar.gz.sha256

macos:
runs-on: macos-14
needs: validate

steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install build dependencies
run: python -m pip install -r requirements.txt pyinstaller
- name: Build macOS release
run: ./build_macos.sh
- uses: actions/upload-artifact@v4
with:
name: release-macos
path: |
dist/UnityScraper-macOS-arm64.tar.gz
dist/UnityScraper-macOS-arm64.tar.gz.sha256

publish:
runs-on: ubuntu-latest
needs: [validate, windows, linux]
needs: [validate, windows, linux, macos]

steps:
- uses: actions/download-artifact@v4
Expand Down
16 changes: 14 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ for CLI and optional REST automation. SQLite is the durable local store.
- `profile_gui.py` renders privacy-aware profile/save inventory and snapshots.
- `collection_gui.py` renders collection analysis, matching, reports, and
repair previews.
- `community_gui.py` renders unified search and the cross-domain community
workspaces.
- `setup_wizard.py` handles first-run storage setup.

GUI operations that can block are dispatched to background threads and return
Expand All @@ -42,6 +44,12 @@ results to Tk's main loop.
snapshots, and PC/console comparisons.
- `database_migrations.py` applies additive schema versions and provides
consistent SQLite backup/restore helpers.
- `community_services.py` coordinates guided sync plans, package workspaces,
artwork, disc and dedup audits, storage, plugins, recovery, compatibility,
and accessibility.
- `unified_search.py` ranks local results across the application domains.
- `structured_knowledge.py` extracts typed records from cached source documents
while retaining document and source relationships.

### Domain and Adapters

Expand Down Expand Up @@ -69,6 +77,9 @@ Main schema groups:
- Profiles: scan runs, profiles, saves, snapshots, snapshot files, GPD
inventories, achievements, comparisons, Xenia migration runs, and auditable
operations
- Community: structured records, guided sync plans, ownership previews, played
titles and images, save comparisons, artwork exports, disc and dedup audits,
storage audits, plugin state, recovery, compatibility, and accessibility

Schema initialization is idempotent. New migrations should preserve existing
data and be covered by tests.
Expand Down Expand Up @@ -183,5 +194,6 @@ See [SECURITY.md](SECURITY.md) for reporting and operational guidance.

`UnityScraper.spec` is the canonical cross-platform PyInstaller definition.
Assets and modules loaded indirectly by the GUI are listed explicitly. GitHub
Actions validates Windows and Linux one-file builds on pull requests. Version
tags publish a Windows ZIP and Linux tarball with separate SHA-256 files.
Actions validates Windows and Linux one-file builds plus an unsigned macOS
Apple Silicon application bundle. Version tags publish platform archives with
separate SHA-256 files.
23 changes: 22 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,26 @@ Notable changes to UnityScraper are documented here. The project follows

### Added

- Community Hub with unified local search across games, source knowledge,
profiles, saves, achievements, files, and structured records.
- Additive schema migration 8 for structured knowledge, guided console plans,
ownership previews, GPD title/image records, save comparisons, artwork,
disc/dedup audits, storage and original Xbox records, plugins, recovery,
dashboard compatibility, and accessibility.
- Structured extraction of hardware, drive, dashboard, exploit, error, format,
repair, and tool records from cached source documents with provenance.
- Confirmed sync-plan queueing through the existing persistent transfer engine.
- Read-only package workspaces, profile dashboards, ownership previews,
block-level save comparison, played-title history, and validated GPD image
export.
- Artwork export presets, multi-disc audits, hash-based duplicate previews, and
recoverable duplicate quarantine/hardlink actions.
- Read-only FATX audits, original Xbox discovery, bounded plugin installation,
recovery actions, and dashboard FTP capability probes.
- High-contrast, large-text, reduced-motion, and keyboard-hint preferences.
- Unsigned macOS Apple Silicon application packaging, checksums, CI validation,
and documentation.
- Runtime repair for historically misencoded translation resources.
- Read-only XDBF/GPD inspection with bounded entry parsing, achievement state,
gamerscore summaries, extracted-folder discovery, and local inventory.
- Profile comparison for save hashes and imported achievement state.
Expand Down Expand Up @@ -63,14 +83,15 @@ Notable changes to UnityScraper are documented here. The project follows

### Changed

- Version advanced to `1.1.0-beta.1`.
- CI now runs the Python suite on Windows, Linux, and macOS.
- Cached XboxUnity titles now resolve immediately in library lists and details,
enrich matching rows page by page, and recover after interrupted refreshes.
- Library rows now show `Unknown game` instead of duplicating the TitleID when
no real game name is known.
- Cached XboxUnity names enrich only blank, unknown, or TitleID-shaped values
and never replace an existing preferred title.
- Library queries now close SQLite handles immediately after use.
- Version advanced to `1.0.0-beta.1`.
- Download queues now use atomic writes and recover interrupted items.
- Update checks select a platform artifact and require its SHA-256 sidecar
before staging it.
Expand Down
78 changes: 78 additions & 0 deletions COMMUNITY_HUB.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Community Hub

The Community Hub brings the wider Xbox and preservation workflows into one
source-attributed, offline-capable workspace. Open it from the desktop sidebar
or press `Ctrl+K` to focus unified search.

## Twenty Integrated Capabilities

1. Unified local search across games, identifiers, wiki knowledge, profiles,
saves, achievements, files, and structured reference records.
2. Structured extraction of motherboards, DVD drives, dashboards, exploits,
errors, formats, repairs, and tools from cached source documents.
3. Guided PC-to-console comparison using a captured, read-only console
inventory and Aurora, Freestyle Dash, XeXMenu, or stock layouts.
4. Explicit queueing of revalidated uploads into the existing persistent,
resumable transfer queue. A preview never starts a transfer by itself.
5. Profile dashboards summarizing saves, played titles, achievements, and
gamerscore from local data.
6. Read-only STFS package inspection and auditable package workspaces that keep
an untouched original and manifest.
7. Ownership-migration previews that record intended profile, console, and
device changes without changing or signing the package.
8. Block-level save comparisons with SHA-256 results and durable audit history.
9. XDBF/GPD played-title history and bounded embedded-image discovery.
10. Validated export of an embedded GPD image without modifying its source.
11. Preferred artwork selection with Aurora, Freestyle Dash, and preservation
archive export layouts plus checksum manifests.
12. Multi-disc completeness audits based on scanned disc number and count.
13. Duplicate previews using size grouping and SHA-256 verification.
14. Recoverable duplicate actions that quarantine the original and can create
a verified hardlink only after a second hash check.
15. Read-only FATX signature and mounted-storage audits.
16. Original Xbox `default.xbe` discovery alongside Xbox 360 collections.
17. Plugin discovery, checksums, permission display, enable/disable state, and
bounded ZIP installation or update with rollback.
18. Recovery scans for partial files, interrupted jobs, incomplete snapshots,
and failed operations, with conservative retry or quarantine actions.
19. Dashboard compatibility probes for login, content-root access, resume, and
advertised remote hash support on a trusted local FTP network.
20. Cross-platform accessibility and packaging: scalable text, high contrast,
reduced-motion preferences, keyboard hints, and Windows, Linux, and macOS
build paths.

## Safety Rules

- Sync plans are previews until the user confirms queueing. Queued jobs still
run through the normal transfer controls.
- Duplicate cleanup never deletes the only retained copy. Quarantined files
remain under `.unityscraper-dedup-quarantine` for manual recovery.
- FATX images are detected read-only. Raw-device and raw-image writes are not
implemented.
- Package workspaces and ownership changes are previews. CON/LIVE/PIRS rebuild,
rehash, signature, and ownership mutation remain disabled until independent
verification and recovery are complete.
- Traditional console FTP is unencrypted and is intended only for a trusted
local network.
- XboxUnity remains HTTP-only. This application does not invent or prefer an
HTTPS endpoint for XboxUnity.

## Data and Provenance

Migration 8 adds durable records for structured knowledge, sync plans, profile
previews, played titles, embedded images, save comparisons, artwork exports,
disc audits, dedup plans, storage audits, original Xbox records, plugin state,
recovery events, dashboard tests, and accessibility preferences. It is additive
and preserves existing databases.

Imported ConsoleMods, XenonLibrary, Free60, Redump, and No-Intro information
continues to retain source, revision, citation, licensing, and conflict data.
Redump and No-Intro DAT files remain user-supplied; copyrighted game content is
never included.

## Platform Notes

Windows and Linux remain the primary tested release targets. macOS CI builds an
unsigned Apple Silicon application bundle and checksum. The macOS artifact is
not notarized, and the documentation does not ask users to disable Gatekeeper.
See [MACOS.md](MACOS.md) for current limitations.
10 changes: 10 additions & 0 deletions DOCS_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@
- [REST API](API.md) - authentication, endpoints, configuration, and safety
- [Linux Support](LINUX.md) - installation, XDG storage, desktop integration,
uninstallation, and troubleshooting
- [macOS Preview](MACOS.md) - Apple Silicon bundle, source setup, checksums,
and signing limitations
- [Community Hub](COMMUNITY_HUB.md) - unified search, profile/package tools,
guided console plans, preservation, plugins, recovery, and safety boundaries
- [Project Status](PROJECT_STATUS.md) - completed work, boundaries, and roadmap
- [Changelog](CHANGELOG.md) - release history

Expand Down Expand Up @@ -53,3 +57,9 @@ Linux packaging:
```bash
./build_linux.sh
```

macOS packaging:

```bash
./build_macos.sh
```
32 changes: 32 additions & 0 deletions MACOS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# macOS Support

UnityScraper supports macOS 11 or newer from source on Intel and Apple silicon
through the same Tk desktop application and native Application Support, Caches,
and Logs paths.

Current preview release archives are built for Apple silicon and contain
`UnityScraper.app` plus a SHA-256 checksum. They are currently unsigned and not
notarized, so macOS may require an explicit **Open** confirmation from Finder.
The project does not ask users to disable Gatekeeper. Intel users should run
from source until a universal release artifact is available.

## Source setup

Install Python 3.10 or newer with Tk support, then run:

```sh
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python desktop_app.py
```

## Build

```sh
chmod +x build_macos.sh
./build_macos.sh
```

Signing and notarization can be enabled by a release maintainer when an Apple
Developer ID certificate and notarization credentials are configured as
repository secrets. Those credentials are never stored in the repository.
14 changes: 11 additions & 3 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,12 @@ backup-management, and source-attributed knowledge application.
launch scripts, and release artifacts.
- Repository contribution, security, architecture, API, and release
documentation.
- Community Hub with unified search, typed knowledge extraction, guided console
upload plans, package/profile workspaces, artwork and disc management,
recoverable deduplication, storage audits, original Xbox discovery, plugin
controls, recovery actions, dashboard probes, and accessibility preferences.
- Additive schema migration 8 and local audit history for every new workspace.
- Windows, Linux, and unsigned Apple Silicon macOS CI packaging.

## Validation

Expand All @@ -65,7 +71,7 @@ backup-management, and source-attributed knowledge application.
end-to-end local workflow.
- Network-backed source syncs remain dependent on each source's availability and
access policy. Cached copies are used when available.
- Windows and Linux executable artifacts are generated by CI and releases
- Windows, Linux, and macOS artifacts are generated by CI and releases
rather than committed to the source tree.

## Deliberate Boundaries
Expand All @@ -86,12 +92,14 @@ backup-management, and source-attributed knowledge application.
- Profile editing, achievement modification, ownership rewriting, and CON
re-signing remain disabled until complete package verification and reliable
cross-platform signing support are available.
- FATX and raw-device access remains read-only. Duplicate actions retain a
quarantine copy, and console plans require explicit queue confirmation.

## Future Work

- Validate console resume and optional hash behavior against a broader matrix
of real dashboard FTP servers.
- Expand read-only GPD coverage with dashboard title-history and safe image
previews after adding a decompression and image validation boundary.
- Add notarization and universal binaries after macOS signing infrastructure is
available.
- Consider package mutation only after complete STFS extraction, rehashing,
signing, verification, and automatic recovery have independent test vectors.
Loading
Loading