Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ results to Tk's main loop.
- `unified_search.py` ranks local results across the application domains.
- `structured_knowledge.py` extracts typed records from cached source documents
while retaining document and source relationships.
- `tool_catalog.py` declares supported community tools, reviewed operations,
platform constraints, executable discovery, saved paths, and checksums.
- `external_tools.py` validates path contracts and runs argument vectors without
a command shell; detached GUI launches remain separate from captured CLI jobs.

### Domain and Adapters

Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ Notable changes to UnityScraper are documented here. The project follows

### Added

- A catalog-backed Tool Center with persistent executable discovery, SHA-256
identification, platform-aware availability, command previews, and CLI parity.
- Native extract-xiso list, extract, create, and guarded rewrite workflows,
plus direct Xenia and Xenia Canary launching.
- Managed launch integrations for user-supplied Velocity, Iso2God, God2ISO,
Xbox Image Browser, and Le Fluffie installations.

- Additive schema migration 11 for offline knowledge archive builds, rendered
document state, and browser-saved wiki import history.
- A self-contained offline Knowledge library with local search, source and
Expand Down
2 changes: 2 additions & 0 deletions DOCS_INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
snapshots, privacy, restore behavior, and Le Fluffie attribution
- [Profile Intelligence and Xenia](PROFILE_INTELLIGENCE.md) - read-only GPD
achievements, profile comparison, and snapshot-first Xenia migration
- [Tool Center](EXTERNAL_TOOLS.md) - supported community utilities, setup,
command-line use, provenance, and safety boundaries
- [Collection Intelligence](COLLECTION_INTELLIGENCE.md) - storage discovery,
XEX identity, Title Update compatibility, preservation, and repair previews
- [Console Sync](CONSOLE_SYNC.md) - persistent transfers, resume, snapshots,
Expand Down
139 changes: 94 additions & 45 deletions EXTERNAL_TOOLS.md
Original file line number Diff line number Diff line change
@@ -1,61 +1,110 @@
# External Tools
# Tool Center

UnityScraper can run trusted Xbox command-line utilities from the desktop
interface. Open **External Tools** from the sidebar.
UnityScraper's **Tool Center** provides one place to discover, configure, and
run trusted Xbox 360 community utilities. Except for the documented XeXTool
copy, users provide their own lawfully obtained executables. UnityScraper does
not download tools, pass commands through a shell, or hide the command that is
about to run.

## XeXTool
## Supported Tools

The Windows build includes XeXTool 6.3, created by **xorloser**, so the XeXTool
preset works without separate setup. The executable was sourced from the
GPL-3.0-licensed
[XboxChef/XexToolGUI](https://github.com/XboxChef/XexToolGUI) project.
UnityScraper preserves its source license and records the exact binary checksum
in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
| Tool | Integration | Platforms | Credit |
| --- | --- | --- | --- |
| XeXTool | XEX information presets and custom arguments | Windows | xorloser |
| extract-xiso | List, extract, create, and guarded rewrite operations | Windows, Linux, macOS | XboxDev |
| Xenia | Launch a selected game or open the emulator | Windows | Xenia Project |
| Xenia Canary | Launch a selected game or open the emulator | Windows | Xenia Canary Project |
| Velocity | Detect, configure, verify, and launch | Windows | Velocity contributors |
| Iso2God | Detect, configure, verify, and launch | Windows | Iso2God contributors |
| God2ISO | Detect, configure, verify, and launch | Windows | Community utility |
| Xbox Image Browser | Detect, configure, verify, and launch | Windows | Community utility |
| Le Fluffie | Detect, configure, verify, and launch | Windows | Dalavin / DJ SkunkieButt |
| Custom CLI tool | User-defined argument vector with input/output placeholders | Current native platform | User supplied |

Use **Browse** to select a different lawfully obtained build at any time. A
missing or invalid saved path automatically falls back to the bundled copy.
Legacy GUI utilities are launch-managed because their command-line contracts
are not stable or publicly documented. UnityScraper deliberately does not
invent arguments for them. Native operation presets are only supplied where a
reviewable command-line interface exists.

The XeXTool preset provides:
FATXplorer and J-Runner are intentionally not integrated.

- **Extended information** using `-l "{input}"`
- **Basic information** using `"{input}"`
- **Custom arguments** for advanced users
## Setup

Choose an XEX file, review the command shown in the output panel, and select
**Run Tool**. Standard output and standard error remain visible in UnityScraper.
Open **Tool Center**, choose a tool, then select **Detect**. Discovery checks a
saved path, bundled resources where applicable, the system path, and a small
set of conventional tool folders. Use **Browse** when a tool is elsewhere.

## Other CLI Tools
The selected executable path is saved in the normal application configuration.
Its SHA-256 checksum is displayed after selection so the same binary can be
identified later. UnityScraper never treats that checksum as a publisher
signature or proof that an executable is safe.

Choose **Custom CLI tool**, select an executable, and enter its argument
template. Two placeholders are supported:
## Native Workflows

```text
{input} selected input file
{output} selected output path
```
### XeXTool

The Windows build includes XeXTool 6.3 by **xorloser** and provides basic and
extended XEX information presets. A different lawfully obtained build can be
selected. Output and errors remain visible in Tool Center.

### extract-xiso

The XboxDev command-line tool receives structured argument vectors for:

Each placeholder becomes part of one argument after parsing. UnityScraper
starts the executable directly with `shell=False`; it does not send the command
through PowerShell, Command Prompt, Bash, or another command shell.
- listing image contents;
- extracting an image to a selected folder;
- creating an image from a selected folder; and
- rewriting an image only after a destructive-operation confirmation.

Custom XeXTool and CLI paths are stored in the normal application
configuration. The bundled executable remains inside UnityScraper's packaged
resources and is not copied into the user data folder.
Keep a backup before any rewrite. The desktop asks for confirmation and the
CLI requires `--tool-allow-modify`.

## Safety
### Xenia and Xenia Canary

- Use tools and files you are legally entitled to use.
- Keep backups before running commands that modify content.
- Prefer the read-only information presets when inspecting an unfamiliar XEX.
- Review custom arguments before running them.
- Do not run executables from an untrusted source.
- Cancellation requests terminate the active process, but a tool may already
have changed its output before termination.
Tool Center can launch either emulator with a selected game path or open it
without a game. Existing Profiles & Saves migration features remain separate:
they preview mappings, create a verified snapshot, and avoid overwriting
different save files.

## Platform Notes
## Command Line

List integrations and their operation IDs:

```powershell
python main.py --list-tools
```

Run a configured operation:

```powershell
python main.py --tool-id extract-xiso --tool-operation list --tool-input game.iso
```

Select and remember an executable explicitly:

```powershell
python main.py --tool-id xenia --tool-operation launch-game `
--tool-executable C:\Tools\Xenia\xenia.exe --tool-input default.xex
```

The external tool must be executable on the current operating system.
The bundled XeXTool executable is enabled on Windows. Windows `.exe` files do
not run natively on Linux, so Linux users can select a native tool or a trusted
wrapper they configured themselves. UnityScraper does not automatically
install or invoke Wine.
`--tool-arg` may be repeated to replace a preset with an advanced argument
vector and requires `--tool-allow-modify`. `{input}` and `{output}` placeholders
are resolved as individual arguments. No PowerShell, Command Prompt, Bash, or
other shell interprets them. Reviewed built-in presets are locked in the GUI;
choose a custom operation when an editable argument vector is required.

## Safety and Platform Boundaries

- Use only software and content you are legally entitled to use.
- Obtain third-party tools from a source you trust and review their licenses.
- Keep backups before conversion or modification operations.
- Review the displayed executable, checksum, operation, and command.
- Cancellation stops a captured command-line process but cannot undo changes
the external program already made.
- Windows executables are not automatically installed or run through Wine on
Linux or macOS. Choose a native build where one exists.
- UnityScraper does not bundle game images, keys, firmware, SDK material, or
closed-source legacy utilities.

See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for project links,
licenses, provenance, and the exact bundled XeXTool checksum.
3 changes: 3 additions & 0 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,9 @@ backup-management, and source-attributed knowledge application.
parity for search and preservation, FATX geometry reports, and bounded STFS
file-table inventory.
- Windows, Linux, and unsigned Apple Silicon macOS CI packaging.
- Catalog-backed Tool Center with native extract-xiso and Xenia workflows,
managed launch support for selected legacy GUI utilities, executable
discovery, checksums, CLI parity, and destructive-operation confirmation.

## Validation

Expand Down
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,10 +88,13 @@ SkunkieButt, and the GPLv3 X360 library and Le Fluffie source. See

### External Tools

- Runs Xbox command-line utilities inside the desktop interface.
- Provides a unified Tool Center for Xbox community utilities.
- Includes XeXTool 6.3 by xorloser for basic or extended XEX information on
Windows.
- Supports custom executables and argument templates for other community tools.
- Adds native workflows for extract-xiso, Xenia, and Xenia Canary.
- Detects, verifies, and launches user-supplied Velocity, Iso2God, God2ISO,
Xbox Image Browser, and Le Fluffie installations.
- Supports custom executables and argument templates for additional tools.
- Shows the exact command, captures standard output and errors, and supports
cancellation and timeouts.
- Executes argument vectors directly without using a command shell.
Expand Down Expand Up @@ -220,7 +223,7 @@ Linux source setup:
| Downloads | Review and manage download activity |
| Backup Manager | Scan, install, verify, export, convert, and transfer owned content |
| Profiles & Saves | Inventory profiles, inspect achievements, compare, snapshot, restore, and migrate to Xenia |
| External Tools | Run XeXTool and other user-supplied command-line utilities |
| Tool Center | Run and manage supported Xbox 360 community utilities |
| Collections | Identify storage, compare Title Updates, verify preservation data, and preview repairs |
| Knowledge | Search sources, facts, citations, imports, and conflicts |
| Community Hub | Unified search, console plans, profiles, preservation, plugins, recovery, compatibility, and release toolkit |
Expand Down
23 changes: 23 additions & 0 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,26 @@ UnityScraper does not include Le Fluffie's executable, updater, embedded key
resources, account-modification code, or artwork. The application credits
Dalavin prominently and links to the archived corresponding source.

## Tool Center Interoperability

The following projects are supported through user-selected executables. Their
binaries and licenses are not bundled by UnityScraper:

- **extract-xiso**, XboxDev: <https://github.com/XboxDev/extract-xiso>
- **Xenia**, Xenia Project: <https://github.com/xenia-project/xenia>
- **Xenia Canary**, Xenia Canary Project:
<https://github.com/xenia-canary/xenia-canary>
- **Velocity**, Velocity contributors:
<https://github.com/Gualdimar/Velocity> (archived GPL-3.0 project)
- **Iso2God**, Iso2God contributors: <https://github.com/r4dius/Iso2God>

Tool Center can also launch user-supplied God2ISO and Xbox Image Browser
installations. These legacy utilities have varied distribution histories, so
UnityScraper does not bundle them or claim a canonical download. Users are
responsible for obtaining lawful copies and reviewing the terms that accompany
their chosen builds.

An integration means that UnityScraper can locate or launch a program; it does
not imply endorsement by, affiliation with, or redistribution permission from
the program's authors.

1 change: 1 addition & 0 deletions UnityScraper.spec
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ a = Analysis(
'knowledge_scheduler',
'knowledge_sync',
'structured_knowledge',
'tool_catalog',
'unified_search',
'plugins',
'plugin_worker',
Expand Down
80 changes: 73 additions & 7 deletions external_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@ class ToolResult:
cancelled: bool


@dataclass(frozen=True)
class ToolLaunch:
"""Details for a detached graphical tool launch."""

command: tuple[str, ...]
pid: int


def split_arguments(value: str, *, windows: bool | None = None) -> list[str]:
"""Split an editable argument template without passing it through a shell."""
use_windows_rules = os.name == "nt" if windows is None else windows
Expand Down Expand Up @@ -64,13 +72,15 @@ def build_command(
*,
input_path: str | Path | None = None,
output_path: str | Path | None = None,
input_kind: str = "file",
output_kind: str = "optional",
) -> tuple[str, ...]:
tool = Path(executable).expanduser().resolve()
if not tool.is_file():
raise ExternalToolError(f"Tool executable was not found: {tool}")

source = self._resolve_input(input_path)
output = self._resolve_output(output_path)
source = self._resolve_input(input_path, input_kind)
output = self._resolve_output(output_path, output_kind)
arguments: list[str] = []
for value in argument_template:
if "{input}" in value and source is None:
Expand All @@ -83,6 +93,40 @@ def build_command(
)
return (str(tool), *arguments)

def launch_detached(
self,
executable: str | Path,
argument_template: Iterable[str] = (),
*,
input_path: str | Path | None = None,
output_path: str | Path | None = None,
input_kind: str = "none",
output_kind: str = "none",
) -> ToolLaunch:
"""Launch a GUI utility without waiting for it to exit."""
command = self.build_command(
executable,
argument_template,
input_path=input_path,
output_path=output_path,
input_kind=input_kind,
output_kind=output_kind,
)
creation_flags = 0
if os.name == "nt":
creation_flags = subprocess.CREATE_NEW_PROCESS_GROUP
try:
process = subprocess.Popen(
command,
cwd=Path(command[0]).parent,
shell=False,
creationflags=creation_flags,
close_fds=os.name != "nt",
)
except OSError as exc:
raise ExternalToolError(f"Could not start external tool: {exc}") from exc
return ToolLaunch(command, process.pid)

def run(
self,
executable: str | Path,
Expand All @@ -91,14 +135,18 @@ def run(
input_path: str | Path | None = None,
output_path: str | Path | None = None,
timeout: float = 300,
input_kind: str = "file",
output_kind: str = "optional",
) -> ToolResult:
command = self.build_command(
executable,
argument_template,
input_path=input_path,
output_path=output_path,
input_kind=input_kind,
output_kind=output_kind,
)
source = self._resolve_input(input_path)
source = self._resolve_input(input_path, input_kind)
working_directory = source.parent if source else Path(command[0]).parent
creation_flags = subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0
started = time.monotonic()
Expand Down Expand Up @@ -153,19 +201,37 @@ def cancel(self) -> bool:
return True

@staticmethod
def _resolve_input(value: str | Path | None) -> Path | None:
def _resolve_input(value: str | Path | None, kind: str = "file") -> Path | None:
if kind not in {"file", "directory", "any", "optional", "none"}:
raise ExternalToolError(f"Unsupported input path kind: {kind}")
if kind == "none":
return None
if value is None or not str(value).strip():
if kind not in {"none", "optional"}:
raise ExternalToolError("This command requires an input path")
return None
path = Path(value).expanduser().resolve()
if not path.is_file():
if kind == "file" and not path.is_file():
raise ExternalToolError(f"Input file was not found: {path}")
if kind == "directory" and not path.is_dir():
raise ExternalToolError(f"Input folder was not found: {path}")
if kind in {"any", "optional"} and not path.exists():
raise ExternalToolError(f"Input path was not found: {path}")
return path

@staticmethod
def _resolve_output(value: str | Path | None) -> Path | None:
def _resolve_output(value: str | Path | None, kind: str = "file") -> Path | None:
if kind not in {"file", "directory", "optional", "none"}:
raise ExternalToolError(f"Unsupported output path kind: {kind}")
if kind == "none":
return None
if value is None or not str(value).strip():
if kind not in {"none", "optional"}:
raise ExternalToolError("This command requires an output path")
return None
path = Path(value).expanduser().resolve()
if not path.parent.is_dir():
if kind == "directory" and not path.is_dir():
raise ExternalToolError(f"Output folder was not found: {path}")
if kind != "directory" and not path.parent.is_dir():
raise ExternalToolError(f"Output folder was not found: {path.parent}")
return path
Loading
Loading