Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ Notable changes to UnityScraper are documented here. The project follows

### Added

- External Tools workspace with XeXTool inspection presets, custom CLI
arguments, command previews, captured output, cancellation, and timeouts.
- Project creator attribution and a community message from TrapEmAll in About.
- Offline XboxUnity title catalog with background refresh, sync history,
TitleID/name autocomplete, and a manual/CLI refresh path.
- Additive schema migration 5 for cached XboxUnity titles and catalog sync
Expand Down
54 changes: 54 additions & 0 deletions EXTERNAL_TOOLS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# External Tools

UnityScraper can run trusted Xbox command-line utilities from the desktop
interface. Open **External Tools** from the sidebar.

## XeXTool

XeXTool is not distributed with UnityScraper. Its redistribution terms are not
clear enough for this project to package the executable. Obtain it lawfully,
review it with your usual security tools, and select your local copy using
**Browse**.

The XeXTool preset provides:

- **Extended information** using `-l "{input}"`
- **Basic information** using `"{input}"`
- **Custom arguments** for advanced users

Choose an XEX file, review the command shown in the output panel, and select
**Run Tool**. Standard output and standard error remain visible in UnityScraper.

## Other CLI Tools

Choose **Custom CLI tool**, select an executable, and enter its argument
template. Two placeholders are supported:

```text
{input} selected input file
{output} selected output path
```

Each placeholder becomes part of one argument after parsing. UnityScraper
starts the executable directly with `shell=False`; it does not send the command
through PowerShell, Command Prompt, Bash, or another command shell.

The configured XeXTool path is stored in the normal application configuration.
Tool binaries are never copied into UnityScraper's data folder.

## Safety

- Use tools and files you are legally entitled to use.
- Keep backups before running commands that modify content.
- Prefer the read-only information presets when inspecting an unfamiliar XEX.
- Review custom arguments before running them.
- Do not run executables from an untrusted source.
- Cancellation requests terminate the active process, but a tool may already
have changed its output before termination.

## Platform Notes

The external tool must be executable on the current operating system.
Windows `.exe` files do not run natively on Linux. UnityScraper does not
automatically install or invoke Wine. Linux users can select native tools or a
trusted wrapper executable they configured themselves.
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,19 @@ service exposes. UnityScraper does not silently substitute HTTPS URLs.
- Captures read-only console inventories and compares PC and console content.
- Runs a user-selected external ISO converter without bundling converter code.

### External Tools

- Runs user-supplied Xbox command-line utilities inside the desktop interface.
- Includes a XeXTool workspace for basic or extended XEX information.
- Supports custom executables and argument templates for other community tools.
- Shows the exact command, captures standard output and errors, and supports
cancellation and timeouts.
- Executes argument vectors directly without using a command shell.

Third-party binaries such as XeXTool are not bundled. Select an executable you
obtained lawfully and trust. See [EXTERNAL_TOOLS.md](EXTERNAL_TOOLS.md) for
setup, placeholders, platform notes, and safety guidance.

### Collection Intelligence and Preservation

- Discovers mounted console, USB, and archive storage.
Expand Down Expand Up @@ -132,6 +145,7 @@ Linux source setup:
| Add Games | Search cached game names, select TitleIDs, or import lists |
| Downloads | Review and manage download activity |
| Backup Manager | Scan, install, verify, export, convert, and transfer owned content |
| External Tools | Run XeXTool and other user-supplied command-line utilities |
| Collections | Identify storage, compare Title Updates, verify preservation data, and preview repairs |
| Knowledge | Search sources, facts, citations, imports, and conflicts |
| Archive Health | Find missing or inconsistent downloaded files |
Expand Down
171 changes: 171 additions & 0 deletions external_tools.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
"""Safe process runner for user-supplied Xbox command-line tools."""

from __future__ import annotations

import os
import shlex
import subprocess
import threading
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable


class ExternalToolError(RuntimeError):
"""Raised when an external tool cannot be configured or launched."""


@dataclass(frozen=True)
class ToolResult:
"""Captured result from one external tool invocation."""

command: tuple[str, ...]
returncode: int
stdout: str
stderr: str
duration_seconds: float
cancelled: bool


def split_arguments(value: str, *, windows: bool | None = None) -> list[str]:
"""Split an editable argument template without passing it through a shell."""
use_windows_rules = os.name == "nt" if windows is None else windows
arguments = shlex.split(value, posix=not use_windows_rules)
if use_windows_rules:
return [
argument[1:-1]
if len(argument) >= 2 and argument[0] == argument[-1] == '"'
else argument
for argument in arguments
]
return arguments


def format_command(command: Iterable[str], *, windows: bool | None = None) -> str:
"""Format an argument vector for display only."""
values = list(command)
use_windows_rules = os.name == "nt" if windows is None else windows
return subprocess.list2cmdline(values) if use_windows_rules else shlex.join(values)


class ExternalToolRunner:
"""Run one selected executable at a time without shell interpretation."""

def __init__(self) -> None:
self._lock = threading.Lock()
self._process: subprocess.Popen[str] | None = None
self._cancel_requested = False

def build_command(
self,
executable: str | Path,
argument_template: Iterable[str],
*,
input_path: str | Path | None = None,
output_path: str | Path | None = None,
) -> tuple[str, ...]:
tool = Path(executable).expanduser().resolve()
if not tool.is_file():
raise ExternalToolError(f"Tool executable was not found: {tool}")

source = self._resolve_input(input_path)
output = self._resolve_output(output_path)
arguments: list[str] = []
for value in argument_template:
if "{input}" in value and source is None:
raise ExternalToolError("This command requires an input file")
if "{output}" in value and output is None:
raise ExternalToolError("This command requires an output path")
arguments.append(
value.replace("{input}", str(source) if source else "")
.replace("{output}", str(output) if output else "")
)
return (str(tool), *arguments)

def run(
self,
executable: str | Path,
argument_template: Iterable[str],
*,
input_path: str | Path | None = None,
output_path: str | Path | None = None,
timeout: float = 300,
) -> ToolResult:
command = self.build_command(
executable,
argument_template,
input_path=input_path,
output_path=output_path,
)
source = self._resolve_input(input_path)
working_directory = source.parent if source else Path(command[0]).parent
creation_flags = subprocess.CREATE_NO_WINDOW if os.name == "nt" else 0
started = time.monotonic()

with self._lock:
if self._process is not None:
raise ExternalToolError("Another external tool is already running")
self._cancel_requested = False
try:
self._process = subprocess.Popen(
command,
cwd=working_directory,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
shell=False,
creationflags=creation_flags,
)
except OSError as exc:
raise ExternalToolError(f"Could not start external tool: {exc}") from exc
process = self._process

try:
stdout, stderr = process.communicate(timeout=max(1, timeout))
except subprocess.TimeoutExpired as exc:
process.kill()
stdout, stderr = process.communicate()
raise ExternalToolError(
f"External tool exceeded the {timeout:g}-second timeout"
) from exc
finally:
with self._lock:
cancelled = self._cancel_requested
self._process = None

return ToolResult(
command,
process.returncode,
stdout,
stderr,
time.monotonic() - started,
cancelled,
)

def cancel(self) -> bool:
"""Terminate the active process, returning whether one was running."""
with self._lock:
if self._process is None:
return False
self._cancel_requested = True
self._process.terminate()
return True

@staticmethod
def _resolve_input(value: str | Path | None) -> Path | None:
if value is None or not str(value).strip():
return None
path = Path(value).expanduser().resolve()
if not path.is_file():
raise ExternalToolError(f"Input file was not found: {path}")
return path

@staticmethod
def _resolve_output(value: str | Path | None) -> Path | None:
if value is None or not str(value).strip():
return None
path = Path(value).expanduser().resolve()
if not path.parent.is_dir():
raise ExternalToolError(f"Output folder was not found: {path.parent}")
return path
Loading
Loading