feat(gateway): add system registry support and source indicators

[alexclewontin]

Summary

Add a system-managed gateway registry under /etc/openshell and surface whether each gateway comes from user or system config in the CLI and TUI.

Related Issue

N/A

Changes

• add bootstrap path handling for installer-provided gateway metadata and active-gateway fallback under /etc/openshell, with OPENSHELL_SYSTEM_GATEWAY_DIR as an override
• make per-user gateways shadow system entries, and keep system registrations read-only from the CLI
• show gateway config source in openshell gateway list, list JSON output, and openshell term
• add CLI smoke coverage for gateway list table and JSON source rendering
• update gateway architecture and manage-gateways docs for the new registry and source indicators

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[johntmyers]

gator-agent

/ok to test d1736c7

[github-actions]

Label test:e2e applied, but pull-request/1625 is at {"messa while the PR head is d1736c7. A maintainer needs to comment /ok to test d1736c73fa7558b48ac1d31f9e1e295342344c09 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

[johntmyers]

/ok to test d1736c7

[johntmyers]

gator-agent

PR Review Status

Validation: project-valid as a concentrated gateway/CLI/TUI packaging-config improvement. The PR has a clear user path for installer-seeded system gateways, user entries shadowing system entries, read-only system registrations, and source indicators in CLI/TUI surfaces.
Head SHA: d1736c73fa7558b48ac1d31f9e1e295342344c09

Review findings:

• Blocking: crates/openshell-bootstrap/src/metadata.rs in list_gateways_with_source only records a gateway name as shadowed after user metadata parses successfully. A corrupt or unreadable user gateways/<name>/metadata.json can therefore allow the same-named system gateway to appear in gateway list as system, which contradicts the stated user-over-system shadowing contract and differs from load_gateway_metadata, which resolves the user path first. Please mark the user directory name as seen before parsing, or otherwise ensure invalid user entries cannot reveal same-named system entries, and add a regression test for corrupt user metadata shadowing a valid system entry.
• Process blocker: GitHub REST still reports mergeable_state: dirty; the PR needs to be updated with main or otherwise resolve merge conflicts.

Docs: direct UX changes are documented in docs/sandboxes/manage-gateways.mdx; no docs/index.yml navigation change appears needed because this updates an existing page.

Tests: applied test:e2e; test:e2e-gpu and test:e2e-kubernetes are not required for this change. I verified maintainer authority and posted /ok to test d1736c73fa7558b48ac1d31f9e1e295342344c09 for the current head SHA after the label-helper requested it.

Next state: gator:blocked until merge conflicts and the blocking review finding are resolved.

[johntmyers]

gator-agent

CI Update

Branch Checks are now failing on the current head SHA d1736c73fa7558b48ac1d31f9e1e295342344c09:

• Rust (linux-amd64-cpu8) and Rust (linux-arm64-cpu8) fail in mise run rust:format:check; cargo fmt --all -- --check reports formatting diffs in the changed Rust files.
• mise Lockfile fails because mise lock rewrites mise.lock for updated Python 3.14.5 standalone artifact URLs/checksums.

This still leaves the PR in gator:blocked along with the existing merge-conflict state and the blocking review finding. E2E is still running, with at least one Kubernetes smoke job already failed; I will keep monitoring the workflow result.

[johntmyers]

gator-agent

E2E Update

The required OpenShell / E2E status has now concluded failure for head SHA d1736c73fa7558b48ac1d31f9e1e295342344c09.

Observed failures:

• kubernetes-e2e / Kubernetes E2E (Rust smoke) fails during tool setup because github:mozilla/sccache@0.14.0 is not in the lockfile, consistent with the stale mise.lock failure in Branch Checks.
• e2e / E2E (python) has broad sandbox/policy failures with repeated supervisor relay h2 protocol error stream warnings and many tests failing soon after startup.
• e2e / E2E (rust-docker) and e2e / E2E (rust-podman) passed.

The PR remains in gator:blocked: it still reports merge conflicts, has the blocking list_gateways_with_source review finding, and now has failed required Branch Checks/E2E. The next useful action is still for the branch author to update the PR with main, fix formatting/lockfile drift, and address the review finding.

[alexclewontin]

@johntmyers I've rebased on main and fixed the formatting, then added a commit to ensure that broken user config still shadows system config. Locally, all the e2e failures I see are reproducible on both main and this branch, so I am not sure they are being introduced by these changes

[johntmyers]

/ok to test 5b2bb33

[johntmyers]

gator-agent

PR Review Status

Validation: still project-valid as a concentrated gateway/CLI/TUI packaging-config improvement. The PR has a clear user path for installer-seeded system gateways, user entries shadowing system entries, read-only system registrations, and source indicators in CLI/TUI surfaces.
Head SHA: 5b2bb335302864be7b8ed82f1e28d99e3e3d3cbb

Review findings:

• Blocking: crates/openshell-bootstrap/src/metadata.rs in list_gateways_with_source now records a user gateway directory as seen before checking whether metadata.json exists. That fixes corrupt user metadata shadowing, but an empty or stale user gateways/<name>/ directory can still hide a valid system gateway. This is reachable after remove_gateway_metadata, which removes the file but not necessarily the directory. Please only shadow system config when the user metadata.json exists or cannot be statted, and add a regression test where an empty user gateway directory does not hide a valid system gateway.
• Blocking: crates/openshell-bootstrap/src/paths.rs accepts OPENSHELL_SYSTEM_GATEWAY_DIR verbatim, including empty or relative values. Because this is presented as a package-managed system registry and can provide the active-gateway fallback, please require a non-empty absolute override path, or provide an explicit maintainer rationale for allowing relative/current-directory system registry lookup.

Docs: direct UX changes are documented in docs/sandboxes/manage-gateways.mdx; no docs/index.yml navigation change appears needed because this updates an existing page.

Tests: test:e2e remains applied. I verified maintainer authority and posted /ok to test 5b2bb335302864be7b8ed82f1e28d99e3e3d3cbb for the new head SHA; Branch Checks and E2E are now running, Helm Lint is green, and GPU E2E is not required.

Next state: gator:in-review while the blocking review feedback is addressed.

[johntmyers]

gator-agent

Gate Update

Required checks are now green for head SHA 5b2bb335302864be7b8ed82f1e28d99e3e3d3cbb: Branch Checks, Helm Lint, and E2E all pass.

The PR remains blocked because GitHub REST still reports mergeable_state: dirty, so the branch needs to be updated with main or otherwise have merge conflicts resolved. The two blocking review findings from the latest gator review also still need author response or code changes:

• empty/stale user gateway directories should not hide a valid system gateway unless user metadata.json exists or cannot be statted
• OPENSHELL_SYSTEM_GATEWAY_DIR should reject empty or relative override paths, or have an explicit maintainer rationale for allowing them

Next state: gator:blocked until the merge conflict and blocking review feedback are resolved.

[johntmyers]

gator-agent

Blocker Follow-Up Nudge

This PR is still blocked after more than 48 business hours.

Current blockers:

• GitHub REST still reports mergeable_state: dirty, so the branch needs to be updated with main or otherwise have merge conflicts resolved.
• The latest gator review findings still need author response or code changes: empty/stale user gateway directories should not hide a valid system gateway, and OPENSHELL_SYSTEM_GATEWAY_DIR should reject empty or relative override paths or have explicit maintainer rationale.

Checks are currently green for head SHA 5b2bb335302864be7b8ed82f1e28d99e3e3d3cbb.

Next action: @alexclewontin, please update the branch and address or respond to the remaining review feedback.

[johntmyers]

/ok to test 85ea854

[johntmyers]

gator-agent

Re-check After Author Update

I re-evaluated latest head 85ea854d33003808c85b89fa49af0d1dc28b0c4e after @alexclewontin pushed an update on 2026-06-09.

Disposition: partially resolved.

Remaining items:

• Resolved: the previous code review blockers are addressed. Empty/stale user gateway directories no longer hide valid system metadata, malformed user metadata still shadows same-named system metadata, and OPENSHELL_SYSTEM_GATEWAY_DIR now ignores empty or relative override values with warnings.
• Still blocked: GitHub REST reports mergeable_state: dirty, so the branch still needs to be updated with main or otherwise have merge conflicts resolved.
• Checks: required Branch Checks, Helm Lint, and E2E are pending on this new head because the copy-pr mirror was waiting for /ok to test; I posted /ok to test 85ea854d33003808c85b89fa49af0d1dc28b0c4e as a standalone command.

Next state: gator:blocked until the merge conflict is resolved. Once the branch is mergeable and required checks complete, gator can move this back to pipeline watch or maintainer approval as appropriate.

[johntmyers]

gator-agent

Gate Update

Current head: 85ea854d33003808c85b89fa49af0d1dc28b0c4e.

State remains gator:blocked.

• Mergeability: GitHub REST still reports mergeable_state: dirty, so the branch still needs to be updated with main or otherwise have merge conflicts resolved.
• Branch Checks: passed on this head.
• Helm Lint: failed before linting in Detect Helm changes while running ./.github/actions/pr-merge-base; the job hit HTTP 401: Requires authentication from a GraphQL-backed gh pr view call.
• E2E: still queued/running for this head.

Next action: keep this PR blocked while the merge conflict and current required-check state are unresolved; gator will continue watching.

[johntmyers]

gator-agent

Gate Update

Current head: 85ea854d33003808c85b89fa49af0d1dc28b0c4e.

State remains gator:blocked.

• Mergeability: GitHub REST still reports mergeable_state: dirty, so the branch still needs to be updated with main or otherwise have merge conflicts resolved.
• Branch Checks: passed on this head.
• E2E: passed on this head.
• Helm Lint: failed before linting in Detect Helm changes; the setup action runs gh pr view, which hit HTTP 401: Requires authentication from https://api.github.com/graphql. This appears to be a workflow/auth setup failure rather than a Helm lint regression in this PR.

Next action: @alexclewontin, please resolve the merge conflict. Once the branch is mergeable, gator can re-check the required Helm gate and move the PR forward if no blockers remain.

[johntmyers]

/ok to test 23aec7a

[johntmyers]

gator-agent

Re-check After Author Update

I re-evaluated latest head 23aec7a9dac9ebdfaa78cd909045185bfcfd634c after @alexclewontin pushed the June 11 update.

Disposition: partially resolved.

Remaining items:

• Resolved: the prior code review blockers are addressed. Empty/stale user gateway directories no longer hide valid system metadata, malformed user metadata still shadows same-named system metadata, and OPENSHELL_SYSTEM_GATEWAY_DIR now ignores empty or relative override values with warnings. The PR is mergeable against current main; the previous dirty merge-conflict blocker is no longer present.
• Still needs author follow-up: docs/reference/gateway-auth.mdx still describes only the user config active gateway and metadata layout. Please update that reference to mention the /etc/openshell system fallback for active_gateway and gateways/<name>/metadata.json.
• Still needs author follow-up: please add direct CLI smoke or unit coverage for rejecting openshell gateway remove <system-only-name> while preserving the system entry, or get a maintainer waiver for that test gap.
• Checks: OpenShell / Branch Checks, OpenShell / Helm Lint, and OpenShell / E2E are pending with “Waiting for /ok to test mirror”; I posted /ok to test 23aec7a9dac9ebdfaa78cd909045185bfcfd634c as a standalone command. DCO and required-status publication are green.

Next state: gator:in-review while the remaining docs/test feedback is addressed and the freshly authorized checks run.