diff --git a/AGENTS.md b/AGENTS.md index dc44c8d..6f4f521 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,6 +21,7 @@ This is the contributor guide for AI agents (and humans!) modifying this repo. T - `nsc/cache/` — disk cache for generated command-models. - `nsc/auth/` — login verification helpers (pre-flight probes, token rotate). - `nsc/aliases/` — curated alias resolver (`ls`, `get`, `rm`, `search`). Framework-free. +- `nsc/completion/` — dynamic shell-completion: framework-free providers + cheap on-disk cache probe + Typer `shell_complete` callbacks. - `nsc/skill/` — bundle-path helper for the portable `SKILL.md` shipped in the wheel. - `nsc/schemas/bundled/` — versioned NetBox OpenAPI snapshots, fallback when offline. diff --git a/CHANGELOG.md b/CHANGELOG.md index c8d82df..22d427a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,63 @@ All notable changes to netbox-super-cli are tracked here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) loosely. From v1.0.0 onward, releases follow [Semantic Versioning](https://semver.org/) and the version in `pyproject.toml` matches the git tag. Pre-1.0 milestones (Phase 1-5) were pinned by tag while `pyproject.toml` stayed at `0.0.1`. +## v1.3.0 — unreleased + +Minor release. A batch of user-facing features — singular alias forms, parallel +bulk writes, dynamic shell completion, and a strict `audit_redaction: full` +mode — plus a round of state-directory, workflow, and supply-chain hardening. + +### Added + +- **Singular alias forms** ([#6]). `nsc ls device` now resolves like + `nsc ls devices` for a curated set of nine resources: `device`, `prefix`, + `tenant`, `vlan`, `site`, `rack`, `interface`, `cable`, and `tag`. A + non-curated singular that does not resolve suggests the plural + (e.g. "Did you mean `devices`?") rather than guessing. +- **`--workers N` on bulk writes** ([#3]). Bulk write commands + (`create --ndjson`, `update --ndjson`, …) accept `--workers N` to run up to N + concurrent in-flight requests (default `1`, max `32`). Per-record + `--on-error` semantics are preserved, and the audit log stays one well-formed + line per record (writes are thread-safe). With `--on-error stop`, the bound + applies to *new* submissions — in-flight requests (up to N) are allowed to + complete. +- **Dynamic shell completion** ([#2]). Completion now reads the cached schema to + complete dynamic values: `nsc ls dev` → + `devices device-roles device-types`; `nsc --profile ` → profile names; + `nsc list --status ` → enum values. This complements the + existing static `--install-completion` stubs. +- **`audit_redaction: full` config mode** ([#5]). A new strict redaction mode + redacts ALL request/response bodies, leaving only + `{method, url, status_code, timestamp, profile}` per audit line. The `url` is + sanitized — the query string and any `user:pass@` credentials are stripped. + The default stays `safe`. + +### Changed + +- **`~/.nsc` state root hardened to `0700`** ([#90]). Defense-in-depth on the + state directory itself; config and audit files were already `0600` and + `logs/` already `0700`. +- **CI runners on Node.js 24** ([#9]). Bumped `actions/checkout` to v5, + `astral-sh/setup-uv` to v7, and `deploy-pages` to v5. +- **Least-privilege workflow permissions** ([#12]). Every GitHub Actions + workflow now declares an explicit least-privilege `permissions:` block. +- **Release workflow actions pinned to commit SHAs** ([#16]). `release.yml` + pins `checkout` and `setup-uv` to commit SHAs (the PyPA publish action stays + on its floating tag per PyPA guidance); added `.github/dependabot.yml` + (github-actions ecosystem, weekly). +- **E2E CI matrix over NetBox v4.5 and v4.6** ([#4]). The end-to-end job now + runs against both NetBox lines (`fail-fast: false`, per-version labels). + +[#6]: https://github.com/thomaschristory/netbox-super-cli/issues/6 +[#3]: https://github.com/thomaschristory/netbox-super-cli/issues/3 +[#2]: https://github.com/thomaschristory/netbox-super-cli/issues/2 +[#5]: https://github.com/thomaschristory/netbox-super-cli/issues/5 +[#90]: https://github.com/thomaschristory/netbox-super-cli/issues/90 +[#9]: https://github.com/thomaschristory/netbox-super-cli/issues/9 +[#12]: https://github.com/thomaschristory/netbox-super-cli/issues/12 +[#16]: https://github.com/thomaschristory/netbox-super-cli/issues/16 +[#4]: https://github.com/thomaschristory/netbox-super-cli/issues/4 + ## v1.2.1 — 2026-06-25 Patch release. Fixes foreign-key handling in the interactive TUI. diff --git a/CLAUDE.md b/CLAUDE.md index 087561a..3705dbe 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,6 +14,7 @@ This is the contributor guide for AI agents (and humans!) modifying this repo. T - `nsc/cache/` — disk cache for generated command-models. - `nsc/auth/` — login verification helpers (pre-flight probes, token rotate). - `nsc/aliases/` — curated alias resolver (`ls`, `get`, `rm`, `search`). Framework-free. +- `nsc/completion/` — dynamic shell-completion: framework-free providers + cheap on-disk cache probe + Typer `shell_complete` callbacks. - `nsc/skill/` — bundle-path helper for the portable `SKILL.md` shipped in the wheel. - `nsc/schemas/bundled/` — versioned NetBox OpenAPI snapshots, fallback when offline. diff --git a/README.md b/README.md index 289c2fe..3212c90 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,10 @@ uv run nsc dcim devices create -f device.yaml --apply uv run nsc dcim devices create -f devices.yaml --apply uv run nsc dcim devices create -f devices.yaml --no-bulk --on-error continue --apply +# Speed up a loop-mode bulk write with concurrent requests (default 1, max 32). +# Per-record --on-error semantics hold; the audit log stays one line per record. +uv run nsc dcim devices create -f devices.ndjson --workers 8 --apply + # NDJSON / JSONL — one record per line; parse failures abort the whole batch # before any wire request fires (`type: input_error`, exit 4, `details.bad_lines`). uv run nsc dcim devices create -f devices.ndjson --apply @@ -108,7 +112,26 @@ Stdin is sniffed from the first 512 bytes (first non-whitespace byte plus a one- - HTTP headers in the `SENSITIVE_HEADERS` set (e.g. `Authorization`, `X-API-Key`) — replaced with `""`. - Request-body fields whose OpenAPI definition has `format: password` OR whose name (case-insensitive) is one of: `password`, `secret`, `token`, `api_key`, `apikey`, `private_key`, `passphrase`, `client_secret`. Nested fields and arrays of objects are walked recursively. -The wire body sent to NetBox is **not** redacted — only the audit log. A failed write still records the redacted body; redaction is irreversible. Treat `audit.jsonl` like a verbose application log: gate it behind your home-directory permissions and rotate / archive accordingly. A "redact everything" mode is on the post-v1.0 roadmap. +The wire body sent to NetBox is **not** redacted — only the audit log. A failed write still records the redacted body; redaction is irreversible. Treat `audit.jsonl` like a verbose application log: gate it behind your home-directory permissions and rotate / archive accordingly. + +For a stricter posture, set `audit_redaction: full` in `~/.nsc/config.yaml` (the default is `safe`, the field-level redaction described above). In `full` mode every audit line keeps only `{method, url, status_code, timestamp, profile}` — all request and response bodies are dropped, and the `url` is sanitized (query string and any `user:pass@` credentials stripped). + +## Curated aliases + +`nsc ls`, `nsc get`, `nsc rm`, and `nsc search` are short verbs over the dynamic +command tree. + +```sh +nsc ls devices # list a resource by plural name +nsc ls device # singular also works (curated set, see below) +nsc get devices 7 # one record by id or name +nsc rm devices 7 --apply # delete (dry-run by default) +nsc search "rack-42" # /api/search/ on NetBox 4.5+ +``` + +Singular forms resolve for a curated set of nine resources: `device`, `prefix`, +`tenant`, `vlan`, `site`, `rack`, `interface`, `cable`, and `tag`. Any other +singular that does not resolve suggests its plural ("Did you mean `devices`?"). ## Output and errors @@ -154,7 +177,16 @@ nsc --install-completion # auto-detects $SHELL nsc --show-completion # prints the script instead of installing ``` -Typer supports `bash`, `zsh`, `fish`, and `pwsh`. Completion of dynamic values (resource names, profile names, filter keys) is on the post-1.0 roadmap. +Typer supports `bash`, `zsh`, `fish`, and `pwsh`. + +Completion of dynamic values reads the cached schema and complements the static +stubs above: + +```sh +nsc ls dev # devices device-roles device-types +nsc --profile # configured profile names +nsc dcim devices list --status # enum values for the field +``` ## Bundled Skill for AI agents diff --git a/docs/architecture/caching.md b/docs/architecture/caching.md index 18d3a40..ee34375 100644 --- a/docs/architecture/caching.md +++ b/docs/architecture/caching.md @@ -70,3 +70,12 @@ nsc cache prune --apply # actually delete nsc cache prune --max-age 30 --apply nsc cache prune --output json # structured envelope ``` + +## Shell completion reads the cache + +Dynamic shell completion (`nsc/completion/`) is backed by this same on-disk +cache. At TAB time the completion providers probe the cached command-model for +the active profile (`/.json`) — never the network — so +resource-name, profile, and enum suggestions are cheap. A missing or corrupt +cache degrades to no suggestions rather than blocking the prompt; the cache is +populated by the first real command (or `nsc login --fetch-schema`). diff --git a/docs/architecture/http-client.md b/docs/architecture/http-client.md index 17b4e1c..57107ed 100644 --- a/docs/architecture/http-client.md +++ b/docs/architecture/http-client.md @@ -22,6 +22,17 @@ - Pagination helper that follows `next` URLs (used by `--all`). - Audit log appender at `~/.nsc/logs/audit.jsonl` — written for writes always, and for any request when `--debug` is set. + +## Concurrency (`--workers N`) + +Bulk write commands accept `--workers N` (default 1, max 32) to keep up to N +requests in flight. Concurrency is **thread-based**: a `ThreadPoolExecutor` +fans the per-record loop out over the single sync `httpx.Client` — there is no +async path. Per-record `--on-error` semantics are preserved regardless of +worker count. Audit appends are serialized by a module-level lock +(`_APPEND_LOCK` in `nsc/http/audit.py`) wrapped around the whole +open/write/close, so concurrent workers can never interleave a partial line — +each record is one well-formed JSON line. - A "last request" snapshot at `~/.nsc/logs/last-request.json` (overwritten every call, regardless of `--debug`) — handy for triage. @@ -55,9 +66,26 @@ Each line of `audit.jsonl` is a JSON object with: - `duration_ms`, `attempt_n`, `final_attempt`, `error_kind` - `dry_run`, `preflight_blocked`, `record_indices`, `applied`, `explain` +## Redaction modes + +Redaction is applied when each entry is serialized (`nsc/http/audit.py`), so a +failed retry never unredacts. The `defaults.audit_redaction` config setting +selects the mode: + +- `safe` (default) — the full audit shape above, with sensitive headers and + `sensitive_paths` body fields masked to `""` and bodies over 256 KB + truncated. +- `full` — compliance escalation that drops every body, header, and query + string entirely. Each line keeps exactly five keys — + `{method, url, status_code, timestamp, profile}` — and `url` is sanitized to + scheme + host[:port] + path so neither query params nor `user:pass@` userinfo + can leak through the one remaining string field. + The audit file is append-only and rotates to `audit.jsonl.1` at 10 MB; it is created owner-only (`0600`) inside a `0700` logs directory, and failed writes do -NOT unredact. See +NOT unredact. The state root (`~/.nsc`) and its subdirectories are clamped to +`0700` via a shared `ensure_private_dir()` (`nsc/config/settings.py`) used by +both the config writer and the audit-dir code. See [Writes and safety](../guides/writes-and-safety.md) for the full redaction contract. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 629f2d3..804ef9f 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -17,6 +17,7 @@ nsc/ ├── cache/ # On-disk cache for generated CommandModels ├── auth/ # Login verification helpers (verify probe, token rotate) ├── aliases/ # Curated alias resolver (ls/get/rm/search) +├── completion/ # Dynamic shell-completion: providers + cache probe + Typer callbacks ├── skill/ # Bundle-path helper for the portable SKILL.md └── schemas/bundled/ # Versioned NetBox OpenAPI snapshots (offline fallback) ``` diff --git a/docs/contributing/development.md b/docs/contributing/development.md index ed3d7ad..d51f234 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -41,6 +41,23 @@ just docs-build # build the site, fail on broken links / missing nav underlying issue and create a new commit. If `ruff format` modifies a file during the hook, re-`git add` and re-commit. +## CI + +GitHub Actions mirrors the local `just` targets, plus an end-to-end suite: + +- **E2E is multi-version.** The e2e workflow runs a matrix over NetBox **4.5 + and 4.6** (latest patch of each minor) with `fail-fast: false`, so a + regression on one version surfaces as a per-dimension failure rather than a + sweeping red. +- **Runners are on Node.js 24** — the JS-based actions are pinned to their + Node-24 majors (`actions/checkout@v5`, `astral-sh/setup-uv@v7`, + `actions/deploy-pages@v5`). +- **Least-privilege permissions.** Every workflow declares an explicit + top-level `permissions:` block; the release workflow additionally SHA-pins + its actions (the PyPA publish action excepted). +- **Dependabot** watches the `github-actions` ecosystem on a weekly schedule + (`.github/dependabot.yml`) so action pins stay current. + ## Where the design lives - `docs/superpowers/specs/` — design specs (gitignored; local-only). diff --git a/docs/getting-started/concepts.md b/docs/getting-started/concepts.md index 33c0b03..3d87616 100644 --- a/docs/getting-started/concepts.md +++ b/docs/getting-started/concepts.md @@ -49,6 +49,12 @@ endpoints — without per-plugin code in `nsc`**. A short curated alias layer sits on top: `nsc ls`, `nsc get`, `nsc rm`, `nsc search`. +For the curated resources — `device`, `prefix`, `tenant`, `vlan`, `site`, +`rack`, `interface`, `cable`, `tag` — the singular works identically to the +plural: `nsc ls device` is the same as `nsc ls devices` (likewise for `get` and +`rm` where supported). A non-curated singular that doesn't resolve suggests the +plural instead ("Did you mean `devices`?"). + ## Dry-run / apply Writes default to dry-run. The CLI shows the resolved request and exits 0 diff --git a/docs/getting-started/first-run.md b/docs/getting-started/first-run.md index b378381..9432de6 100644 --- a/docs/getting-started/first-run.md +++ b/docs/getting-started/first-run.md @@ -74,6 +74,7 @@ nsc dcim devices list # default page (page_size=50) nsc dcim devices list --all # paginate to completion nsc dcim devices get 7 # get by id nsc ls devices # alias — matches devices.list +nsc ls device # singular works too for curated resources nsc dcim devices list --output json --all # canonical machine-readable form ``` diff --git a/docs/getting-started/install.md b/docs/getting-started/install.md index c8bab99..19ee45e 100644 --- a/docs/getting-started/install.md +++ b/docs/getting-started/install.md @@ -50,8 +50,16 @@ nsc --install-completion=pwsh Restart your shell, then verify with `nsc `. -> Dynamic completion (resource names, profile names, filter keys) is on the -> post-1.0 roadmap. +### Dynamic completion + +Once completion is installed, TAB expands real values from the cached schema — +no network call at TAB time: + +```sh +nsc ls dev # → devices device-roles device-types +nsc --profile # → profile names from ~/.nsc/config.yaml +nsc dcim devices list --status # → schema enum values for that filter +``` ## Requirements diff --git a/docs/guides/ci-and-automation.md b/docs/guides/ci-and-automation.md index 3d9f452..bc4bc27 100644 --- a/docs/guides/ci-and-automation.md +++ b/docs/guides/ci-and-automation.md @@ -21,6 +21,22 @@ This: - Emits machine-readable output to stdout. - Returns a stable exit code. +## Faster bulk loads with `--workers` + +When a payload loops record-by-record, `--workers N` runs up to N requests +concurrently (default `1`, max `32`), which can sharply cut wall-clock time for +large NDJSON loads: + +```sh +nsc dcim devices create -f new-devices.ndjson --apply --workers 8 \ + --output json --on-error continue +``` + +Raise it cautiously: NetBox may rate-limit or strain under high concurrency, so +tune N to what your instance tolerates. The `--on-error` semantics and the +one-line-per-record audit log are unchanged. See +[Writes and safety](writes-and-safety.md#parallel-bulk-writes-workers). + ## Exit-code-driven control flow ```sh diff --git a/docs/guides/managing-profiles.md b/docs/guides/managing-profiles.md index 0956176..247c53e 100644 --- a/docs/guides/managing-profiles.md +++ b/docs/guides/managing-profiles.md @@ -96,3 +96,9 @@ NSC_URL=https://other NSC_TOKEN=$TOK nsc dcim devices list # env vars also wor ``` Override precedence (highest first): CLI flag > env var > config profile. + +## Shell completion + +With shell completion installed, `nsc --profile ` completes configured +profile names (read from the cached command-model and config — no network at +TAB time). diff --git a/docs/guides/working-with-plugins.md b/docs/guides/working-with-plugins.md index ab995aa..7cb02b1 100644 --- a/docs/guides/working-with-plugins.md +++ b/docs/guides/working-with-plugins.md @@ -30,6 +30,16 @@ nsc commands --schema -o json | jq '.tags.my_plugin' # full subt document) and emits JSON only. The dumped tree is `{info_title, info_version, schema_hash, tags: {: {resources: ...}}}`. +## Aliases across plugins + +The tag-skipping aliases (`ls`, `get`, `rm`, `search`) resolve a resource name +across **every** tag in the schema, plugin tags included — so +`nsc ls widgets` finds `my_plugin widgets` without naming the tag, as long as +the name is unambiguous. Singular forms only apply to the curated core +resources (`device`, `prefix`, `tenant`, `vlan`, `site`, `rack`, `interface`, +`cable`, `tag`); plugin resources are matched by their plural name as it +appears in the schema. + ## Schema cache and plugin upgrades The cache is keyed by the schema's SHA-256 hash, stored at diff --git a/docs/guides/writes-and-safety.md b/docs/guides/writes-and-safety.md index 473087b..103aa3f 100644 --- a/docs/guides/writes-and-safety.md +++ b/docs/guides/writes-and-safety.md @@ -47,6 +47,24 @@ nsc dcim devices create -f devices.ndjson --on-error continue --apply # don't `--on-error continue` collects per-record failures, reports them in a summary envelope, and exits with the worst error type's code. +### Parallel bulk writes (`--workers`) + +When `nsc` loops record-by-record (no bulk endpoint, or `--no-bulk`), +`--workers N` runs up to N concurrent in-flight requests: + +```sh +nsc dcim devices create -f devices.ndjson --workers 8 --apply +``` + +The default is `1` (sequential — unchanged behaviour); the maximum is **32** +(a higher value is rejected). The per-record `--on-error continue`/`stop` +semantics are preserved, and the audit log still gets one well-formed JSON line +per record. + +Note that `--on-error stop` bounds only *new* submissions: in-flight requests +(up to N) still complete. So when the record count is ≤ `--workers`, every +record is sent regardless of `stop`. + ## Input formats | Extension | Format | diff --git a/docs/index.md b/docs/index.md index 648dce5..1e4ad62 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,8 +8,15 @@ because the schema — not hand-written code — defines the surface. - **Plugins just work.** New endpoints from any installed plugin appear as commands automatically. - **Multi-instance.** Named profiles per NetBox instance, plus env-var overrides. -- **Safe by default.** Writes preview as dry-runs unless you pass `--apply`. +- **Safe by default.** Writes preview as dry-runs unless you pass `--apply`. An + `audit_redaction: full` mode strips every request/response body from the audit + log, leaving only `{method, url, status_code, timestamp, profile}`. +- **Fast bulk writes.** `--workers N` runs up to N concurrent requests on a + bulk write (default `1`, max `32`), with per-record error handling intact. - **Agent-friendly.** Deterministic command shape, machine-readable JSON output, stable error envelope with documented exit codes. +- **Curated aliases + completion.** `nsc ls/get/rm/search` over any resource — + with singular forms for common ones (`nsc ls device`) — and dynamic shell + completion of resource, profile, and enum values from the cached schema. - **Interactive TUI.** `nsc tui` opens a keyboard-driven UI to browse, filter, edit, bulk-edit and search — see the [Interactive TUI guide](guides/interactive-tui.md). ## Three killer examples diff --git a/skills/netbox-super-cli/SKILL.md b/skills/netbox-super-cli/SKILL.md index 2832ee2..d49b2c7 100644 --- a/skills/netbox-super-cli/SKILL.md +++ b/skills/netbox-super-cli/SKILL.md @@ -15,7 +15,11 @@ endpoint list to go stale. - `nsc` is on PATH after `pip install netbox-super-cli`. - Config lives at `~/.nsc/config.yaml`. - Audit log lives at `~/.nsc/logs/audit.jsonl` (one JSON-line per request/response; - passwords are redacted). + passwords are redacted). `defaults.audit_redaction` controls how much is + kept: `safe` (default) masks sensitive headers/fields but keeps bodies; + `full` drops all bodies/headers/query and leaves only + `{method, url, status_code, timestamp, profile}` (url stripped of query and + `user:pass@`). - Cache lives at `~/.nsc/cache//.json`. ## Command shape @@ -52,6 +56,11 @@ Plus interactive meta-commands (not aliases): - `nsc commands --schema ` — dump the entire generated command tree as JSON (useful for discovery; `--schema` is required here). +With shell completion installed, TAB now expands resource names, profiles +(`nsc --profile `), and `--status` enum values. Completion reads the +on-disk cache only — no network call at TAB time — and silently offers +nothing if the cache is missing. + ## Dry-run / apply discipline ALL writes are dry-runs by default. The `--apply` flag is the only path to @@ -192,6 +201,16 @@ operation. ``` - **Generate the NDJSON once** with whatever scripting tool you prefer (`jq`, Python, awk). Don't re-run `list` between every patch. +- **Speed up large bulk writes with `--workers N`.** Bulk write commands + accept `--workers N` (default **1**, max **32**) to keep N requests in + flight at once instead of one-at-a-time: + ```bash + nsc dcim interfaces update -f changes.ndjson --workers 8 --apply + ``` + Per-record `--on-error` semantics are unchanged at any worker count, and + the audit log stays intact — appends are serialized so each record is + still exactly one well-formed JSON line. Start at the default and raise + it only when the record count is large; respect the 32 ceiling. ### Schema fetches