From d1d11b8a2048eb90094c97977ba60257c13520f8 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Sat, 1 Aug 2026 00:28:41 +0530 Subject: [PATCH 01/16] docs: add VM discovery phase 0 design, ticket plan, and epic draft Design for making VMs first-class Nudgebee resources: per-segment forager discovery (sweep, hypervisor, LDAP, SSH inventory), signed content packs, server-side reconciliation and coverage reporting. Referenced by epic nudgebee/nudgebee-enterprise#35404 and tickets #113-#116. --- docs/design/vm-discovery-phase0-epic.md | 100 +++++ docs/design/vm-discovery-phase0-tickets.md | 163 +++++++ docs/design/vm-discovery-phase0.md | 489 +++++++++++++++++++++ 3 files changed, 752 insertions(+) create mode 100644 docs/design/vm-discovery-phase0-epic.md create mode 100644 docs/design/vm-discovery-phase0-tickets.md create mode 100644 docs/design/vm-discovery-phase0.md diff --git a/docs/design/vm-discovery-phase0-epic.md b/docs/design/vm-discovery-phase0-epic.md new file mode 100644 index 0000000..9a79a1a --- /dev/null +++ b/docs/design/vm-discovery-phase0-epic.md @@ -0,0 +1,100 @@ +# Epic 1: VM Discovery & Package Inventory + +First epic of the **VM Patch & Vulnerability Management** initiative: + +| Epic | What ships | Customer's naming | +|------|-----------|-------------------| +| **1. Discovery & package inventory (this one)** | Every VM found, packages cached, coverage report | first half of their "Phase 1" | +| 2. Vulnerability matching | Packages → CVEs (vendor advisories), escalation policy (CVSS/urgency config) | second half of their "Phase 1" | +| 3. Patching | Subscription-based patch apply, approval flow, maintenance windows | their "Phase 2" | +| 4. Hardening & insight | CIS benchmarking, AI impact analysis of vulns on applications | later asks | + +Epics 2–4 are sequenced, not speculative — 2 needs 1's package data, +3 needs 2's vuln data. We only break tickets down for epic 1 now. + +## Summary + +Make VMs first-class resources in Nudgebee — the same way we treat +cloud resources and Kubernetes workloads today. This epic answers +three questions for every customer: + +1. How many VMs do you have? +2. What packages are installed on each one? +3. For every VM we couldn't inventory — why not? + +This is the foundation for patch & vulnerability management (the +customer ask), and later for observability and automation on VMs. + +## Why + +A customer wants patch and vulnerability management for their VMs, +including on-premise machines where no cloud API exists. Nothing else +in that product can ship until we reliably know which VMs exist and +what's installed on them. No vendor today does this for Linux without +installing an agent on every VM — that gap is our opening. + +## How (in one paragraph) + +We install nothing on the VMs. One forager node per network segment +does all the work: it pings the network to find machines, asks the +hypervisor and Active Directory for their lists, and logs into each VM +over SSH with a read-only user to collect the OS and package list. +The commands it runs come from a signed, versioned "content pack" +downloaded from our server — so changing what we collect never +requires a new forager release. All data flows to the server, which +merges the different sightings into one asset list and shows a +coverage report. + +## Definition of done + +The coverage report is live and correct for a real fleet: + +- Every VM appears with a state: **discovered** (we know it exists), + **reachable** (SSH login works), or **inventoried** (package list + collected). +- Every gap has a reason (no credential, powered off, EOL OS, stale…). +- Works on CentOS/RHEL, Ubuntu/Debian, SUSE, Alpine. Windows is out + of scope for now. + +## Out of scope (later phases) + +CVE matching, patching, escalation policy, CIS benchmarking, Windows, +application-level (non-OS) packages. + +## Tickets + +| # | Ticket | Where | Note | +|---|--------|-------|------| +| P1 | Forager: SSH inventory + signed content-pack runner | forager | no dependencies | +| P2 | Server: asset storage + ingest + parsers + dedup logic | nudgebee | no dependencies | +| P3 | Forager: network sweep + Active Directory lookup | forager | after P1+P2 demo | +| P4 | Forager: hypervisor connector (vCenter/Proxmox/libvirt) | forager | **blocked: customer must tell us their hypervisor** | +| P5 | Server: scheduler + coverage states + merge cloud VMs | nudgebee | after P1+P2 demo | +| P6 | Content pack v1 + signing/publish pipeline | content | with P5 | +| P7 | UI: asset list + coverage report screen | nudgebee | last | +| P8 | Customer setup docs (SSH user, firewall/IDS allowlist) | nudgebee-docs | alongside P6/P7 | + +Milestone 0 = P1 + P2 together: one VM inventoried end-to-end +(signed pack → SSH collection → server → package list in the DB). +Prove the pipe before building breadth. + +Storage is an open discussion, not decided: prefer extending the +existing resource model (`cloud_resources` etc.) so features that +already work on resources — troubleshooting, automation, UI — +pick up VMs automatically; add new tables only for data that has no +home today (package cache, identity merge keys). + +## Open questions for the customer (send now) + +1. Which hypervisor do you run — VMware, Proxmox, Hyper-V, plain KVM? + (blocks P4) +2. Roughly how many VMs and network segments? +3. Can we get one read-only SSH credential per segment, or do you use + a vault (CyberArk etc.)? +4. What share of the fleet is Windows? +5. Any restriction on how the forager downloads content packs + (HTTPS out, or must everything ride the existing relay channel)? + +Details behind this epic: `vm-discovery-phase0.md` (design), +`vm-discovery-phase0-tickets.md` (full ticket breakdown with +acceptance criteria). diff --git a/docs/design/vm-discovery-phase0-tickets.md b/docs/design/vm-discovery-phase0-tickets.md new file mode 100644 index 0000000..9920c1e --- /dev/null +++ b/docs/design/vm-discovery-phase0-tickets.md @@ -0,0 +1,163 @@ +# Ticket Plan: VM Discovery & Package Inventory (Phase 0) + +Derived from `vm-discovery-phase0.md`. One epic, 8 tickets. Boundary +rule: one ticket = one owner + one repo/service + one demoable +deliverable. Spikes are the first tasks inside a ticket, not tickets. + +**Epic: VMs as first-class Nudgebee resources — Phase 0 discovery & +package inventory.** Done when the coverage report (§11) is correct +against a real fleet: every VM is `discovered`, `reachable`, or +`inventoried`, and every gap has a reason. + +**Prerequisite action (not a ticket):** customer answers to §13 — +hypervisor stack (hard-blocks P4), fleet size, credential model, +Windows %, pack distribution channel. Send immediately; only P4 +waits on the answers. + +**Open decisions — explore inside the ticket, nothing pre-decided:** + +- Storage model: extend `cloud_resources`/resource model vs new + tables (P2, design doc §8.1). +- Content-pack format & schema: the P1 spike *proposes* a format for + team review; the YAML sketch in §7 is illustrative only. +- Pack distribution channel: HTTPS vs existing relay WSS (P6 + + customer question 5). +- Sweep implementation: plain TCP-connect vs a library like naabu; + safe-scan defaults (P3). +- Hypervisor connector: which one first (P4, customer answer). +- Scheduler: reuse the scan_orchestrator pattern vs something new + (P5). +- Concurrency/cadence/staleness defaults: tune from customer fleet + size, not fixed in this doc. + +Non-negotiable invariants (correctness, not approach — these hold +whatever we pick): weak identifiers never merge assets alone; raw +observations stay immutable/replayable; package versions stored +verbatim incl. epoch/release; content packs signed + pinnable; no +software installed on target VMs. + +--- + +## P1 — Forager: discovery module + content-pack runner + SSH inventory +Repo: forager. New `pkg/proxy/discovery` module, registered like +existing proxies. + +Tasks: (a) spike the content-pack design — output is a *proposed* +format for team review (candidate fields: version, signature, +collectors, `when` exprs, a `kind` axis so observability/automation +can reuse the runner later); signing reuses the existing Ed25519 +trust root; verbatim execution, framed per-collector output; +(b) spike SSH executor concurrency on `pkg/proxy/ssh` plumbing — +target iteration, per-target timeout, output caps, bounded +concurrency; (c) `discovery_inventory` action (targets[], +credential_ref, content_pack_ref) wiring both; credentials via +`pkg/secrets`. + +AC: tampered pack rejected, unknown `when` var fails closed; 50–100 +concurrent targets without fd/memory blowup, per-target status on +partial failure; a signed action from relay inventories a static +target list on ≥2 distro families; credentials never in logs or +responses. + +## P2 — Server: asset model, ingest, parsers, reconciler +Repo: nudgebee (api-server + relay-server). + +Tasks: (a) storage decision first (design doc §8.1 is OPEN): extend +the existing `cloud_resources`/resource model so existing features +(troubleshooting, automation, UI) pick VMs up automatically, vs new +dedicated tables — then migrations only for what has no home today +(identities for merge logic, raw observations, packages, +subscriptions, static EOL data); +(b) relay `asset_inventory` handler (new upward action, not +`datasource_inventory`) — batch insert, size limits; (c) parsers: +os-release, rpm (epoch/release verbatim), dpkg, apk, +subscription/repo, reboot-pending, identity — golden-file tests from +real distro output (CentOS 7, RHEL 9, Ubuntu 20.04/22.04/24.04, +Debian 12, SLES 15, Alpine); (d) reconciler v1 — strong-id merge, +weak-id corroborate only, staleness decay, replay from observations. + +AC: weak identifiers cannot merge alone (schema-enforced); two +observations sharing machine-id merge, two sharing only IP don't; +rule fix + replay loses no data; epoch/release survive round-trip; +malformed observation rejected without dropping its batch. + +**M0 exit = P1 + P2 demo:** one host end-to-end — signed pack, SSH +collection, relay ingest, parsed packages in Postgres. + +## P3 — Forager: discovery sources — sweep + LDAP +Repo: forager. + +Tasks: (a) `discovery_sweep` — ARP local L2, ICMP, TCP 22/3389/5985 +over configured CIDRs; opt-in per CIDR, exclusions, hard rate cap +(default ≤100 pps), well-formed packets, windows honored; decide +TCP-connect vs naabu inside the ticket; (b) `discovery_ldap` — +read-only AD bind (go-ldap), filter `lastLogonTimestamp` tombstones. + +AC: /24 sweep returns IP/MAC/rDNS/open-ports; rate cap verified by +packet capture; exclusions honored; objectGUID lands as STRONG +identity. + +## P4 — Forager: hypervisor connector (first one only) +Repo: forager. **Blocked on customer answer.** Build exactly one of: +vcenter (govmomi) | proxmox (REST) | libvirt-over-SSH. Datasource +type + read-only credential via `pkg/secrets`. + +AC: returns UUID, name, power state, guest OS, IPs **including +powered-off VMs**; powered-off reaches `assets.power_state`. + +## P5 — Server: scheduler, coverage states, cloud projection +Repo: nudgebee (api-server). + +Tasks: (a) scheduler — server-side, forager holds zero schedule +state; candidate: reuse the scan_orchestrator pattern (confirm at +kickoff): sweep cadence per CIDR, hypervisor poll, per-asset +inventory cadence, windows, concurrency caps; (b) coverage-state machine — `discovered → +reachable → inventoried` tracked independently, gap-reason taxonomy +(ssh-refused, ssh-auth-failed, stale, powered-off, unsupported-os, +eol-no-repo) denormalized onto assets; (c) project existing +cloud_resources VMs into `asset_observations` (instance-id STRONG) so +cloud and on-prem VMs merge into one asset universe. + +AC: window/cap violations impossible by construction; every +non-inventoried asset has exactly one primary gap reason; an EC2 +instance also seen via SSH is one asset, not two. + +## P6 — Content pack v1 + publish pipeline +Repo: content (+ server config). `linux-inventory` covering the §6 +matrix (RHEL-like, Debian-like, SUSE, Alpine); CI signing; +distribution channel (default HTTPS unless customer answer says +otherwise); tenant pinning + ring rollout server-side. + +AC: pack version visible per tenant; pinned tenant never receives a +newer pack; new version reaches ring 0 before ring 1. + +## P7 — UI: asset list + coverage report +Repo: nudgebee (UI). The §11 screen: totals, per-state counts, gap +breakdown, per-asset drill-down (identities, packages, subscription +binding, sources, last_seen), EOL flags. + +AC: numbers reconcile with DB counts; gap rows link to affected +assets. + +## P8 — Onboarding docs +Repo: nudgebee-docs. `nudgebee-ro` user + SSH key setup (optional +narrow sudoers for dmidecode), hypervisor read-only role, IDS +whitelisting of forager IP, CIDR opt-in guidance. + +AC: a customer goes from zero to first coverage report using only +this doc. Not an afterthought — Rapid7's lesson (§2 finding 9) is +that credential management is the operational cost center. + +--- + +## Dependencies + +``` +P1 ─┬─► M0 exit ─► P3, P5 ─► P6, P7 +P2 ─┘ │ +customer answer ─► P4┘ P8 alongside P6/P7 +``` + +P1 and P2 have no dependencies and run in parallel. P4 is the only +customer-blocked ticket. P8 can start once P1's credential model is +settled. diff --git a/docs/design/vm-discovery-phase0.md b/docs/design/vm-discovery-phase0.md new file mode 100644 index 0000000..7d8735f --- /dev/null +++ b/docs/design/vm-discovery-phase0.md @@ -0,0 +1,489 @@ +# Design: VM Discovery & Package Inventory (Phase 0) + +Status: DRAFT +Scope: Phase 0 — VM discovery, package inventory, reconciliation, +coverage reporting. Discovery makes VMs first-class Nudgebee +resources; patch & vulnerability management is the first consumer. +Out of scope (later phases): CVE matching, escalation policy, patching, +CIS benchmarking, application (non-OS-package) inventory, Windows. + +--- + +## 1. Problem + +Customers need patch & vulnerability management for VMs, starting with +on-premise / self-hosted fleets where no cloud control plane exists. +Before anything else can ship, we must answer: + +> How many VMs exist, which packages are installed on each, and for +> every VM we can't inventory — why not? + +Constraints: + +- **No agent on target VMs.** Rollout friction kills adoption; agent + upgrade management is a liability (post-CrowdStrike, staged rollout + and version pinning are procurement questions). +- **Minimal, rarely-upgraded footprint.** The only installed software + is one forager node per network segment. +- **On-prem first.** Cloud APIs (EC2/Azure/GCE) are optional + enrichment, not a dependency. + +Product framing: a discovered VM becomes a first-class Nudgebee +resource — the same status cloud resources and Kubernetes workloads +have today. Discovery is the foundation; independent capability +tracks build on the VM resource: + +1. **Observability** — node health, metrics, logs, service state. +2. **Security** — package inventory → CVE matching, patching, CIS + benchmarking (the customer ask driving this doc). +3. **Automation** — signed, audited command execution on VMs through + the per-segment forager; patching is one instance of it. + +Phase 0 therefore avoids patch-specific shortcuts: the asset tables +are a general resource catalog (packages are one satellite dataset), +and signed content packs are the generic execution vehicle all three +tracks reuse. + +## 2. Industry research summary + +Surveyed: Qualys, Tenable, Rapid7, Tanium, osquery/Fleet, AWS SSM, +Red Hat Satellite, WSUS, Ansible, Lansweeper, runZero, Wiz/Orca, +Axonius, PatchMon, CrowdStrike, JFrog, and the dedicated +patch-management pack (Automox, Action1, NinjaOne, ManageEngine, +Ivanti). Findings that drive this design: + +1. **Every mature tool runs the same four-stage pipeline:** seed from + authoritative sources (hypervisor/AD/cloud) → unauthenticated sweep + → authenticated inventory → continuous reconciliation. Tools differ + only in how stage 3 reaches the box. +2. **Sweeps find, they don't inventory.** Unauthenticated scanning + tops out at fingerprints. Package data requires credentials or a + resident channel — no exceptions in the industry. +3. **Only control planes see powered-off machines** (hypervisor, + cloud, AD last-logon). Scanning can't; tools handle absence with + staleness decay. +4. **Reconciliation is the product; collection is commodity.** Asset + duplication is the #1 operational complaint against Qualys/Tenable; + Axonius built a company purely on reconciling other tools' + inventories. +5. **Nobody serious ships a full-logic agent per VM anymore.** Red Hat + deprecated and removed katello-agent in favor of OS-native + reporting (`subscription-manager` package-profile upload). +6. **The converged answer to "minimal agent, no upgrades" is three + layers:** a per-segment node (Nessus scanner model) + agentless + push over SSH (Ansible model) + **signed content-driven execution** + (Qualys manifests, Tanium sensors, osquery query packs) — the agent + binary is a frozen interpreter; capability changes ship as signed + content. +7. **Updates need pinning + staged rollout as first-class features** — + even content updates, post-CrowdStrike. The Channel File 291 RCA is + the proof: content updates **bypassed** the N-1/N-2 pinning + customers had on sensor binaries; the post-incident fixes were + staged/canary content rollout and customer control over content + scheduling — exactly the pack pinning + ring rollout in §7 of this + doc. The split is industry consensus: Rapid7's Insight Agent keeps + shipping content when binary updates are disabled; Ivanti moved + Linux to "contentless" detection (query the distro's own repos); + ManageEngine syncs a central patch DB independent of agent + versions. +8. **Dedicated patch vendors don't solve discovery.** Automox, + Action1, NinjaOne, and ManageEngine are per-endpoint agent + + enrollment; their answer to "machines we don't know about" is + scanning AD in order to push more agents. Ivanti Security Controls + is the only central-node agentless inventory — Windows-only, over + admin shares/remote registry. **No vendor ships agentless SSH + Linux inventory from a per-segment collector** — the exact + combination this design proposes is open ground. +9. **Rapid7 validates the per-segment model and flags its two + hazards.** "Assigning a Scan Engine to each subnetwork is a best + practice" (stateless engines, hub-and-spoke to the console). But: + (a) remote credential management hurt enough that they shipped + Scan Assistant, a tiny cert-authenticated on-host helper — expect + credentials to be the operational cost center and design the vault + integration well; (b) their agent/engine asset correlation relies + on a UDP-31400 probe plus hostname/IP/MAC/UUID matching, and + produces duplicates when it fails — deterministic identity + (machine-id/SMBIOS first) must be built in up front, which §8.2 + does. +10. **Confirmed dead ends.** Snapshot side-scanning (Wiz/Orca) does + not extend on-prem: Wiz's vSphere support is vCenter-API-only + with guest package scanning unshipped, and Orca covered on-prem + by introducing a runtime agent — abandoning agentless. JFrog has + no machine discovery anywhere in its portfolio (Xray scans + artifacts in Artifactory; Runtime is a Helm-installed K8s eBPF + sensor; Connect is per-device token enrollment) — it is an + artifact-layer product, not a VM-discovery comparable. + +Forager already implements the per-segment-node and SSH-push layers +(outbound-only WSS, Ed25519-signed actions, SSH executor, datasource +auto-registration). The missing pattern is signed-content collection. + +## 3. Build vs fork: why not adopt an existing OSS scanner + +Evaluated forking/adopting an open-source project instead of extending +forager. Conclusion: **no fork; reuse at the library and server +layers.** + +### Candidates + +| Project | What it does | Why not a fork base | +|---|---|---| +| **Vuls** (future-architect) | Closest match: Go, agentless SSH scan, package collection + CVE matching for CentOS/Ubuntu/RHEL/Debian/SUSE/Alpine/Amazon | **GPL-3.0** — forking into our agent makes the combined work GPL. Local TOML config model (ours is cloud-pushed), no relay/transport, per-distro logic baked into the binary — the exact thing we're avoiding | +| **cnquery** (Mondoo) | Agentless SSH inventory, polished | **BUSL-1.1** — source-available license that prohibits building a competing commercial product on it. We are the prohibited use | +| **Wazuh** | Inventory + vuln detection | Per-VM agent (violates constraint), GPLv2, huge | +| **OpenVAS/GVM** | Full scanner appliance | GPL, C, enormous; a second heavyweight deployable in the customer env kills the "one small thing per segment" story | +| **osquery/Fleet** | Best-in-class inventory | Per-VM agent — wrong model | +| **Ansible** | Agentless push | GPLv3, Python runtime dependency on the node, we'd use 2% of it | +| **PatchMon** | Patch monitoring/automation: pending updates, patch policies with windows + dry-run, approval-gated execution, OpenSCAP CIS | **AGPL-3.0** — network copyleft, binds on SaaS use, worst license in the survey. Per-host agent (10 binary variants to maintain fleet-wide) — the rejected model. And **no discovery at all**: hosts exist only once the agent is installed; coverage accounting is impossible in its model | + +### Why fork economics are bad here + +1. **A fork saves the cheap part.** The collection layer is + `rpm -qa` / `dpkg-query -W` / `cat /etc/os-release` over SSH — a + few hundred lines against forager's existing SSH client. +2. **A fork doesn't help with the hard part.** Reconciliation + + coverage accounting is the product (finding #4), it's server-side, + and no OSS project ships it in reusable form. +3. **We'd port the fork onto forager anyway.** Transport, signing, + secrets, outbound-only WSS, datasource registration — the fork has + none of it; forager has all of it. Integration work exceeds the + collection code saved. +4. **A fork works against the no-upgrade goal.** Vuls' per-distro + logic lives in the binary — every distro quirk is a binary release. + The content-pack architecture (§7) is strictly better. +5. **Forks rot.** We would own the divergence forever; upstream fixes + conflict with our patches. + +### Where OSS reuse is right + +- **Libraries in forager** (license-clean): `govmomi` (Apache-2.0) + for vCenter, `go-ldap` (MIT) for AD, `naabu` (MIT) or ~300 + hand-rolled lines for the sweep. Do **not** shell out to nmap — its + NPSL license is a known problem for commercial redistribution. +- **Server-side, Phase 1:** Trivy or Grype (both Apache-2.0) for CVE + matching against the package cache — Trivy is already in + scan_orchestrator. Biggest genuine reuse win; zero agent impact. +- **Vuls as a reference, not a dependency:** its code encodes years of + per-distro edge cases (epoch handling, CentOS vault, backport + version formats). Read to design content packs; don't copy code. +- **SBOM output** (CycloneDX/SPDX) for the package cache so + third-party tools interop for free. +- Acceptable one-off: running Vuls standalone for a pre-sales demo. + Throwaway only, not a foundation. +- **PatchMon as a phases-1–3 UX reference** (same status as Vuls — + read, don't reuse): its policy model (immediate/delayed/maintenance + window), dry-run → approval → full-shell-output audit flow, and + per-package-manager pending-update handling are a good benchmark. + Also a competitor to track. + +## 4. Architecture summary + +``` +┌─────────────────────────┐ ┌────────────────────────────────────┐ +│ Nudgebee Cloud │ │ Customer Site / Segment │ +│ │ wss │ │ +│ Scheduler ─► Relay ◄───┼──────────┼─► Forager (1 per segment) │ +│ │ ▲ │ │ │ │ +│ Reconciler │ │ │ ├─ sweep: ARP/ICMP/TCP probe │ +│ │ asset_ │ │ ├─ hypervisor: vCenter/Proxmox │ +│ assets DB inventory │ │ ├─ ldap: AD computer objects │ +│ │ │ └─ inventory: SSH push ──► VMs │ +└─────────────────────────┘ │ (nothing installed) │ + └────────────────────────────────────┘ +``` + +- **Forager = per-segment discovery node.** Sweeps its segment, + queries the hypervisor/directory, collects per-VM inventory over + SSH (run commands, capture output, leave nothing behind). +- **Target VMs get nothing installed.** Their "agent" is sshd + the + package manager the OS already has. +- **Collection logic ships as signed content packs** (§7); the forager + binary stays frozen. +- **All policy lives server-side:** schedules, CIDR scopes, rate + limits, reconciliation. Forager initiates nothing on its own. +- Per-VM forager install remains a *fallback* for hosts where SSH is + blocked by policy — an option, not the model. + +## 5. Discovery sources (authority ladder) + +Reconciled, not either/or. Ordered by authority: + +| # | Source | Sees | Identifier quality | Notes | +|---|--------|------|--------------------|-------| +| 1 | Hypervisor API — vCenter, Proxmox, libvirt-via-SSH | All VMs incl. **powered-off** | Strong (instance/SMBIOS UUID) | The on-prem "control plane"; only true denominator | +| 2 | Directory/inventory — AD LDAP, DNS zones, DHCP leases, Ansible inventory, node_exporter targets | Recorded hosts | Weak (FQDN/IP/MAC) | Cheap, high yield; corroboration only | +| 3 | Network sweep — ARP (local L2), ICMP + TCP 22/3389/5985 across configured CIDRs | Live, reachable hosts | Weak until inventoried | Finds the unrecorded box; noisiest source | +| 4 | Cloud APIs — existing cloud-collector (EC2/Azure/GCE) | Cloud VMs | Strong (instance-id) | Optional enrichment where accounts exist | +| 5 | Authenticated inventory (SSH) | Everything about a reachable host | **Strong (machine-id, SMBIOS UUID)** | The only source of package data | + +Sweep safety (product requirements, not niceties): + +- Opt-in per CIDR; explicit exclusion lists. +- Hard rate cap (default ≤100 pps); well-formed packets only — + malformed nmap-style probes destabilize embedded/OT gear. +- Configurable scan windows. +- Onboarding docs must tell customers to whitelist the forager IP in + their IDS — an unannounced sweep trips Darktrace/Suricata and + generates a security incident. + +**Powered-off VMs:** only source 1 sees them. Without a hypervisor +integration, an off host does not exist to us. Model this honestly +with staleness decay: `last_seen` ages → `stale` after N days → +`presumed-retired` after M, surfaced in the coverage report. The claim +is "not observed since X," never a false "doesn't exist." + +## 6. Targeted OS matrix (v1) + +Linux only. Windows (WinRM/WMI) explicitly deferred. + +| Family | Distros | Package query | Patch channel binding | +|--------|---------|---------------|----------------------| +| RHEL-like | CentOS 7/Stream, RHEL 7–9, Rocky, Alma, Oracle, Amazon Linux 2/2023 | `rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}\t%{RELEASE}\t%{ARCH}\t%{INSTALLTIME}\n'` | `subscription-manager status` + `identity` (RHEL); `yum repolist -q` / `dnf repolist` | +| Debian-like | Ubuntu 18.04–24.04, Debian 10–12 | `dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${db:Status-Status}\n'` | `pro status --format json` (Ubuntu ESM); `/etc/apt/sources.list{,.d/}` | +| SUSE | SLES 12/15, openSUSE Leap | `rpm -qa` (as RHEL-like) | `SUSEConnect --status-text`; `zypper lr -u` | +| Alpine | 3.x | `apk info -v` | `/etc/apk/repositories` | + +Full per-host collection set (one SSH session, one batched script from +the content pack): + +- **Identity:** `/etc/machine-id`; SMBIOS UUID via + `/sys/class/dmi/id/product_uuid` (root) with `dmidecode -s + system-uuid` fallback if sudo-permitted — degrade gracefully to + machine-id only; hostname/FQDN; MACs (`ip -o link`). +- **OS:** `/etc/os-release` (ID, VERSION_ID, PRETTY_NAME), + `uname -r -m`. +- **Packages:** per family above. Version strings kept **verbatim** + incl. epoch and release — required for backport-aware CVE matching + in Phase 1 (vendor OVAL, not NVD CPE); never normalize away the + distro release suffix. +- **Subscription/repo binding:** per family above. Captured now + because a lapsed subscription or dead repo means the host **cannot + be patched** — that surfaces in Phase 0's gap report, not at patch + time in Phase 2. +- **Reboot-pending signal:** `/var/run/reboot-required` (Debian-like), + `dnf needs-restarting -r` exit code (RHEL-like, if present). Cheap + now, needed by Phase 2. + +Requirements on targets: reachable sshd + one scoped credential (§9). +No python dependency (unlike Ansible), no temp files — commands run +directly, output streams back. + +**EOL reality check:** CentOS 7 (EOL June 2024) and CentOS 8 are +common in exactly the fleets that buy this. Inventory works +identically; Phase 0 records `os_eol: true` from a small static EOL +table so the coverage report can say "N hosts are EOL — no vendor +patches exist for anything on them." + +## 7. Signed content packs (the no-upgrade mechanism) + +The per-OS collection commands in §6 do **not** live in the forager +binary. They ship as a versioned, Ed25519-signed content pack fetched +from the cloud (same trust root as action signing). Format below is +illustrative — the actual schema is the P1 spike's output, reviewed +by the team: + +```yaml +# content-pack: linux-inventory v3 +version: 3 +signature: +collectors: + - id: os-release # every host + cmd: cat /etc/os-release + - id: pkgs-rpm + when: os_family == "rhel" + cmd: rpm -qa --qf '...' + - id: pkgs-dpkg + when: os_family == "debian" + cmd: dpkg-query -W -f '...' + # ... +``` + +Rules: + +- Forager validates the signature, then executes collectors verbatim; + it contains **no per-distro logic** beyond a tiny `when`-expression + evaluator and output framing. +- Adding a distro, fixing a collector, adding a fact = publish a new + pack version. No binary release. +- **Pinning + staged rollout are first-class:** tenants can pin a pack + version; new versions roll out by ring. Content is an update surface + too — treat it with the same care as binaries. +- **Output parsing happens server-side.** The agent returns raw framed + output per collector. Parsers (the code that actually churns) never + ship in the agent. + +## 8. Server side + +### 8.1 Data model (OPEN — for discussion, not final) + +Two directions; decide at P2 kickoff, not in this doc: + +- **Option A — reuse the existing resource model.** VMs become + another resource kind in `cloud_resources` and the existing + datasource/inventory plumbing, the way other resource types + already work. Big advantage: everything that already consumes + resources — troubleshooting, automation, UI listing — infers VMs + with little or no new plumbing. +- **Option B — dedicated asset tables.** The sketch below. Cleaner + for identity reconciliation and package data, but new plumbing + everywhere. + +Likely landing point: A for the VM resource itself, with new +satellite tables only where nothing exists today (identities for +merge logic, packages, subscriptions). The sketch below shows the +data we must hold, not a schema decision. + +``` +assets -- one row per real machine + id, tenant_id, cloud_account_id? + hostname, fqdn, os_family, os_version, os_eol, kernel, arch + env, criticality, owner -- from tags/CMDB when known + power_state, first_seen, last_seen + coverage_state, gap_reason -- denormalized for the report + +asset_identities -- merge keys; many per asset + asset_id, kind, value, source, observed_at + -- kinds: machine_id | smbios_uuid | instance_id | ad_objectguid (STRONG) + -- fqdn | ip | mac (WEAK) + -- unique (tenant_id, kind, value) enforced for STRONG kinds only + +asset_observations -- raw per-source records, never merged/edited + asset_id?, tenant_id, source, source_ref, observed_at, payload jsonb + +asset_packages -- the package cache + asset_id, name, epoch, version, release, arch, pkg_type, repo, installed_at + pk (asset_id, name, arch) + +asset_subscriptions -- patch-channel binding (used by Phase 2) + asset_id, kind (rhsm|pro|scc|apt|yum), status, channels jsonb +``` + +### 8.2 Reconciliation rules + +The hard part (research finding #4). + +- **Strong identifiers may merge on their own:** machine-id, + SMBIOS/instance UUID, AD objectGUID. +- **Weak identifiers never merge alone** (FQDN, IP, MAC) — they + corroborate. Merging on a recycled DHCP lease silently fuses two + hosts and is nearly undetectable afterwards. Enforced by schema, + not convention. +- Observations are immutable; `assets` is a projection. Bad merge → + fix rule → re-project. No data loss. +- Reaping: full-snapshot semantics per (agent, source) with staleness + decay — same pattern as the existing `removeStaleAgentDatasources`. + +### 8.3 Ingest path + +New upward action `asset_inventory` (do not overload the existing +`datasource_inventory`, which means "what datasources does this agent +have"): + +```json +{"action": "asset_inventory", + "source": "sweep|vcenter|ldap|libvirt|host_inventory", + "content_pack_version": 3, + "observations": [{"identities": {...}, "facts": {...}, + "collectors": {"pkgs-rpm": ""}}]} +``` + +Relay handler → insert `asset_observations` → reconciler projects into +`assets`/`asset_packages`. Collector-output parsers live here. + +### 8.4 Scheduler + +Server-side cron pushing actions to forager (same pattern as +scan_orchestrator job scheduling): sweep cadence per CIDR, hypervisor +poll cadence, inventory cadence per asset, scan windows, concurrency +caps. Forager holds zero schedule state. + +## 9. Forager changes + +New proxy module `pkg/proxy/discovery`, registered like existing +modules. New datasource types configured like db proxies (credentials +via existing `pkg/secrets` local/cloud-push): + +- `vcenter` (govmomi, read-only role) +- `proxmox` (REST `/cluster/resources`) +- `libvirt` (over SSH to the KVM host: `virsh list --all --uuid`) +- `ldap` (AD computer objects; filter `lastLogonTimestamp` to skip + tombstones) + +Actions (existing signed `ActionRequest` scheme): + +| Action | Params | Returns | +|--------|--------|---------| +| `discovery_sweep` | cidrs, ports, rate_pps, timeout, exclusions | live hosts: IP, MAC (L2), rDNS, open ports | +| `discovery_hypervisor` | datasource_id | VM list: UUID, name, power state, guest OS, IPs | +| `discovery_ldap` | datasource_id, active_within | computer objects: name, DNS name, lastLogon, objectGUID | +| `discovery_inventory` | targets[], credential_ref, content_pack_ref | per-host facts + raw collector output (§6) | + +SSH execution reuses `pkg/proxy/ssh` client plumbing (connection, +auth, timeouts); the discovery module owns target iteration, +concurrency caps, and content-pack execution. + +## 10. Credentials & security + +- One **read-only inventory credential** per segment: SSH key for a + dedicated `nudgebee-ro` user. None of the §6 commands require root + except SMBIOS UUID (graceful fallback; optional narrow sudoers entry + for `dmidecode` if the customer wants it). +- Delivered via existing `pkg/secrets` (local or cloud-push); never + logged, never included in action responses. +- Hypervisor/LDAP credentials: read-only roles (vCenter read-only, + unprivileged AD bind). +- All actions Ed25519-signed; content packs signed with the same trust + root. +- No new inbound ports on forager; no new ports on targets beyond + existing sshd. + +## 11. Coverage report — the Phase 0 deliverable + +One screen, per tenant: + +``` +Discovered: 412 Reachable: 391 Inventoried: 380 +Gaps (32): + 11 ssh-refused (no credential / firewall) + 8 ssh-auth-failed (bad credential) + 6 stale (not observed in 14d) + 4 powered-off (per vCenter) + 2 unsupported-os (fingerprint: FreeBSD) + 1 eol-no-repo (CentOS 7, vault not configured) +``` + +Coverage states per asset, tracked independently: `discovered` (exists +per any source) → `reachable` (SSH auth ok) → `inventoried` (package +list collected). `discovered − inventoried` is a feature, not an error +state. Nothing else in Phase 0 ships until this screen is correct. + +## 12. Build list + +1. Migrations: `assets`, `asset_identities`, `asset_observations`, + `asset_packages`, `asset_subscriptions` + static EOL table. +2. Forager `pkg/proxy/discovery`: sweep, hypervisor (first connector + per open question 1), ldap, SSH inventory executor; content-pack + fetch/verify. +3. Content pack v1: linux-inventory (RHEL-like, Debian-like, SUSE, + Alpine collectors). +4. Relay: `asset_inventory` handler; server-side collector parsers; + reconciler. +5. Scheduler: per-tenant CIDRs/cadences/windows/rate caps. +6. UI: asset list + coverage report. +7. Cloud-collector VM resources wired in as observer source (existing + data, new projection). + +## 13. Open questions + +1. **Customer hypervisor stack** — VMware, Proxmox, Hyper-V, or + unmanaged KVM/bare metal? Sizes item 2; build only that connector + first. +2. Approximate fleet size and segment count — sets sweep/inventory + cadence defaults and forager concurrency caps. +3. Shared SSH credential per segment, per-host credentials, or an + existing vault (CyberArk etc.)? +4. Windows timeline — deferred from v1, but if the fleet is >30% + Windows, WinRM support moves up. +5. Content-pack distribution: piggyback on the relay WSS channel vs + HTTPS fetch from cloud API (relay keeps one channel; HTTPS is + simpler to cache/CDN). From f1179b2a69528fe95c98d4fd396f53816a281a61 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Sat, 1 Aug 2026 00:37:41 +0530 Subject: [PATCH 02/16] docs: fold P5 into P2 (coverage states, cloud merge); scheduler to kickoff Matches the issue restructure after #35406 was removed. --- docs/design/vm-discovery-phase0-epic.md | 5 ++-- docs/design/vm-discovery-phase0-tickets.md | 29 ++++++++-------------- 2 files changed, 12 insertions(+), 22 deletions(-) diff --git a/docs/design/vm-discovery-phase0-epic.md b/docs/design/vm-discovery-phase0-epic.md index 9a79a1a..bfffb11 100644 --- a/docs/design/vm-discovery-phase0-epic.md +++ b/docs/design/vm-discovery-phase0-epic.md @@ -66,11 +66,10 @@ application-level (non-OS) packages. | # | Ticket | Where | Note | |---|--------|-------|------| | P1 | Forager: SSH inventory + signed content-pack runner | forager | no dependencies | -| P2 | Server: asset storage + ingest + parsers + dedup logic | nudgebee | no dependencies | +| P2 | Server: asset storage + ingest + parsers + dedup + coverage states | nudgebee | no dependencies | | P3 | Forager: network sweep + Active Directory lookup | forager | after P1+P2 demo | | P4 | Forager: hypervisor connector (vCenter/Proxmox/libvirt) | forager | **blocked: customer must tell us their hypervisor** | -| P5 | Server: scheduler + coverage states + merge cloud VMs | nudgebee | after P1+P2 demo | -| P6 | Content pack v1 + signing/publish pipeline | content | with P5 | +| P6 | Content pack v1 + signing/publish pipeline | content | after P1 | | P7 | UI: asset list + coverage report screen | nudgebee | last | | P8 | Customer setup docs (SSH user, firewall/IDS allowlist) | nudgebee-docs | alongside P6/P7 | diff --git a/docs/design/vm-discovery-phase0-tickets.md b/docs/design/vm-discovery-phase0-tickets.md index 9920c1e..06ae969 100644 --- a/docs/design/vm-discovery-phase0-tickets.md +++ b/docs/design/vm-discovery-phase0-tickets.md @@ -105,22 +105,13 @@ type + read-only credential via `pkg/secrets`. AC: returns UUID, name, power state, guest OS, IPs **including powered-off VMs**; powered-off reaches `assets.power_state`. -## P5 — Server: scheduler, coverage states, cloud projection -Repo: nudgebee (api-server). - -Tasks: (a) scheduler — server-side, forager holds zero schedule -state; candidate: reuse the scan_orchestrator pattern (confirm at -kickoff): sweep cadence per CIDR, hypervisor poll, per-asset -inventory cadence, windows, concurrency caps; (b) coverage-state machine — `discovered → -reachable → inventoried` tracked independently, gap-reason taxonomy -(ssh-refused, ssh-auth-failed, stale, powered-off, unsupported-os, -eol-no-repo) denormalized onto assets; (c) project existing -cloud_resources VMs into `asset_observations` (instance-id STRONG) so -cloud and on-prem VMs merge into one asset universe. - -AC: window/cap violations impossible by construction; every -non-inventoried asset has exactly one primary gap reason; an EC2 -instance also seen via SSH is one asset, not two. +## P5 — removed (folded into P2 and kickoff decisions) +Coverage states + gap-reason taxonomy and the cloud-VM merge +(instance-id as STRONG identity) moved into P2 — they're part of the +same projection logic. Scheduling of recurring runs is a kickoff +decision (candidate: reuse the scan_orchestrator pattern — see +`RunOne`/`ScanAccount` usage in recommendation/service.go); forager +still holds zero schedule state either way. ## P6 — Content pack v1 + publish pipeline Repo: content (+ server config). `linux-inventory` covering the §6 @@ -153,9 +144,9 @@ that credential management is the operational cost center. ## Dependencies ``` -P1 ─┬─► M0 exit ─► P3, P5 ─► P6, P7 -P2 ─┘ │ -customer answer ─► P4┘ P8 alongside P6/P7 +P1 ─┬─► M0 exit ─► P3 ─► P6, P7 +P2 ─┘ │ +customer answer ─► P4 P8 alongside P6/P7 ``` P1 and P2 have no dependencies and run in parallel. P4 is the only From 385645c934e47dccbc1b064fbc96c898390d3ee4 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Sat, 1 Aug 2026 00:39:11 +0530 Subject: [PATCH 03/16] docs: datasource config from UI (server-pushed), credential residency open --- docs/design/vm-discovery-phase0.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/docs/design/vm-discovery-phase0.md b/docs/design/vm-discovery-phase0.md index 7d8735f..bb4fad8 100644 --- a/docs/design/vm-discovery-phase0.md +++ b/docs/design/vm-discovery-phase0.md @@ -400,8 +400,27 @@ caps. Forager holds zero schedule state. ## 9. Forager changes New proxy module `pkg/proxy/discovery`, registered like existing -modules. New datasource types configured like db proxies (credentials -via existing `pkg/secrets` local/cloud-push): +modules. + +**Datasource configuration: from the Nudgebee UI, pushed down** +(recommendation — matches every surveyed tool: Rapid7 discovery +connections live in the console and are assigned to engines; Qualys +and Tenable manage scanners centrally; the per-segment node only +enrolls). Two reasons beyond precedent: the coverage report needs the +server to know *intended* scope (CIDRs, datasources) or gaps can't be +computed; and the existing integrations config-push path +(`integration_config.go` / `proxy_config_push.go`) plus `pkg/secrets` +cloud-push already deliver exactly this. Forager keeps only bootstrap +config (relay URL, pairing) locally. + +Open point — credential residency: datasource *existence and scope* +is always UI-configured, but the credential *value* is either +UI-entered (cloud-push, default) or a local/vault reference on the +forager for customers who won't let credentials transit our cloud +(`pkg/secrets` local mode exists for this). + +New datasource types (credentials via existing `pkg/secrets` +local/cloud-push): - `vcenter` (govmomi, read-only role) - `proxmox` (REST `/cluster/resources`) From 357e1f781e7031f111312d36882be128f92027ed Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Sat, 1 Aug 2026 00:42:37 +0530 Subject: [PATCH 04/16] docs: explain customer-phase mapping column in epic --- docs/design/vm-discovery-phase0-epic.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/design/vm-discovery-phase0-epic.md b/docs/design/vm-discovery-phase0-epic.md index bfffb11..565d83b 100644 --- a/docs/design/vm-discovery-phase0-epic.md +++ b/docs/design/vm-discovery-phase0-epic.md @@ -2,15 +2,19 @@ First epic of the **VM Patch & Vulnerability Management** initiative: -| Epic | What ships | Customer's naming | -|------|-----------|-------------------| +| Epic | What ships | Customer's phase | +|------|-----------|------------------| | **1. Discovery & package inventory (this one)** | Every VM found, packages cached, coverage report | first half of their "Phase 1" | | 2. Vulnerability matching | Packages → CVEs (vendor advisories), escalation policy (CVSS/urgency config) | second half of their "Phase 1" | | 3. Patching | Subscription-based patch apply, approval flow, maintenance windows | their "Phase 2" | | 4. Hardening & insight | CIS benchmarking, AI impact analysis of vulns on applications | later asks | -Epics 2–4 are sequenced, not speculative — 2 needs 1's package data, -3 needs 2's vuln data. We only break tickets down for epic 1 now. +The last column maps our epics to the phase numbers the customer used +in their ask ("Phase 1 - discovery of VM -> packages and +vulnerability, phase 2 - applying patch") — their "Phase 1" spans our +epics 1+2, so on customer calls the numbers differ. Epics 2–4 are +sequenced, not speculative — 2 needs 1's package data, 3 needs 2's +vuln data. We only break tickets down for epic 1 now. ## Summary From 300f4517c185a10efceb02b4fd49d6e6e4d5f5c8 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Sat, 1 Aug 2026 00:47:51 +0530 Subject: [PATCH 05/16] =?UTF-8?q?docs:=20epic=20in=20product=20voice=20?= =?UTF-8?q?=E2=80=94=20drop=20customer-phase=20column,=20reword=20why?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/vm-discovery-phase0-epic.md | 34 ++++++++++++------------- 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/docs/design/vm-discovery-phase0-epic.md b/docs/design/vm-discovery-phase0-epic.md index 565d83b..1bafcc7 100644 --- a/docs/design/vm-discovery-phase0-epic.md +++ b/docs/design/vm-discovery-phase0-epic.md @@ -2,19 +2,15 @@ First epic of the **VM Patch & Vulnerability Management** initiative: -| Epic | What ships | Customer's phase | -|------|-----------|------------------| -| **1. Discovery & package inventory (this one)** | Every VM found, packages cached, coverage report | first half of their "Phase 1" | -| 2. Vulnerability matching | Packages → CVEs (vendor advisories), escalation policy (CVSS/urgency config) | second half of their "Phase 1" | -| 3. Patching | Subscription-based patch apply, approval flow, maintenance windows | their "Phase 2" | -| 4. Hardening & insight | CIS benchmarking, AI impact analysis of vulns on applications | later asks | - -The last column maps our epics to the phase numbers the customer used -in their ask ("Phase 1 - discovery of VM -> packages and -vulnerability, phase 2 - applying patch") — their "Phase 1" spans our -epics 1+2, so on customer calls the numbers differ. Epics 2–4 are -sequenced, not speculative — 2 needs 1's package data, 3 needs 2's -vuln data. We only break tickets down for epic 1 now. +| Epic | What ships | +|------|-----------| +| **1. Discovery & package inventory (this one)** | Every VM found, packages cached, coverage report | +| 2. Vulnerability matching | Packages → CVEs (vendor advisories), escalation policy (CVSS/urgency config) | +| 3. Patching | Subscription-based patch apply, approval flow, maintenance windows | +| 4. Hardening & insight | CIS benchmarking, AI impact analysis of vulns on applications | + +Epics 2–4 are sequenced, not speculative — 2 needs 1's package data, +3 needs 2's vuln data. We only break tickets down for epic 1 now. ## Summary @@ -31,11 +27,13 @@ customer ask), and later for observability and automation on VMs. ## Why -A customer wants patch and vulnerability management for their VMs, -including on-premise machines where no cloud API exists. Nothing else -in that product can ship until we reliably know which VMs exist and -what's installed on them. No vendor today does this for Linux without -installing an agent on every VM — that gap is our opening. +Customers running self-hosted and on-premise VM fleets need patch & +vulnerability management, and cloud APIs don't cover those machines. +Nothing else in the product can ship until we reliably know which VMs +exist and what's installed on them. No vendor today does agentless +Linux inventory from a per-segment collector — every competitor +requires an agent on each VM — so this is both the prerequisite and +the differentiation. A design-partner customer is waiting on it. ## How (in one paragraph) From 57fbbdc5c8ff24109b3d2ad6f9f859e61d99a671 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Sat, 1 Aug 2026 00:49:12 +0530 Subject: [PATCH 06/16] docs: product voice for security-track framing bullet --- docs/design/vm-discovery-phase0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/vm-discovery-phase0.md b/docs/design/vm-discovery-phase0.md index bb4fad8..b88bb91 100644 --- a/docs/design/vm-discovery-phase0.md +++ b/docs/design/vm-discovery-phase0.md @@ -35,7 +35,7 @@ tracks build on the VM resource: 1. **Observability** — node health, metrics, logs, service state. 2. **Security** — package inventory → CVE matching, patching, CIS - benchmarking (the customer ask driving this doc). + benchmarking (the initiative's first deliverable). 3. **Automation** — signed, audited command execution on VMs through the per-segment forager; patching is one instance of it. From a5b9eec3f6346ac51c4c63d4f729b20cae6af69b Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Mon, 3 Aug 2026 14:25:32 +0530 Subject: [PATCH 07/16] docs: add VM discovery deployment plan Covers what can ship now (an rc image) and the four gaps blocking end-to-end discovery: local config cannot configure a discovery datasource, a sweep-only datasource still demands SSH credentials, no signed content pack exists (#116), and nothing triggers discovery yet. --- docs/design/vm-discovery-deployment.md | 191 +++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 docs/design/vm-discovery-deployment.md diff --git a/docs/design/vm-discovery-deployment.md b/docs/design/vm-discovery-deployment.md new file mode 100644 index 0000000..80ddef9 --- /dev/null +++ b/docs/design/vm-discovery-deployment.md @@ -0,0 +1,191 @@ +# VM Discovery — Deployment Plan + +Status: DRAFT +Covers: getting the merged discovery module (forager#118) into an +environment where it can actually run. + +## Where things stand + +Merged to `main` as c2ad9c7. **Not released** — `v0.1.3` is the latest +tag and discovery is one commit ahead of it. The release workflow fires +on a `v*.*.*` tag and publishes `ghcr.io/nudgebee/forager` plus the +chart to `oci://ghcr.io/nudgebee/charts`. + +Deploying the new image is safe on its own: nothing about existing +datasource types changed, so current foragers keep working. But +**end-to-end discovery cannot run yet** — four gaps, listed below with +what each one blocks. + +## Existing deployment pattern (what we build on) + +`nudgebee-infra/deploy/customers/rackspace/agents/forager-values-dev.yaml` +is the working precedent: + +- Chart `deploy/helm/forager` from this repo, release + `nudgebee-forager-dev`, namespace `nudgebee-agent-dev`. +- Points at `wss://relay.dev.nudgebee.pollux.in/register`. +- Secrets SOPS-encrypted with AWS KMS; `encrypted_regex` already + covers `accessKey|accessSecret|password|username`. +- Deployed via `installation.sh -f values.yaml -k`. + +The chart takes us most of the way: `forager.datasources` is passed +through with `toYaml`, so arbitrary datasource fields reach +`forager.yaml`, and `extraVolumes`/`extraVolumeMounts` exist for +mounting a pack directory. No chart changes are strictly required. + +## Blockers + +### 1. Local config cannot configure a discovery datasource + +`cmd/app.go` maps only `allowed_hosts` → `allowed_cidrs`. There is no +path from `forager.yaml` to `pack_public_key`, `pack_dir`, `ldap`, +`known_hosts_file`, `max_rate_pps`, or the port/concurrency/timeout +settings — `config.LocalDatasource` has no fields for them. + +Consequence per action: + +| Action | Works from local YAML today | +|---|---| +| `discovery_sweep` | yes — only needs `allowed_cidrs` | +| `discovery_ldap` | no — directory config unreachable | +| `discovery_inventory` | no — without a pack key nothing runs | + +The production intent (design §9) is that the server pushes datasource +config, which would sidestep this. But that path needs the server-side +work that does not exist yet, so local config is what we have. + +**Fix:** add the discovery fields to `config.LocalDatasource` and map +them in `app.go`. Small, self-contained. + +### 2. A sweep-only datasource still demands SSH credentials + +`Configure` returns `ssh username is required` before it looks at +anything else. A datasource intended purely for sweeping — which needs +no credentials at all — cannot be configured without inventing dummy +ones. + +**Fix:** only require SSH credentials when they are needed, and let +`HealthCheck` report which actions a given datasource can serve. +Otherwise the first thing an operator does is put fake credentials in +a values file, which is a bad habit to teach. + +### 3. No signed content pack exists + +`docs/content-packs/linux-inventory-example.yaml` is deliberately +unsigned and is a format reference, not a shippable pack. The real +pack, CI signing, and pin/ring rollout are #116. + +`discovery_inventory` refuses to run without a verified pack, so this +blocks inventory entirely — by design. + +**Fix for a first deployment:** generate a keypair, sign the example +pack by hand, mount it via `extraVolumes`, set `pack_public_key`. Not +a substitute for #116. + +### 4. Nothing triggers discovery + +There is no scheduler and no UI: the server never sends +`discovery_sweep`, `discovery_ldap`, or `discovery_inventory`. A +deployed forager would connect and sit idle. That is +nudgebee-enterprise#35405 plus the scheduling decision. + +Relay signing is *not* a blocker — `DefaultSigningFields` +(`action`, `datasource_id`, `params`) already matches all three +actions. Adding them explicitly to `SigningFields` in +`relay-server/pkg/signing/signer.go` is worth doing for clarity, but +nothing breaks without it. + +## Plan + +### Phase A — ship a release candidate (no dependencies) + +Cut `v0.1.4-rc.1`, not `v0.1.4`. The workflow already treats a tag +containing `-` as a prerelease, and the precedent exists (`v0.1.2-rc.1`). + +What an rc tag does and does not touch: + +| | rc tag | final tag | +|---|---|---| +| `ghcr.io/nudgebee/forager:0.1.4-rc.1` | pushed | — | +| `ghcr.io/nudgebee/forager:latest` | **untouched** (`latest=auto` skips prereleases) | moves | +| `0.1` shorthand tag | not applied to prereleases | moves | +| GitHub Release | marked prerelease | full release | +| `registry.nudgebee.com` mirror (ECR + S3) | **skipped** — the mirror jobs are gated on `!contains(github.ref, '-')` | synced | +| Chart `oci://ghcr.io/nudgebee/charts/forager:0.1.4-rc.1` | new version, does not overwrite 0.1.3 | new version | + +So no customer-facing pointer moves. Existing installs consuming +`:latest` or the mirror stay on 0.1.3 until we cut a final tag. + +1. Tag `v0.1.4-rc.1` on `main`. +2. Roll the Rackspace **dev** forager onto it by bumping `image.tag` + in `forager-values-dev.yaml`. +3. Confirm the existing `rackspace-pg` datasource still works — a pure + regression check, no discovery involved. + +Value: the binary is available and exercised without committing to a +release, and any discovery rollout afterwards is a config change +rather than an image change. Phases B–D can iterate through +`-rc.2`, `-rc.3` at no cost to anyone downstream. + +### Phase B — make discovery configurable (blockers 1 and 2) + +One forager PR: + +- Add discovery fields to `config.LocalDatasource` and map them in + `app.go`. +- Require SSH credentials only when the datasource will serve + inventory. +- Extend `values-enterprise.yaml.example` with a documented discovery + datasource. + +### Phase C — first real sweep (needs A + B) + +Deploy a second forager release scoped to sweeping only, in the +Rackspace cluster: + +- `allowed_cidrs` limited to a **known-safe range** — the cluster's own + pod/service CIDR, not a customer network. +- Exclusions for anything that should never be probed. +- `ports: [22]` to start, so no RDP/WinRM probes appear in security + monitoring. +- Rate left at the default 100 pps. + +Because nothing schedules actions yet, trigger one by hand through the +relay and read the response. That is the smallest thing that proves the +network path, signing, and scope enforcement all work together. + +Before this runs anywhere near a customer network, the forager's IP +needs allowlisting in their IDS — an unannounced sweep looks exactly +like an attacker, and a bare TCP connect still makes `sshd` log +`Did not receive identification string`, which some fail2ban configs +act on. + +### Phase D — inventory (needs C + a pack) + +- Generate a pack signing keypair; store the private key where CI can + reach it for #116, and put the public key in the values file. +- Sign the example pack, mount it read-only at `pack_dir` via + `extraVolumes`. +- Create a `nudgebee-ro` user on one or two test VMs. +- Run `discovery_inventory` by hand against those hosts and check the + raw collector output. + +### Phase E — scheduled and server-driven + +Everything above is manual. Real operation needs +nudgebee-enterprise#35405 (asset storage and ingest) plus the +scheduler, at which point datasource config moves to server-push and +local YAML becomes a bootstrap detail. + +## Open questions + +1. Which cluster for the first sweep — Rackspace dev, or a dedicated + test environment? Rackspace is real customer infrastructure, so the + CIDR scope needs deciding with care. +2. Where does the pack signing private key live? CI secret for #116 is + the obvious answer, but it should be settled before we hand-sign + anything, so the throwaway key does not become load-bearing. +3. Do we reuse the existing `nudgebee-forager-dev` release for + discovery, or run a separate one? Separate is cleaner — different + blast radius, different credentials, and a sweep misconfiguration + should not be able to disturb the working pg datasource. From 23a02369233330f0a81640ddf7e08a2eefa59b60 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Mon, 3 Aug 2026 17:11:23 +0530 Subject: [PATCH 08/16] docs: record AWS testbed finding on SMBIOS UUID availability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit product_uuid is root-only on stock Linux, so the unprivileged nudgebee-ro credential never gets an SMBIOS UUID. That leaves a hypervisor record and an SSH record for the same on-prem VM with no strong identifier in common, which blocks the merge §8.2 depends on. board_asset_tag is world-readable and carries the EC2 instance id, so cloud VMs are unaffected. --- docs/design/vm-discovery-phase0.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/design/vm-discovery-phase0.md b/docs/design/vm-discovery-phase0.md index b88bb91..37009b6 100644 --- a/docs/design/vm-discovery-phase0.md +++ b/docs/design/vm-discovery-phase0.md @@ -250,6 +250,24 @@ the content pack): `/sys/class/dmi/id/product_uuid` (root) with `dmidecode -s system-uuid` fallback if sudo-permitted — degrade gracefully to machine-id only; hostname/FQDN; MACs (`ip -o link`). + + **Verified on the AWS testbed (2026-08-03):** `product_uuid` is mode + `-r--------` root-only on stock Linux, so an unprivileged + `nudgebee-ro` gets **nothing** — SMBIOS UUID is unavailable in the + default credential model, not merely sometimes. `product_serial` is + root-only too. `board_asset_tag` *is* world-readable and on EC2 + carries the instance id, so cloud VMs still yield a strong + identifier. + + This has a consequence for §8.2 that is easy to miss: a hypervisor + record (source 1) knows a VM's SMBIOS UUID, and an SSH record + (source 5) knows its machine-id. On-prem, with no sudo, **they share + no strong identifier** — so the two sources cannot be merged on one, + and every VM would appear twice. The narrow sudoers entry for + `dmidecode` is therefore not the optional nicety §10 implies; it is + what makes hypervisor↔SSH reconciliation possible at all. Either + require it, or accept that on-prem merging leans on corroborated + weak identifiers, which §8.2 currently forbids. Decide before P4. - **OS:** `/etc/os-release` (ID, VERSION_ID, PRETTY_NAME), `uname -r -m`. - **Packages:** per family above. Version strings kept **verbatim** From de6ac355952f6bd8577cedf5ca00d98ed73174dc Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Mon, 3 Aug 2026 18:12:16 +0530 Subject: [PATCH 09/16] docs: AWS testbed runbook with executed results Records the setup, what a correct sweep and inventory look like on real hosts, and the three findings: SMBIOS UUID is root-only so hypervisor-to-SSH merging has no shared strong identifier on-prem, rpm reports a missing epoch as (none), and board_asset_tag gives cloud VMs a strong identifier unprivileged. --- docs/design/vm-discovery-aws-testbed.md | 210 ++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 docs/design/vm-discovery-aws-testbed.md diff --git a/docs/design/vm-discovery-aws-testbed.md b/docs/design/vm-discovery-aws-testbed.md new file mode 100644 index 0000000..8390d69 --- /dev/null +++ b/docs/design/vm-discovery-aws-testbed.md @@ -0,0 +1,210 @@ +# VM Discovery — AWS testbed setup + +Status: RUNBOOK — executed 2026-08-03, results inline +Goal: first real sweep and inventory, on our own AWS, with a locally built +binary. No dependency on releases or on the server side. + +## What runs where + +Three EC2 instances in one small subnet: + +``` +subnet 10.x.y.0/28 (isolated, us-east-1) +┌──────────────────────────────────────────────┐ +│ forager-discovery ← runs the forager │ +│ │ sweeps the subnet, SSHes into targets │ +│ ├──► target-ubuntu (Debian-like) │ +│ └──► target-al2023 (RHEL-like) │ +└──────────────────────────────────────────────┘ +``` + +**The forager runs on its own EC2 instance, not as a pod on EKS.** Two +reasons, and the second is the one that matters: + +1. It matches how this is actually deployed — one node per network segment. + Testing a different topology than the product ships would prove less than + it appears to. +2. MAC enrichment reads the kernel neighbour cache after probing. On EKS with + the VPC CNI, a pod's neighbour cache does not see other instances' L2 + entries, so MACs would come back empty and we would not know whether that + is a bug or the environment. + +Targets are two distros on purpose: `dpkg-query` and `rpm -qa` are different +collectors in the pack, and a run that only exercises one proves half the +matrix. Amazon Linux 2023 is RHEL-like (dnf/rpm), so it covers that family +without needing a RHEL subscription. + +## 1. Create the instances + +Anything small is fine — `t3.micro` throughout. Requirements: + +- All three in the **same subnet**, so the sweep's L2 neighbour lookup works. +- Security group allowing `forager-discovery` → targets on TCP 22. +- The forager host needs outbound 443 to reach the relay. It needs **no + inbound** at all. +- Tag them so they are obviously disposable. + +A `/28` gives 11 usable addresses — enough to see the sweep correctly report +"3 responded out of 11 scanned", which is the result that tells you the +address expansion and exclusion logic are right. + +## 2. Build and copy the binary + +```bash +# from the forager repo +make build-all # or: CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ + # go build -o build/nudgebee-forager-linux-amd64 ./cmd +scp build/nudgebee-forager-linux-amd64 ec2-user@:/tmp/ +``` + +Use `-linux-arm64` instead if you pick Graviton instances. + +```bash +sudo install -m 0755 /tmp/nudgebee-forager-linux-amd64 /usr/local/bin/nudgebee-forager +sudo mkdir -p /etc/nudgebee /var/lib/nudgebee +``` + +## 3. Configure + +`/etc/nudgebee/forager.yaml` on the forager host: + +```yaml +relay_url: wss://relay.dev.nudgebee.pollux.in/register +access_key: REPLACE_ME +access_secret: REPLACE_ME +data_dir: /var/lib/nudgebee +signing_public_key: 'fA+zl69KzfyYZ8+f722PGR8h8TUP/u76nngOGKUUpYo=' + +datasources: + - name: aws-testbed + type: discovery + + # Scope ceiling. The subnet and nothing else. + allowed_hosts: + - 10.x.y.0/28 + + discovery: + max_rate_pps: 50 + concurrency: 10 + + # Omit credentials entirely for the sweep-only stage. The node will + # report it can serve discovery_sweep and not discovery_inventory, + # which is the honest state — and is only possible from rc.2 onward. +``` + +Point it at the **dev** relay, not prod: a half-configured node should not +appear in the prod account while we are shaking it out. + +Run it in the foreground first rather than as a service — the startup logs +say whether the datasource configured, and that is most of what stage 1 is +checking: + +```bash +sudo nudgebee-forager --config /etc/nudgebee/forager.yaml +``` + +Expect `discovery proxy configured` with `allowed_cidrs=1`, and +`ssh host key verification disabled` (correct — no known_hosts yet). + +## 4. Sweep + +Nothing schedules discovery yet, so trigger it by hand through the relay: + +```json +{"action": "discovery_sweep", + "datasource_id": "local:aws-testbed", + "params": {"cidrs": ["10.x.y.0/28"], "ports": [22], "rate_pps": 50}} +``` + +`ports: [22]` deliberately — the default also probes 3389/5985, which on a +Linux-only testbed generates RDP/WinRM probes for no benefit. + +What a correct result looks like: `addresses_scanned` of 11 for a /28 (16 +minus network, broadcast, and AWS's three reserved addresses will still be +scanned — AWS reserves them but they simply will not answer), three hosts +returned with `open_ports: [22]`, and MACs present since everything is on the +same L2. + +If MACs come back empty, that is the finding — it means the neighbour-cache +read is not working in this environment, which is exactly what running on EC2 +rather than EKS was meant to establish. + +## 5. Inventory + +Needs a signed pack, which does not exist yet (#116). For the testbed, +generate a throwaway keypair and sign the example pack: + +```bash +# keypair +go run ./cmd/... # no helper yet — see pkg/signing.GenerateKeypair +``` + +Then on each target: + +```bash +sudo useradd --system --create-home --shell /bin/sh nudgebee-ro +sudo install -d -m 700 -o nudgebee-ro ~nudgebee-ro/.ssh +echo '' | sudo tee ~nudgebee-ro/.ssh/authorized_keys +sudo chmod 600 ~nudgebee-ro/.ssh/authorized_keys +sudo chown nudgebee-ro: ~nudgebee-ro/.ssh/authorized_keys +``` + +Add to the datasource: `pack_public_key`, `pack_dir`, and the SSH credential. +Then: + +```json +{"action": "discovery_inventory", + "datasource_id": "local:aws-testbed", + "params": {"targets": ["10.x.y.4", "10.x.y.5"], "content_pack_version": 1}} +``` + +The check that matters is not "did it return data" but **whether the two +distros ran different collectors** — `pkgs-dpkg` on Ubuntu, `pkgs-rpm` on +AL2023 — and whether version strings came back with epoch and release intact +(`5.14.0-362.8.1.el9_3`, not `5.14.0`). Backport-aware CVE matching in epic 2 +depends on that, and it is the thing most likely to be silently wrong. + +**The throwaway signing key must not become load-bearing.** Where the real +key lives is an open question on #116; generate this one, use it, and do not +put it anywhere durable. + +## What actually happened (2026-08-03) + +Built on account 864186153326 (sandbox, not the prod 740395098545), +default VPC, subnet `subnet-0e08ebf5c41c11e4d`, instances pinned to +`172.31.0.10/.11/.12`. The default security group needed one added +rule — its self-referencing rule covered only 5432, so nothing could +reach port 22 and the sweep would have returned zero hosts for reasons +having nothing to do with the code. + +**Sweep** — 14 addresses scanned, 3 found, 1.2s. `172.31.0.10` came +back without a MAC while the other two had one: a host does not ARP +for its own address, so it is absent from its own neighbour cache. +That is the result that shows the MAC enrichment reads the kernel +cache rather than inventing values — and is precisely what running on +EKS would have obscured. + +**Inventory** — both hosts in 1.6s, 609 and 494 packages. The right +collectors ran on each family and neither ran the other's. Epoch and +release survived intact (`openssl 1 3.5.5 1.amzn2023.0.5`; Debian's +`acpid 1:2.0.33-1ubuntu1` kept its `1:`). + +**Findings worth carrying forward:** + +1. `/sys/class/dmi/id/product_uuid` is `-r--------` root-only, so the + unprivileged credential never reads an SMBIOS UUID. See §6 of the + design doc — this breaks hypervisor↔SSH merging on-prem. +2. `rpm` prints `(none)` for a missing epoch, not `0` or empty, and + the two are not equivalent for version comparison. Reported on + nudgebee-enterprise#35405. +3. `board_asset_tag` is world-readable and carries the EC2 instance + id, so the cloud-VM merge path works unprivileged. + +Nothing schedules discovery yet, so both stages were driven through +the exported proxy API by throwaway harnesses rather than the relay. + +## Cost and teardown + +Three t3.micro instances are a few dollars a month. They are disposable — +terminate them once the coverage report reads correctly, and rebuild from +this document if needed. From af1284b4464434162918c209ce42159ab38f0c18 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Mon, 3 Aug 2026 18:30:15 +0530 Subject: [PATCH 10/16] docs: plan after the AWS testbed run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The agent half is proven and does nothing, because everything user-visible is behind the server side. Records the one design decision the testbed forced — SMBIOS UUID is root-only, so on-prem a hypervisor record and an SSH record share no strong identifier — with three options and the tradeoff of each. --- docs/design/vm-discovery-next.md | 114 +++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 docs/design/vm-discovery-next.md diff --git a/docs/design/vm-discovery-next.md b/docs/design/vm-discovery-next.md new file mode 100644 index 0000000..890863a --- /dev/null +++ b/docs/design/vm-discovery-next.md @@ -0,0 +1,114 @@ +# VM Discovery — what's next + +Status: PLAN, 2026-08-03 +Written after the AWS testbed run, which changed the picture in two ways. + +## Where we are + +The agent half is done and proven on real hosts. #113 and #114 are +merged and closed; `v0.1.4-rc.2` is released with a green pipeline; a +forager is deployed on EC2, registered as `aws-dev-proxy`, and both +sweep and inventory were validated against Ubuntu and Amazon Linux +targets (609 and 494 packages, correct per-family collectors, epoch and +release intact). + +**And it does nothing.** The agent is registered and idle, because +nothing can send it a discovery action. Everything user-visible is +behind the server side. + +That is the honest summary: we have a working collector and no product. + +## The one decision that blocks design, not code + +**SMBIOS UUID is unreadable by the unprivileged credential.** Verified +on the testbed: `/sys/class/dmi/id/product_uuid` is mode `-r--------`. + +The consequence is not obvious and is easy to build straight past. A +hypervisor knows a VM by its SMBIOS UUID. An SSH inventory knows the +same VM by its machine-id. On-prem, with no sudo, **those two sources +share no strong identifier** — so §8.2's rule that only strong +identifiers may merge means the same VM lands in the asset list twice, +once from each source, forever. + +Three ways out, and someone has to pick before #115 or the #35405 +schema hardens: + +1. **Require the narrow `dmidecode` sudoers entry.** Cleanest data + model; costs a privilege we currently advertise as unnecessary, and + it is a per-host change customers must roll out. +2. **Merge hypervisor↔SSH on corroborated weak identifiers** (hostname + plus IP, say). Keeps the credential unprivileged; weakens the rule + that exists precisely because DHCP recycles IPs. If we do this, the + corroboration threshold needs writing down, not left to judgement. +3. **Accept the duplication** and reconcile in the UI. Cheapest now, + worst to live with — "how many VMs do I have" is the question this + epic exists to answer, and it would answer it wrong. + +Cloud VMs are unaffected: `board_asset_tag` is world-readable and +carries the instance id, so the cloud↔SSH merge works today. + +## Critical path + +### 1. Server side — nudgebee-enterprise#35405 + +The only thing between us and something demoable. Nothing else on this +list changes what a user can see. + +Carries two testbed findings already reported on the ticket: +`rpm` prints `(none)` for a missing epoch (and `(none)` ≠ `0` for +version comparison), and golden files for the parsers can be captured +from the testbed rather than hand-written. + +Also needs the identity decision above, since it determines what the +merge logic is allowed to do. + +### 2. Scheduling + +Deliberately left undecided in the ticket breakdown. Now it is the +thing standing between a deployed agent and a running one. Candidate +remains reusing `scan_orchestrator`; confirm or reject at #35405 +kickoff rather than carrying it as an open question indefinitely. + +### 3. Content pack pipeline — forager#116 + +Required before any customer deployment. The testbed runs on a +hand-signed pack and a throwaway key that must not become +load-bearing. Where the real signing key lives is still unanswered and +should be settled as part of this. + +### 4. Hypervisor connector — forager#115 + +Still blocked on which hypervisor the customer runs, and now also on +the identity decision. Worth noting it is the only source that sees +powered-off VMs, so the coverage report's denominator is wrong without +it. + +## Small fixes worth doing while the above runs + +- **`install.sh` fails cryptically on a stale `/tmp` file.** It + downloads to `/tmp/nudgebee-forager`; if that path exists owned by + another user, `fs.protected_regular=1` (default on AL2023 and most + modern distros) stops root writing it and curl reports a bare error + 23. Download to `mktemp -d` instead. Cost us two failed installs. +- **No `private_key_file` credential option.** SSH keys must be inlined + into config, so key material sits in values files. Supporting a path + would let Kubernetes mount a secret directly. +- **Agent credentials are plaintext in the infra repo.** All four + `proxy-agent/values-*.yaml` carry `accessKey`/`accessSecret` in + clear, while the Rackspace file is properly SOPS-encrypted with an + `encrypted_regex` that already covers those fields. Same pattern, + applied to the customer and not to us. Worth encrypting and rotating. +- **Stale IP allowances in the AWS default security group.** Two + `/32` rules for addresses nobody is using, plus one granting all TCP + ports. Worth pruning. + +## Testbed + +Three `t3.micro` in account 864186153326, tagged +`Purpose=vm-discovery-testbed`, `Disposable=true`. A few dollars a +month. Keep while #115 and #35405 are in flight — rebuilding costs +more than running them — and terminate once the coverage report is +correct. + +Note the security group pins SSH to specific source IPs, so access +breaks whenever someone's address changes. From 549aa4c7f624215320ae144fa2f730f2af0a492d Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Mon, 3 Aug 2026 19:08:37 +0530 Subject: [PATCH 11/16] docs: action and relay wire reference from the testbed run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exact requests and verbatim responses for discovery_sweep and discovery_inventory, plus parsing notes taken from real output rather than from the collector definitions — rpm reports a missing epoch as (none), smbios-uuid comes back empty under the unprivileged credential. Marks clearly which parts were exercised and which were not: the actions ran against real hosts, the relay envelope around them has never carried a discovery action. --- docs/design/vm-discovery-wire-reference.md | 165 +++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 docs/design/vm-discovery-wire-reference.md diff --git a/docs/design/vm-discovery-wire-reference.md b/docs/design/vm-discovery-wire-reference.md new file mode 100644 index 0000000..9ddf185 --- /dev/null +++ b/docs/design/vm-discovery-wire-reference.md @@ -0,0 +1,165 @@ +# VM Discovery — action and relay wire reference + +Status: REFERENCE, from the AWS testbed run 2026-08-03 +Audience: whoever builds the server side (nudgebee-enterprise#35405) and +needs to know exactly what to send and what comes back. + +Everything below is either what was actually run, or the envelope the +relay produces — both marked as such, because the distinction matters: +**the actions were exercised, the relay envelope around them was not.** + +## What was actually run + +Driven through the discovery proxy's exported Go API by a throwaway +harness on the forager host, because nothing can dispatch a discovery +action through the relay yet. + +### Datasource configuration + +```go +p.Configure(map[string]any{ + "allowed_cidrs": []any{"172.31.0.0/28"}, + "max_rate_pps": float64(50), + // inventory additionally needed: + "pack_public_key": "", + "pack_dir": "/etc/nudgebee/packs", +}, map[string]string{ + "username": "nudgebee-ro", + "private_key": "", +}) +``` + +### discovery_sweep + +```go +&proxy.ActionRequest{ + Action: "discovery_sweep", + Params: map[string]any{ + "cidrs": []any{"172.31.0.0/28"}, + "ports": []any{float64(22)}, + "rate_pps": float64(50), + "timeout_ms": float64(1000), + }, +} +``` + +Response (`ActionResponse.Data`, verbatim): + +```json +{ + "cidrs": ["172.31.0.0/28"], + "addresses_scanned": 14, + "addresses_excluded": 0, + "rate_pps": 50, + "duration_seconds": 1.184975093, + "hosts": [ + {"ip": "172.31.0.10", "open_ports": [22], + "rdns": "ip-172-31-0-10.ec2.internal", "sources": ["tcp"]}, + {"ip": "172.31.0.11", "mac": "02:4d:07:48:c4:87", "open_ports": [22], + "rdns": "ip-172-31-0-11.ec2.internal", "sources": ["tcp","arp"]}, + {"ip": "172.31.0.12", "mac": "02:16:a7:ae:cb:8b", "open_ports": [22], + "rdns": "ip-172-31-0-12.ec2.internal", "sources": ["tcp","arp"]} + ] +} +``` + +`172.31.0.10` has no MAC because it is the forager itself — a host does +not ARP for its own address. Do not treat a missing MAC as an error; +it is also normal for anything reached through a router. + +`addresses_scanned` is 14, not 16: network and broadcast are skipped. + +### discovery_inventory + +```go +&proxy.ActionRequest{ + Action: "discovery_inventory", + Params: map[string]any{ + "targets": []any{"172.31.0.11", "172.31.0.12"}, + "content_pack_version": float64(1), + }, +} +``` + +Response shape, with the fields that matter to ingest: + +```json +{ + "content_pack_version": 1, + "targets": [ + { + "host": "172.31.0.11", + "status": "ok", + "duration_seconds": 1.5, + "facts": {"os_family":"debian","os_id":"ubuntu","os_major":"22","arch":"x86_64"}, + "collectors": { + "os-release": "NAME=\"Ubuntu\"\nID=ubuntu\n...", + "pkgs-dpkg": "acpid\t1:2.0.33-1ubuntu1\tamd64\tinstalled\n...", + "machine-id": "ec2403e319a2f3f0ae53a05e3daf084b\n", + "smbios-uuid": "" + }, + "collector_errors": {}, + "skipped_collectors": [] + } + ] +} +``` + +Per-target `status` is one of `ok`, `ssh-refused`, `ssh-auth-failed`, +`timeout`, `error`. **A failed target is a result, not an error** — the +batch still returns 200. These map onto coverage gap reasons. + +## Parsing notes from real output + +Taken from the actual runs, not from the collector definitions. + +- **`rpm` prints `(none)` for a missing epoch**, not `0` or empty: + `publicsuffix-list-dafsa\t(none)\t2026...`. `(none)` and `0` are not + equivalent for version comparison and must not be collapsed. +- **Epoch and release survive intact** and must stay that way: + `openssl\t1\t3.5.5\t1.amzn2023.0.5\tx86_64\t1784931981`, and Debian's + `acpid\t1:2.0.33-1ubuntu1` keeps its `1:` prefix. +- **`smbios-uuid` comes back empty** under the unprivileged credential — + `/sys/class/dmi/id/product_uuid` is mode `-r--------`. Plan for + machine-id as the only strong identifier on-prem. +- Collector output is raw text exactly as the command emitted it, + including a trailing newline. Where a collector wrote to stderr, it + is appended after a `\n[stderr]\n` marker. + +Real output from both distro families is available on the testbed for +capturing golden files rather than hand-writing them. + +## The relay envelope — shape only, NOT yet exercised + +What the relay wraps an action in. Reproduced here from the relay's +signer and pinned by tests in `pkg/ws/discovery_wire_test.go`, but no +discovery action has actually travelled this path: the relay's dispatch +endpoint is hardcoded for `http_request`, so sending one means +publishing to the agent's queue, which is server-side work. + +```json +{ + "request_id": "", + "datasource_id": "local:aws-testbed", + "action": "discovery_sweep", + "params": { "cidrs": ["172.31.0.0/28"], "ports": [22] }, + + "signed_payload": "{\"action\":\"discovery_sweep\",\"datasource_id\":\"local:aws-testbed\",\"params\":{...}}", + "signature": "", + "signed_at": "", + "nonce": "" +} +``` + +Two things to get right: + +1. **Discovery actions are not in the relay's explicit `SigningFields` + map**, so they fall through to `DefaultSigningFields` — + `action`, `datasource_id`, `params`. That is correct for all three, + and is now pinned by a test rather than assumed. +2. **All three require a signature.** They were absent from the + forager's `signedActions` allowlist and therefore unverified; fixed + in forager#122. An unsigned or tampered discovery action is now + rejected with 403. + +`datasource_id` is `local:` for a locally configured datasource. From 2f2bb7e2f5694fba401284f922277ee2a1b8815d Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Tue, 4 Aug 2026 09:50:55 +0530 Subject: [PATCH 12/16] docs: customer-facing discovery document MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plain-language explanation of what discovery does, written for a non-technical reader, with the joint-build split made explicit: what we need answered, what only they can set up, and the identification trade-off that is genuinely their call. States current status honestly rather than as 'in progress' — what works and was tested on real machines, what is not built, and the limits we are not hiding (powered-off machines are invisible without the hypervisor; no-SSH machines can be found but not inventoried). --- docs/design/vm-discovery-customer.md | 219 +++++++++++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 docs/design/vm-discovery-customer.md diff --git a/docs/design/vm-discovery-customer.md b/docs/design/vm-discovery-customer.md new file mode 100644 index 0000000..701b249 --- /dev/null +++ b/docs/design/vm-discovery-customer.md @@ -0,0 +1,219 @@ +# VM Discovery — how it works, and what we need from you + +Draft for discussion. This is a joint build: some of it is ours, some +of it only you can do, and a few choices we should make together. + +--- + +## 1. The problem this solves + +Before you can patch anything, you have to know what you have. Most +teams have a list somewhere, and most of those lists are wrong — a +spreadsheet, a CMDB nobody updates, a monitoring tool that only sees +what someone installed an agent on. + +We want to answer three questions honestly: + +1. **How many machines do you actually have?** +2. **What software is installed on each one?** +3. **For anything we could not check — why not?** + +The third one matters as much as the first two. A tool that reports +"400 machines, all healthy" while quietly missing 50 is worse than one +that says "we found 450, we could not get into 50, here is why." + +## 2. How it works + +**We do not install anything on your machines.** + +Instead, you run one small program — we call it the collector — on a +single machine in each network. That collector: + +- looks around the network to see what is there, +- asks your virtualisation platform and Active Directory what they + know about, +- logs into each machine over SSH using a read-only account you + create, runs a few read-only commands, and reads the answers. + +``` + ┌── one collector per network ──┐ + │ · finds machines │ + │ · logs in over SSH │ + └──────────┬────────────────────┘ + │ SSH, read-only + ┌───────────┼───────────┐ + ▼ ▼ ▼ + your VM your VM your VM ← nothing installed here +``` + +The commands it runs are things like "list the installed packages" — +the same commands your own administrators would type. Nothing is +changed, nothing is written, nothing is restarted. + +**Why no agent on each machine?** Because installing software on +hundreds of machines is a project in itself, and keeping it updated is +a permanent cost. One collector per network is something you can +approve, place, and watch. + +## 3. What you get + +A single view, per network, that looks like this: + +``` +Found: 412 machines + Reachable: 391 (we can log in) + Inventoried: 380 (we have the software list) + +Cannot inventory 32: + 11 no credentials or blocked by firewall + 8 login rejected + 6 not seen for two weeks + 4 powered off + 2 operating system we do not support yet + 1 running an OS with no security updates available +``` + +Plus, for each machine: its name, operating system, the full list of +installed software with exact versions, and whether it is still +receiving security updates from its vendor. + +That last point is worth calling out. A machine whose vendor +subscription has lapsed, or which runs an OS past end-of-life, **cannot +be patched at all**. Better to learn that now than when you are trying +to fix an urgent vulnerability. + +## 4. What we need from you + +### 4a. Answers — so we build the right thing first + +| Question | Why it matters | +|---|---| +| Which virtualisation platform? VMware, Proxmox, Hyper-V, plain KVM, or a mix | It is the only thing that knows about **powered-off** machines. We will build support for yours first | +| Roughly how many machines, across how many networks? | Sets how fast we scan and how many collectors you need | +| Do you use Active Directory? | It knows machines exist even when they are switched off or firewalled | +| Roughly what share is Windows? | We support Linux first. If a large part of the fleet is Windows, we reprioritise | +| Which Linux distributions, and any very old ones? | Old systems like CentOS 7 still work for inventory, but have no vendor patches — we want to flag them clearly rather than surprise you | + +### 4b. Access — things only you can set up + +1. **A read-only login on the machines.** A user (we suggest + `nudgebee-ro`) with an SSH key, which can run read-only commands and + nothing else. You would normally push this with whatever tool you + already use to manage machines. We will give you the exact commands. + +2. **A read-only account on your virtualisation platform.** View + permissions only — enough to list machines and see whether they are + on or off. + +3. **A read-only Active Directory account**, if you want us to use it. + An ordinary user with no special group membership. + +4. **A place to run the collector** — one small VM per network. It + needs outbound internet access on port 443 and no incoming access + at all. + +### 4c. Permission to look — the bit people forget + +Scanning a network looks exactly like an attack, because technically it +is the same activity. Before we start we need: + +- **Your security team to expect it**, and to allow the collector's + address in whatever intrusion detection you run. Without this, the + first scan generates a security incident. +- **Agreement on which address ranges** we may look at, and which we + must not touch — printers, industrial controllers, medical devices, + anything fragile. +- **Agreement on when.** We can restrict scanning to specific hours. + +We deliberately scan gently: normal connections only, at a slow rate +you control, with no unusual traffic of the sort that can upset older +equipment. + +## 5. Decisions we should make together + +### 5a. Identifying a machine reliably + +This one has a real trade-off and we would like your view. + +To avoid listing the same machine twice, we need something that +uniquely identifies it. There are two candidates: + +- A **machine ID** file, readable by an ordinary user. +- A **hardware ID** from the virtual BIOS, which on Linux can only be + read by an administrator. + +Your virtualisation platform reports the *hardware* ID. Our read-only +login can only read the *machine* ID. If we only have those two, we +cannot tell that "the machine VMware calls X" and "the machine we +logged into" are the same box — so it appears twice. + +Three ways forward: + +1. **Grant one narrow extra permission** — allow the read-only user to + run a single command that reads the hardware ID, and nothing else. + Cleanest result; costs one small permission on each machine. +2. **Match on name and address instead.** No extra permission, but less + reliable — addresses get reused, and occasionally we would get it + wrong. +3. **Accept some duplicates** and let you merge them in the interface. + No setup cost, but the machine count is then approximate. + +We lean towards option 1, but it is your estimate of the effort that +should decide it. + +### 5b. How often, and how fast + +Defaults we suggest, all adjustable: look for new machines daily, +re-check software weekly, scan slowly enough to be invisible in +network monitoring. + +### 5c. What happens to the data + +The collector sends what it read — machine names, operating systems, +installed software lists. It does not read your files, your databases, +or your application data. If there are categories you do not want +leaving your network, tell us now rather than later. + +## 6. Where we actually are + +Being straight about status, because "in progress" covers too much. + +**Working now, tested on real machines:** + +- Finding machines on a network +- Reading the full installed-software list over SSH, on both major + Linux families +- Reading Active Directory +- The collector runs as a service and communicates securely + +**Not built yet:** + +- Storing the results and presenting them to you — this is the next + piece of work and the biggest remaining one +- Reading your virtualisation platform — we build this once you tell + us which one you use +- Automatic scheduling, so it runs by itself rather than on request +- Windows machines +- Matching software against known vulnerabilities. That is the next + phase, and this one has to be right first + +**Known limits we are not hiding:** + +- A machine that is switched off is invisible until we connect to your + virtualisation platform. Nothing that looks at a network can see a + machine that is off. +- A machine with no SSH access can be found, but not inventoried. It + will show in the report as a gap with the reason. +- Very old systems can be inventoried, but there may be no patches + available for them. We will say so plainly. + +## 7. What we would like next + +1. The answers in section 4a — mainly which virtualisation platform. +2. Your view on the identification trade-off in 5a. +3. A single network to try first. Ideally a small one, with machines + you know, so that when we show you the result you can tell whether + it is right. + +The last point is the important one. The first run is only useful if +you can check it against what you already know is there. From a8d2d01331bf59896c8aec7000322cad29eb57e4 Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Tue, 4 Aug 2026 10:03:05 +0530 Subject: [PATCH 13/16] docs: cut repetition from the customer document Four points were each made three times: machines that are switched off needing the hypervisor, old systems having no patches, nothing being installed on customer machines, and machines that cannot be logged into. Each now appears once, in the section that owns it, with cross-references where another section needs to point at it. Also drops the closing section that restated the questions and the decision already asked for above. --- docs/design/vm-discovery-customer.md | 247 +++++++++++---------------- 1 file changed, 97 insertions(+), 150 deletions(-) diff --git a/docs/design/vm-discovery-customer.md b/docs/design/vm-discovery-customer.md index 701b249..c91a555 100644 --- a/docs/design/vm-discovery-customer.md +++ b/docs/design/vm-discovery-customer.md @@ -1,39 +1,29 @@ # VM Discovery — how it works, and what we need from you Draft for discussion. This is a joint build: some of it is ours, some -of it only you can do, and a few choices we should make together. +only you can do, and one choice we should make together. --- -## 1. The problem this solves +## 1. What this solves Before you can patch anything, you have to know what you have. Most -teams have a list somewhere, and most of those lists are wrong — a -spreadsheet, a CMDB nobody updates, a monitoring tool that only sees -what someone installed an agent on. +teams have a list somewhere, and most of those lists are wrong. -We want to answer three questions honestly: - -1. **How many machines do you actually have?** -2. **What software is installed on each one?** -3. **For anything we could not check — why not?** - -The third one matters as much as the first two. A tool that reports -"400 machines, all healthy" while quietly missing 50 is worse than one -that says "we found 450, we could not get into 50, here is why." +We want to answer three questions honestly: how many machines you have, +what is installed on each, and — for anything we could not check — why +not. The third matters as much as the others. A tool reporting "400 +machines, all healthy" while quietly missing 50 is worse than one that +says "we found 450, we could not get into 50, here is why." ## 2. How it works -**We do not install anything on your machines.** - -Instead, you run one small program — we call it the collector — on a -single machine in each network. That collector: +**Nothing is installed on your machines.** -- looks around the network to see what is there, -- asks your virtualisation platform and Active Directory what they - know about, -- logs into each machine over SSH using a read-only account you - create, runs a few read-only commands, and reads the answers. +You run one small program — the collector — on a single machine in each +network. It looks around the network to see what is there, asks your +virtualisation platform and Active Directory what they know about, and +logs into each machine over SSH using a read-only account you create. ``` ┌── one collector per network ──┐ @@ -43,22 +33,19 @@ single machine in each network. That collector: │ SSH, read-only ┌───────────┼───────────┐ ▼ ▼ ▼ - your VM your VM your VM ← nothing installed here + your VM your VM your VM ``` -The commands it runs are things like "list the installed packages" — -the same commands your own administrators would type. Nothing is -changed, nothing is written, nothing is restarted. +The commands it runs are ones your own administrators would type — list +the installed packages, read the OS version. Nothing is changed, +written, or restarted. -**Why no agent on each machine?** Because installing software on -hundreds of machines is a project in itself, and keeping it updated is -a permanent cost. One collector per network is something you can -approve, place, and watch. +Installing software on hundreds of machines is a project in itself, and +keeping it updated is a permanent cost. One collector per network is +something you can approve, place, and watch. ## 3. What you get -A single view, per network, that looks like this: - ``` Found: 412 machines Reachable: 391 (we can log in) @@ -73,147 +60,107 @@ Cannot inventory 32: 1 running an OS with no security updates available ``` -Plus, for each machine: its name, operating system, the full list of -installed software with exact versions, and whether it is still -receiving security updates from its vendor. +Plus, per machine: name, operating system, the full list of installed +software with exact versions, and whether it is still receiving +security updates from its vendor. -That last point is worth calling out. A machine whose vendor +That last field is worth the attention. A machine whose vendor subscription has lapsed, or which runs an OS past end-of-life, **cannot -be patched at all**. Better to learn that now than when you are trying -to fix an urgent vulnerability. +be patched at all** — better to know now than while trying to fix an +urgent vulnerability. ## 4. What we need from you -### 4a. Answers — so we build the right thing first +### Answers | Question | Why it matters | |---|---| -| Which virtualisation platform? VMware, Proxmox, Hyper-V, plain KVM, or a mix | It is the only thing that knows about **powered-off** machines. We will build support for yours first | -| Roughly how many machines, across how many networks? | Sets how fast we scan and how many collectors you need | -| Do you use Active Directory? | It knows machines exist even when they are switched off or firewalled | -| Roughly what share is Windows? | We support Linux first. If a large part of the fleet is Windows, we reprioritise | -| Which Linux distributions, and any very old ones? | Old systems like CentOS 7 still work for inventory, but have no vendor patches — we want to flag them clearly rather than surprise you | - -### 4b. Access — things only you can set up - -1. **A read-only login on the machines.** A user (we suggest - `nudgebee-ro`) with an SSH key, which can run read-only commands and - nothing else. You would normally push this with whatever tool you - already use to manage machines. We will give you the exact commands. - -2. **A read-only account on your virtualisation platform.** View - permissions only — enough to list machines and see whether they are - on or off. - +| Which virtualisation platform — VMware, Proxmox, Hyper-V, plain KVM, or a mix? | We build support for yours first; see the first limit in section 6 | +| Roughly how many machines, across how many networks? | Sets scanning pace and how many collectors you need | +| Do you use Active Directory? | It knows machines exist even when they are unreachable | +| Roughly what share is Windows? | We support Linux first; a large Windows estate changes our order of work | +| Which Linux distributions, and any very old ones? | Changes what we can promise — see section 3 | + +### Access + +1. **A read-only login on the machines** — a user (we suggest + `nudgebee-ro`) with an SSH key, able to run read-only commands and + nothing else. Normally pushed with whatever tool you already use to + manage machines. We will give you the exact commands. +2. **A read-only account on your virtualisation platform** — view + permissions only. 3. **A read-only Active Directory account**, if you want us to use it. - An ordinary user with no special group membership. + An ordinary user, no special groups. +4. **One small VM per network** to run the collector. Outbound internet + on port 443, no incoming access at all. -4. **A place to run the collector** — one small VM per network. It - needs outbound internet access on port 443 and no incoming access - at all. - -### 4c. Permission to look — the bit people forget +### Permission to look Scanning a network looks exactly like an attack, because technically it -is the same activity. Before we start we need: - -- **Your security team to expect it**, and to allow the collector's - address in whatever intrusion detection you run. Without this, the - first scan generates a security incident. -- **Agreement on which address ranges** we may look at, and which we - must not touch — printers, industrial controllers, medical devices, - anything fragile. -- **Agreement on when.** We can restrict scanning to specific hours. - -We deliberately scan gently: normal connections only, at a slow rate -you control, with no unusual traffic of the sort that can upset older +is the same activity. Before we start: + +- **Your security team needs to expect it**, and to allow the + collector's address in your intrusion detection. Without this, the + first scan becomes a security incident. +- **Agree which address ranges** we may look at, and which we must not + touch — printers, industrial controllers, medical devices, anything + fragile. +- **Agree when.** Scanning can be restricted to specific hours. + +We scan gently by design: ordinary connections only, at a slow rate you +control, with none of the unusual traffic that can upset older equipment. -## 5. Decisions we should make together - -### 5a. Identifying a machine reliably +## 5. The decision we need from you -This one has a real trade-off and we would like your view. +To avoid listing a machine twice, we need something that uniquely +identifies it. There are two candidates: a **machine ID** file, readable +by an ordinary user, and a **hardware ID** from the virtual BIOS, which +on Linux only an administrator can read. -To avoid listing the same machine twice, we need something that -uniquely identifies it. There are two candidates: +Your virtualisation platform reports the hardware ID. Our read-only +login can only read the machine ID. With just those two, we cannot tell +that "the machine VMware calls X" and "the machine we logged into" are +the same box — so it appears twice. -- A **machine ID** file, readable by an ordinary user. -- A **hardware ID** from the virtual BIOS, which on Linux can only be - read by an administrator. +| Option | Result | Cost | +|---|---|---| +| Allow the read-only user to run one extra command that reads the hardware ID | Accurate count | One small permission per machine | +| Match on name and network address instead | No setup | Less reliable; addresses get reused and we would occasionally be wrong | +| Accept duplicates, merge them in the interface | No setup | The machine count is approximate | -Your virtualisation platform reports the *hardware* ID. Our read-only -login can only read the *machine* ID. If we only have those two, we -cannot tell that "the machine VMware calls X" and "the machine we -logged into" are the same box — so it appears twice. +We lean towards the first, but your estimate of the effort should +decide it. -Three ways forward: - -1. **Grant one narrow extra permission** — allow the read-only user to - run a single command that reads the hardware ID, and nothing else. - Cleanest result; costs one small permission on each machine. -2. **Match on name and address instead.** No extra permission, but less - reliable — addresses get reused, and occasionally we would get it - wrong. -3. **Accept some duplicates** and let you merge them in the interface. - No setup cost, but the machine count is then approximate. - -We lean towards option 1, but it is your estimate of the effort that -should decide it. - -### 5b. How often, and how fast - -Defaults we suggest, all adjustable: look for new machines daily, -re-check software weekly, scan slowly enough to be invisible in -network monitoring. - -### 5c. What happens to the data - -The collector sends what it read — machine names, operating systems, -installed software lists. It does not read your files, your databases, -or your application data. If there are categories you do not want -leaving your network, tell us now rather than later. +Two smaller ones, both with sensible defaults you can change: how often +we look (daily for new machines, weekly for software), and whether any +categories of data must not leave your network. The collector reads +machine names, operating systems and software lists — not your files, +databases, or application data. ## 6. Where we actually are -Being straight about status, because "in progress" covers too much. - -**Working now, tested on real machines:** - -- Finding machines on a network -- Reading the full installed-software list over SSH, on both major - Linux families -- Reading Active Directory -- The collector runs as a service and communicates securely - -**Not built yet:** - -- Storing the results and presenting them to you — this is the next - piece of work and the biggest remaining one -- Reading your virtualisation platform — we build this once you tell - us which one you use -- Automatic scheduling, so it runs by itself rather than on request -- Windows machines -- Matching software against known vulnerabilities. That is the next - phase, and this one has to be right first +**Working now, tested on real machines:** finding machines on a +network; reading the full installed-software list over SSH on both +major Linux families; reading Active Directory; running as a service +and communicating securely. -**Known limits we are not hiding:** +**Not built yet:** storing the results and showing them to you — the +next and largest piece; reading your virtualisation platform, which we +build once you tell us which one; automatic scheduling, so it runs by +itself; Windows; and matching software against known vulnerabilities, +which is the following phase and needs this one to be right first. -- A machine that is switched off is invisible until we connect to your - virtualisation platform. Nothing that looks at a network can see a - machine that is off. -- A machine with no SSH access can be found, but not inventoried. It - will show in the report as a gap with the reason. -- Very old systems can be inventoried, but there may be no patches - available for them. We will say so plainly. +**Limits we are not hiding:** -## 7. What we would like next +- A machine that is switched off cannot be seen by anything that looks + at a network. It stays invisible until we connect to your + virtualisation platform. +- A machine we cannot log into can still be found and counted, but not + inventoried. It appears in the report with the reason. -1. The answers in section 4a — mainly which virtualisation platform. -2. Your view on the identification trade-off in 5a. -3. A single network to try first. Ideally a small one, with machines - you know, so that when we show you the result you can tell whether - it is right. +## 7. Next step -The last point is the important one. The first run is only useful if -you can check it against what you already know is there. +Pick one network to try — ideally a small one whose contents you +already know. The first run is only useful if you can check the result +against what you believe is there. From 785aae68b9b67274468fda5c23f80e1834c9aa0d Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Tue, 4 Aug 2026 10:07:20 +0530 Subject: [PATCH 14/16] docs: rewrite the customer document in a plainer register MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous version read as generated: an em-dash in nearly every paragraph, rule-of-three constructions, a rhetorical turn per section, bold lead-ins on every bullet, and headings like 'Limits we are not hiding' that perform honesty rather than being honest. Same content, flatter voice. No em-dashes, no bold, no tables, and the editorialising cut — the section on why per-machine agents are costly was selling rather than explaining, and the customer did not ask to be sold. --- docs/design/vm-discovery-customer.md | 255 ++++++++++++++------------- 1 file changed, 133 insertions(+), 122 deletions(-) diff --git a/docs/design/vm-discovery-customer.md b/docs/design/vm-discovery-customer.md index c91a555..7be1032 100644 --- a/docs/design/vm-discovery-customer.md +++ b/docs/design/vm-discovery-customer.md @@ -1,50 +1,48 @@ -# VM Discovery — how it works, and what we need from you +# VM Discovery -Draft for discussion. This is a joint build: some of it is ours, some -only you can do, and one choice we should make together. +Draft for discussion. Some of this we build, some of it only you can +set up, and there's one choice we'd like your opinion on. ---- +## What it does -## 1. What this solves +Finds the machines on your network and lists the software installed on +each one, so that patching has something accurate to work from. -Before you can patch anything, you have to know what you have. Most -teams have a list somewhere, and most of those lists are wrong. +For any machine it can't check, it says which machine and why. That +part is deliberate. An inventory that silently skips 50 machines is +harder to trust than one that reports 50 gaps. -We want to answer three questions honestly: how many machines you have, -what is installed on each, and — for anything we could not check — why -not. The third matters as much as the others. A tool reporting "400 -machines, all healthy" while quietly missing 50 is worse than one that -says "we found 450, we could not get into 50, here is why." +## How it works -## 2. How it works - -**Nothing is installed on your machines.** - -You run one small program — the collector — on a single machine in each -network. It looks around the network to see what is there, asks your -virtualisation platform and Active Directory what they know about, and -logs into each machine over SSH using a read-only account you create. +You run one small program on a single machine in each network. We call +it the collector. Nothing gets installed on the machines being +inventoried. ``` - ┌── one collector per network ──┐ - │ · finds machines │ - │ · logs in over SSH │ - └──────────┬────────────────────┘ + ┌── collector (one per network) ──┐ + │ finds machines │ + │ logs in over SSH │ + └──────────┬──────────────────────┘ │ SSH, read-only ┌───────────┼───────────┐ ▼ ▼ ▼ your VM your VM your VM ``` -The commands it runs are ones your own administrators would type — list -the installed packages, read the OS version. Nothing is changed, -written, or restarted. +The collector does three things: + +1. Probes the network to see which addresses respond. +2. Asks your virtualisation platform and Active Directory what they + know about. +3. Logs into each machine over SSH with a read-only account you create, + and runs read-only commands: list installed packages, read the OS + version, check whether a reboot is pending. -Installing software on hundreds of machines is a project in itself, and -keeping it updated is a permanent cost. One collector per network is -something you can approve, place, and watch. +Nothing is written, changed or restarted on your machines. -## 3. What you get +## What you get + +A per-network summary: ``` Found: 412 machines @@ -56,111 +54,124 @@ Cannot inventory 32: 8 login rejected 6 not seen for two weeks 4 powered off - 2 operating system we do not support yet - 1 running an OS with no security updates available + 2 operating system we don't support yet + 1 no security updates available for this OS ``` -Plus, per machine: name, operating system, the full list of installed -software with exact versions, and whether it is still receiving -security updates from its vendor. +And per machine: name, operating system, every installed package with +its exact version, and whether the machine is still entitled to +security updates. -That last field is worth the attention. A machine whose vendor -subscription has lapsed, or which runs an OS past end-of-life, **cannot -be patched at all** — better to know now than while trying to fix an -urgent vulnerability. +That last one is easy to overlook. If a machine's vendor subscription +has lapsed, or it runs an OS past end of life, no patch exists for it. +You want to find that out during an inventory, not during an incident. -## 4. What we need from you +## What we need from you -### Answers +### Questions -| Question | Why it matters | -|---|---| -| Which virtualisation platform — VMware, Proxmox, Hyper-V, plain KVM, or a mix? | We build support for yours first; see the first limit in section 6 | -| Roughly how many machines, across how many networks? | Sets scanning pace and how many collectors you need | -| Do you use Active Directory? | It knows machines exist even when they are unreachable | -| Roughly what share is Windows? | We support Linux first; a large Windows estate changes our order of work | -| Which Linux distributions, and any very old ones? | Changes what we can promise — see section 3 | +- Which virtualisation platform do you run: VMware, Proxmox, Hyper-V, + plain KVM, or a mix? We'll build support for yours first. It also + affects one of the limitations below. +- Roughly how many machines, and how many separate networks? This sets + how fast we scan and how many collectors you need. +- Do you use Active Directory? +- Roughly what proportion is Windows? We're doing Linux first, but a + large Windows estate would change that. +- Which Linux distributions, and are any of them old? See the note on + end of life above. -### Access +### Access you'd need to set up -1. **A read-only login on the machines** — a user (we suggest - `nudgebee-ro`) with an SSH key, able to run read-only commands and - nothing else. Normally pushed with whatever tool you already use to - manage machines. We will give you the exact commands. -2. **A read-only account on your virtualisation platform** — view +1. A read-only user on each machine (we suggest `nudgebee-ro`) with an + SSH key. It only needs to run read-only commands. You'd normally + push this with whatever you already use to manage machines. We'll + give you the commands. +2. A read-only account on your virtualisation platform. View permissions only. -3. **A read-only Active Directory account**, if you want us to use it. - An ordinary user, no special groups. -4. **One small VM per network** to run the collector. Outbound internet - on port 443, no incoming access at all. +3. A read-only Active Directory account, if you want us to use it. An + ordinary user account, no group memberships. +4. One small VM per network to run the collector on. It needs outbound + internet on port 443. It needs no inbound access at all. + +### Before we scan anything -### Permission to look +Network scanning looks like an attack to security tooling, so: -Scanning a network looks exactly like an attack, because technically it -is the same activity. Before we start: +- Your security team needs to know, and to allow the collector's + address in your intrusion detection. Otherwise the first scan + becomes an incident. +- We need to agree which address ranges are in scope, and which are + off limits. Printers, industrial controllers and medical devices are + the usual exclusions. +- We can restrict scanning to particular hours if you'd prefer. -- **Your security team needs to expect it**, and to allow the - collector's address in your intrusion detection. Without this, the - first scan becomes a security incident. -- **Agree which address ranges** we may look at, and which we must not - touch — printers, industrial controllers, medical devices, anything - fragile. -- **Agree when.** Scanning can be restricted to specific hours. +The scan itself uses ordinary connections at a rate you set, not the +sort of traffic that upsets older equipment. -We scan gently by design: ordinary connections only, at a slow rate you -control, with none of the unusual traffic that can upset older -equipment. +## The choice we'd like your opinion on -## 5. The decision we need from you +To avoid counting a machine twice, we need something that identifies it +uniquely. There are two candidates: -To avoid listing a machine twice, we need something that uniquely -identifies it. There are two candidates: a **machine ID** file, readable -by an ordinary user, and a **hardware ID** from the virtual BIOS, which -on Linux only an administrator can read. +- A machine ID file, which any user can read. +- A hardware ID from the virtual BIOS, which on Linux only an + administrator can read. Your virtualisation platform reports the hardware ID. Our read-only -login can only read the machine ID. With just those two, we cannot tell -that "the machine VMware calls X" and "the machine we logged into" are -the same box — so it appears twice. - -| Option | Result | Cost | -|---|---|---| -| Allow the read-only user to run one extra command that reads the hardware ID | Accurate count | One small permission per machine | -| Match on name and network address instead | No setup | Less reliable; addresses get reused and we would occasionally be wrong | -| Accept duplicates, merge them in the interface | No setup | The machine count is approximate | - -We lean towards the first, but your estimate of the effort should -decide it. - -Two smaller ones, both with sensible defaults you can change: how often -we look (daily for new machines, weekly for software), and whether any -categories of data must not leave your network. The collector reads -machine names, operating systems and software lists — not your files, -databases, or application data. - -## 6. Where we actually are - -**Working now, tested on real machines:** finding machines on a -network; reading the full installed-software list over SSH on both -major Linux families; reading Active Directory; running as a service -and communicating securely. - -**Not built yet:** storing the results and showing them to you — the -next and largest piece; reading your virtualisation platform, which we -build once you tell us which one; automatic scheduling, so it runs by -itself; Windows; and matching software against known vulnerabilities, -which is the following phase and needs this one to be right first. - -**Limits we are not hiding:** - -- A machine that is switched off cannot be seen by anything that looks - at a network. It stays invisible until we connect to your - virtualisation platform. -- A machine we cannot log into can still be found and counted, but not - inventoried. It appears in the report with the reason. - -## 7. Next step - -Pick one network to try — ideally a small one whose contents you -already know. The first run is only useful if you can check the result -against what you believe is there. +login can only get the machine ID. So if we rely on those alone, we +can't tell that the machine VMware calls X is the same one we logged +into, and it gets counted twice. + +Three ways to handle it: + +1. Let the read-only user run one extra command that reads the hardware + ID. Gives an accurate count. Costs one narrow permission on each + machine. +2. Match on hostname and IP instead. Nothing to set up, but less + reliable, since IPs get reused. +3. Live with duplicates and merge them in the interface. Nothing to set + up, but the machine count is approximate. + +We'd pick the first, but it depends how much work that permission is in +your environment, which you'd know better than us. + +Two smaller things, both defaults you can change: how often we look +(daily for new machines, weekly for software), and whether any of this +data shouldn't leave your network. The collector reads machine names, +OS versions and package lists. It doesn't read files, databases or +application data. + +## Where this actually is + +Working, and tested against real machines: + +- Finding machines on a network +- Reading the full package list over SSH on both major Linux families +- Reading Active Directory +- Running as a service, talking to us securely + +Not built yet: + +- Storing the results and showing them to you. This is the next piece + of work and the largest. +- Reading your virtualisation platform. We build this once you tell us + which one you have. +- Scheduling, so it runs on its own rather than on request. +- Windows. +- Matching packages against known vulnerabilities. That's the next + phase and it depends on this one being right. + +## What it can't do + +- A machine that's switched off can't be found by anything that looks + at a network. It stays invisible until we can read your + virtualisation platform, which is the only thing that knows about it. +- A machine we can't log into gets found and counted, but not + inventoried. It shows in the report with the reason. + +## Suggested next step + +Pick one network to start with, ideally a small one where you already +know roughly what's there. The first run is only useful if you can +check the answer against something. From a9716e105e63161946cc977d6e2af3d2b2da398c Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Tue, 4 Aug 2026 10:50:38 +0530 Subject: [PATCH 15/16] docs: fix the Active Directory framing in the customer document MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asked 'do you use Active Directory?', which answers nothing — everyone uses it for user authentication. The question that decides whether it helps discovery is whether their Linux servers are domain-joined, and most are not. AD holds a computer object per domain-joined machine, which is what we query (objectCategory=computer, not users). But domain-joining Linux is a minority setup, so for a Linux-first scope AD will usually return little. The doc now says so instead of listing it beside the hypervisor as an equal source. --- docs/design/vm-discovery-customer.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/design/vm-discovery-customer.md b/docs/design/vm-discovery-customer.md index 7be1032..123755a 100644 --- a/docs/design/vm-discovery-customer.md +++ b/docs/design/vm-discovery-customer.md @@ -32,8 +32,9 @@ inventoried. The collector does three things: 1. Probes the network to see which addresses respond. -2. Asks your virtualisation platform and Active Directory what they - know about. +2. Asks your virtualisation platform what machines exist, including ones + that are switched off. If your Linux servers are joined to an Active + Directory domain, it can ask that too. 3. Logs into each machine over SSH with a read-only account you create, and runs read-only commands: list installed packages, read the OS version, check whether a reboot is pending. @@ -75,7 +76,10 @@ You want to find that out during an inventory, not during an incident. affects one of the limitations below. - Roughly how many machines, and how many separate networks? This sets how fast we scan and how many collectors you need. -- Do you use Active Directory? +- Are your Linux servers joined to an Active Directory domain? Most + aren't, and if yours aren't, AD won't help here. AD holds a record + for each domain-joined machine, which is useful when it applies, but + it is normally Windows machines that are joined rather than Linux. - Roughly what proportion is Windows? We're doing Linux first, but a large Windows estate would change that. - Which Linux distributions, and are any of them old? See the note on @@ -89,8 +93,9 @@ You want to find that out during an inventory, not during an incident. give you the commands. 2. A read-only account on your virtualisation platform. View permissions only. -3. A read-only Active Directory account, if you want us to use it. An - ordinary user account, no group memberships. +3. A read-only Active Directory account, but only if your Linux + machines are domain-joined. An ordinary user account, no group + memberships. 4. One small VM per network to run the collector on. It needs outbound internet on port 443. It needs no inbound access at all. @@ -148,7 +153,7 @@ Working, and tested against real machines: - Finding machines on a network - Reading the full package list over SSH on both major Linux families -- Reading Active Directory +- Reading Active Directory, for domain-joined machines - Running as a service, talking to us securely Not built yet: From 247f102e744d33ce872bece1c607661267a0f92b Mon Sep 17 00:00:00 2001 From: mayankpande88 Date: Tue, 4 Aug 2026 10:55:16 +0530 Subject: [PATCH 16/16] docs: name the three identifiers instead of assuming they are understood Reviewer asked whether machine ID and MAC address are the same thing, which they are not, and the section did not say so. It now names all three with real example values and states which side can see each. Also corrects the second option. It said 'match on hostname and IP', having overlooked that both the hypervisor and an SSH login can see MAC addresses, which are more stable than IPs. That makes the no-setup option meaningfully better than described, and the customer was being asked to choose against a weaker version of it than actually exists. --- docs/design/vm-discovery-customer.md | 33 +++++++++++++++++----------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/docs/design/vm-discovery-customer.md b/docs/design/vm-discovery-customer.md index 123755a..64bb6b8 100644 --- a/docs/design/vm-discovery-customer.md +++ b/docs/design/vm-discovery-customer.md @@ -116,25 +116,32 @@ sort of traffic that upsets older equipment. ## The choice we'd like your opinion on -To avoid counting a machine twice, we need something that identifies it -uniquely. There are two candidates: +The same machine can be seen twice: once by your virtualisation +platform, and once when we log into it. To show it once, we need +something that tells us both sightings are the same box. -- A machine ID file, which any user can read. -- A hardware ID from the virtual BIOS, which on Linux only an - administrator can read. +Three identifiers are in play, and they are not the same thing: -Your virtualisation platform reports the hardware ID. Our read-only -login can only get the machine ID. So if we rely on those alone, we -can't tell that the machine VMware calls X is the same one we logged -into, and it gets counted twice. +- **Machine ID.** A random string written when the OS was installed, + for example `ec2403e319a2f3f0ae53a05e3daf084b`. Any user can read + it. Your virtualisation platform has never heard of it. +- **Hardware ID.** The virtual BIOS serial number, set when the VM was + created. Your virtualisation platform knows every VM by this. On + Linux only an administrator can read it, so our read-only login + cannot. +- **MAC address.** The hardware address of a network card, for example + `02:4d:07:48:c4:87`. Both sides can see this one. -Three ways to handle it: +So the platform knows the hardware ID, we know the machine ID, and +neither recognises the other's. Three ways to bridge that: 1. Let the read-only user run one extra command that reads the hardware - ID. Gives an accurate count. Costs one narrow permission on each + ID. Exact match, no guessing. Costs one narrow permission on each machine. -2. Match on hostname and IP instead. Nothing to set up, but less - reliable, since IPs get reused. +2. Match on MAC address, plus hostname and IP as a cross-check. Nothing + to set up. Usually right, but not guaranteed: a machine can have + several network cards, and a cloned VM can end up sharing a MAC with + the machine it was cloned from. 3. Live with duplicates and merge them in the interface. Nothing to set up, but the machine count is approximate.