diff --git a/docs/design/vm-discovery-aws-testbed.md b/docs/design/vm-discovery-aws-testbed.md new file mode 100644 index 0000000..8390d69 --- /dev/null +++ b/docs/design/vm-discovery-aws-testbed.md @@ -0,0 +1,210 @@ +# VM Discovery — AWS testbed setup + +Status: RUNBOOK — executed 2026-08-03, results inline +Goal: first real sweep and inventory, on our own AWS, with a locally built +binary. No dependency on releases or on the server side. + +## What runs where + +Three EC2 instances in one small subnet: + +``` +subnet 10.x.y.0/28 (isolated, us-east-1) +┌──────────────────────────────────────────────┐ +│ forager-discovery ← runs the forager │ +│ │ sweeps the subnet, SSHes into targets │ +│ ├──► target-ubuntu (Debian-like) │ +│ └──► target-al2023 (RHEL-like) │ +└──────────────────────────────────────────────┘ +``` + +**The forager runs on its own EC2 instance, not as a pod on EKS.** Two +reasons, and the second is the one that matters: + +1. It matches how this is actually deployed — one node per network segment. + Testing a different topology than the product ships would prove less than + it appears to. +2. MAC enrichment reads the kernel neighbour cache after probing. On EKS with + the VPC CNI, a pod's neighbour cache does not see other instances' L2 + entries, so MACs would come back empty and we would not know whether that + is a bug or the environment. + +Targets are two distros on purpose: `dpkg-query` and `rpm -qa` are different +collectors in the pack, and a run that only exercises one proves half the +matrix. Amazon Linux 2023 is RHEL-like (dnf/rpm), so it covers that family +without needing a RHEL subscription. + +## 1. Create the instances + +Anything small is fine — `t3.micro` throughout. Requirements: + +- All three in the **same subnet**, so the sweep's L2 neighbour lookup works. +- Security group allowing `forager-discovery` → targets on TCP 22. +- The forager host needs outbound 443 to reach the relay. It needs **no + inbound** at all. +- Tag them so they are obviously disposable. + +A `/28` gives 11 usable addresses — enough to see the sweep correctly report +"3 responded out of 11 scanned", which is the result that tells you the +address expansion and exclusion logic are right. + +## 2. Build and copy the binary + +```bash +# from the forager repo +make build-all # or: CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ + # go build -o build/nudgebee-forager-linux-amd64 ./cmd +scp build/nudgebee-forager-linux-amd64 ec2-user@:/tmp/ +``` + +Use `-linux-arm64` instead if you pick Graviton instances. + +```bash +sudo install -m 0755 /tmp/nudgebee-forager-linux-amd64 /usr/local/bin/nudgebee-forager +sudo mkdir -p /etc/nudgebee /var/lib/nudgebee +``` + +## 3. Configure + +`/etc/nudgebee/forager.yaml` on the forager host: + +```yaml +relay_url: wss://relay.dev.nudgebee.pollux.in/register +access_key: REPLACE_ME +access_secret: REPLACE_ME +data_dir: /var/lib/nudgebee +signing_public_key: 'fA+zl69KzfyYZ8+f722PGR8h8TUP/u76nngOGKUUpYo=' + +datasources: + - name: aws-testbed + type: discovery + + # Scope ceiling. The subnet and nothing else. + allowed_hosts: + - 10.x.y.0/28 + + discovery: + max_rate_pps: 50 + concurrency: 10 + + # Omit credentials entirely for the sweep-only stage. The node will + # report it can serve discovery_sweep and not discovery_inventory, + # which is the honest state — and is only possible from rc.2 onward. +``` + +Point it at the **dev** relay, not prod: a half-configured node should not +appear in the prod account while we are shaking it out. + +Run it in the foreground first rather than as a service — the startup logs +say whether the datasource configured, and that is most of what stage 1 is +checking: + +```bash +sudo nudgebee-forager --config /etc/nudgebee/forager.yaml +``` + +Expect `discovery proxy configured` with `allowed_cidrs=1`, and +`ssh host key verification disabled` (correct — no known_hosts yet). + +## 4. Sweep + +Nothing schedules discovery yet, so trigger it by hand through the relay: + +```json +{"action": "discovery_sweep", + "datasource_id": "local:aws-testbed", + "params": {"cidrs": ["10.x.y.0/28"], "ports": [22], "rate_pps": 50}} +``` + +`ports: [22]` deliberately — the default also probes 3389/5985, which on a +Linux-only testbed generates RDP/WinRM probes for no benefit. + +What a correct result looks like: `addresses_scanned` of 11 for a /28 (16 +minus network, broadcast, and AWS's three reserved addresses will still be +scanned — AWS reserves them but they simply will not answer), three hosts +returned with `open_ports: [22]`, and MACs present since everything is on the +same L2. + +If MACs come back empty, that is the finding — it means the neighbour-cache +read is not working in this environment, which is exactly what running on EC2 +rather than EKS was meant to establish. + +## 5. Inventory + +Needs a signed pack, which does not exist yet (#116). For the testbed, +generate a throwaway keypair and sign the example pack: + +```bash +# keypair +go run ./cmd/... # no helper yet — see pkg/signing.GenerateKeypair +``` + +Then on each target: + +```bash +sudo useradd --system --create-home --shell /bin/sh nudgebee-ro +sudo install -d -m 700 -o nudgebee-ro ~nudgebee-ro/.ssh +echo '' | sudo tee ~nudgebee-ro/.ssh/authorized_keys +sudo chmod 600 ~nudgebee-ro/.ssh/authorized_keys +sudo chown nudgebee-ro: ~nudgebee-ro/.ssh/authorized_keys +``` + +Add to the datasource: `pack_public_key`, `pack_dir`, and the SSH credential. +Then: + +```json +{"action": "discovery_inventory", + "datasource_id": "local:aws-testbed", + "params": {"targets": ["10.x.y.4", "10.x.y.5"], "content_pack_version": 1}} +``` + +The check that matters is not "did it return data" but **whether the two +distros ran different collectors** — `pkgs-dpkg` on Ubuntu, `pkgs-rpm` on +AL2023 — and whether version strings came back with epoch and release intact +(`5.14.0-362.8.1.el9_3`, not `5.14.0`). Backport-aware CVE matching in epic 2 +depends on that, and it is the thing most likely to be silently wrong. + +**The throwaway signing key must not become load-bearing.** Where the real +key lives is an open question on #116; generate this one, use it, and do not +put it anywhere durable. + +## What actually happened (2026-08-03) + +Built on account 864186153326 (sandbox, not the prod 740395098545), +default VPC, subnet `subnet-0e08ebf5c41c11e4d`, instances pinned to +`172.31.0.10/.11/.12`. The default security group needed one added +rule — its self-referencing rule covered only 5432, so nothing could +reach port 22 and the sweep would have returned zero hosts for reasons +having nothing to do with the code. + +**Sweep** — 14 addresses scanned, 3 found, 1.2s. `172.31.0.10` came +back without a MAC while the other two had one: a host does not ARP +for its own address, so it is absent from its own neighbour cache. +That is the result that shows the MAC enrichment reads the kernel +cache rather than inventing values — and is precisely what running on +EKS would have obscured. + +**Inventory** — both hosts in 1.6s, 609 and 494 packages. The right +collectors ran on each family and neither ran the other's. Epoch and +release survived intact (`openssl 1 3.5.5 1.amzn2023.0.5`; Debian's +`acpid 1:2.0.33-1ubuntu1` kept its `1:`). + +**Findings worth carrying forward:** + +1. `/sys/class/dmi/id/product_uuid` is `-r--------` root-only, so the + unprivileged credential never reads an SMBIOS UUID. See §6 of the + design doc — this breaks hypervisor↔SSH merging on-prem. +2. `rpm` prints `(none)` for a missing epoch, not `0` or empty, and + the two are not equivalent for version comparison. Reported on + nudgebee-enterprise#35405. +3. `board_asset_tag` is world-readable and carries the EC2 instance + id, so the cloud-VM merge path works unprivileged. + +Nothing schedules discovery yet, so both stages were driven through +the exported proxy API by throwaway harnesses rather than the relay. + +## Cost and teardown + +Three t3.micro instances are a few dollars a month. They are disposable — +terminate them once the coverage report reads correctly, and rebuild from +this document if needed. diff --git a/docs/design/vm-discovery-customer.md b/docs/design/vm-discovery-customer.md new file mode 100644 index 0000000..64bb6b8 --- /dev/null +++ b/docs/design/vm-discovery-customer.md @@ -0,0 +1,189 @@ +# VM Discovery + +Draft for discussion. Some of this we build, some of it only you can +set up, and there's one choice we'd like your opinion on. + +## What it does + +Finds the machines on your network and lists the software installed on +each one, so that patching has something accurate to work from. + +For any machine it can't check, it says which machine and why. That +part is deliberate. An inventory that silently skips 50 machines is +harder to trust than one that reports 50 gaps. + +## How it works + +You run one small program on a single machine in each network. We call +it the collector. Nothing gets installed on the machines being +inventoried. + +``` + ┌── collector (one per network) ──┐ + │ finds machines │ + │ logs in over SSH │ + └──────────┬──────────────────────┘ + │ SSH, read-only + ┌───────────┼───────────┐ + ▼ ▼ ▼ + your VM your VM your VM +``` + +The collector does three things: + +1. Probes the network to see which addresses respond. +2. Asks your virtualisation platform what machines exist, including ones + that are switched off. If your Linux servers are joined to an Active + Directory domain, it can ask that too. +3. Logs into each machine over SSH with a read-only account you create, + and runs read-only commands: list installed packages, read the OS + version, check whether a reboot is pending. + +Nothing is written, changed or restarted on your machines. + +## What you get + +A per-network summary: + +``` +Found: 412 machines + Reachable: 391 (we can log in) + Inventoried: 380 (we have the software list) + +Cannot inventory 32: + 11 no credentials or blocked by firewall + 8 login rejected + 6 not seen for two weeks + 4 powered off + 2 operating system we don't support yet + 1 no security updates available for this OS +``` + +And per machine: name, operating system, every installed package with +its exact version, and whether the machine is still entitled to +security updates. + +That last one is easy to overlook. If a machine's vendor subscription +has lapsed, or it runs an OS past end of life, no patch exists for it. +You want to find that out during an inventory, not during an incident. + +## What we need from you + +### Questions + +- Which virtualisation platform do you run: VMware, Proxmox, Hyper-V, + plain KVM, or a mix? We'll build support for yours first. It also + affects one of the limitations below. +- Roughly how many machines, and how many separate networks? This sets + how fast we scan and how many collectors you need. +- Are your Linux servers joined to an Active Directory domain? Most + aren't, and if yours aren't, AD won't help here. AD holds a record + for each domain-joined machine, which is useful when it applies, but + it is normally Windows machines that are joined rather than Linux. +- Roughly what proportion is Windows? We're doing Linux first, but a + large Windows estate would change that. +- Which Linux distributions, and are any of them old? See the note on + end of life above. + +### Access you'd need to set up + +1. A read-only user on each machine (we suggest `nudgebee-ro`) with an + SSH key. It only needs to run read-only commands. You'd normally + push this with whatever you already use to manage machines. We'll + give you the commands. +2. A read-only account on your virtualisation platform. View + permissions only. +3. A read-only Active Directory account, but only if your Linux + machines are domain-joined. An ordinary user account, no group + memberships. +4. One small VM per network to run the collector on. It needs outbound + internet on port 443. It needs no inbound access at all. + +### Before we scan anything + +Network scanning looks like an attack to security tooling, so: + +- Your security team needs to know, and to allow the collector's + address in your intrusion detection. Otherwise the first scan + becomes an incident. +- We need to agree which address ranges are in scope, and which are + off limits. Printers, industrial controllers and medical devices are + the usual exclusions. +- We can restrict scanning to particular hours if you'd prefer. + +The scan itself uses ordinary connections at a rate you set, not the +sort of traffic that upsets older equipment. + +## The choice we'd like your opinion on + +The same machine can be seen twice: once by your virtualisation +platform, and once when we log into it. To show it once, we need +something that tells us both sightings are the same box. + +Three identifiers are in play, and they are not the same thing: + +- **Machine ID.** A random string written when the OS was installed, + for example `ec2403e319a2f3f0ae53a05e3daf084b`. Any user can read + it. Your virtualisation platform has never heard of it. +- **Hardware ID.** The virtual BIOS serial number, set when the VM was + created. Your virtualisation platform knows every VM by this. On + Linux only an administrator can read it, so our read-only login + cannot. +- **MAC address.** The hardware address of a network card, for example + `02:4d:07:48:c4:87`. Both sides can see this one. + +So the platform knows the hardware ID, we know the machine ID, and +neither recognises the other's. Three ways to bridge that: + +1. Let the read-only user run one extra command that reads the hardware + ID. Exact match, no guessing. Costs one narrow permission on each + machine. +2. Match on MAC address, plus hostname and IP as a cross-check. Nothing + to set up. Usually right, but not guaranteed: a machine can have + several network cards, and a cloned VM can end up sharing a MAC with + the machine it was cloned from. +3. Live with duplicates and merge them in the interface. Nothing to set + up, but the machine count is approximate. + +We'd pick the first, but it depends how much work that permission is in +your environment, which you'd know better than us. + +Two smaller things, both defaults you can change: how often we look +(daily for new machines, weekly for software), and whether any of this +data shouldn't leave your network. The collector reads machine names, +OS versions and package lists. It doesn't read files, databases or +application data. + +## Where this actually is + +Working, and tested against real machines: + +- Finding machines on a network +- Reading the full package list over SSH on both major Linux families +- Reading Active Directory, for domain-joined machines +- Running as a service, talking to us securely + +Not built yet: + +- Storing the results and showing them to you. This is the next piece + of work and the largest. +- Reading your virtualisation platform. We build this once you tell us + which one you have. +- Scheduling, so it runs on its own rather than on request. +- Windows. +- Matching packages against known vulnerabilities. That's the next + phase and it depends on this one being right. + +## What it can't do + +- A machine that's switched off can't be found by anything that looks + at a network. It stays invisible until we can read your + virtualisation platform, which is the only thing that knows about it. +- A machine we can't log into gets found and counted, but not + inventoried. It shows in the report with the reason. + +## Suggested next step + +Pick one network to start with, ideally a small one where you already +know roughly what's there. The first run is only useful if you can +check the answer against something. diff --git a/docs/design/vm-discovery-deployment.md b/docs/design/vm-discovery-deployment.md new file mode 100644 index 0000000..80ddef9 --- /dev/null +++ b/docs/design/vm-discovery-deployment.md @@ -0,0 +1,191 @@ +# VM Discovery — Deployment Plan + +Status: DRAFT +Covers: getting the merged discovery module (forager#118) into an +environment where it can actually run. + +## Where things stand + +Merged to `main` as c2ad9c7. **Not released** — `v0.1.3` is the latest +tag and discovery is one commit ahead of it. The release workflow fires +on a `v*.*.*` tag and publishes `ghcr.io/nudgebee/forager` plus the +chart to `oci://ghcr.io/nudgebee/charts`. + +Deploying the new image is safe on its own: nothing about existing +datasource types changed, so current foragers keep working. But +**end-to-end discovery cannot run yet** — four gaps, listed below with +what each one blocks. + +## Existing deployment pattern (what we build on) + +`nudgebee-infra/deploy/customers/rackspace/agents/forager-values-dev.yaml` +is the working precedent: + +- Chart `deploy/helm/forager` from this repo, release + `nudgebee-forager-dev`, namespace `nudgebee-agent-dev`. +- Points at `wss://relay.dev.nudgebee.pollux.in/register`. +- Secrets SOPS-encrypted with AWS KMS; `encrypted_regex` already + covers `accessKey|accessSecret|password|username`. +- Deployed via `installation.sh -f values.yaml -k`. + +The chart takes us most of the way: `forager.datasources` is passed +through with `toYaml`, so arbitrary datasource fields reach +`forager.yaml`, and `extraVolumes`/`extraVolumeMounts` exist for +mounting a pack directory. No chart changes are strictly required. + +## Blockers + +### 1. Local config cannot configure a discovery datasource + +`cmd/app.go` maps only `allowed_hosts` → `allowed_cidrs`. There is no +path from `forager.yaml` to `pack_public_key`, `pack_dir`, `ldap`, +`known_hosts_file`, `max_rate_pps`, or the port/concurrency/timeout +settings — `config.LocalDatasource` has no fields for them. + +Consequence per action: + +| Action | Works from local YAML today | +|---|---| +| `discovery_sweep` | yes — only needs `allowed_cidrs` | +| `discovery_ldap` | no — directory config unreachable | +| `discovery_inventory` | no — without a pack key nothing runs | + +The production intent (design §9) is that the server pushes datasource +config, which would sidestep this. But that path needs the server-side +work that does not exist yet, so local config is what we have. + +**Fix:** add the discovery fields to `config.LocalDatasource` and map +them in `app.go`. Small, self-contained. + +### 2. A sweep-only datasource still demands SSH credentials + +`Configure` returns `ssh username is required` before it looks at +anything else. A datasource intended purely for sweeping — which needs +no credentials at all — cannot be configured without inventing dummy +ones. + +**Fix:** only require SSH credentials when they are needed, and let +`HealthCheck` report which actions a given datasource can serve. +Otherwise the first thing an operator does is put fake credentials in +a values file, which is a bad habit to teach. + +### 3. No signed content pack exists + +`docs/content-packs/linux-inventory-example.yaml` is deliberately +unsigned and is a format reference, not a shippable pack. The real +pack, CI signing, and pin/ring rollout are #116. + +`discovery_inventory` refuses to run without a verified pack, so this +blocks inventory entirely — by design. + +**Fix for a first deployment:** generate a keypair, sign the example +pack by hand, mount it via `extraVolumes`, set `pack_public_key`. Not +a substitute for #116. + +### 4. Nothing triggers discovery + +There is no scheduler and no UI: the server never sends +`discovery_sweep`, `discovery_ldap`, or `discovery_inventory`. A +deployed forager would connect and sit idle. That is +nudgebee-enterprise#35405 plus the scheduling decision. + +Relay signing is *not* a blocker — `DefaultSigningFields` +(`action`, `datasource_id`, `params`) already matches all three +actions. Adding them explicitly to `SigningFields` in +`relay-server/pkg/signing/signer.go` is worth doing for clarity, but +nothing breaks without it. + +## Plan + +### Phase A — ship a release candidate (no dependencies) + +Cut `v0.1.4-rc.1`, not `v0.1.4`. The workflow already treats a tag +containing `-` as a prerelease, and the precedent exists (`v0.1.2-rc.1`). + +What an rc tag does and does not touch: + +| | rc tag | final tag | +|---|---|---| +| `ghcr.io/nudgebee/forager:0.1.4-rc.1` | pushed | — | +| `ghcr.io/nudgebee/forager:latest` | **untouched** (`latest=auto` skips prereleases) | moves | +| `0.1` shorthand tag | not applied to prereleases | moves | +| GitHub Release | marked prerelease | full release | +| `registry.nudgebee.com` mirror (ECR + S3) | **skipped** — the mirror jobs are gated on `!contains(github.ref, '-')` | synced | +| Chart `oci://ghcr.io/nudgebee/charts/forager:0.1.4-rc.1` | new version, does not overwrite 0.1.3 | new version | + +So no customer-facing pointer moves. Existing installs consuming +`:latest` or the mirror stay on 0.1.3 until we cut a final tag. + +1. Tag `v0.1.4-rc.1` on `main`. +2. Roll the Rackspace **dev** forager onto it by bumping `image.tag` + in `forager-values-dev.yaml`. +3. Confirm the existing `rackspace-pg` datasource still works — a pure + regression check, no discovery involved. + +Value: the binary is available and exercised without committing to a +release, and any discovery rollout afterwards is a config change +rather than an image change. Phases B–D can iterate through +`-rc.2`, `-rc.3` at no cost to anyone downstream. + +### Phase B — make discovery configurable (blockers 1 and 2) + +One forager PR: + +- Add discovery fields to `config.LocalDatasource` and map them in + `app.go`. +- Require SSH credentials only when the datasource will serve + inventory. +- Extend `values-enterprise.yaml.example` with a documented discovery + datasource. + +### Phase C — first real sweep (needs A + B) + +Deploy a second forager release scoped to sweeping only, in the +Rackspace cluster: + +- `allowed_cidrs` limited to a **known-safe range** — the cluster's own + pod/service CIDR, not a customer network. +- Exclusions for anything that should never be probed. +- `ports: [22]` to start, so no RDP/WinRM probes appear in security + monitoring. +- Rate left at the default 100 pps. + +Because nothing schedules actions yet, trigger one by hand through the +relay and read the response. That is the smallest thing that proves the +network path, signing, and scope enforcement all work together. + +Before this runs anywhere near a customer network, the forager's IP +needs allowlisting in their IDS — an unannounced sweep looks exactly +like an attacker, and a bare TCP connect still makes `sshd` log +`Did not receive identification string`, which some fail2ban configs +act on. + +### Phase D — inventory (needs C + a pack) + +- Generate a pack signing keypair; store the private key where CI can + reach it for #116, and put the public key in the values file. +- Sign the example pack, mount it read-only at `pack_dir` via + `extraVolumes`. +- Create a `nudgebee-ro` user on one or two test VMs. +- Run `discovery_inventory` by hand against those hosts and check the + raw collector output. + +### Phase E — scheduled and server-driven + +Everything above is manual. Real operation needs +nudgebee-enterprise#35405 (asset storage and ingest) plus the +scheduler, at which point datasource config moves to server-push and +local YAML becomes a bootstrap detail. + +## Open questions + +1. Which cluster for the first sweep — Rackspace dev, or a dedicated + test environment? Rackspace is real customer infrastructure, so the + CIDR scope needs deciding with care. +2. Where does the pack signing private key live? CI secret for #116 is + the obvious answer, but it should be settled before we hand-sign + anything, so the throwaway key does not become load-bearing. +3. Do we reuse the existing `nudgebee-forager-dev` release for + discovery, or run a separate one? Separate is cleaner — different + blast radius, different credentials, and a sweep misconfiguration + should not be able to disturb the working pg datasource. diff --git a/docs/design/vm-discovery-next.md b/docs/design/vm-discovery-next.md new file mode 100644 index 0000000..890863a --- /dev/null +++ b/docs/design/vm-discovery-next.md @@ -0,0 +1,114 @@ +# VM Discovery — what's next + +Status: PLAN, 2026-08-03 +Written after the AWS testbed run, which changed the picture in two ways. + +## Where we are + +The agent half is done and proven on real hosts. #113 and #114 are +merged and closed; `v0.1.4-rc.2` is released with a green pipeline; a +forager is deployed on EC2, registered as `aws-dev-proxy`, and both +sweep and inventory were validated against Ubuntu and Amazon Linux +targets (609 and 494 packages, correct per-family collectors, epoch and +release intact). + +**And it does nothing.** The agent is registered and idle, because +nothing can send it a discovery action. Everything user-visible is +behind the server side. + +That is the honest summary: we have a working collector and no product. + +## The one decision that blocks design, not code + +**SMBIOS UUID is unreadable by the unprivileged credential.** Verified +on the testbed: `/sys/class/dmi/id/product_uuid` is mode `-r--------`. + +The consequence is not obvious and is easy to build straight past. A +hypervisor knows a VM by its SMBIOS UUID. An SSH inventory knows the +same VM by its machine-id. On-prem, with no sudo, **those two sources +share no strong identifier** — so §8.2's rule that only strong +identifiers may merge means the same VM lands in the asset list twice, +once from each source, forever. + +Three ways out, and someone has to pick before #115 or the #35405 +schema hardens: + +1. **Require the narrow `dmidecode` sudoers entry.** Cleanest data + model; costs a privilege we currently advertise as unnecessary, and + it is a per-host change customers must roll out. +2. **Merge hypervisor↔SSH on corroborated weak identifiers** (hostname + plus IP, say). Keeps the credential unprivileged; weakens the rule + that exists precisely because DHCP recycles IPs. If we do this, the + corroboration threshold needs writing down, not left to judgement. +3. **Accept the duplication** and reconcile in the UI. Cheapest now, + worst to live with — "how many VMs do I have" is the question this + epic exists to answer, and it would answer it wrong. + +Cloud VMs are unaffected: `board_asset_tag` is world-readable and +carries the instance id, so the cloud↔SSH merge works today. + +## Critical path + +### 1. Server side — nudgebee-enterprise#35405 + +The only thing between us and something demoable. Nothing else on this +list changes what a user can see. + +Carries two testbed findings already reported on the ticket: +`rpm` prints `(none)` for a missing epoch (and `(none)` ≠ `0` for +version comparison), and golden files for the parsers can be captured +from the testbed rather than hand-written. + +Also needs the identity decision above, since it determines what the +merge logic is allowed to do. + +### 2. Scheduling + +Deliberately left undecided in the ticket breakdown. Now it is the +thing standing between a deployed agent and a running one. Candidate +remains reusing `scan_orchestrator`; confirm or reject at #35405 +kickoff rather than carrying it as an open question indefinitely. + +### 3. Content pack pipeline — forager#116 + +Required before any customer deployment. The testbed runs on a +hand-signed pack and a throwaway key that must not become +load-bearing. Where the real signing key lives is still unanswered and +should be settled as part of this. + +### 4. Hypervisor connector — forager#115 + +Still blocked on which hypervisor the customer runs, and now also on +the identity decision. Worth noting it is the only source that sees +powered-off VMs, so the coverage report's denominator is wrong without +it. + +## Small fixes worth doing while the above runs + +- **`install.sh` fails cryptically on a stale `/tmp` file.** It + downloads to `/tmp/nudgebee-forager`; if that path exists owned by + another user, `fs.protected_regular=1` (default on AL2023 and most + modern distros) stops root writing it and curl reports a bare error + 23. Download to `mktemp -d` instead. Cost us two failed installs. +- **No `private_key_file` credential option.** SSH keys must be inlined + into config, so key material sits in values files. Supporting a path + would let Kubernetes mount a secret directly. +- **Agent credentials are plaintext in the infra repo.** All four + `proxy-agent/values-*.yaml` carry `accessKey`/`accessSecret` in + clear, while the Rackspace file is properly SOPS-encrypted with an + `encrypted_regex` that already covers those fields. Same pattern, + applied to the customer and not to us. Worth encrypting and rotating. +- **Stale IP allowances in the AWS default security group.** Two + `/32` rules for addresses nobody is using, plus one granting all TCP + ports. Worth pruning. + +## Testbed + +Three `t3.micro` in account 864186153326, tagged +`Purpose=vm-discovery-testbed`, `Disposable=true`. A few dollars a +month. Keep while #115 and #35405 are in flight — rebuilding costs +more than running them — and terminate once the coverage report is +correct. + +Note the security group pins SSH to specific source IPs, so access +breaks whenever someone's address changes. diff --git a/docs/design/vm-discovery-phase0-epic.md b/docs/design/vm-discovery-phase0-epic.md new file mode 100644 index 0000000..1bafcc7 --- /dev/null +++ b/docs/design/vm-discovery-phase0-epic.md @@ -0,0 +1,101 @@ +# Epic 1: VM Discovery & Package Inventory + +First epic of the **VM Patch & Vulnerability Management** initiative: + +| Epic | What ships | +|------|-----------| +| **1. Discovery & package inventory (this one)** | Every VM found, packages cached, coverage report | +| 2. Vulnerability matching | Packages → CVEs (vendor advisories), escalation policy (CVSS/urgency config) | +| 3. Patching | Subscription-based patch apply, approval flow, maintenance windows | +| 4. Hardening & insight | CIS benchmarking, AI impact analysis of vulns on applications | + +Epics 2–4 are sequenced, not speculative — 2 needs 1's package data, +3 needs 2's vuln data. We only break tickets down for epic 1 now. + +## Summary + +Make VMs first-class resources in Nudgebee — the same way we treat +cloud resources and Kubernetes workloads today. This epic answers +three questions for every customer: + +1. How many VMs do you have? +2. What packages are installed on each one? +3. For every VM we couldn't inventory — why not? + +This is the foundation for patch & vulnerability management (the +customer ask), and later for observability and automation on VMs. + +## Why + +Customers running self-hosted and on-premise VM fleets need patch & +vulnerability management, and cloud APIs don't cover those machines. +Nothing else in the product can ship until we reliably know which VMs +exist and what's installed on them. No vendor today does agentless +Linux inventory from a per-segment collector — every competitor +requires an agent on each VM — so this is both the prerequisite and +the differentiation. A design-partner customer is waiting on it. + +## How (in one paragraph) + +We install nothing on the VMs. One forager node per network segment +does all the work: it pings the network to find machines, asks the +hypervisor and Active Directory for their lists, and logs into each VM +over SSH with a read-only user to collect the OS and package list. +The commands it runs come from a signed, versioned "content pack" +downloaded from our server — so changing what we collect never +requires a new forager release. All data flows to the server, which +merges the different sightings into one asset list and shows a +coverage report. + +## Definition of done + +The coverage report is live and correct for a real fleet: + +- Every VM appears with a state: **discovered** (we know it exists), + **reachable** (SSH login works), or **inventoried** (package list + collected). +- Every gap has a reason (no credential, powered off, EOL OS, stale…). +- Works on CentOS/RHEL, Ubuntu/Debian, SUSE, Alpine. Windows is out + of scope for now. + +## Out of scope (later phases) + +CVE matching, patching, escalation policy, CIS benchmarking, Windows, +application-level (non-OS) packages. + +## Tickets + +| # | Ticket | Where | Note | +|---|--------|-------|------| +| P1 | Forager: SSH inventory + signed content-pack runner | forager | no dependencies | +| P2 | Server: asset storage + ingest + parsers + dedup + coverage states | nudgebee | no dependencies | +| P3 | Forager: network sweep + Active Directory lookup | forager | after P1+P2 demo | +| P4 | Forager: hypervisor connector (vCenter/Proxmox/libvirt) | forager | **blocked: customer must tell us their hypervisor** | +| P6 | Content pack v1 + signing/publish pipeline | content | after P1 | +| P7 | UI: asset list + coverage report screen | nudgebee | last | +| P8 | Customer setup docs (SSH user, firewall/IDS allowlist) | nudgebee-docs | alongside P6/P7 | + +Milestone 0 = P1 + P2 together: one VM inventoried end-to-end +(signed pack → SSH collection → server → package list in the DB). +Prove the pipe before building breadth. + +Storage is an open discussion, not decided: prefer extending the +existing resource model (`cloud_resources` etc.) so features that +already work on resources — troubleshooting, automation, UI — +pick up VMs automatically; add new tables only for data that has no +home today (package cache, identity merge keys). + +## Open questions for the customer (send now) + +1. Which hypervisor do you run — VMware, Proxmox, Hyper-V, plain KVM? + (blocks P4) +2. Roughly how many VMs and network segments? +3. Can we get one read-only SSH credential per segment, or do you use + a vault (CyberArk etc.)? +4. What share of the fleet is Windows? +5. Any restriction on how the forager downloads content packs + (HTTPS out, or must everything ride the existing relay channel)? + +Details behind this epic: `vm-discovery-phase0.md` (design), +`vm-discovery-phase0-tickets.md` (full ticket breakdown with +acceptance criteria). diff --git a/docs/design/vm-discovery-phase0-tickets.md b/docs/design/vm-discovery-phase0-tickets.md new file mode 100644 index 0000000..06ae969 --- /dev/null +++ b/docs/design/vm-discovery-phase0-tickets.md @@ -0,0 +1,154 @@ +# Ticket Plan: VM Discovery & Package Inventory (Phase 0) + +Derived from `vm-discovery-phase0.md`. One epic, 8 tickets. Boundary +rule: one ticket = one owner + one repo/service + one demoable +deliverable. Spikes are the first tasks inside a ticket, not tickets. + +**Epic: VMs as first-class Nudgebee resources — Phase 0 discovery & +package inventory.** Done when the coverage report (§11) is correct +against a real fleet: every VM is `discovered`, `reachable`, or +`inventoried`, and every gap has a reason. + +**Prerequisite action (not a ticket):** customer answers to §13 — +hypervisor stack (hard-blocks P4), fleet size, credential model, +Windows %, pack distribution channel. Send immediately; only P4 +waits on the answers. + +**Open decisions — explore inside the ticket, nothing pre-decided:** + +- Storage model: extend `cloud_resources`/resource model vs new + tables (P2, design doc §8.1). +- Content-pack format & schema: the P1 spike *proposes* a format for + team review; the YAML sketch in §7 is illustrative only. +- Pack distribution channel: HTTPS vs existing relay WSS (P6 + + customer question 5). +- Sweep implementation: plain TCP-connect vs a library like naabu; + safe-scan defaults (P3). +- Hypervisor connector: which one first (P4, customer answer). +- Scheduler: reuse the scan_orchestrator pattern vs something new + (P5). +- Concurrency/cadence/staleness defaults: tune from customer fleet + size, not fixed in this doc. + +Non-negotiable invariants (correctness, not approach — these hold +whatever we pick): weak identifiers never merge assets alone; raw +observations stay immutable/replayable; package versions stored +verbatim incl. epoch/release; content packs signed + pinnable; no +software installed on target VMs. + +--- + +## P1 — Forager: discovery module + content-pack runner + SSH inventory +Repo: forager. New `pkg/proxy/discovery` module, registered like +existing proxies. + +Tasks: (a) spike the content-pack design — output is a *proposed* +format for team review (candidate fields: version, signature, +collectors, `when` exprs, a `kind` axis so observability/automation +can reuse the runner later); signing reuses the existing Ed25519 +trust root; verbatim execution, framed per-collector output; +(b) spike SSH executor concurrency on `pkg/proxy/ssh` plumbing — +target iteration, per-target timeout, output caps, bounded +concurrency; (c) `discovery_inventory` action (targets[], +credential_ref, content_pack_ref) wiring both; credentials via +`pkg/secrets`. + +AC: tampered pack rejected, unknown `when` var fails closed; 50–100 +concurrent targets without fd/memory blowup, per-target status on +partial failure; a signed action from relay inventories a static +target list on ≥2 distro families; credentials never in logs or +responses. + +## P2 — Server: asset model, ingest, parsers, reconciler +Repo: nudgebee (api-server + relay-server). + +Tasks: (a) storage decision first (design doc §8.1 is OPEN): extend +the existing `cloud_resources`/resource model so existing features +(troubleshooting, automation, UI) pick VMs up automatically, vs new +dedicated tables — then migrations only for what has no home today +(identities for merge logic, raw observations, packages, +subscriptions, static EOL data); +(b) relay `asset_inventory` handler (new upward action, not +`datasource_inventory`) — batch insert, size limits; (c) parsers: +os-release, rpm (epoch/release verbatim), dpkg, apk, +subscription/repo, reboot-pending, identity — golden-file tests from +real distro output (CentOS 7, RHEL 9, Ubuntu 20.04/22.04/24.04, +Debian 12, SLES 15, Alpine); (d) reconciler v1 — strong-id merge, +weak-id corroborate only, staleness decay, replay from observations. + +AC: weak identifiers cannot merge alone (schema-enforced); two +observations sharing machine-id merge, two sharing only IP don't; +rule fix + replay loses no data; epoch/release survive round-trip; +malformed observation rejected without dropping its batch. + +**M0 exit = P1 + P2 demo:** one host end-to-end — signed pack, SSH +collection, relay ingest, parsed packages in Postgres. + +## P3 — Forager: discovery sources — sweep + LDAP +Repo: forager. + +Tasks: (a) `discovery_sweep` — ARP local L2, ICMP, TCP 22/3389/5985 +over configured CIDRs; opt-in per CIDR, exclusions, hard rate cap +(default ≤100 pps), well-formed packets, windows honored; decide +TCP-connect vs naabu inside the ticket; (b) `discovery_ldap` — +read-only AD bind (go-ldap), filter `lastLogonTimestamp` tombstones. + +AC: /24 sweep returns IP/MAC/rDNS/open-ports; rate cap verified by +packet capture; exclusions honored; objectGUID lands as STRONG +identity. + +## P4 — Forager: hypervisor connector (first one only) +Repo: forager. **Blocked on customer answer.** Build exactly one of: +vcenter (govmomi) | proxmox (REST) | libvirt-over-SSH. Datasource +type + read-only credential via `pkg/secrets`. + +AC: returns UUID, name, power state, guest OS, IPs **including +powered-off VMs**; powered-off reaches `assets.power_state`. + +## P5 — removed (folded into P2 and kickoff decisions) +Coverage states + gap-reason taxonomy and the cloud-VM merge +(instance-id as STRONG identity) moved into P2 — they're part of the +same projection logic. Scheduling of recurring runs is a kickoff +decision (candidate: reuse the scan_orchestrator pattern — see +`RunOne`/`ScanAccount` usage in recommendation/service.go); forager +still holds zero schedule state either way. + +## P6 — Content pack v1 + publish pipeline +Repo: content (+ server config). `linux-inventory` covering the §6 +matrix (RHEL-like, Debian-like, SUSE, Alpine); CI signing; +distribution channel (default HTTPS unless customer answer says +otherwise); tenant pinning + ring rollout server-side. + +AC: pack version visible per tenant; pinned tenant never receives a +newer pack; new version reaches ring 0 before ring 1. + +## P7 — UI: asset list + coverage report +Repo: nudgebee (UI). The §11 screen: totals, per-state counts, gap +breakdown, per-asset drill-down (identities, packages, subscription +binding, sources, last_seen), EOL flags. + +AC: numbers reconcile with DB counts; gap rows link to affected +assets. + +## P8 — Onboarding docs +Repo: nudgebee-docs. `nudgebee-ro` user + SSH key setup (optional +narrow sudoers for dmidecode), hypervisor read-only role, IDS +whitelisting of forager IP, CIDR opt-in guidance. + +AC: a customer goes from zero to first coverage report using only +this doc. Not an afterthought — Rapid7's lesson (§2 finding 9) is +that credential management is the operational cost center. + +--- + +## Dependencies + +``` +P1 ─┬─► M0 exit ─► P3 ─► P6, P7 +P2 ─┘ │ +customer answer ─► P4 P8 alongside P6/P7 +``` + +P1 and P2 have no dependencies and run in parallel. P4 is the only +customer-blocked ticket. P8 can start once P1's credential model is +settled. diff --git a/docs/design/vm-discovery-phase0.md b/docs/design/vm-discovery-phase0.md new file mode 100644 index 0000000..37009b6 --- /dev/null +++ b/docs/design/vm-discovery-phase0.md @@ -0,0 +1,526 @@ +# Design: VM Discovery & Package Inventory (Phase 0) + +Status: DRAFT +Scope: Phase 0 — VM discovery, package inventory, reconciliation, +coverage reporting. Discovery makes VMs first-class Nudgebee +resources; patch & vulnerability management is the first consumer. +Out of scope (later phases): CVE matching, escalation policy, patching, +CIS benchmarking, application (non-OS-package) inventory, Windows. + +--- + +## 1. Problem + +Customers need patch & vulnerability management for VMs, starting with +on-premise / self-hosted fleets where no cloud control plane exists. +Before anything else can ship, we must answer: + +> How many VMs exist, which packages are installed on each, and for +> every VM we can't inventory — why not? + +Constraints: + +- **No agent on target VMs.** Rollout friction kills adoption; agent + upgrade management is a liability (post-CrowdStrike, staged rollout + and version pinning are procurement questions). +- **Minimal, rarely-upgraded footprint.** The only installed software + is one forager node per network segment. +- **On-prem first.** Cloud APIs (EC2/Azure/GCE) are optional + enrichment, not a dependency. + +Product framing: a discovered VM becomes a first-class Nudgebee +resource — the same status cloud resources and Kubernetes workloads +have today. Discovery is the foundation; independent capability +tracks build on the VM resource: + +1. **Observability** — node health, metrics, logs, service state. +2. **Security** — package inventory → CVE matching, patching, CIS + benchmarking (the initiative's first deliverable). +3. **Automation** — signed, audited command execution on VMs through + the per-segment forager; patching is one instance of it. + +Phase 0 therefore avoids patch-specific shortcuts: the asset tables +are a general resource catalog (packages are one satellite dataset), +and signed content packs are the generic execution vehicle all three +tracks reuse. + +## 2. Industry research summary + +Surveyed: Qualys, Tenable, Rapid7, Tanium, osquery/Fleet, AWS SSM, +Red Hat Satellite, WSUS, Ansible, Lansweeper, runZero, Wiz/Orca, +Axonius, PatchMon, CrowdStrike, JFrog, and the dedicated +patch-management pack (Automox, Action1, NinjaOne, ManageEngine, +Ivanti). Findings that drive this design: + +1. **Every mature tool runs the same four-stage pipeline:** seed from + authoritative sources (hypervisor/AD/cloud) → unauthenticated sweep + → authenticated inventory → continuous reconciliation. Tools differ + only in how stage 3 reaches the box. +2. **Sweeps find, they don't inventory.** Unauthenticated scanning + tops out at fingerprints. Package data requires credentials or a + resident channel — no exceptions in the industry. +3. **Only control planes see powered-off machines** (hypervisor, + cloud, AD last-logon). Scanning can't; tools handle absence with + staleness decay. +4. **Reconciliation is the product; collection is commodity.** Asset + duplication is the #1 operational complaint against Qualys/Tenable; + Axonius built a company purely on reconciling other tools' + inventories. +5. **Nobody serious ships a full-logic agent per VM anymore.** Red Hat + deprecated and removed katello-agent in favor of OS-native + reporting (`subscription-manager` package-profile upload). +6. **The converged answer to "minimal agent, no upgrades" is three + layers:** a per-segment node (Nessus scanner model) + agentless + push over SSH (Ansible model) + **signed content-driven execution** + (Qualys manifests, Tanium sensors, osquery query packs) — the agent + binary is a frozen interpreter; capability changes ship as signed + content. +7. **Updates need pinning + staged rollout as first-class features** — + even content updates, post-CrowdStrike. The Channel File 291 RCA is + the proof: content updates **bypassed** the N-1/N-2 pinning + customers had on sensor binaries; the post-incident fixes were + staged/canary content rollout and customer control over content + scheduling — exactly the pack pinning + ring rollout in §7 of this + doc. The split is industry consensus: Rapid7's Insight Agent keeps + shipping content when binary updates are disabled; Ivanti moved + Linux to "contentless" detection (query the distro's own repos); + ManageEngine syncs a central patch DB independent of agent + versions. +8. **Dedicated patch vendors don't solve discovery.** Automox, + Action1, NinjaOne, and ManageEngine are per-endpoint agent + + enrollment; their answer to "machines we don't know about" is + scanning AD in order to push more agents. Ivanti Security Controls + is the only central-node agentless inventory — Windows-only, over + admin shares/remote registry. **No vendor ships agentless SSH + Linux inventory from a per-segment collector** — the exact + combination this design proposes is open ground. +9. **Rapid7 validates the per-segment model and flags its two + hazards.** "Assigning a Scan Engine to each subnetwork is a best + practice" (stateless engines, hub-and-spoke to the console). But: + (a) remote credential management hurt enough that they shipped + Scan Assistant, a tiny cert-authenticated on-host helper — expect + credentials to be the operational cost center and design the vault + integration well; (b) their agent/engine asset correlation relies + on a UDP-31400 probe plus hostname/IP/MAC/UUID matching, and + produces duplicates when it fails — deterministic identity + (machine-id/SMBIOS first) must be built in up front, which §8.2 + does. +10. **Confirmed dead ends.** Snapshot side-scanning (Wiz/Orca) does + not extend on-prem: Wiz's vSphere support is vCenter-API-only + with guest package scanning unshipped, and Orca covered on-prem + by introducing a runtime agent — abandoning agentless. JFrog has + no machine discovery anywhere in its portfolio (Xray scans + artifacts in Artifactory; Runtime is a Helm-installed K8s eBPF + sensor; Connect is per-device token enrollment) — it is an + artifact-layer product, not a VM-discovery comparable. + +Forager already implements the per-segment-node and SSH-push layers +(outbound-only WSS, Ed25519-signed actions, SSH executor, datasource +auto-registration). The missing pattern is signed-content collection. + +## 3. Build vs fork: why not adopt an existing OSS scanner + +Evaluated forking/adopting an open-source project instead of extending +forager. Conclusion: **no fork; reuse at the library and server +layers.** + +### Candidates + +| Project | What it does | Why not a fork base | +|---|---|---| +| **Vuls** (future-architect) | Closest match: Go, agentless SSH scan, package collection + CVE matching for CentOS/Ubuntu/RHEL/Debian/SUSE/Alpine/Amazon | **GPL-3.0** — forking into our agent makes the combined work GPL. Local TOML config model (ours is cloud-pushed), no relay/transport, per-distro logic baked into the binary — the exact thing we're avoiding | +| **cnquery** (Mondoo) | Agentless SSH inventory, polished | **BUSL-1.1** — source-available license that prohibits building a competing commercial product on it. We are the prohibited use | +| **Wazuh** | Inventory + vuln detection | Per-VM agent (violates constraint), GPLv2, huge | +| **OpenVAS/GVM** | Full scanner appliance | GPL, C, enormous; a second heavyweight deployable in the customer env kills the "one small thing per segment" story | +| **osquery/Fleet** | Best-in-class inventory | Per-VM agent — wrong model | +| **Ansible** | Agentless push | GPLv3, Python runtime dependency on the node, we'd use 2% of it | +| **PatchMon** | Patch monitoring/automation: pending updates, patch policies with windows + dry-run, approval-gated execution, OpenSCAP CIS | **AGPL-3.0** — network copyleft, binds on SaaS use, worst license in the survey. Per-host agent (10 binary variants to maintain fleet-wide) — the rejected model. And **no discovery at all**: hosts exist only once the agent is installed; coverage accounting is impossible in its model | + +### Why fork economics are bad here + +1. **A fork saves the cheap part.** The collection layer is + `rpm -qa` / `dpkg-query -W` / `cat /etc/os-release` over SSH — a + few hundred lines against forager's existing SSH client. +2. **A fork doesn't help with the hard part.** Reconciliation + + coverage accounting is the product (finding #4), it's server-side, + and no OSS project ships it in reusable form. +3. **We'd port the fork onto forager anyway.** Transport, signing, + secrets, outbound-only WSS, datasource registration — the fork has + none of it; forager has all of it. Integration work exceeds the + collection code saved. +4. **A fork works against the no-upgrade goal.** Vuls' per-distro + logic lives in the binary — every distro quirk is a binary release. + The content-pack architecture (§7) is strictly better. +5. **Forks rot.** We would own the divergence forever; upstream fixes + conflict with our patches. + +### Where OSS reuse is right + +- **Libraries in forager** (license-clean): `govmomi` (Apache-2.0) + for vCenter, `go-ldap` (MIT) for AD, `naabu` (MIT) or ~300 + hand-rolled lines for the sweep. Do **not** shell out to nmap — its + NPSL license is a known problem for commercial redistribution. +- **Server-side, Phase 1:** Trivy or Grype (both Apache-2.0) for CVE + matching against the package cache — Trivy is already in + scan_orchestrator. Biggest genuine reuse win; zero agent impact. +- **Vuls as a reference, not a dependency:** its code encodes years of + per-distro edge cases (epoch handling, CentOS vault, backport + version formats). Read to design content packs; don't copy code. +- **SBOM output** (CycloneDX/SPDX) for the package cache so + third-party tools interop for free. +- Acceptable one-off: running Vuls standalone for a pre-sales demo. + Throwaway only, not a foundation. +- **PatchMon as a phases-1–3 UX reference** (same status as Vuls — + read, don't reuse): its policy model (immediate/delayed/maintenance + window), dry-run → approval → full-shell-output audit flow, and + per-package-manager pending-update handling are a good benchmark. + Also a competitor to track. + +## 4. Architecture summary + +``` +┌─────────────────────────┐ ┌────────────────────────────────────┐ +│ Nudgebee Cloud │ │ Customer Site / Segment │ +│ │ wss │ │ +│ Scheduler ─► Relay ◄───┼──────────┼─► Forager (1 per segment) │ +│ │ ▲ │ │ │ │ +│ Reconciler │ │ │ ├─ sweep: ARP/ICMP/TCP probe │ +│ │ asset_ │ │ ├─ hypervisor: vCenter/Proxmox │ +│ assets DB inventory │ │ ├─ ldap: AD computer objects │ +│ │ │ └─ inventory: SSH push ──► VMs │ +└─────────────────────────┘ │ (nothing installed) │ + └────────────────────────────────────┘ +``` + +- **Forager = per-segment discovery node.** Sweeps its segment, + queries the hypervisor/directory, collects per-VM inventory over + SSH (run commands, capture output, leave nothing behind). +- **Target VMs get nothing installed.** Their "agent" is sshd + the + package manager the OS already has. +- **Collection logic ships as signed content packs** (§7); the forager + binary stays frozen. +- **All policy lives server-side:** schedules, CIDR scopes, rate + limits, reconciliation. Forager initiates nothing on its own. +- Per-VM forager install remains a *fallback* for hosts where SSH is + blocked by policy — an option, not the model. + +## 5. Discovery sources (authority ladder) + +Reconciled, not either/or. Ordered by authority: + +| # | Source | Sees | Identifier quality | Notes | +|---|--------|------|--------------------|-------| +| 1 | Hypervisor API — vCenter, Proxmox, libvirt-via-SSH | All VMs incl. **powered-off** | Strong (instance/SMBIOS UUID) | The on-prem "control plane"; only true denominator | +| 2 | Directory/inventory — AD LDAP, DNS zones, DHCP leases, Ansible inventory, node_exporter targets | Recorded hosts | Weak (FQDN/IP/MAC) | Cheap, high yield; corroboration only | +| 3 | Network sweep — ARP (local L2), ICMP + TCP 22/3389/5985 across configured CIDRs | Live, reachable hosts | Weak until inventoried | Finds the unrecorded box; noisiest source | +| 4 | Cloud APIs — existing cloud-collector (EC2/Azure/GCE) | Cloud VMs | Strong (instance-id) | Optional enrichment where accounts exist | +| 5 | Authenticated inventory (SSH) | Everything about a reachable host | **Strong (machine-id, SMBIOS UUID)** | The only source of package data | + +Sweep safety (product requirements, not niceties): + +- Opt-in per CIDR; explicit exclusion lists. +- Hard rate cap (default ≤100 pps); well-formed packets only — + malformed nmap-style probes destabilize embedded/OT gear. +- Configurable scan windows. +- Onboarding docs must tell customers to whitelist the forager IP in + their IDS — an unannounced sweep trips Darktrace/Suricata and + generates a security incident. + +**Powered-off VMs:** only source 1 sees them. Without a hypervisor +integration, an off host does not exist to us. Model this honestly +with staleness decay: `last_seen` ages → `stale` after N days → +`presumed-retired` after M, surfaced in the coverage report. The claim +is "not observed since X," never a false "doesn't exist." + +## 6. Targeted OS matrix (v1) + +Linux only. Windows (WinRM/WMI) explicitly deferred. + +| Family | Distros | Package query | Patch channel binding | +|--------|---------|---------------|----------------------| +| RHEL-like | CentOS 7/Stream, RHEL 7–9, Rocky, Alma, Oracle, Amazon Linux 2/2023 | `rpm -qa --qf '%{NAME}\t%{EPOCH}\t%{VERSION}\t%{RELEASE}\t%{ARCH}\t%{INSTALLTIME}\n'` | `subscription-manager status` + `identity` (RHEL); `yum repolist -q` / `dnf repolist` | +| Debian-like | Ubuntu 18.04–24.04, Debian 10–12 | `dpkg-query -W -f '${Package}\t${Version}\t${Architecture}\t${db:Status-Status}\n'` | `pro status --format json` (Ubuntu ESM); `/etc/apt/sources.list{,.d/}` | +| SUSE | SLES 12/15, openSUSE Leap | `rpm -qa` (as RHEL-like) | `SUSEConnect --status-text`; `zypper lr -u` | +| Alpine | 3.x | `apk info -v` | `/etc/apk/repositories` | + +Full per-host collection set (one SSH session, one batched script from +the content pack): + +- **Identity:** `/etc/machine-id`; SMBIOS UUID via + `/sys/class/dmi/id/product_uuid` (root) with `dmidecode -s + system-uuid` fallback if sudo-permitted — degrade gracefully to + machine-id only; hostname/FQDN; MACs (`ip -o link`). + + **Verified on the AWS testbed (2026-08-03):** `product_uuid` is mode + `-r--------` root-only on stock Linux, so an unprivileged + `nudgebee-ro` gets **nothing** — SMBIOS UUID is unavailable in the + default credential model, not merely sometimes. `product_serial` is + root-only too. `board_asset_tag` *is* world-readable and on EC2 + carries the instance id, so cloud VMs still yield a strong + identifier. + + This has a consequence for §8.2 that is easy to miss: a hypervisor + record (source 1) knows a VM's SMBIOS UUID, and an SSH record + (source 5) knows its machine-id. On-prem, with no sudo, **they share + no strong identifier** — so the two sources cannot be merged on one, + and every VM would appear twice. The narrow sudoers entry for + `dmidecode` is therefore not the optional nicety §10 implies; it is + what makes hypervisor↔SSH reconciliation possible at all. Either + require it, or accept that on-prem merging leans on corroborated + weak identifiers, which §8.2 currently forbids. Decide before P4. +- **OS:** `/etc/os-release` (ID, VERSION_ID, PRETTY_NAME), + `uname -r -m`. +- **Packages:** per family above. Version strings kept **verbatim** + incl. epoch and release — required for backport-aware CVE matching + in Phase 1 (vendor OVAL, not NVD CPE); never normalize away the + distro release suffix. +- **Subscription/repo binding:** per family above. Captured now + because a lapsed subscription or dead repo means the host **cannot + be patched** — that surfaces in Phase 0's gap report, not at patch + time in Phase 2. +- **Reboot-pending signal:** `/var/run/reboot-required` (Debian-like), + `dnf needs-restarting -r` exit code (RHEL-like, if present). Cheap + now, needed by Phase 2. + +Requirements on targets: reachable sshd + one scoped credential (§9). +No python dependency (unlike Ansible), no temp files — commands run +directly, output streams back. + +**EOL reality check:** CentOS 7 (EOL June 2024) and CentOS 8 are +common in exactly the fleets that buy this. Inventory works +identically; Phase 0 records `os_eol: true` from a small static EOL +table so the coverage report can say "N hosts are EOL — no vendor +patches exist for anything on them." + +## 7. Signed content packs (the no-upgrade mechanism) + +The per-OS collection commands in §6 do **not** live in the forager +binary. They ship as a versioned, Ed25519-signed content pack fetched +from the cloud (same trust root as action signing). Format below is +illustrative — the actual schema is the P1 spike's output, reviewed +by the team: + +```yaml +# content-pack: linux-inventory v3 +version: 3 +signature: +collectors: + - id: os-release # every host + cmd: cat /etc/os-release + - id: pkgs-rpm + when: os_family == "rhel" + cmd: rpm -qa --qf '...' + - id: pkgs-dpkg + when: os_family == "debian" + cmd: dpkg-query -W -f '...' + # ... +``` + +Rules: + +- Forager validates the signature, then executes collectors verbatim; + it contains **no per-distro logic** beyond a tiny `when`-expression + evaluator and output framing. +- Adding a distro, fixing a collector, adding a fact = publish a new + pack version. No binary release. +- **Pinning + staged rollout are first-class:** tenants can pin a pack + version; new versions roll out by ring. Content is an update surface + too — treat it with the same care as binaries. +- **Output parsing happens server-side.** The agent returns raw framed + output per collector. Parsers (the code that actually churns) never + ship in the agent. + +## 8. Server side + +### 8.1 Data model (OPEN — for discussion, not final) + +Two directions; decide at P2 kickoff, not in this doc: + +- **Option A — reuse the existing resource model.** VMs become + another resource kind in `cloud_resources` and the existing + datasource/inventory plumbing, the way other resource types + already work. Big advantage: everything that already consumes + resources — troubleshooting, automation, UI listing — infers VMs + with little or no new plumbing. +- **Option B — dedicated asset tables.** The sketch below. Cleaner + for identity reconciliation and package data, but new plumbing + everywhere. + +Likely landing point: A for the VM resource itself, with new +satellite tables only where nothing exists today (identities for +merge logic, packages, subscriptions). The sketch below shows the +data we must hold, not a schema decision. + +``` +assets -- one row per real machine + id, tenant_id, cloud_account_id? + hostname, fqdn, os_family, os_version, os_eol, kernel, arch + env, criticality, owner -- from tags/CMDB when known + power_state, first_seen, last_seen + coverage_state, gap_reason -- denormalized for the report + +asset_identities -- merge keys; many per asset + asset_id, kind, value, source, observed_at + -- kinds: machine_id | smbios_uuid | instance_id | ad_objectguid (STRONG) + -- fqdn | ip | mac (WEAK) + -- unique (tenant_id, kind, value) enforced for STRONG kinds only + +asset_observations -- raw per-source records, never merged/edited + asset_id?, tenant_id, source, source_ref, observed_at, payload jsonb + +asset_packages -- the package cache + asset_id, name, epoch, version, release, arch, pkg_type, repo, installed_at + pk (asset_id, name, arch) + +asset_subscriptions -- patch-channel binding (used by Phase 2) + asset_id, kind (rhsm|pro|scc|apt|yum), status, channels jsonb +``` + +### 8.2 Reconciliation rules + +The hard part (research finding #4). + +- **Strong identifiers may merge on their own:** machine-id, + SMBIOS/instance UUID, AD objectGUID. +- **Weak identifiers never merge alone** (FQDN, IP, MAC) — they + corroborate. Merging on a recycled DHCP lease silently fuses two + hosts and is nearly undetectable afterwards. Enforced by schema, + not convention. +- Observations are immutable; `assets` is a projection. Bad merge → + fix rule → re-project. No data loss. +- Reaping: full-snapshot semantics per (agent, source) with staleness + decay — same pattern as the existing `removeStaleAgentDatasources`. + +### 8.3 Ingest path + +New upward action `asset_inventory` (do not overload the existing +`datasource_inventory`, which means "what datasources does this agent +have"): + +```json +{"action": "asset_inventory", + "source": "sweep|vcenter|ldap|libvirt|host_inventory", + "content_pack_version": 3, + "observations": [{"identities": {...}, "facts": {...}, + "collectors": {"pkgs-rpm": ""}}]} +``` + +Relay handler → insert `asset_observations` → reconciler projects into +`assets`/`asset_packages`. Collector-output parsers live here. + +### 8.4 Scheduler + +Server-side cron pushing actions to forager (same pattern as +scan_orchestrator job scheduling): sweep cadence per CIDR, hypervisor +poll cadence, inventory cadence per asset, scan windows, concurrency +caps. Forager holds zero schedule state. + +## 9. Forager changes + +New proxy module `pkg/proxy/discovery`, registered like existing +modules. + +**Datasource configuration: from the Nudgebee UI, pushed down** +(recommendation — matches every surveyed tool: Rapid7 discovery +connections live in the console and are assigned to engines; Qualys +and Tenable manage scanners centrally; the per-segment node only +enrolls). Two reasons beyond precedent: the coverage report needs the +server to know *intended* scope (CIDRs, datasources) or gaps can't be +computed; and the existing integrations config-push path +(`integration_config.go` / `proxy_config_push.go`) plus `pkg/secrets` +cloud-push already deliver exactly this. Forager keeps only bootstrap +config (relay URL, pairing) locally. + +Open point — credential residency: datasource *existence and scope* +is always UI-configured, but the credential *value* is either +UI-entered (cloud-push, default) or a local/vault reference on the +forager for customers who won't let credentials transit our cloud +(`pkg/secrets` local mode exists for this). + +New datasource types (credentials via existing `pkg/secrets` +local/cloud-push): + +- `vcenter` (govmomi, read-only role) +- `proxmox` (REST `/cluster/resources`) +- `libvirt` (over SSH to the KVM host: `virsh list --all --uuid`) +- `ldap` (AD computer objects; filter `lastLogonTimestamp` to skip + tombstones) + +Actions (existing signed `ActionRequest` scheme): + +| Action | Params | Returns | +|--------|--------|---------| +| `discovery_sweep` | cidrs, ports, rate_pps, timeout, exclusions | live hosts: IP, MAC (L2), rDNS, open ports | +| `discovery_hypervisor` | datasource_id | VM list: UUID, name, power state, guest OS, IPs | +| `discovery_ldap` | datasource_id, active_within | computer objects: name, DNS name, lastLogon, objectGUID | +| `discovery_inventory` | targets[], credential_ref, content_pack_ref | per-host facts + raw collector output (§6) | + +SSH execution reuses `pkg/proxy/ssh` client plumbing (connection, +auth, timeouts); the discovery module owns target iteration, +concurrency caps, and content-pack execution. + +## 10. Credentials & security + +- One **read-only inventory credential** per segment: SSH key for a + dedicated `nudgebee-ro` user. None of the §6 commands require root + except SMBIOS UUID (graceful fallback; optional narrow sudoers entry + for `dmidecode` if the customer wants it). +- Delivered via existing `pkg/secrets` (local or cloud-push); never + logged, never included in action responses. +- Hypervisor/LDAP credentials: read-only roles (vCenter read-only, + unprivileged AD bind). +- All actions Ed25519-signed; content packs signed with the same trust + root. +- No new inbound ports on forager; no new ports on targets beyond + existing sshd. + +## 11. Coverage report — the Phase 0 deliverable + +One screen, per tenant: + +``` +Discovered: 412 Reachable: 391 Inventoried: 380 +Gaps (32): + 11 ssh-refused (no credential / firewall) + 8 ssh-auth-failed (bad credential) + 6 stale (not observed in 14d) + 4 powered-off (per vCenter) + 2 unsupported-os (fingerprint: FreeBSD) + 1 eol-no-repo (CentOS 7, vault not configured) +``` + +Coverage states per asset, tracked independently: `discovered` (exists +per any source) → `reachable` (SSH auth ok) → `inventoried` (package +list collected). `discovered − inventoried` is a feature, not an error +state. Nothing else in Phase 0 ships until this screen is correct. + +## 12. Build list + +1. Migrations: `assets`, `asset_identities`, `asset_observations`, + `asset_packages`, `asset_subscriptions` + static EOL table. +2. Forager `pkg/proxy/discovery`: sweep, hypervisor (first connector + per open question 1), ldap, SSH inventory executor; content-pack + fetch/verify. +3. Content pack v1: linux-inventory (RHEL-like, Debian-like, SUSE, + Alpine collectors). +4. Relay: `asset_inventory` handler; server-side collector parsers; + reconciler. +5. Scheduler: per-tenant CIDRs/cadences/windows/rate caps. +6. UI: asset list + coverage report. +7. Cloud-collector VM resources wired in as observer source (existing + data, new projection). + +## 13. Open questions + +1. **Customer hypervisor stack** — VMware, Proxmox, Hyper-V, or + unmanaged KVM/bare metal? Sizes item 2; build only that connector + first. +2. Approximate fleet size and segment count — sets sweep/inventory + cadence defaults and forager concurrency caps. +3. Shared SSH credential per segment, per-host credentials, or an + existing vault (CyberArk etc.)? +4. Windows timeline — deferred from v1, but if the fleet is >30% + Windows, WinRM support moves up. +5. Content-pack distribution: piggyback on the relay WSS channel vs + HTTPS fetch from cloud API (relay keeps one channel; HTTPS is + simpler to cache/CDN). diff --git a/docs/design/vm-discovery-wire-reference.md b/docs/design/vm-discovery-wire-reference.md new file mode 100644 index 0000000..9ddf185 --- /dev/null +++ b/docs/design/vm-discovery-wire-reference.md @@ -0,0 +1,165 @@ +# VM Discovery — action and relay wire reference + +Status: REFERENCE, from the AWS testbed run 2026-08-03 +Audience: whoever builds the server side (nudgebee-enterprise#35405) and +needs to know exactly what to send and what comes back. + +Everything below is either what was actually run, or the envelope the +relay produces — both marked as such, because the distinction matters: +**the actions were exercised, the relay envelope around them was not.** + +## What was actually run + +Driven through the discovery proxy's exported Go API by a throwaway +harness on the forager host, because nothing can dispatch a discovery +action through the relay yet. + +### Datasource configuration + +```go +p.Configure(map[string]any{ + "allowed_cidrs": []any{"172.31.0.0/28"}, + "max_rate_pps": float64(50), + // inventory additionally needed: + "pack_public_key": "", + "pack_dir": "/etc/nudgebee/packs", +}, map[string]string{ + "username": "nudgebee-ro", + "private_key": "", +}) +``` + +### discovery_sweep + +```go +&proxy.ActionRequest{ + Action: "discovery_sweep", + Params: map[string]any{ + "cidrs": []any{"172.31.0.0/28"}, + "ports": []any{float64(22)}, + "rate_pps": float64(50), + "timeout_ms": float64(1000), + }, +} +``` + +Response (`ActionResponse.Data`, verbatim): + +```json +{ + "cidrs": ["172.31.0.0/28"], + "addresses_scanned": 14, + "addresses_excluded": 0, + "rate_pps": 50, + "duration_seconds": 1.184975093, + "hosts": [ + {"ip": "172.31.0.10", "open_ports": [22], + "rdns": "ip-172-31-0-10.ec2.internal", "sources": ["tcp"]}, + {"ip": "172.31.0.11", "mac": "02:4d:07:48:c4:87", "open_ports": [22], + "rdns": "ip-172-31-0-11.ec2.internal", "sources": ["tcp","arp"]}, + {"ip": "172.31.0.12", "mac": "02:16:a7:ae:cb:8b", "open_ports": [22], + "rdns": "ip-172-31-0-12.ec2.internal", "sources": ["tcp","arp"]} + ] +} +``` + +`172.31.0.10` has no MAC because it is the forager itself — a host does +not ARP for its own address. Do not treat a missing MAC as an error; +it is also normal for anything reached through a router. + +`addresses_scanned` is 14, not 16: network and broadcast are skipped. + +### discovery_inventory + +```go +&proxy.ActionRequest{ + Action: "discovery_inventory", + Params: map[string]any{ + "targets": []any{"172.31.0.11", "172.31.0.12"}, + "content_pack_version": float64(1), + }, +} +``` + +Response shape, with the fields that matter to ingest: + +```json +{ + "content_pack_version": 1, + "targets": [ + { + "host": "172.31.0.11", + "status": "ok", + "duration_seconds": 1.5, + "facts": {"os_family":"debian","os_id":"ubuntu","os_major":"22","arch":"x86_64"}, + "collectors": { + "os-release": "NAME=\"Ubuntu\"\nID=ubuntu\n...", + "pkgs-dpkg": "acpid\t1:2.0.33-1ubuntu1\tamd64\tinstalled\n...", + "machine-id": "ec2403e319a2f3f0ae53a05e3daf084b\n", + "smbios-uuid": "" + }, + "collector_errors": {}, + "skipped_collectors": [] + } + ] +} +``` + +Per-target `status` is one of `ok`, `ssh-refused`, `ssh-auth-failed`, +`timeout`, `error`. **A failed target is a result, not an error** — the +batch still returns 200. These map onto coverage gap reasons. + +## Parsing notes from real output + +Taken from the actual runs, not from the collector definitions. + +- **`rpm` prints `(none)` for a missing epoch**, not `0` or empty: + `publicsuffix-list-dafsa\t(none)\t2026...`. `(none)` and `0` are not + equivalent for version comparison and must not be collapsed. +- **Epoch and release survive intact** and must stay that way: + `openssl\t1\t3.5.5\t1.amzn2023.0.5\tx86_64\t1784931981`, and Debian's + `acpid\t1:2.0.33-1ubuntu1` keeps its `1:` prefix. +- **`smbios-uuid` comes back empty** under the unprivileged credential — + `/sys/class/dmi/id/product_uuid` is mode `-r--------`. Plan for + machine-id as the only strong identifier on-prem. +- Collector output is raw text exactly as the command emitted it, + including a trailing newline. Where a collector wrote to stderr, it + is appended after a `\n[stderr]\n` marker. + +Real output from both distro families is available on the testbed for +capturing golden files rather than hand-writing them. + +## The relay envelope — shape only, NOT yet exercised + +What the relay wraps an action in. Reproduced here from the relay's +signer and pinned by tests in `pkg/ws/discovery_wire_test.go`, but no +discovery action has actually travelled this path: the relay's dispatch +endpoint is hardcoded for `http_request`, so sending one means +publishing to the agent's queue, which is server-side work. + +```json +{ + "request_id": "", + "datasource_id": "local:aws-testbed", + "action": "discovery_sweep", + "params": { "cidrs": ["172.31.0.0/28"], "ports": [22] }, + + "signed_payload": "{\"action\":\"discovery_sweep\",\"datasource_id\":\"local:aws-testbed\",\"params\":{...}}", + "signature": "", + "signed_at": "", + "nonce": "" +} +``` + +Two things to get right: + +1. **Discovery actions are not in the relay's explicit `SigningFields` + map**, so they fall through to `DefaultSigningFields` — + `action`, `datasource_id`, `params`. That is correct for all three, + and is now pinned by a test rather than assumed. +2. **All three require a signature.** They were absent from the + forager's `signedActions` allowlist and therefore unverified; fixed + in forager#122. An unsigned or tampered discovery action is now + rejected with 403. + +`datasource_id` is `local:` for a locally configured datasource.