docs: VM discovery design, tickets, deployment plan, and testbed results - #117
docs: VM discovery design, tickets, deployment plan, and testbed results#117mayankpande88 wants to merge 16 commits into
Conversation
Design for making VMs first-class Nudgebee resources: per-segment forager discovery (sweep, hypervisor, LDAP, SSH inventory), signed content packs, server-side reconciliation and coverage reporting. Referenced by epic nudgebee/nudgebee-enterprise#35404 and tickets #113-#116.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the design, epic, and ticket breakdown documents for Phase 0 of the VM Discovery & Package Inventory initiative, which aims to make VMs first-class resources in Nudgebee using an agentless, per-segment forager model. The review feedback highlights several critical technical issues in the design: the asset_packages primary key must include version and release to support co-installed packages like kernels; the asset_identities table is missing the tenant_id column required for its unique constraint; the virsh command specified for libvirt discovery is insufficient to retrieve all required VM metadata; collecting SMBIOS UUIDs requires root privileges which conflicts with the unprivileged SSH user; and ARP sweeps are Layer 2 only and cannot be routed across arbitrary CIDRs.
| asset_packages -- the package cache | ||
| asset_id, name, epoch, version, release, arch, pkg_type, repo, installed_at | ||
| pk (asset_id, name, arch) |
There was a problem hiding this comment.
Setting the primary key of asset_packages to pk (asset_id, name, arch) will cause primary key violations on systems that have multiple versions of the same package installed concurrently. For example, Linux kernels (e.g., kernel or linux-image-* packages) are routinely co-installed during updates. Additionally, RPM-based systems allow co-installation of different versions/releases of the same package (such as gpg-pubkey or dual-arch libraries).
To support multi-version co-installation, the primary key should include the version and release fields.
Suggested schema update:
asset_packages -- the package cache
asset_id, name, epoch, version, release, arch, pkg_type, repo, installed_at
pk (asset_id, name, version, release, arch)|
|
||
| - `vcenter` (govmomi, read-only role) | ||
| - `proxmox` (REST `/cluster/resources`) | ||
| - `libvirt` (over SSH to the KVM host: `virsh list --all --uuid`) |
There was a problem hiding this comment.
The command virsh list --all --uuid only outputs a list of raw UUIDs. It does not provide the VM name, power state, guest OS, or IP addresses. Furthermore, virsh on its own does not natively know the guest OS or guest IP addresses unless the QEMU guest agent is installed and queried (e.g., via virsh domifaddr <dom> --source agent or virsh qemu-agent-command).
To satisfy the discovery_hypervisor return requirements (UUID, name, power state, guest OS, IPs), the libvirt connector will need to perform more complex queries, such as parsing virsh list --all for names/states, and inspecting domain XMLs (virsh dumpxml) or querying network DHCP leases/ARP tables to resolve IPs.
| - **Identity:** `/etc/machine-id`; SMBIOS UUID via | ||
| `/sys/class/dmi/id/product_uuid` (root) with `dmidecode -s | ||
| system-uuid` fallback if sudo-permitted — degrade gracefully to | ||
| machine-id only; hostname/FQDN; MACs (`ip -o link`). |
There was a problem hiding this comment.
Both /sys/class/dmi/id/product_uuid and dmidecode require root/sudo privileges on most standard Linux distributions (permissions on /sys/class/dmi/id/product_uuid are typically 0400 owned by root). Since the SSH user nudgebee-ro is unprivileged, reading the SMBIOS UUID via the primary path will fail without sudo. The design should explicitly note that the primary path also requires sudo/root, or accept that SMBIOS UUID collection will always fall back to machine-id for unprivileged SSH sessions.
| asset_identities -- merge keys; many per asset | ||
| asset_id, kind, value, source, observed_at | ||
| -- kinds: machine_id | smbios_uuid | instance_id | ad_objectguid (STRONG) | ||
| -- fqdn | ip | mac (WEAK) | ||
| -- unique (tenant_id, kind, value) enforced for STRONG kinds only |
There was a problem hiding this comment.
The asset_identities table definition is missing the tenant_id column, but the unique constraint references it: unique (tenant_id, kind, value). To enforce this multi-column unique constraint, tenant_id must be added as a column to the asset_identities table.
Suggested schema update:
asset_identities -- merge keys; many per asset
asset_id, tenant_id, kind, value, source, observed_at
-- kinds: machine_id | smbios_uuid | instance_id | ad_objectguid (STRONG)
-- fqdn | ip | mac (WEAK)
-- unique (tenant_id, kind, value) enforced for STRONG kinds only| Tasks: (a) `discovery_sweep` — ARP local L2, ICMP, TCP 22/3389/5985 | ||
| over configured CIDRs; opt-in per CIDR, exclusions, hard rate cap | ||
| (default ≤100 pps), well-formed packets, windows honored; decide | ||
| TCP-connect vs naabu inside the ticket; (b) `discovery_ldap` — | ||
| read-only AD bind (go-ldap), filter `lastLogonTimestamp` tombstones. |
There was a problem hiding this comment.
ARP (Address Resolution Protocol) is a Layer 2 protocol and cannot be routed across subnets. Therefore, performing an ARP sweep over arbitrary "configured CIDRs" is technically impossible if those CIDRs belong to routed networks outside the forager's local broadcast domain.
The ticket description should clarify that ARP sweeps are strictly limited to the local subnet of the forager's active network interface, whereas ICMP and TCP port probes (22/3389/5985) are used for routed CIDRs.
…ckoff Matches the issue restructure after #35406 was removed.
Covers what can ship now (an rc image) and the four gaps blocking end-to-end discovery: local config cannot configure a discovery datasource, a sweep-only datasource still demands SSH credentials, no signed content pack exists (#116), and nothing triggers discovery yet.
product_uuid is root-only on stock Linux, so the unprivileged nudgebee-ro credential never gets an SMBIOS UUID. That leaves a hypervisor record and an SSH record for the same on-prem VM with no strong identifier in common, which blocks the merge §8.2 depends on. board_asset_tag is world-readable and carries the EC2 instance id, so cloud VMs are unaffected.
Records the setup, what a correct sweep and inventory look like on real hosts, and the three findings: SMBIOS UUID is root-only so hypervisor-to-SSH merging has no shared strong identifier on-prem, rpm reports a missing epoch as (none), and board_asset_tag gives cloud VMs a strong identifier unprivileged.
The agent half is proven and does nothing, because everything user-visible is behind the server side. Records the one design decision the testbed forced — SMBIOS UUID is root-only, so on-prem a hypervisor record and an SSH record share no strong identifier — with three options and the tradeoff of each.
mayankpande88
changed the title
Exact requests and verbatim responses for discovery_sweep and discovery_inventory, plus parsing notes taken from real output rather than from the collector definitions — rpm reports a missing epoch as (none), smbios-uuid comes back empty under the unprivileged credential. Marks clearly which parts were exercised and which were not: the actions ran against real hosts, the relay envelope around them has never carried a discovery action.
Plain-language explanation of what discovery does, written for a non-technical reader, with the joint-build split made explicit: what we need answered, what only they can set up, and the identification trade-off that is genuinely their call. States current status honestly rather than as 'in progress' — what works and was tested on real machines, what is not built, and the limits we are not hiding (powered-off machines are invisible without the hypervisor; no-SSH machines can be found but not inventoried).
Four points were each made three times: machines that are switched off needing the hypervisor, old systems having no patches, nothing being installed on customer machines, and machines that cannot be logged into. Each now appears once, in the section that owns it, with cross-references where another section needs to point at it. Also drops the closing section that restated the questions and the decision already asked for above.
The previous version read as generated: an em-dash in nearly every paragraph, rule-of-three constructions, a rhetorical turn per section, bold lead-ins on every bullet, and headings like 'Limits we are not hiding' that perform honesty rather than being honest. Same content, flatter voice. No em-dashes, no bold, no tables, and the editorialising cut — the section on why per-machine agents are costly was selling rather than explaining, and the customer did not ask to be sold.
Asked 'do you use Active Directory?', which answers nothing — everyone uses it for user authentication. The question that decides whether it helps discovery is whether their Linux servers are domain-joined, and most are not. AD holds a computer object per domain-joined machine, which is what we query (objectCategory=computer, not users). But domain-joining Linux is a minority setup, so for a Linux-first scope AD will usually return little. The doc now says so instead of listing it beside the hypervisor as an equal source.
Reviewer asked whether machine ID and MAC address are the same thing, which they are not, and the section did not say so. It now names all three with real example values and states which side can see each. Also corrects the second option. It said 'match on hostname and IP', having overlooked that both the hypervisor and an SSH login can see MAC addresses, which are more stable than IPs. That makes the no-setup option meaningfully better than described, and the customer was being asked to choose against a weaker version of it than actually exists.
1 participant
Description
The design and planning record for VM discovery (epic nudgebee/nudgebee-enterprise#35404), including what the AWS testbed run actually found.
Six documents:
vm-discovery-phase0.mdvm-discovery-phase0-tickets.mdvm-discovery-phase0-epic.mdvm-discovery-deployment.mdvm-discovery-aws-testbed.mdvm-discovery-next.mdData model and pack format are deliberately marked OPEN rather than settled — the storage question in particular may resolve toward reusing the existing resource model, and the docs say so instead of implying a decision that has not been made.
The finding worth reading if nothing else:
/sys/class/dmi/id/product_uuidis root-only, so the unprivileged credential never reads an SMBIOS UUID. A hypervisor knows a VM by SMBIOS UUID and SSH inventory knows it by machine-id, so on-prem those two sources share no strong identifier — under the "only strong identifiers may merge" rule, the same VM lands in the asset list twice, permanently. Three options with their costs are written up invm-discovery-next.md; it needs deciding before #115 or the #35405 schema hardens.Type of change
How Has This Been Tested?
Not code. The claims in the testbed document were produced by running the thing: a forager deployed on EC2 swept a /28 and inventoried Ubuntu 22.04 and Amazon Linux 2023 targets, and the numbers, collector names and identity findings recorded there are from those runs rather than from reading the source.
Checklist
make validatepasses (fmt + lint + test)