diff --git a/.vale/styles/spelling-exceptions.txt b/.vale/styles/spelling-exceptions.txt index 11e2d78..712ab6a 100644 --- a/.vale/styles/spelling-exceptions.txt +++ b/.vale/styles/spelling-exceptions.txt @@ -113,6 +113,7 @@ Redoc repo ressources REST +RIRs rowcount Rowcount schema_mapping diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx new file mode 100644 index 0000000..afeb222 --- /dev/null +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -0,0 +1,333 @@ +--- +title: Sync NetBox to Infrahub +description: Set up a local Infrahub instance, load a data model, and synchronize infrastructure data from the public NetBox demo. +--- + +This tutorial is for people getting started with Infrahub: it starts from a blank page, with no existing Infrahub instance or schema required. We will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) into Infrahub, installing and configuring everything we need along the way. This tutorial covers the fundamental work needed to get started; more advanced topics will be covered in subsequent guides. + +:::tip Already running Infrahub? +If you already have an Infrahub instance running, you can skip straight to the [NetBox adapter documentation](../adapters/netbox.mdx). +::: + +By the end of this tutorial, you will know how to: + +- run Infrahub in a Docker container +- install Infrahub Sync +- load a schema into Infrahub +- create a NetBox → Infrahub sync project +- synchronize NetBox objects into Infrahub +- review the import in a branch and open a proposed change + +**Time**: ~30 minutes + +**What you will build**: A running Infrahub instance with a production-grade schema covering the same core domains as NetBox (locations, devices, interfaces, IPAM, and organizations), populated with data synchronized from the public NetBox demo into a branch, ready to review as a proposed change. + +**Prerequisites**: + +- [Docker](https://docs.docker.com/get-docker/) installed and running (Docker Desktop or OrbStack) +- [uv](https://docs.astral.sh/uv/getting-started/installation/) (Python package manager) +- [Python 3.11+](https://www.python.org/downloads/) + +:::warning Public demo data +The NetBox demo instance is public and resets regularly. Object counts, names, and sample data may differ from the examples in this tutorial. Because anyone can edit it, it can also contain malformed or unexpected data that breaks the sync — see [Troubleshooting](#troubleshooting) if you run into errors. +::: + +## Create a project + +[Copier](https://copier.readthedocs.io/) is a project scaffolding tool. Use it to create a new Infrahub project from the official template, which includes the standard file structure, task definitions, and a `schemas/` folder: + +1. Run the following command to create a new project directory: + +```bash +uv tool run --from 'copier' copier copy https://github.com/opsmill/infrahub-template infrahub-automation +``` + +When prompted, enter a project name (for example, `infrahub-automation`), then press **Enter** to accept the default for every remaining prompt (they all default to **No**). + +2. Navigate to the project directory: + +```bash +cd infrahub-automation +``` + +3. Open the project in your IDE. If you have Visual Studio Code installed, you can run: + +```bash +code . +``` + +:::success Verification +Run `ls` in the project directory. You should see files including `pyproject.toml`, `tasks.py`, and a `schemas/` folder. +::: + +## Start Infrahub + +The project template includes [Invoke](https://www.pyinvoke.org/) tasks that wrap Docker Compose commands. + +1. Start all Infrahub services with a single command: + +```bash +uv run invoke start +``` + +The first run takes a few minutes while Docker downloads the container images. + +2. Open your browser and go to [http://localhost:8000](http://localhost:8000). + +3. Log in from the bottom-left corner using the default credentials: + + - Username: `admin` + - Password: `infrahub` + +:::success Verification +You should see the Infrahub web interface with a navigation menu on the left side. +::: + +## Install infrahub-sync + +1. Install the Python dependencies with uv: + +```bash +uv add "infrahub-sync==2.0.0" pynetbox +``` + +This installs the `infrahub-sync` command. `pynetbox` is required by the NetBox adapter. + +2. Verify infrahub-sync command is available: + +```bash +uv run infrahub-sync --help +``` + +For more about supported adapters and their Python requirements, see the [NetBox adapter documentation](../adapters/netbox.mdx). + +## Load a schema into Infrahub + +Infrahub stores data according to its schema. Before we can import any data, we need Infrahub to know the kinds of objects that the sync will create. + +We provide a production-grade Infrahub schema covering DCIM and IPAM features similar to NetBox's. It is not a one-to-one port of NetBox's own data model, and that's intentional. + +:::info + +This is a production-grade Infrahub schema, not a copy of NetBox's data model. Infrahub gives you a flexible graph model, so a real migration can preserve the parts of NetBox that matter to you while adapting the model to your own workflows. + +::: + +This collection of schemas is still a work in progress; eventually it will be available through the marketplace. For now, download the files directly: + +```bash +BRANCH="bgi-schema-library-v2" +DEST="schemas" +BASE_URL="https://raw.githubusercontent.com/opsmill/schema-library/${BRANCH}" + +FILES=( + "base/dcim.yml" + "base/location.yml" + "base/ipam.yml" + "base/organization.yml" + "extensions/aggregate/aggregate.yml" + "extensions/cable/cable.yml" + "extensions/circuit/circuit.yml" + "extensions/compute/compute.yml" + "extensions/cluster/cluster.yml" + "extensions/hosting_cluster/hosting_cluster.yml" + "extensions/lag/lag.yml" + "extensions/location_site/location_site.yml" + "extensions/vlan/vlan.yml" + "extensions/qinq/qinq.yml" + "extensions/rack/rack.yml" + "extensions/vrf/vrf.yml" +) + +for f in "${FILES[@]}"; do + curl -sSL --create-dirs -o "${DEST}/${f}" "${BASE_URL}/${f}" +done +``` + +Export the local Infrahub address and API token. Infrahub Sync will need them later to authenticate against your instance: + +```bash +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" +``` + +This is the default admin token that the project's Docker Compose stack seeds automatically, and it matches the default already configured in `infrahubctl.toml`. Do not reuse it for an internet-facing or shared Infrahub instance. + +Then load the schema into Infrahub using the project's `load-schema` task: + +```bash +uv run invoke load-schema +``` + +Refresh [the local Infrahub web interface](http://localhost:8000). You should see additional schema objects in the left navigation. You can also open the schema view in the UI to explore the kinds and relationships that were loaded. + +## Create a NetBox API token + +Create a token in the public NetBox demo instance so Infrahub Sync can read data from NetBox. + +1. Open the [public NetBox demo](https://demo.netbox.dev). +2. Log in with username `admin` and password `admin`. +3. Open your user profile. +4. Create an API token. +5. Copy the complete generated token value. + +Export the token locally: + +```bash +export NETBOX_URL="https://demo.netbox.dev" +export NETBOX_TOKEN="nbt_..." +``` + +Export the complete `nbt_...` token value. Do not include an authorization prefix such as `Bearer`, and do not copy only the short **Key** field. + +## Create the sync project + +A sync project is a directory containing a `config.yml` file. The configuration names the source and destination adapters, maps source fields to destination fields, and includes references that let Infrahub Sync compute write order. + +From your project directory (`infrahub-automation`), create a sync project directory and download the example NetBox to Infrahub configuration. + +```bash +mkdir -p sync-projects/netbox-demo +curl -L \ + https://raw.githubusercontent.com/opsmill/infrahub-sync/refs/heads/main/examples/netbox_to_infrahub/config.yml \ + -o sync-projects/netbox-demo/config.yml +``` + +The downloaded configuration is named `netbox-demo-to-infrahub`. It maps selected NetBox objects onto the Infrahub schema. + +Open `sync-projects/netbox-demo/config.yml` in an editor and scan the top-level keys: + +- `name` identifies the sync project. +- `source` configures the NetBox adapter. +- `destination` configures the Infrahub adapter. +- `schema_mapping` defines how NetBox API resources become Infrahub objects. + +The configuration includes default endpoint values, but the environment variables exported above take precedence for tokens and URLs. + +The example configuration comments also mention `apply`, `--run-id`, and `--parallel`. This tutorial uses the beginner path: preview with `diff`, then write with `sync --diff`. See [Run a sync](../running-a-sync.mdx) for cached plans and parallel execution. + +For a fuller explanation of this file, see [Create a sync project](../creating-a-sync-project.mdx) and the [schema mapping reference](../reference/schema-mapping.mdx). + +## Generate the sync code + +Generate the Python models and adapter code for this sync project. + +```bash +uv run infrahub-sync generate --name netbox-demo-to-infrahub --directory sync-projects +``` + +The `generate` command reads the sync configuration and the destination schema, then writes the Python code used at runtime. Run `generate` again any time you edit `config.yml`. + +For more about the command flow, see [Run a sync](../running-a-sync.mdx). + +## Create a branch + +Infrahub tracks changes through branches, so you can review a batch of imported data as a proposed change before it lands on `main`. Create a branch to hold this import: + +```bash +uv run infrahubctl branch create netbox-import +``` + +The rest of this tutorial runs `diff` and `sync` against this branch with `--branch netbox-import`, so the import can be reviewed before merging. For more about branches and proposed changes, see [Infrahub version control](https://docs.infrahub.app/topics/version-control/). + +## Preview the changes + +Run a dry-run diff against the `netbox-import` branch before writing any data to Infrahub. + +```bash +uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects --branch netbox-import +``` + +The `diff` command loads data from NetBox and Infrahub, compares both sides, and prints the planned changes. It also writes a cached plan under `.infrahub-sync-cache/netbox-demo-to-infrahub//plan.parquet`. + +Review the output before continuing. On a first run against an empty branch, most planned changes should be creates. Exact counts depend on the current public NetBox demo data. + +## Sync the data + +After reviewing the diff, run the sync against the same branch. + +```bash +uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --branch netbox-import --diff +``` + +The `--diff` option prints the diff before applying the changes. The first sync can take a few minutes because it writes the imported objects and relationships into the `netbox-import` branch — `main` is untouched until you merge the resulting proposed change. Because this configuration omits `order:`, Infrahub Sync derives the write order from the mapping references. + +## Verify the imported data + +The imported data lives on the `netbox-import` branch, not on `main`. Open [the local Infrahub web interface](http://localhost:8000): + +1. Use the branch selector in the top-left corner to switch from `main` to `netbox-import`. +2. Browse the left navigation for imported objects from the NetBox demo. Depending on the current demo data and the example mapping, you may see objects such as: + + - tags and organizations (manufacturers, providers, RIRs) + - sites and racks + - devices and interfaces + - VRFs, VLANs, and VLAN groups + - prefixes, IP addresses, and aggregates + - circuits + +3. Once you're happy with the data, open a proposed change from `netbox-import` toward `main` so the import can be reviewed before it's merged. + +If you switch back to `main`, none of this data is there yet — that's expected, since it's still isolated in the branch. + +## What happened + +You used Infrahub Sync to move data from NetBox into Infrahub in a controlled sequence: + +1. Infrahub provided the destination graph and schema. +2. NetBox provided the source data. +3. A branch isolated the import from `main` so it could be reviewed first. +4. `config.yml` described the adapters, field mappings, references, filters, and Transformations. +5. `generate` converted the configuration into runnable sync code. +6. `diff` compared the source and destination without writing changes. +7. `sync` applied the reviewed changes to the branch. + +The same pattern applies to larger migrations: start with a clear schema, map a small set of objects, condition source data where needed, review the diff inside a branch, and then synchronize before opening a proposed change toward `main`. + +## Stop the local Infrahub instance + +When you are finished, stop the Docker Compose stack. + +```bash +uv run invoke stop +``` + +## Troubleshooting + +### `infrahub-sync` fails with `Both url and token must be specified` + +```text +ERROR | infrahub_sync.cli | Failed to initialize the Sync Instance: Error initializing InfrahubAdapter: Both url and token must be specified! +``` + +The NetBox and/or Infrahub adapter can't find credentials. `generate`, `diff`, `sync`, and `apply` all read them from the environment, not from `config.yml`, so make sure every variable is exported in the shell you're running the command from: + +```bash +export NETBOX_URL="https://demo.netbox.dev" +export NETBOX_TOKEN="" +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" +``` + +This error names whichever adapter (NetBox or Infrahub) is missing its variables — the same message is raised for either one. + +### `infrahub-sync` fails with `Object ... already present` + +```text +ValueError: An error occurred while loading Netbox: ('Object 172.16.0.2/24__Alpha already present', IpamIPAddress "172.16.0.2/24__Alpha") +``` + +Two source objects map to the same identifier — here, two IP addresses sharing an address and VRF. Delete one of the duplicates in NetBox (**IPAM > IP Addresses**, search the address from the message), then re-run the command. + +## Next steps + +Now that you have completed a first sync, you have covered some of the basic objects in NetBox. + +This is only the first step. You will likely want to bring in the parts that are unique to your own NetBox instance (for example, roles or custom fields). + +Other guides will soon be available to cover: + +- How to sync locations/regions +- How to deal with VLAN/Prefix/Device roles +- How to cover custom attributes / relationships +- Migrate configuration context \ No newline at end of file diff --git a/docs/sidebars.ts b/docs/sidebars.ts index 474f709..e8d9c89 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -12,6 +12,13 @@ const sidebars: SidebarsConfig = { 'running-a-sync', ], }, + { + type: 'category', + label: 'Tutorials', + items: [ + 'tutorials/netbox-demo-to-infrahub', + ], + }, { type: 'category', label: 'Guides', diff --git a/examples/netbox_to_infrahub/config.yml b/examples/netbox_to_infrahub/config.yml index 203dc0b..5ed7286 100644 --- a/examples/netbox_to_infrahub/config.yml +++ b/examples/netbox_to_infrahub/config.yml @@ -1,5 +1,5 @@ --- -# from-netbox — demonstrates the two newest features of infrahub-sync: +# netbox-demo-to-infrahub — demonstrates the two newest features of infrahub-sync: # # 1. Auto-tiered write order. The `order:` list has been removed; the engine # derives tiers from `schema_mapping[].fields[].reference` and groups @@ -7,33 +7,40 @@ # enforce a hard barrier between tiers. # # 2. Parquet sidecar cache. Every `diff` writes per-resource snapshots and -# a `plan.parquet` under `.infrahub-sync-cache/from-netbox//`. +# a `plan.parquet` under `.infrahub-sync-cache/netbox-demo-to-infrahub//`. # The run_id is logged at INFO; pass it to `apply` to re-dispatch the # cached plan without re-extracting the source. # # Usage: -# uv run infrahub-sync diff --name from-netbox --directory examples/ -# uv run infrahub-sync apply --name from-netbox --run-id --directory examples/ -# uv run infrahub-sync sync --name from-netbox --directory examples/ --parallel +# uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects +# uv run infrahub-sync apply --name netbox-demo-to-infrahub --run-id --directory sync-projects +# uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --parallel +# +# --------------------------------------------------------------------------- -name: from-netbox +name: netbox-demo-to-infrahub source: name: netbox settings: url: "https://demo.netbox.dev" - token: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" destination: name: infrahub settings: url: "http://localhost:8000" +# Credentials are read from the environment, not this file: +# export NETBOX_URL="https://demo.netbox.dev" +# export NETBOX_TOKEN="nbt_..." +# export INFRAHUB_ADDRESS="http://localhost:8000" +# export INFRAHUB_API_TOKEN="..." + # order: omitted — infrahub-sync auto-computes tiers from schema_mapping. # Uncomment and edit only if you need to override the computed order: # order: # - BuiltinTag -# - RoleGeneric +# - CoreStandardGroup # ... schema_mapping: @@ -47,135 +54,96 @@ schema_mapping: - name: description mapping: description - # Roles (DCIM, IPAM, Devices) - - name: RoleGeneric - mapping: dcim.device_roles - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: RoleGeneric - mapping: dcim.rack_roles - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: RoleGeneric - mapping: ipam.roles - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - # Tenancy (tenants, tenant groups) - - name: CoreStandardGroup - mapping: tenancy.tenant-groups - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: OrganizationGeneric - mapping: tenancy.tenants - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: group - mapping: group - reference: CoreStandardGroup - - # Sites (Site Groups, Region, Sites, Location) - - name: CoreStandardGroup - mapping: dcim.site-groups - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: LocationGeneric - mapping: dcim.regions - fields: - - name: name - mapping: slug - - name: description - mapping: name - - name: type - static: "Region" - - name: tags - mapping: tags - reference: BuiltinTag - - name: LocationGeneric + # Sites / Regions / Locations + - name: LocationSite mapping: dcim.sites + identifiers: ["name"] fields: - name: name mapping: slug - name: description mapping: name - - name: type - static: "Site" - - name: group - mapping: group - reference: CoreStandardGroup - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: tags - mapping: tags - reference: BuiltinTag - - name: LocationGeneric - mapping: dcim.locations - fields: - - name: name - mapping: slug - - name: description - mapping: name - - name: type - static: "Location" - - name: organization - mapping: tenant - reference: OrganizationGeneric + - name: status + mapping: status.value + - name: facility + mapping: facility + - name: physical_address + mapping: physical_address + - name: timezone + mapping: time_zone - name: tags mapping: tags reference: BuiltinTag + # ----------------------------------------------------------------------- + # Netbox "region" model is relatively loose and could accommodate multiple layers of nested objects (e.g. continent > country > city). + # In Infrahub you have to explicitly define this hierarchy in the schema. + # As every organization organizes this differently, we don't provide out-of-the-box support for it, even though we provide a couple of examples in the schema library / marketplace. + # For the sake of having a first working sync, we've defined a very basic site node in the schema that you can build upon once your hierarchy is clear. + # To fully support region you would need to: + # - Create the hierarchy in the schema + # - Uncomment the lines below and adjust to your specific data <> schema + # + # For example, here the region is top-level (it has no parent) + # - name: LocationRegion + # mapping: dcim.regions + # identifiers: ["name"] + # fields: + # - name: name + # mapping: name + # - name: description + # mapping: description + # filters: + # - field: parent + # operation: "is_empty" + # + # For example, here countries sit directly below the region (depth = 1) + # - name: LocationCountry + # mapping: dcim.regions + # identifiers: ["name"] + # fields: + # - name: name + # mapping: name + # - name: description + # mapping: description + # - name: parent + # mapping: region + # reference: LocationRegion + # filters: + # - field: _depth + # operation: "==" + # value: 1 + # ----------------------------------------------------------------------- + # Racks - - name: InfraRack + - name: LocationRack mapping: dcim.racks - identifiers: ["name", "location"] + identifiers: ["name", "site"] fields: - name: name mapping: name - - name: location + - name: site mapping: site - reference: LocationGeneric + reference: LocationSite - name: height mapping: u_height - - name: serial_number - mapping: serial - - name: asset_tag - mapping: asset_tag - name: facility_id mapping: facility_id - - name: role - mapping: role - reference: RoleGeneric + - name: status + mapping: status.value - name: tags mapping: tags reference: BuiltinTag + - name: serial_number + mapping: serial + - name: asset_tag + mapping: asset_tag + # NOTE: current schema doesn't support role for racks (see related note) + # - name: role + # mapping: role.slug - # Device (manufacturer, device types, devices, interfaces) - - name: OrganizationGeneric + # Manufacturers / Platforms / Device Types + - name: OrganizationManufacturer mapping: dcim.manufacturers identifiers: ["name"] fields: @@ -183,7 +151,21 @@ schema_mapping: mapping: name - name: description mapping: description - - name: ChoiceDeviceType + - name: tags + mapping: tags + reference: BuiltinTag + + - name: DcimPlatform + mapping: dcim.platforms + identifiers: ["name"] + fields: + - name: name + mapping: name + - name: manufacturer + mapping: manufacturer + reference: OrganizationManufacturer + + - name: DcimDeviceType mapping: dcim.device-types identifiers: ["name", "manufacturer"] fields: @@ -195,69 +177,126 @@ schema_mapping: mapping: is_full_depth - name: height mapping: integer_height - - name: tags - mapping: tags - reference: BuiltinTag + - name: weight + mapping: integer_weight + # NOTE: Netbox tracks weight_unit separately (kg/lb/g/oz); schema + # has no unit field, so mixed-unit sources will be inconsistent. - name: manufacturer mapping: manufacturer - reference: OrganizationGeneric + reference: OrganizationManufacturer transforms: - field: integer_height expression: "{{ u_height|float|round(0, 'ceil') }}" - # -> The netbox constraint is `dcim_device_unique_name_site_tenant` - # Reusing device_name + site + Organization as identifiers - # /!\ Seem like Netbox allowed device to have the same name if there is a virtual-chassis - - name: InfraDevice - identifiers: ["location", "rack", "organization", "name"] + - field: integer_weight + expression: "{{ weight|float|round(0, 'ceil') }}" + # --------------------------------------------------------------------- + # Netbox roles are user-defined/arbitrary, so there's no guarantee the + # Netbox slug matches one of the fixed choices below — sync will fail for + # values (roles) that don't match one of the schema's dropdown choices. + # TODO: decide whether to (a) map role.slug directly per node below and + # accept failures on non-matching values, (b) extend the schema's choice + # lists to cover your Netbox roles, or (c) drop role sync entirely. + # dcim.rack_roles has no destination at all anymore (Rack has no role + # field in the new schema) — not mapped below. + # --------------------------------------------------------------------- + + # Devices + # ----------------------------------------------------------------------- + # Netbox splits location across `site` (mandatory) and `rack` (optional). + # Infrahub consolidates both under a single `location` relationship that + # can point to either a LocationRack or a LocationSite. Split into two + # entries over the same dcim.devices source with complementary filters: + # racked devices use the rack, everything else falls back to the site. + # TODO: Find a better way to support this + # ----------------------------------------------------------------------- + - name: DcimDevice + identifiers: ["location", "name"] + mapping: dcim.devices + fields: + - name: name # /!\ Netbox allows duplicate device names within a virtual-chassis. + mapping: name + - name: serial + mapping: serial + - name: description + mapping: description + - name: position + mapping: position + - name: rack_face + mapping: face.value + - name: status + mapping: status.value + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug + - name: device_type + mapping: device_type + reference: DcimDeviceType + - name: platform + mapping: platform + reference: DcimPlatform + - name: location + mapping: rack + reference: LocationRack + - name: tags + mapping: tags + reference: BuiltinTag + - name: primary_address + mapping: primary_ip + reference: IpamIPAddress + filters: + - field: parent_device # For sub-devices (e.g. stacks), you'd most likely have a dedicated node in Infrahub + operation: "is_empty" + - field: rack + operation: "is_not_empty" + - field: name # /!\ Netbox allows empty name for devices + operation: "is_not_empty" + + - name: DcimDevice + identifiers: ["location", "name"] mapping: dcim.devices fields: - name: name mapping: name - - name: serial_number + - name: serial mapping: serial - - name: asset_tag - mapping: asset_tag - name: description mapping: description - - name: model + - name: position + mapping: position + - name: rack_face + mapping: face.value + - name: status + mapping: status.value + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug + - name: device_type mapping: device_type - reference: ChoiceDeviceType - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: role - mapping: role - reference: RoleGeneric - # - name: status - # mapping: status - # reference: StatusGeneric + reference: DcimDeviceType + - name: platform + mapping: platform + reference: DcimPlatform - name: location mapping: site - reference: LocationGeneric - - name: rack - mapping: rack - reference: InfraRack + reference: LocationSite - name: tags mapping: tags reference: BuiltinTag - # Showcase Filters - # It will import the ones with dmu01 in their name *AND* without pdu + - name: primary_address + mapping: primary_ip + reference: IpamIPAddress filters: - - field: name - operation: "contains" - value: "dmi01" - - field: name - operation: "not contains" - value: "pdu" - # Showcase Transforms - transforms: - - field: name - expression: "{{ name.lower() if name else '' }}" - - field: serial_number - expression: "{{ serial.lower() if serial else '' }}}" + - field: parent_device + operation: "is_empty" + - field: rack + operation: "is_empty" + - field: name # /!\ Netbox allows empty name for devices + operation: "is_not_empty" + + # TODO: Add support for cluster, VM, servers ... - # Interfaces (interfaces, rear port, front port) - - name: InfraInterfaceL2L3 + # Interfaces (Physical / Virtual / LAG) + - name: InterfacePhysical identifiers: ["device", "name"] mapping: dcim.interfaces fields: @@ -265,73 +304,137 @@ schema_mapping: mapping: name - name: description mapping: description - - name: interface_type - mapping: type.label - - name: l2_mode - mapping: mode.label - name: mac_address mapping: mac_address - # - name: ip_addresses - # mapping: ip_addresses - # reference: InfraIPAddress - - name: mgmt_only - mapping: mgmt_only - - name: untagged_vlan - mapping: untagged_vlan - reference: InfraVLAN - - name: tagged_vlan - mapping: tagged_vlans - reference: InfraVLAN + - name: l2_mode + mapping: mode.value + - name: mtu + mapping: mtu + - name: ip_addresses + mapping: ip_addresses + reference: IpamIPAddress + # TODO: Make sure no vlan is skipped due to missing group then enable this + # - name: untagged_vlan + # mapping: untagged_vlan + # reference: IpamVLAN + # - name: tagged_vlan + # mapping: tagged_vlans + # reference: IpamVLAN - name: device mapping: device - reference: InfraDevice - - name: tags - mapping: tags - reference: BuiltinTag - # - name: status - # mapping: status - # reference: StatusGeneric - filters: - - field: device.name - operation: "contains" - value: "dmi01" - - field: device.name + reference: DcimDevice + - name: bundle + mapping: lag + reference: InterfaceLag + # TODO: many fields are not implemented in the schema (mgmt_only, type, duplex ...) + filters: # Filters to only get the "physical interfaces" + # NOTE: Netbox's `type` is a nested {value, label} object, not a plain + # string — filtering on bare `type` compares against the dict itself + # (dict membership checks keys, dict == string is always False), which + # silently let every interface through here and matched none below. + - field: type.value + operation: "not contains" + value: "virtual" + - field: type.value operation: "not contains" - value: "pdu" + value: "lag" + # NOTE: must mirror the DcimDevice-level exclusions — otherwise an + # interface can reference a device that was filtered out of the sync, + # and the `device` reference lookup fails ("Unable to locate the node"). + # Only `name` can be checked here: Netbox's interface payload embeds a + # *brief* device object (id/url/display/name/description only), which + # never carries `parent_device` — so we can't replicate the + # `parent_device is_empty` exclusion at this level. Not an issue today + # (no devices in this dataset have parent_device set), but if you add + # sub-devices later, their interfaces will need filtering out some + # other way (e.g. a follow-up id-based exclusion list). + - field: device.name + operation: "is_not_empty" - # Circuits (Provider, Provider Network, Circuits Types, Circuits) - - name: OrganizationGeneric - mapping: circuits.providers - identifiers: ["name"] + - name: InterfaceVirtual + identifiers: ["device", "name"] + mapping: dcim.interfaces fields: - name: name mapping: name - name: description mapping: description - - name: InfraProviderNetwork - mapping: circuits.provider-networks - identifiers: ["name"] + - name: mac_address + mapping: mac_address + - name: l2_mode + mapping: mode.value + - name: ip_addresses + mapping: ip_addresses + reference: IpamIPAddress + - name: untagged_vlan + mapping: untagged_vlan + reference: IpamVLAN + - name: tagged_vlan + mapping: tagged_vlans + reference: IpamVLAN + - name: device + mapping: device + reference: DcimDevice + # TODO: many fields are not implemented in the schema (mgmt_only, type, duplex ...) + filters: # Filters to only get the "virtual interfaces" + - field: type.value + operation: "==" + value: "virtual" + # NOTE: must mirror the DcimDevice-level `name` exclusion — see + # InterfacePhysical for why `parent_device` can't be checked here too. + - field: device.name + operation: "is_not_empty" + + - name: InterfaceLag + identifiers: ["device", "name"] + mapping: dcim.interfaces fields: - name: name mapping: name - - name: vendor_id - mapping: service_id - - name: provider - mapping: provider - reference: OrganizationGeneric + - name: bundle_number + mapping: bundle_number + # Capture the number at the end of the name (e.g. Po1 -> 1) - name: description mapping: description - # - name: status - # mapping: status - # reference: StatusGeneric - - name: tags - mapping: tags - reference: BuiltinTag - - name: provider - mapping: provider - reference: OrganizationGeneric - - name: ChoiceCircuitType - mapping: circuits.circuit-types + - name: mac_address + mapping: mac_address + - name: l2_mode + mapping: mode.value + - name: ip_addresses + mapping: ip_addresses + reference: IpamIPAddress + - name: untagged_vlan + mapping: untagged_vlan + reference: IpamVLAN + - name: tagged_vlan + mapping: tagged_vlans + reference: IpamVLAN + - name: device + mapping: device + reference: DcimDevice + # TODO: many fields are not implemented in the schema (mgmt_only, type, duplex ...) + # TODO: many fields are not available in Netbox + # - name: lacp_rate + # - name: lacp_mode + transforms: + - field: bundle_number + expression: |- + {% set suffix = name[(name.rstrip('0123456789'))|length:] %} + {{ suffix | int if suffix else None }} + filters: # Filters to only get the "lag interfaces" + - field: type.value + operation: "==" + value: "lag" + # NOTE: must mirror the DcimDevice-level `name` exclusion — see + # InterfacePhysical for why `parent_device` can't be checked here too. + - field: device.name + operation: "is_not_empty" + + # TODO: Many other types of interface to handle + + # Provider + - name: OrganizationProvider + mapping: circuits.providers identifiers: ["name"] fields: - name: name @@ -341,44 +444,38 @@ schema_mapping: - name: tags mapping: tags reference: BuiltinTag - - name: InfraCircuit + + # Circuit + - name: DcimCircuit mapping: circuits.circuits identifiers: ["circuit_id"] fields: - name: circuit_id mapping: cid - - name: vendor_id - mapping: cid - - name: provider - mapping: provider - reference: OrganizationGeneric - name: description mapping: description - # - name: status - # mapping: status - # reference: StatusGeneric - - name: type - mapping: type.name + - name: commit_rate + mapping: commit_rate + - name: status + mapping: status.value + # NOTE: similar to roles, circuit types have to be defined in the schema + # - name: circuit_type + # mapping: type.slug - name: provider mapping: provider - reference: OrganizationGeneric - - name: tags - mapping: tags - reference: BuiltinTag + reference: OrganizationProvider -# IPAM (VRF, VLANs Groups, VLANs, Prefixes, IPs) - - name: InfraRouteTarget + # IPAM (VRF, Route Targets, VLAN Groups, VLANs, Prefixes, IPs) + - name: IpamRouteTarget mapping: ipam.route-targets - identifiers: ["name", "organization"] + identifiers: ["name"] fields: - name: name mapping: name - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: InfraVRF + + - name: IpamVRF mapping: ipam.vrfs identifiers: ["name"] fields: @@ -386,18 +483,18 @@ schema_mapping: mapping: name - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - name: vrf_rd mapping: rd + - name: enforce_unique + mapping: enforce_unique - name: import_rt mapping: import_targets - reference: InfraRouteTarget + reference: IpamRouteTarget - name: export_rt mapping: export_targets - reference: InfraRouteTarget - - name: CoreStandardGroup + reference: IpamRouteTarget + + - name: IpamVLANGroup mapping: ipam.vlan-groups identifiers: ["name"] fields: @@ -405,26 +502,42 @@ schema_mapping: mapping: name - name: description mapping: description - - name: InfraVLAN + # TODO: `scope` (peer IpamVLANGroupScope: Region/Site/Rack/etc.) needs + # resolving Netbox's polymorphic scope_type/scope_id — not mapped yet. + # - name: scope + # mapping: scope + # reference: LocationSite + + - name: IpamVLAN mapping: ipam.vlans - identifiers: ["name", "vlan_id", "location", "vlan_group"] + identifiers: ["name", "vlan_id", "vlan_group"] fields: - name: name mapping: name - name: vlan_id mapping: vid - - name: organization - mapping: tenant - reference: OrganizationGeneric - name: description mapping: description - - name: location - mapping: site - reference: LocationGeneric + - name: status + mapping: status.value + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug - name: vlan_group mapping: group - reference: CoreStandardGroup - - name: InfraPrefix + reference: IpamVLANGroup + # vlan_group is required (optional: false) in the schema, but Netbox + # VLANs can have no group at all — skip those rather than fail the sync. + filters: + - field: group + operation: "is_not_empty" + + # TODO: add support for provider-network + # TODO: add support for tenant + # TODO: add support for device types -> object templates + # TODO: add support for DcimCircuitEndpoint + + - name: IpamPrefix mapping: ipam.prefixes identifiers: ["prefix", "vrf"] fields: @@ -432,26 +545,32 @@ schema_mapping: mapping: prefix - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: role - mapping: role - reference: RoleGeneric + - name: status + mapping: resolved_status + - name: member_type + mapping: resolved_member_type + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug - name: vrf mapping: vrf - reference: InfraVRF - - name: location - mapping: site - reference: LocationGeneric - # - name: status - # mapping: status - # reference: StatusGeneric + reference: IpamVRF + # TODO: Make sure no vlan is skipped due to missing group then enable this # - name: vlan - # mapping: vlan.name - # - name: location - # mapping: location.slug - - name: InfraIPAddress + # mapping: vlan + # reference: IpamVLAN + # TODO: `scope` (peer IpamPrefixScope: Region/Site/Rack/etc.) needs + # resolving Netbox's polymorphic scope_type/scope_id — not mapped yet. + # - name: scope + # mapping: scope + # reference: LocationSite + transforms: # These transforms accommodate different ways of dealing with containers + - field: resolved_status + expression: "{{ 'active' if status.value == 'container' else status.value }}" + - field: resolved_member_type + expression: "{{ 'prefix' if status.value == 'container' else 'address' }}" + + - name: IpamIPAddress mapping: ipam.ip-addresses identifiers: ["address", "vrf"] fields: @@ -459,15 +578,70 @@ schema_mapping: mapping: address - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - # Not the same as ipam.role - # - name: role - # mapping: role.value - # - name: status - # mapping: status - # reference: StatusGeneric + - name: status + mapping: status.value + # NOTE: Netbox also has dhcp/slaac statuses with no schema equivalent. - name: vrf mapping: vrf - reference: InfraVRF + reference: IpamVRF + # NOTE: This isn't enforced on the Netbox side, and might create validation issues + # - name: fqdn + # mapping: dns_name + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.value + + # IPAM — Aggregates / RIRs (new in schema, no prior mapping existed) + - name: OrganizationRIR + mapping: ipam.rirs + identifiers: ["name"] + fields: + - name: name + mapping: name + - name: description + mapping: description + - name: is_private + mapping: is_private + - name: tags + mapping: tags + reference: BuiltinTag + + - name: IpamAggregate + mapping: ipam.aggregates + identifiers: ["prefix"] + fields: + - name: prefix + mapping: prefix + - name: description + mapping: description + - name: date_added + mapping: date_added + - name: rir + mapping: rir + reference: OrganizationRIR + + # ----------------------------------------------------------------------- + # New schema extensions with NO mapping below — none of these existed in + # the previous config, and each needs a real design decision rather than + # a guess: + # + # - ClusterGeneric / ClusterHosting (schemas/extensions/cluster, + # hosting_cluster): Netbox source would be virtualization.clusters, + # but `technology` is a fixed Dropdown (vmware/kvm/hyper_v/xen/ + # nutanix/openstack/kubernetes/public_cloud) while Netbox's + # cluster.type is a free-form ClusterType object — needs a mapping + # table, not a direct field copy. + # - ComputePhysicalServer / VirtualizationVirtualMachine + # (schemas/extensions/compute): virtualization.virtual-machines would + # feed VirtualizationVirtualMachine, but `host` is polymorphic + # (ClusterHosting or ComputePhysicalServer) and memory units differ + # (Netbox MB vs schema's documented GB) — needs a transform. + # ComputePhysicalServer has no obvious Netbox source at all (would + # require picking out dcim.devices by role, which is arbitrary). + # - DcimCable (schemas/extensions/cable): Netbox dcim.cables uses + # polymorphic multi-sided terminations (cable-terminations endpoint) + # that don't map cleanly onto DcimCable's simpler two-endpoint + # Connector model — needs real design work. + # - IpamSVLAN / IpamCVLAN (schemas/extensions/qinq): Infrahub-only + # concept (802.1ad Q-in-Q split), no equivalent Netbox source field. + # -----------------------------------------------------------------------