From 0785d60a1b3cf93974b5748bbe348b6ebb24dcb2 Mon Sep 17 00:00:00 2001 From: Blake Ellis Date: Thu, 18 Jun 2026 11:37:14 -0400 Subject: [PATCH 01/16] Add NetBox to Infrahub sync tutorial Adds a tutorial for syncing public NetBox demo data into a local Infrahub instance and exposes it in the docs sidebar. Validation: - uv run pytest -q - npx markdownlint-cli2 "docs/docs/**/*.{md,mdx}" - uv run invoke docs.docusaurus Generated with Codex. --- .../tutorials/netbox-demo-to-infrahub.mdx | 496 ++++++++++++++++++ docs/sidebars.ts | 7 + 2 files changed, 503 insertions(+) create mode 100644 docs/docs/tutorials/netbox-demo-to-infrahub.mdx diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx new file mode 100644 index 0000000..b055a96 --- /dev/null +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -0,0 +1,496 @@ +--- +title: Sync NetBox to Infrahub +--- + +In this tutorial we will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) +into Infrahub. We will start from scratch, installing and configuring everything we need along the way. + +By the end of this tutorial, you will know how to: + +- install Infrahub Sync +- run Infrahub in a Docker container; +- load the example NetBox schema into Infrahub; +- create a NetBox → Infrahub sync project; +- use `transforms` to reshape source values before import; +- use `filters` to control which source objects are synchronized; +- review the planned changes with `infrahub-sync diff`; and +- synchronize NetBox objects into Infrahub. + +This tutorial is self-contained. When a related page provides more detail, we +link to it so you can continue exploring after the first successful sync. + +:::warning Public demo data + +The NetBox demo instance is public and resets regularly. Object counts, names, +and sample data may differ from the examples in this tutorial. + +::: + +## Prerequisites + +We walk through everything in this tutorial except Docker, Python, and basic command line +tooling like Git and curl. You also don't need to register for any accounts, we +will use open source software and the public NetBox demo to get the job done. + +Specific requirements are: + +- [Docker](https://www.docker.com/) installed and running +- [Python](https://www.python.org/) 3.10 or later +- [Git](https://git-scm.com/) +- [curl](https://curl.se/) +- A web browser + +## Prepare a local workspace + +Create a directory for this tutorial. We will install Infrahub Sync and store the +sync project here. + +```bash +mkdir -p ~/repos/demo-infrahub-sync +cd ~/repos/demo-infrahub-sync +``` + +## Install Infrahub Sync + +Create a Python virtual environment and install Infrahub Sync from PyPI. + +```bash +python3 -m venv .venv +source .venv/bin/activate +pip install infrahub-sync pynetbox +``` + +`infrahub-sync` installs the Infrahub SDK, including the `infrahubctl` command. +`pynetbox` is required by the NetBox adapter. + +Verify both commands are available: + +```bash +infrahub-sync --help +infrahubctl --help +``` + +For more about supported adapters and their Python requirements, see the +[NetBox adapter documentation](../adapters/netbox.mdx). + +## Start Infrahub locally + +Clone Infrahub and start it with Docker Compose. + +```bash +cd ~/repos +git clone https://github.com/opsmill/infrahub.git +cd infrahub +docker compose up -d +``` + +Wait for the containers to start and then open +[the local Infrahub web interface](http://localhost:8000). + +Log in with the default local credentials: + +- Username: `admin` +- Password: `infrahub` + +## Create an Infrahub API token + +Infrahub Sync needs an API token to write data into Infrahub. + +1. If the header shows **Log in anonymous**, open it and log in as `admin`. +2. In the Infrahub web interface, click the user menu. +3. Open **Account settings**. +4. Open the **Tokens** tab. +5. Click **Add account token**. +6. Enter a token name, such as `infrahub-sync tutorial`. +7. Click **Save**. +8. Copy the generated token before you confirm the dialog. Infrahub shows this + token only once. + +Export the token in the shell where you will run `infrahub-sync`: + +```bash +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="" +``` + +Do not print or commit API tokens. Keep them in your shell environment or secret +manager. + +## Load the NetBox schema into Infrahub + +Infrahub stores data according to its schema. Before we can import NetBox data, +we need Infrahub to know the kinds of objects that the sync will create. + +The Infrahub repository includes an example schema for data imported from +NetBox. Load that schema into your local Infrahub instance. + +```bash +cd ~/repos/demo-infrahub-sync +infrahubctl schema load ~/repos/infrahub/models/examples/netbox/netbox.yml +``` + +Refresh [the local Infrahub web interface](http://localhost:8000). You should +see additional schema objects in the left navigation. + +:::info + +The example schema is a practical starting point. It does not try to reproduce +NetBox one-to-one. Infrahub gives you a flexible graph model, so a real migration +can preserve the parts of NetBox that matter while adapting the model to your +future workflows. + +For more about the schema used by this adapter, see the +[NetBox adapter schema notes](../adapters/netbox.mdx#schema). + +::: + +## Create a NetBox demo API token + +Create a token in the public NetBox demo instance so Infrahub Sync can read data +from NetBox. + +1. Open the [public NetBox demo](https://demo.netbox.dev). +2. Log in using the credentials shown on the NetBox demo site. +3. Open your user profile. +4. Create an API token. +5. Copy the generated token. + +Export the token locally: + +```bash +export NETBOX_URL="https://demo.netbox.dev" +export NETBOX_TOKEN="" +``` + +Export only the token value. Do not include an authorization prefix such as +`Bearer`. + +## Create the sync project + +A sync project is a directory containing a `config.yml` file. The configuration +names the source and destination adapters, defines which objects move first, and +maps source fields to destination fields. + +Create a sync project directory and download the example NetBox to Infrahub +configuration. + +```bash +cd ~/repos/demo-infrahub-sync +mkdir -p sync-projects/netbox-demo +curl -L \ + https://raw.githubusercontent.com/opsmill/infrahub-sync/refs/heads/main/examples/netbox_to_infrahub/config.yml \ + -o sync-projects/netbox-demo/config.yml +``` + +The downloaded configuration is named `from-netbox`. It maps selected NetBox +objects to the example Infrahub NetBox schema. + +Open `sync-projects/netbox-demo/config.yml` in an editor and scan the top-level +keys: + +- `name` identifies the sync project. +- `source` configures the NetBox adapter. +- `destination` configures the Infrahub adapter. +- `order` defines the sequence for writing related objects. +- `schema_mapping` defines how NetBox API resources become Infrahub objects. + +The configuration includes default endpoint values, but the environment +variables exported above take precedence for tokens and URLs. + +For a fuller explanation of this file, see [Create a sync project](../creating-a-sync-project.mdx) +and the [schema mapping reference](../reference/schema-mapping.mdx). + +## Understand the normalization step + +Public demo data is useful because it is real enough to exercise the sync, but +it can also be messy. The NetBox demo resets regularly, many people use it, and +some values that are valid in NetBox may collide with identifiers in the example +Infrahub schema. + +For this tutorial, we normalize a few fields before generating the sync code. +This shows an important Infrahub Sync pattern: a sync project is not only a field +mapping. It can also condition data as it moves between systems. + +We will use two configuration features: + +- `transforms` change a source value before Infrahub Sync uses it. For example, + a tenant called `Engineering` can become `Engineering (engineering)` so it is + unique and easier to trace back to the NetBox slug. +- `filters` exclude source objects that should not be imported in this tutorial. + For example, we will ignore IP addresses already assigned to another NetBox + object so the first import path remains predictable. + +These adjustments do not modify NetBox. They only change how this sync project +prepares data before writing it to Infrahub. + +## Normalize public demo identifiers + +Run the following script from the tutorial workspace. It edits +`sync-projects/netbox-demo/config.yml` and adds the normalization rules described +above. + +```bash +python - <<'PY' +from pathlib import Path + +import yaml + +path = Path("sync-projects/netbox-demo/config.yml") +config = yaml.safe_load(path.read_text()) + +for item in config["schema_mapping"]: + name = item["name"] + mapping = item.get("mapping") + + if name == "OrganizationGeneric" and mapping == "tenancy.tenants": + item["transforms"] = [ + {"field": "name", "expression": "{{ name }} ({{ slug }})"}, + ] + elif name == "OrganizationGeneric" and mapping == "dcim.manufacturers": + item["transforms"] = [ + {"field": "name", "expression": "Manufacturer: {{ name }}"}, + ] + elif name == "OrganizationGeneric" and mapping == "circuits.providers": + item["transforms"] = [ + {"field": "name", "expression": "Provider: {{ name }}"}, + ] + elif name == "InfraCircuit" and mapping == "circuits.circuits": + item["transforms"] = [ + {"field": "cid", "expression": "{{ cid }} (netbox-{{ id }})"}, + ] + elif name == "InfraIPAddress" and mapping == "ipam.ip-addresses": + item["filters"] = [ + {"field": "assigned_object", "operation": "is_empty"}, + *item.get("filters", []), + ] + +path.write_text("---\n" + yaml.safe_dump(config, sort_keys=False)) +PY +``` + +Run this normalization before `generate`, because `generate` reads `config.yml` +and creates Python code from it. + +### Optional: inspect the normalization manually + +Open `sync-projects/netbox-demo/config.yml` again and search for +`tenancy.tenants`. You should see a transform like this: + +```yaml +transforms: + - field: name + expression: "{{ name }} ({{ slug }})" +``` + +This tells Infrahub Sync to build the destination `name` from two NetBox values. +If the NetBox demo contains duplicate tenant names, the slug helps distinguish +them in Infrahub. + +Now search for `ipam.ip-addresses`. You should see a filter like this: + +```yaml +filters: + - field: assigned_object + operation: is_empty +``` + +This tells Infrahub Sync to import only unassigned IP addresses for this tutorial +path. + +If a future run fails because the public demo contains another duplicate value, +you can use the same pattern: identify the destination field used as an +identifier, add enough source context with a `transforms` expression to make it +unique, run `generate` again, and then rerun `diff`. + +## List the sync project + +Confirm that Infrahub Sync can find the project. + +```bash +infrahub-sync list --directory sync-projects +``` + +You should see a project named `from-netbox`. + +## Generate the sync code + +Generate the Python models and adapter code for this sync project. + +```bash +infrahub-sync generate --name from-netbox --directory sync-projects +``` + +The `generate` command reads the sync configuration and the destination schema, +then writes the Python code used at runtime. Run `generate` again any time you +edit `config.yml`. + +For more about the command flow, see [Run a sync](../running-a-sync.mdx). + +## Preview the changes + +Run a dry-run diff before writing data to Infrahub. + +```bash +infrahub-sync diff --name from-netbox --directory sync-projects +``` + +The `diff` command loads data from NetBox and Infrahub, compares both sides, and +prints the planned changes. It also writes a cached plan under +`.infrahub-sync-cache/`. + +Review the output before continuing. On a first run against an empty local +Infrahub instance, most planned changes should be creates. Exact counts depend +on the current public NetBox demo data. + +## Sync the data + +After reviewing the diff, run the sync. + +```bash +infrahub-sync sync --name from-netbox --directory sync-projects --diff +``` + +The `--diff` option prints the diff before applying the changes. The first sync +can take a few minutes because it writes the imported objects and relationships +into Infrahub. + +## Verify the imported data + +Open [the local Infrahub web interface](http://localhost:8000) and check for +imported objects from the NetBox demo. Depending on the current demo data and +the example mapping, you may see objects such as: + +- tags +- organizations +- locations +- racks +- devices +- interfaces +- prefixes +- IP addresses +- VLANs + +You can also verify from the CLI: + +```bash +infrahubctl object get InfraDevice --limit 5 +infrahubctl object get OrganizationGeneric --limit 5 +``` + +If these commands return objects, the sync imported data successfully. + +## What happened + +You used Infrahub Sync to move data from NetBox into Infrahub in a controlled +sequence: + +1. Infrahub provided the destination graph and schema. +2. NetBox provided the source data. +3. `config.yml` described the adapters, object order, field mappings, filters, + and transforms. +4. `generate` converted the configuration into runnable sync code. +5. `diff` compared the source and destination without writing changes. +6. `sync` applied the reviewed changes to Infrahub. + +The same pattern applies to larger migrations: start with a clear schema, map a +small set of objects, condition source data where needed, review the diff, and +then synchronize. + +## Stop the local Infrahub instance + +When you are finished, stop the Docker Compose stack. + +```bash +cd ~/repos/infrahub +docker compose down +``` + +## Troubleshooting + +### Docker reports that port 8000 is already allocated + +Another container or local service is already using `localhost:8000`. Check the +running containers: + +```bash +docker ps --format 'table {{.Names}}\t{{.Ports}}\t{{.Status}}' +``` + +If the port is held by an older Infrahub container from the same local compose +project, remove stale containers and retry: + +```bash +cd ~/repos/infrahub +docker compose down --remove-orphans +docker compose up -d +``` + +### NetBox authentication fails + +The public NetBox demo resets regularly. If authentication fails, create a new +NetBox API token and export `NETBOX_TOKEN` again. + +### Infrahub authentication fails + +Confirm that the token is exported in the shell where you run Infrahub Sync: + +```bash +echo "$INFRAHUB_ADDRESS" +``` + +Do not print API tokens in shared terminals, logs, or support requests. + +### Schema errors occur during generate, diff, or sync + +Make sure the NetBox schema was loaded before running `generate`, `diff`, or +`sync`: + +```bash +infrahubctl schema load ~/repos/infrahub/models/examples/netbox/netbox.yml +``` + +The schema load may print `default_filter is deprecated` notices. Those notices +are non-fatal if the command says the schema loaded successfully. + +### Diff fails with `ObjectAlreadyExists` + +The public NetBox demo data is mutable and can contain duplicate tenant names, +circuit IDs, or IP addresses. Make sure you ran the normalization step before +`infrahub-sync generate`. Then rerun: + +```bash +infrahub-sync generate --name from-netbox --directory sync-projects +infrahub-sync diff --name from-netbox --directory sync-projects +``` + +If the error identifies a different duplicate field, inspect the related mapping +in `config.yml`, add a targeted `transforms` expression, regenerate, and diff +again. + +### A second diff after sync fails with `PeerIdentifierError` + +After data is synced, a follow-up `diff` may fail while reading back some +route-target relationships from Infrahub. The initial `sync` can still complete +successfully; verify imported data with `infrahubctl object get` or in the +Infrahub UI. + +### NetBox adapter import fails + +Install the NetBox SDK in the same virtual environment: + +```bash +pip install pynetbox +``` + +## Next steps + +Now that you have completed a first sync, continue with these pages: + +- [Create a sync project](../creating-a-sync-project.mdx) to learn the structure + of `config.yml` in more detail. +- [Run a sync](../running-a-sync.mdx) to learn more about `list`, `generate`, + `diff`, and `sync`. +- [NetBox adapter](../adapters/netbox.mdx) to review adapter requirements, + direction, and configuration. +- [Schema mapping reference](../reference/schema-mapping.mdx) to learn more + about identifiers, references, filters, and transforms. diff --git a/docs/sidebars.ts b/docs/sidebars.ts index 15499d2..ea07342 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -12,6 +12,13 @@ const sidebars: SidebarsConfig = { 'running-a-sync', ], }, + { + type: 'category', + label: 'Tutorials', + items: [ + 'tutorials/netbox-demo-to-infrahub', + ], + }, { type: 'category', label: 'Guides', From f22e9d383ae80937e800d76c2eae665cc43252c5 Mon Sep 17 00:00:00 2001 From: Blake Ellis Date: Mon, 22 Jun 2026 09:44:43 -0400 Subject: [PATCH 02/16] Update NetBox demo tutorial Refreshes the tutorial after local validation: pins commands to the published 2.0.0 package, documents demo token and startup behavior, aligns verification with current public NetBox data, and keeps the order guidance focused on auto-derived write ordering. Also fixes the NetBox example config serial_number Transformation expression by removing a stray trailing brace and corrects the dmi01 comment so reviewers can inspect that config change directly. Generated with Codex. --- .../tutorials/netbox-demo-to-infrahub.mdx | 133 ++++++++++++------ examples/netbox_to_infrahub/config.yml | 4 +- 2 files changed, 93 insertions(+), 44 deletions(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index b055a96..60a9208 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -11,7 +11,7 @@ By the end of this tutorial, you will know how to: - run Infrahub in a Docker container; - load the example NetBox schema into Infrahub; - create a NetBox → Infrahub sync project; -- use `transforms` to reshape source values before import; +- use Transformation rules to reshape source values before import; - use `filters` to control which source objects are synchronized; - review the planned changes with `infrahub-sync diff`; and - synchronize NetBox objects into Infrahub. @@ -40,6 +40,9 @@ Specific requirements are: - [curl](https://curl.se/) - A web browser +The local Infrahub stack runs several containers. Configure Docker with at least +8 GB of memory before starting the tutorial. + ## Prepare a local workspace Create a directory for this tutorial. We will install Infrahub Sync and store the @@ -52,12 +55,14 @@ cd ~/repos/demo-infrahub-sync ## Install Infrahub Sync -Create a Python virtual environment and install Infrahub Sync from PyPI. +Create a Python virtual environment and install Infrahub Sync from PyPI. This +tutorial pins Infrahub Sync and the example configuration to the same release so +the commands and configuration comments stay aligned. ```bash python3 -m venv .venv source .venv/bin/activate -pip install infrahub-sync pynetbox +pip install "infrahub-sync==2.0.0" pynetbox ``` `infrahub-sync` installs the Infrahub SDK, including the `infrahubctl` command. @@ -75,16 +80,25 @@ For more about supported adapters and their Python requirements, see the ## Start Infrahub locally -Clone Infrahub and start it with Docker Compose. +Clone Infrahub and start it with Docker Compose. The token below is a well-known +local demo value. Do not use it for an internet-facing or shared Infrahub +instance. ```bash cd ~/repos git clone https://github.com/opsmill/infrahub.git cd infrahub +export INFRAHUB_INITIAL_ADMIN_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" docker compose up -d ``` -Wait for the containers to start and then open +Wait until the API answers: + +```bash +until curl -s -f -o /dev/null http://localhost:8000/api/config; do sleep 2; done +``` + +Then open [the local Infrahub web interface](http://localhost:8000). Log in with the default local credentials: @@ -92,27 +106,22 @@ Log in with the default local credentials: - Username: `admin` - Password: `infrahub` -## Create an Infrahub API token +## Configure the Infrahub API token Infrahub Sync needs an API token to write data into Infrahub. -1. If the header shows **Log in anonymous**, open it and log in as `admin`. -2. In the Infrahub web interface, click the user menu. -3. Open **Account settings**. -4. Open the **Tokens** tab. -5. Click **Add account token**. -6. Enter a token name, such as `infrahub-sync tutorial`. -7. Click **Save**. -8. Copy the generated token before you confirm the dialog. Infrahub shows this - token only once. - -Export the token in the shell where you will run `infrahub-sync`: +Export the local demo token in the shell where you will run `infrahub-sync`: ```bash export INFRAHUB_ADDRESS="http://localhost:8000" -export INFRAHUB_API_TOKEN="" +export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" ``` +`INFRAHUB_INITIAL_ADMIN_TOKEN` is read when Infrahub initializes a fresh +database. If you already started this local stack without that variable, create +an account token in the Infrahub web interface and export that token value +instead. + Do not print or commit API tokens. Keep them in your shell environment or secret manager. @@ -130,7 +139,8 @@ infrahubctl schema load ~/repos/infrahub/models/examples/netbox/netbox.yml ``` Refresh [the local Infrahub web interface](http://localhost:8000). You should -see additional schema objects in the left navigation. +see additional schema objects in the left navigation. You can also open the +schema view in the UI to explore the kinds and relationships that were loaded. :::info @@ -150,10 +160,10 @@ Create a token in the public NetBox demo instance so Infrahub Sync can read data from NetBox. 1. Open the [public NetBox demo](https://demo.netbox.dev). -2. Log in using the credentials shown on the NetBox demo site. +2. Log in with username `admin` and password `admin`. 3. Open your user profile. 4. Create an API token. -5. Copy the generated token. +5. Copy the complete generated token value. Export the token locally: @@ -162,14 +172,14 @@ export NETBOX_URL="https://demo.netbox.dev" export NETBOX_TOKEN="" ``` -Export only the token value. Do not include an authorization prefix such as -`Bearer`. +Export the complete `nbt_...` token value. Do not include an authorization +prefix such as `Bearer`, and do not copy only the short **Key** field. ## Create the sync project A sync project is a directory containing a `config.yml` file. The configuration -names the source and destination adapters, defines which objects move first, and -maps source fields to destination fields. +names the source and destination adapters, maps source fields to destination +fields, and includes references that let Infrahub Sync compute write order. Create a sync project directory and download the example NetBox to Infrahub configuration. @@ -178,7 +188,7 @@ configuration. cd ~/repos/demo-infrahub-sync mkdir -p sync-projects/netbox-demo curl -L \ - https://raw.githubusercontent.com/opsmill/infrahub-sync/refs/heads/main/examples/netbox_to_infrahub/config.yml \ + https://raw.githubusercontent.com/opsmill/infrahub-sync/refs/tags/2.0.0/examples/netbox_to_infrahub/config.yml \ -o sync-projects/netbox-demo/config.yml ``` @@ -191,12 +201,20 @@ keys: - `name` identifies the sync project. - `source` configures the NetBox adapter. - `destination` configures the Infrahub adapter. -- `order` defines the sequence for writing related objects. - `schema_mapping` defines how NetBox API resources become Infrahub objects. +In Infrahub Sync 2.0 and later, `order:` is usually omitted. The engine derives +write order from `reference:` fields in `schema_mapping`. Use an explicit +`order:` list only when you need to override the computed order. + The configuration includes default endpoint values, but the environment variables exported above take precedence for tokens and URLs. +The example configuration comments also mention `apply`, `--run-id`, and +`--parallel`. This tutorial uses the beginner path: preview with `diff`, then +write with `sync --diff`. See [Run a sync](../running-a-sync.mdx) for cached +plans and parallel execution. + For a fuller explanation of this file, see [Create a sync project](../creating-a-sync-project.mdx) and the [schema mapping reference](../reference/schema-mapping.mdx). @@ -213,13 +231,19 @@ mapping. It can also condition data as it moves between systems. We will use two configuration features: -- `transforms` change a source value before Infrahub Sync uses it. For example, - a tenant called `Engineering` can become `Engineering (engineering)` so it is - unique and easier to trace back to the NetBox slug. +- Entries under `transforms` change a source value before Infrahub Sync uses it. + For example, a tenant called `Engineering` can become + `Engineering (engineering)` so it is unique and easier to trace back to the + NetBox slug. - `filters` exclude source objects that should not be imported in this tutorial. For example, we will ignore IP addresses already assigned to another NetBox object so the first import path remains predictable. +The example schema maps NetBox tenants, manufacturers, and circuit providers to +the same Infrahub kind, `OrganizationGeneric`. The normalization rules add source +context to names that might otherwise collide inside that shared destination +kind. + These adjustments do not modify NetBox. They only change how this sync project prepares data before writing it to Infrahub. @@ -274,7 +298,7 @@ and creates Python code from it. ### Optional: inspect the normalization manually Open `sync-projects/netbox-demo/config.yml` again and search for -`tenancy.tenants`. You should see a transform like this: +`tenancy.tenants`. You should see an entry like this: ```yaml transforms: @@ -299,7 +323,7 @@ path. If a future run fails because the public demo contains another duplicate value, you can use the same pattern: identify the destination field used as an -identifier, add enough source context with a `transforms` expression to make it +identifier, add enough source context with a Transformation expression to make it unique, run `generate` again, and then rerun `diff`. ## List the sync project @@ -324,6 +348,9 @@ The `generate` command reads the sync configuration and the destination schema, then writes the Python code used at runtime. Run `generate` again any time you edit `config.yml`. +The command may print Pydantic warnings about `local_id` shadowing a parent +attribute. Those warnings are non-fatal. + For more about the command flow, see [Run a sync](../running-a-sync.mdx). ## Preview the changes @@ -336,7 +363,7 @@ infrahub-sync diff --name from-netbox --directory sync-projects The `diff` command loads data from NetBox and Infrahub, compares both sides, and prints the planned changes. It also writes a cached plan under -`.infrahub-sync-cache/`. +`.infrahub-sync-cache/from-netbox//plan.parquet`. Review the output before continuing. On a first run against an empty local Infrahub instance, most planned changes should be creates. Exact counts depend @@ -352,7 +379,8 @@ infrahub-sync sync --name from-netbox --directory sync-projects --diff The `--diff` option prints the diff before applying the changes. The first sync can take a few minutes because it writes the imported objects and relationships -into Infrahub. +into Infrahub. Because this configuration omits `order:`, Infrahub Sync derives +the write order from the mapping references. ## Verify the imported data @@ -364,17 +392,18 @@ the example mapping, you may see objects such as: - organizations - locations - racks -- devices -- interfaces - prefixes - IP addresses - VLANs +The example configuration also filters devices and interfaces to names that +contain `dmi01`. The public NetBox demo often has no matching devices. + You can also verify from the CLI: ```bash -infrahubctl object get InfraDevice --limit 5 infrahubctl object get OrganizationGeneric --limit 5 +infrahubctl object get InfraPrefix --limit 5 ``` If these commands return objects, the sync imported data successfully. @@ -386,8 +415,8 @@ sequence: 1. Infrahub provided the destination graph and schema. 2. NetBox provided the source data. -3. `config.yml` described the adapters, object order, field mappings, filters, - and transforms. +3. `config.yml` described the adapters, field mappings, references, filters, and + Transformations. 4. `generate` converted the configuration into runnable sync code. 5. `diff` compared the source and destination without writing changes. 6. `sync` applied the reviewed changes to Infrahub. @@ -407,6 +436,21 @@ docker compose down ## Troubleshooting +### Infrahub does not become available on first startup + +The first startup can take several minutes. `infrahub-server` may restart while +it waits for Neo4j to become ready. + +If the API never answers after a few minutes, or if you reused an old local +Infrahub stack, remove the local volumes and start again: + +```bash +cd ~/repos/infrahub +docker compose down -v +export INFRAHUB_INITIAL_ADMIN_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" +docker compose up -d +``` + ### Docker reports that port 8000 is already allocated Another container or local service is already using `localhost:8000`. Check the @@ -435,7 +479,7 @@ NetBox API token and export `NETBOX_TOKEN` again. Confirm that the token is exported in the shell where you run Infrahub Sync: ```bash -echo "$INFRAHUB_ADDRESS" +test -n "$INFRAHUB_API_TOKEN" && echo "Token is set" || echo "Token is missing" ``` Do not print API tokens in shared terminals, logs, or support requests. @@ -464,7 +508,7 @@ infrahub-sync diff --name from-netbox --directory sync-projects ``` If the error identifies a different duplicate field, inspect the related mapping -in `config.yml`, add a targeted `transforms` expression, regenerate, and diff +in `config.yml`, add a targeted Transformation expression, regenerate, and diff again. ### A second diff after sync fails with `PeerIdentifierError` @@ -486,6 +530,9 @@ pip install pynetbox Now that you have completed a first sync, continue with these pages: +- Try syncing to an Infrahub branch with `--branch netbox-import`, then review + the imported data in Infrahub before merging it to `main`. See + [Infrahub version control](https://docs.infrahub.app/topics/version-control/). - [Create a sync project](../creating-a-sync-project.mdx) to learn the structure of `config.yml` in more detail. - [Run a sync](../running-a-sync.mdx) to learn more about `list`, `generate`, @@ -493,4 +540,6 @@ Now that you have completed a first sync, continue with these pages: - [NetBox adapter](../adapters/netbox.mdx) to review adapter requirements, direction, and configuration. - [Schema mapping reference](../reference/schema-mapping.mdx) to learn more - about identifiers, references, filters, and transforms. + about identifiers, references, filters, and Transformations. +- [Infrahub schema](https://docs.infrahub.app/topics/schema/) to learn how + Infrahub models infrastructure data. diff --git a/examples/netbox_to_infrahub/config.yml b/examples/netbox_to_infrahub/config.yml index 203dc0b..0370752 100644 --- a/examples/netbox_to_infrahub/config.yml +++ b/examples/netbox_to_infrahub/config.yml @@ -241,7 +241,7 @@ schema_mapping: mapping: tags reference: BuiltinTag # Showcase Filters - # It will import the ones with dmu01 in their name *AND* without pdu + # It will import the ones with dmi01 in their name *AND* without pdu filters: - field: name operation: "contains" @@ -254,7 +254,7 @@ schema_mapping: - field: name expression: "{{ name.lower() if name else '' }}" - field: serial_number - expression: "{{ serial.lower() if serial else '' }}}" + expression: "{{ serial.lower() if serial else '' }}" # Interfaces (interfaces, rear port, front port) - name: InfraInterfaceL2L3 From 441c33a683c392c4c1d0190adee6542f9f5a1ec4 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Tue, 7 Jul 2026 14:06:21 +0200 Subject: [PATCH 03/16] feat: New netbox config file based on updated schema library schema --- examples/netbox_to_infrahub/config.yml | 701 +++++++++++++++---------- 1 file changed, 436 insertions(+), 265 deletions(-) diff --git a/examples/netbox_to_infrahub/config.yml b/examples/netbox_to_infrahub/config.yml index 0370752..a0c3855 100644 --- a/examples/netbox_to_infrahub/config.yml +++ b/examples/netbox_to_infrahub/config.yml @@ -15,25 +15,32 @@ # uv run infrahub-sync diff --name from-netbox --directory examples/ # uv run infrahub-sync apply --name from-netbox --run-id --directory examples/ # uv run infrahub-sync sync --name from-netbox --directory examples/ --parallel +# +# --------------------------------------------------------------------------- -name: from-netbox +name: netbox-demo-to-infrahub source: name: netbox settings: url: "https://demo.netbox.dev" - token: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" destination: name: infrahub settings: url: "http://localhost:8000" +# Credentials are read from the environment, not this file: +# export NETBOX_URL="https://demo.netbox.dev" +# export NETBOX_TOKEN="nbt_..." +# export INFRAHUB_ADDRESS="http://localhost:8000" +# export INFRAHUB_API_TOKEN="..." + # order: omitted — infrahub-sync auto-computes tiers from schema_mapping. # Uncomment and edit only if you need to override the computed order: # order: # - BuiltinTag -# - RoleGeneric +# - CoreStandardGroup # ... schema_mapping: @@ -47,135 +54,96 @@ schema_mapping: - name: description mapping: description - # Roles (DCIM, IPAM, Devices) - - name: RoleGeneric - mapping: dcim.device_roles - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: RoleGeneric - mapping: dcim.rack_roles - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: RoleGeneric - mapping: ipam.roles - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - # Tenancy (tenants, tenant groups) - - name: CoreStandardGroup - mapping: tenancy.tenant-groups - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: OrganizationGeneric - mapping: tenancy.tenants - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: group - mapping: group - reference: CoreStandardGroup - - # Sites (Site Groups, Region, Sites, Location) - - name: CoreStandardGroup - mapping: dcim.site-groups - identifiers: ["name"] - fields: - - name: name - mapping: name - - name: description - mapping: description - - name: LocationGeneric - mapping: dcim.regions - fields: - - name: name - mapping: slug - - name: description - mapping: name - - name: type - static: "Region" - - name: tags - mapping: tags - reference: BuiltinTag - - name: LocationGeneric + # Sites / Regions / Locations + - name: LocationSite mapping: dcim.sites + identifiers: ["name"] fields: - name: name mapping: slug - name: description mapping: name - - name: type - static: "Site" - - name: group - mapping: group - reference: CoreStandardGroup - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: tags - mapping: tags - reference: BuiltinTag - - name: LocationGeneric - mapping: dcim.locations - fields: - - name: name - mapping: slug - - name: description - mapping: name - - name: type - static: "Location" - - name: organization - mapping: tenant - reference: OrganizationGeneric + - name: status + mapping: status.value + - name: facility + mapping: facility + - name: physical_address + mapping: physical_address + - name: timezone + mapping: time_zone - name: tags mapping: tags reference: BuiltinTag + # ----------------------------------------------------------------------- + # Netbox "region" model is relatively loose and could accommodate multiple layers of nested objects (e.g. continent > country > city). + # In Infrahub you have to explicitly define this hierarchy in the schema. + # As every organization has a different way to organize this we do not provide out of the box support for that even tho we provide a couple of example in the schema library / marketplace. + # For the sake of having a first working sync we have a very basic site node defined in the schema that will be able to build upon once your hierarchy is clear. + # To fully support region you would need to: + # - Create the hierarchy in the schema + # - Uncomment the lines below and adjust to your specific data <> schema + # + # For example here region is top level (so has no parent) + # - name: LocationRegion + # mapping: dcim.regions + # identifiers: ["name"] + # fields: + # - name: name + # mapping: name + # - name: description + # mapping: description + # filters: + # - field: parent + # operation: "is_empty" + # + # For example here country are right below the region (so has depth = 1) + # - name: LocationCountry + # mapping: dcim.regions + # identifiers: ["name"] + # fields: + # - name: name + # mapping: name + # - name: description + # mapping: description + # - name: parent + # mapping: region + # reference: LocationRegion + # filters: + # - field: _depth + # operation: "==" + # value: 1 + # ----------------------------------------------------------------------- + # Racks - - name: InfraRack + - name: LocationRack mapping: dcim.racks - identifiers: ["name", "location"] + identifiers: ["name", "site"] fields: - name: name mapping: name - - name: location + - name: site mapping: site - reference: LocationGeneric + reference: LocationSite - name: height mapping: u_height - - name: serial_number - mapping: serial - - name: asset_tag - mapping: asset_tag - name: facility_id mapping: facility_id - - name: role - mapping: role - reference: RoleGeneric + - name: status + mapping: status.value - name: tags mapping: tags reference: BuiltinTag + - name: serial_number + mapping: serial + - name: asset_tag + mapping: asset_tag + # NOTE: current schema doesn't support role for racks (see related note) + # - name: role + # mapping: role.slug - # Device (manufacturer, device types, devices, interfaces) - - name: OrganizationGeneric + # Manufacturers / Platforms / Device Types + - name: OrganizationManufacturer mapping: dcim.manufacturers identifiers: ["name"] fields: @@ -183,7 +151,21 @@ schema_mapping: mapping: name - name: description mapping: description - - name: ChoiceDeviceType + - name: tags + mapping: tags + reference: BuiltinTag + + - name: DcimPlatform + mapping: dcim.platforms + identifiers: ["name"] + fields: + - name: name + mapping: name + - name: manufacturer + mapping: manufacturer + reference: OrganizationManufacturer + + - name: DcimDeviceType mapping: dcim.device-types identifiers: ["name", "manufacturer"] fields: @@ -195,69 +177,125 @@ schema_mapping: mapping: is_full_depth - name: height mapping: integer_height - - name: tags - mapping: tags - reference: BuiltinTag + - name: weight + mapping: weight + # NOTE: Netbox tracks weight_unit separately (kg/lb/g/oz); schema + # has no unit field, so mixed-unit sources will be inconsistent. - name: manufacturer mapping: manufacturer - reference: OrganizationGeneric + reference: OrganizationManufacturer transforms: - field: integer_height expression: "{{ u_height|float|round(0, 'ceil') }}" - # -> The netbox constraint is `dcim_device_unique_name_site_tenant` - # Reusing device_name + site + Organization as identifiers - # /!\ Seem like Netbox allowed device to have the same name if there is a virtual-chassis - - name: InfraDevice - identifiers: ["location", "rack", "organization", "name"] + + # --------------------------------------------------------------------- + # Netbox roles are user-defined/arbitrary, so there's no guarantee the + # netbox slug matches one of the fixed choices below — sync will fail for + # values (roles) that don't match one of the schema's dropdown choices. + # TODO: decide whether to (a) map role.slug directly per node below and + # accept failures on non-matching values, (b) extend the schema's choice + # lists to cover your netbox roles, or (c) drop role sync entirely. + # dcim.rack_roles has no destination at all anymore (Rack has no role + # field in the new schema) — not mapped below. + # --------------------------------------------------------------------- + + # Devices + # ----------------------------------------------------------------------- + # Netbox splits location across `site` (mandatory) and `rack` (optional). + # Infrahub consolidates both under a single `location` relationship that + # can point to either a LocationRack or a LocationSite. Split into two + # entries over the same dcim.devices source with complementary filters: + # racked devices use the rack, everything else falls back to the site. + # TODO: Find a better way to support this + # ----------------------------------------------------------------------- + - name: DcimDevice + identifiers: ["location", "name"] + mapping: dcim.devices + fields: + - name: name # /!\ Netbox allows duplicate device names within a virtual-chassis. + mapping: name + - name: serial + mapping: serial + - name: description + mapping: description + - name: position + mapping: position + - name: rack_face + mapping: face.value + - name: status + mapping: status.value + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug + - name: device_type + mapping: device_type + reference: DcimDeviceType + - name: platform + mapping: platform + reference: DcimPlatform + - name: location + mapping: rack + reference: LocationRack + - name: tags + mapping: tags + reference: BuiltinTag + - name: primary_address + mapping: primary_ip + reference: IpamIPAddress + filters: + - field: parent_device # Most lickely for sub-devices (e.g. stacks...) you'd have a dedicated node in Infrahub + operation: "is_empty" + - field: rack + operation: "is_not_empty" + - field: name # /!\ Netbox allows empty name for devices + operation: "is_not_empty" + + - name: DcimDevice + identifiers: ["location", "name"] mapping: dcim.devices fields: - name: name mapping: name - - name: serial_number + - name: serial mapping: serial - - name: asset_tag - mapping: asset_tag - name: description mapping: description - - name: model + - name: position + mapping: position + - name: rack_face + mapping: face.value + - name: status + mapping: status.value + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug + - name: device_type mapping: device_type - reference: ChoiceDeviceType - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: role - mapping: role - reference: RoleGeneric - # - name: status - # mapping: status - # reference: StatusGeneric + reference: DcimDeviceType + - name: platform + mapping: platform + reference: DcimPlatform - name: location mapping: site - reference: LocationGeneric - - name: rack - mapping: rack - reference: InfraRack + reference: LocationSite - name: tags mapping: tags reference: BuiltinTag - # Showcase Filters - # It will import the ones with dmi01 in their name *AND* without pdu + - name: primary_address + mapping: primary_ip + reference: IpamIPAddress filters: - - field: name - operation: "contains" - value: "dmi01" - - field: name - operation: "not contains" - value: "pdu" - # Showcase Transforms - transforms: - - field: name - expression: "{{ name.lower() if name else '' }}" - - field: serial_number - expression: "{{ serial.lower() if serial else '' }}" + - field: parent_device + operation: "is_empty" + - field: rack + operation: "is_empty" + - field: name # /!\ Netbox allows empty name for devices + operation: "is_not_empty" + + # TODO: Add support for cluster, VM, servers ... - # Interfaces (interfaces, rear port, front port) - - name: InfraInterfaceL2L3 + # Interfaces (Physical / Virtual / LAG) + - name: InterfacePhysical identifiers: ["device", "name"] mapping: dcim.interfaces fields: @@ -265,73 +303,136 @@ schema_mapping: mapping: name - name: description mapping: description - - name: interface_type - mapping: type.label - - name: l2_mode - mapping: mode.label - name: mac_address mapping: mac_address - # - name: ip_addresses - # mapping: ip_addresses - # reference: InfraIPAddress - - name: mgmt_only - mapping: mgmt_only + - name: l2_mode + mapping: mode.value + - name: mtu + mapping: mtu + - name: ip_addresses + mapping: ip_addresses + reference: IpamIPAddress - name: untagged_vlan mapping: untagged_vlan - reference: InfraVLAN + reference: IpamVLAN - name: tagged_vlan mapping: tagged_vlans - reference: InfraVLAN + reference: IpamVLAN - name: device mapping: device - reference: InfraDevice - - name: tags - mapping: tags - reference: BuiltinTag - # - name: status - # mapping: status - # reference: StatusGeneric - filters: - - field: device.name - operation: "contains" - value: "dmi01" - - field: device.name + reference: DcimDevice + - name: bundle + mapping: lag + reference: InterfaceLag + # TODO: many fields are not implemented in the schema (mgmt_only, type, duplex ...) + filters: # Filters to only get the "physical interfaces" + # NOTE: Netbox's `type` is a nested {value, label} object, not a plain + # string — filtering on bare `type` compares against the dict itself + # (dict membership checks keys, dict == string is always False), which + # silently let every interface through here and matched none below. + - field: type.value + operation: "not contains" + value: "virtual" + - field: type.value operation: "not contains" - value: "pdu" + value: "lag" + # NOTE: must mirror the DcimDevice-level exclusions — otherwise an + # interface can reference a device that was filtered out of the sync, + # and the `device` reference lookup fails ("Unable to locate the node"). + # Only `name` can be checked here: Netbox's interface payload embeds a + # *brief* device object (id/url/display/name/description only), which + # never carries `parent_device` — so we can't replicate the + # `parent_device is_empty` exclusion at this level. Not an issue today + # (no devices in this dataset have parent_device set), but if you add + # sub-devices later, their interfaces will need filtering out some + # other way (e.g. a follow-up id-based exclusion list). + - field: device.name + operation: "is_not_empty" - # Circuits (Provider, Provider Network, Circuits Types, Circuits) - - name: OrganizationGeneric - mapping: circuits.providers - identifiers: ["name"] + - name: InterfaceVirtual + identifiers: ["device", "name"] + mapping: dcim.interfaces fields: - name: name mapping: name - name: description mapping: description - - name: InfraProviderNetwork - mapping: circuits.provider-networks - identifiers: ["name"] + - name: mac_address + mapping: mac_address + - name: l2_mode + mapping: mode.value + - name: ip_addresses + mapping: ip_addresses + reference: IpamIPAddress + - name: untagged_vlan + mapping: untagged_vlan + reference: IpamVLAN + - name: tagged_vlan + mapping: tagged_vlans + reference: IpamVLAN + - name: device + mapping: device + reference: DcimDevice + # TODO: many fields are not implemented in the schema (mgmt_only, type, duplex ...) + filters: # Filters to only get the "virtual interfaces" + - field: type.value + operation: "==" + value: "virtual" + # NOTE: must mirror the DcimDevice-level `name` exclusion — see + # InterfacePhysical for why `parent_device` can't be checked here too. + - field: device.name + operation: "is_not_empty" + + - name: InterfaceLag + identifiers: ["device", "name"] + mapping: dcim.interfaces fields: - name: name mapping: name - - name: vendor_id - mapping: service_id - - name: provider - mapping: provider - reference: OrganizationGeneric + - name: bundle_number + mapping: bundle_number + # Capture the number at the end of the name (e.g. Po1 -> 1) - name: description mapping: description - # - name: status - # mapping: status - # reference: StatusGeneric - - name: tags - mapping: tags - reference: BuiltinTag - - name: provider - mapping: provider - reference: OrganizationGeneric - - name: ChoiceCircuitType - mapping: circuits.circuit-types + - name: mac_address + mapping: mac_address + - name: l2_mode + mapping: mode.value + - name: ip_addresses + mapping: ip_addresses + reference: IpamIPAddress + - name: untagged_vlan + mapping: untagged_vlan + reference: IpamVLAN + - name: tagged_vlan + mapping: tagged_vlans + reference: IpamVLAN + - name: device + mapping: device + reference: DcimDevice + # TODO: many fields are not implemented in the schema (mgmt_only, type, duplex ...) + # TODO: many fields are not available in Netbox + # - name: lacp_rate + # - name: lacp_mode + transforms: + - field: bundle_number + expression: |- + {% set suffix = name[(name.rstrip('0123456789'))|length:] %} + {{ suffix | int if suffix else None }} + filters: # Filters to only get the "lag interfaces" + - field: type.value + operation: "==" + value: "lag" + # NOTE: must mirror the DcimDevice-level `name` exclusion — see + # InterfacePhysical for why `parent_device` can't be checked here too. + - field: device.name + operation: "is_not_empty" + + # TODO: Many other types of interface to handle + + # Provider + - name: OrganizationProvider + mapping: circuits.providers identifiers: ["name"] fields: - name: name @@ -341,44 +442,38 @@ schema_mapping: - name: tags mapping: tags reference: BuiltinTag - - name: InfraCircuit + + # Circuit + - name: DcimCircuit mapping: circuits.circuits identifiers: ["circuit_id"] fields: - name: circuit_id mapping: cid - - name: vendor_id - mapping: cid - - name: provider - mapping: provider - reference: OrganizationGeneric - name: description mapping: description - # - name: status - # mapping: status - # reference: StatusGeneric - - name: type - mapping: type.name + - name: commit_rate + mapping: commit_rate + - name: status + mapping: status.value + # NOTE: similar to roles, circuit type have to be defined in the schema + # - name: circuit_type + # mapping: type.slug - name: provider mapping: provider - reference: OrganizationGeneric - - name: tags - mapping: tags - reference: BuiltinTag + reference: OrganizationProvider -# IPAM (VRF, VLANs Groups, VLANs, Prefixes, IPs) - - name: InfraRouteTarget + # IPAM (VRF, Route Targets, VLAN Groups, VLANs, Prefixes, IPs) + - name: IpamRouteTarget mapping: ipam.route-targets - identifiers: ["name", "organization"] + identifiers: ["name"] fields: - name: name mapping: name - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: InfraVRF + + - name: IpamVRF mapping: ipam.vrfs identifiers: ["name"] fields: @@ -386,18 +481,18 @@ schema_mapping: mapping: name - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - name: vrf_rd mapping: rd + - name: enforce_unique + mapping: enforce_unique - name: import_rt mapping: import_targets - reference: InfraRouteTarget + reference: IpamRouteTarget - name: export_rt mapping: export_targets - reference: InfraRouteTarget - - name: CoreStandardGroup + reference: IpamRouteTarget + + - name: IpamVLANGroup mapping: ipam.vlan-groups identifiers: ["name"] fields: @@ -405,26 +500,42 @@ schema_mapping: mapping: name - name: description mapping: description - - name: InfraVLAN + # TODO: `scope` (peer IpamVLANGroupScope: Region/Site/Rack/etc.) needs + # resolving Netbox's polymorphic scope_type/scope_id — not mapped yet. + # - name: scope + # mapping: scope + # reference: LocationSite + + - name: IpamVLAN mapping: ipam.vlans - identifiers: ["name", "vlan_id", "location", "vlan_group"] + identifiers: ["name", "vlan_id", "vlan_group"] fields: - name: name mapping: name - name: vlan_id mapping: vid - - name: organization - mapping: tenant - reference: OrganizationGeneric - name: description mapping: description - - name: location - mapping: site - reference: LocationGeneric + - name: status + mapping: status.value + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug - name: vlan_group mapping: group - reference: CoreStandardGroup - - name: InfraPrefix + reference: IpamVLANGroup + # vlan_group is required (optional: false) in the schema, but Netbox + # VLANs can have no group at all — skip those rather than fail the sync. + filters: + - field: group + operation: "is_not_empty" + + # TODO: add support for provider-network + # TODO: add support for tenant + # TODO: add support for device types -> object templates + # TODO: add support for DcimCircuitEndpoint + + - name: IpamPrefix mapping: ipam.prefixes identifiers: ["prefix", "vrf"] fields: @@ -432,26 +543,31 @@ schema_mapping: mapping: prefix - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - - name: role - mapping: role - reference: RoleGeneric + - name: status + mapping: resolved_status + - name: member_type + mapping: resolved_member_type + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.slug - name: vrf mapping: vrf - reference: InfraVRF - - name: location - mapping: site - reference: LocationGeneric - # - name: status - # mapping: status - # reference: StatusGeneric - # - name: vlan - # mapping: vlan.name - # - name: location - # mapping: location.slug - - name: InfraIPAddress + reference: IpamVRF + - name: vlan + mapping: vlan + reference: IpamVLAN + # TODO: `scope` (peer IpamPrefixScope: Region/Site/Rack/etc.) needs + # resolving Netbox's polymorphic scope_type/scope_id — not mapped yet. + # - name: scope + # mapping: scope + # reference: LocationSite + transforms: # These transforms accomodate different way of dealing with container + - field: resolved_status + expression: "{{ 'active' if status.value == 'container' else status.value }}" + - field: resolved_member_type + expression: "{{ 'prefix' if status.value == 'container' else 'address' }}" + + - name: IpamIPAddress mapping: ipam.ip-addresses identifiers: ["address", "vrf"] fields: @@ -459,15 +575,70 @@ schema_mapping: mapping: address - name: description mapping: description - - name: organization - mapping: tenant - reference: OrganizationGeneric - # Not the same as ipam.role - # - name: role - # mapping: role.value - # - name: status - # mapping: status - # reference: StatusGeneric + - name: status + mapping: status.value + # NOTE: Netbox also has dhcp/slaac statuses with no schema equivalent. - name: vrf mapping: vrf - reference: InfraVRF + reference: IpamVRF + # NOTE: This is not enforced on netbox side, might create validation issues + # - name: fqdn + # mapping: dns_name + # NOTE: schema changes are required to support role sync (see related note) + # - name: role + # mapping: role.value + + # IPAM — Aggregates / RIRs (new in schema, no prior mapping existed) + - name: OrganizationRIR + mapping: ipam.rirs + identifiers: ["name"] + fields: + - name: name + mapping: name + - name: description + mapping: description + - name: is_private + mapping: is_private + - name: tags + mapping: tags + reference: BuiltinTag + + - name: IpamAggregate + mapping: ipam.aggregates + identifiers: ["prefix"] + fields: + - name: prefix + mapping: prefix + - name: description + mapping: description + - name: date_added + mapping: date_added + - name: rir + mapping: rir + reference: OrganizationRIR + + # ----------------------------------------------------------------------- + # New schema extensions with NO mapping below — none of these existed in + # the previous config, and each needs a real design decision rather than + # a guess: + # + # - ClusterGeneric / ClusterHosting (schemas/extensions/cluster, + # hosting_cluster): Netbox source would be virtualization.clusters, + # but `technology` is a fixed Dropdown (vmware/kvm/hyper_v/xen/ + # nutanix/openstack/kubernetes/public_cloud) while Netbox's + # cluster.type is a free-form ClusterType object — needs a mapping + # table, not a direct field copy. + # - ComputePhysicalServer / VirtualizationVirtualMachine + # (schemas/extensions/compute): virtualization.virtual-machines would + # feed VirtualizationVirtualMachine, but `host` is polymorphic + # (ClusterHosting or ComputePhysicalServer) and memory units differ + # (Netbox MB vs schema's documented GB) — needs a transform. + # ComputePhysicalServer has no obvious Netbox source at all (would + # require picking out dcim.devices by role, which is arbitrary). + # - DcimCable (schemas/extensions/cable): Netbox dcim.cables uses + # polymorphic multi-sided terminations (cable-terminations endpoint) + # that don't map cleanly onto DcimCable's simpler two-endpoint + # Connector model — needs real design work. + # - IpamSVLAN / IpamCVLAN (schemas/extensions/qinq): Infrahub-only + # concept (802.1ad Q-in-Q split), no equivalent Netbox source field. + # ----------------------------------------------------------------------- From 1f68de7c50fe7504889cefed3b00da368065c698 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Tue, 7 Jul 2026 14:48:23 +0200 Subject: [PATCH 04/16] fix: improve+cleanup config example --- examples/netbox_to_infrahub/config.yml | 37 +++++++++++++------------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/examples/netbox_to_infrahub/config.yml b/examples/netbox_to_infrahub/config.yml index a0c3855..a4ce94a 100644 --- a/examples/netbox_to_infrahub/config.yml +++ b/examples/netbox_to_infrahub/config.yml @@ -1,5 +1,5 @@ --- -# from-netbox — demonstrates the two newest features of infrahub-sync: +# netbox-demo-to-infrahub — demonstrates the two newest features of infrahub-sync: # # 1. Auto-tiered write order. The `order:` list has been removed; the engine # derives tiers from `schema_mapping[].fields[].reference` and groups @@ -7,14 +7,14 @@ # enforce a hard barrier between tiers. # # 2. Parquet sidecar cache. Every `diff` writes per-resource snapshots and -# a `plan.parquet` under `.infrahub-sync-cache/from-netbox//`. +# a `plan.parquet` under `.infrahub-sync-cache/netbox-demo-to-infrahub//`. # The run_id is logged at INFO; pass it to `apply` to re-dispatch the # cached plan without re-extracting the source. # # Usage: -# uv run infrahub-sync diff --name from-netbox --directory examples/ -# uv run infrahub-sync apply --name from-netbox --run-id --directory examples/ -# uv run infrahub-sync sync --name from-netbox --directory examples/ --parallel +# uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects +# uv run infrahub-sync apply --name netbox-demo-to-infrahub --run-id --directory sync-projects +# uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --parallel # # --------------------------------------------------------------------------- @@ -78,13 +78,13 @@ schema_mapping: # ----------------------------------------------------------------------- # Netbox "region" model is relatively loose and could accommodate multiple layers of nested objects (e.g. continent > country > city). # In Infrahub you have to explicitly define this hierarchy in the schema. - # As every organization has a different way to organize this we do not provide out of the box support for that even tho we provide a couple of example in the schema library / marketplace. - # For the sake of having a first working sync we have a very basic site node defined in the schema that will be able to build upon once your hierarchy is clear. + # As every organization organizes this differently, we don't provide out-of-the-box support for it, even though we provide a couple of examples in the schema library / marketplace. + # For the sake of having a first working sync, we've defined a very basic site node in the schema that you can build upon once your hierarchy is clear. # To fully support region you would need to: # - Create the hierarchy in the schema # - Uncomment the lines below and adjust to your specific data <> schema # - # For example here region is top level (so has no parent) + # For example, here the region is top-level (it has no parent) # - name: LocationRegion # mapping: dcim.regions # identifiers: ["name"] @@ -97,7 +97,7 @@ schema_mapping: # - field: parent # operation: "is_empty" # - # For example here country are right below the region (so has depth = 1) + # For example, here countries sit directly below the region (depth = 1) # - name: LocationCountry # mapping: dcim.regions # identifiers: ["name"] @@ -190,11 +190,11 @@ schema_mapping: # --------------------------------------------------------------------- # Netbox roles are user-defined/arbitrary, so there's no guarantee the - # netbox slug matches one of the fixed choices below — sync will fail for + # Netbox slug matches one of the fixed choices below — sync will fail for # values (roles) that don't match one of the schema's dropdown choices. # TODO: decide whether to (a) map role.slug directly per node below and # accept failures on non-matching values, (b) extend the schema's choice - # lists to cover your netbox roles, or (c) drop role sync entirely. + # lists to cover your Netbox roles, or (c) drop role sync entirely. # dcim.rack_roles has no destination at all anymore (Rack has no role # field in the new schema) — not mapped below. # --------------------------------------------------------------------- @@ -243,7 +243,7 @@ schema_mapping: mapping: primary_ip reference: IpamIPAddress filters: - - field: parent_device # Most lickely for sub-devices (e.g. stacks...) you'd have a dedicated node in Infrahub + - field: parent_device # For sub-devices (e.g. stacks), you'd most likely have a dedicated node in Infrahub operation: "is_empty" - field: rack operation: "is_not_empty" @@ -456,7 +456,7 @@ schema_mapping: mapping: commit_rate - name: status mapping: status.value - # NOTE: similar to roles, circuit type have to be defined in the schema + # NOTE: similar to roles, circuit types have to be defined in the schema # - name: circuit_type # mapping: type.slug - name: provider @@ -553,15 +553,16 @@ schema_mapping: - name: vrf mapping: vrf reference: IpamVRF - - name: vlan - mapping: vlan - reference: IpamVLAN + # TODO: Make sure no vlan is skipped due to missing group then enable this + # - name: vlan + # mapping: vlan + # reference: IpamVLAN # TODO: `scope` (peer IpamPrefixScope: Region/Site/Rack/etc.) needs # resolving Netbox's polymorphic scope_type/scope_id — not mapped yet. # - name: scope # mapping: scope # reference: LocationSite - transforms: # These transforms accomodate different way of dealing with container + transforms: # These transforms accommodate different ways of dealing with containers - field: resolved_status expression: "{{ 'active' if status.value == 'container' else status.value }}" - field: resolved_member_type @@ -581,7 +582,7 @@ schema_mapping: - name: vrf mapping: vrf reference: IpamVRF - # NOTE: This is not enforced on netbox side, might create validation issues + # NOTE: This isn't enforced on the Netbox side, and might create validation issues # - name: fqdn # mapping: dns_name # NOTE: schema changes are required to support role sync (see related note) From 1ced5458f8dc6647c978847dccb944cdb3d94f2b Mon Sep 17 00:00:00 2001 From: Baptiste Date: Tue, 7 Jul 2026 14:52:12 +0200 Subject: [PATCH 05/16] docs: new version of the tutorial --- .../tutorials/netbox-demo-to-infrahub-v2.mdx | 317 ++++++++++++++++++ 1 file changed, 317 insertions(+) create mode 100644 docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx new file mode 100644 index 0000000..af4bcc0 --- /dev/null +++ b/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx @@ -0,0 +1,317 @@ +--- +title: Sync NetBox to Infrahub +description: Set up a local Infrahub instance, load a data model, and synchronize infrastructure data from the public NetBox demo. +--- + +In this tutorial we will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) into Infrahub. We will start from scratch, installing and configuring everything we need along the way. This tutorial covers the fundamental work needed to get started; more advanced topics will be covered in subsequent guides. + +By the end of this tutorial, you will know how to: + +- install Infrahub Sync +- run Infrahub in a Docker container +- load the example NetBox schema into Infrahub +- create a NetBox → Infrahub sync project +- synchronize NetBox objects into Infrahub + +**Time**: ~30 minutes + +**What you will build**: A running Infrahub instance with a NetBox-derived data model, populated with locations, devices, interfaces, IPAM, and organization data synchronized from the public NetBox demo. + +:::note +Running Infrahub locally with Docker is the most convenient setup for development and experimentation, but it requires a machine with adequate resources. See [hardware requirements](../deploy-manage/install-configure/hardware-requirements) for details. If you prefer a lighter option, try the [Infrahub Sandbox](https://sandbox.infrahub.app/) or the [Getting Started Lab](https://opsmill.instruqt.com/pages/labs) instead. +::: + +**Prerequisites**: + +- [Docker](https://docs.docker.com/get-docker/) installed and running (Docker Desktop or OrbStack) +- [uv](https://docs.astral.sh/uv/getting-started/installation/) (Python package manager) +- [Python 3.10+](https://www.python.org/downloads/) + +:::warning Public demo data + +The NetBox demo instance is public and resets regularly. Object counts, names, and sample data may differ from the examples in this tutorial. + +::: + +## Create a project + +[Copier](https://copier.readthedocs.io/) is a project scaffolding tool. Use it to create a new Infrahub project from the official template, which includes the standard file structure, task definitions, and a `schemas/` folder: + +1. Run the following command to create a new project directory: + +```bash +uv tool run --from 'copier' copier copy https://github.com/opsmill/infrahub-template infrahub-automation +``` + +If you are unsure about the options, accept the default values for all prompts by pressing **Y**. + +2. Navigate to the project directory: + +```bash +cd infrahub-automation +``` + +3. Open the project in your IDE. If you have Visual Studio Code installed, you can run: + +```bash +code . +``` + +:::success Verification +Run `ls` in the project directory. You should see files including `pyproject.toml`, `tasks.py`, and a `schemas/` folder. +::: + +## Start Infrahub + +The project template includes [Invoke](https://www.pyinvoke.org/) tasks that wrap Docker Compose commands. + +1. Start all Infrahub services with a single command: + +```bash +uv run invoke start +``` + +The first run takes a few minutes while Docker downloads the container images. + +2. Open your browser and go to [http://localhost:8000](http://localhost:8000). + +3. Log in from the bottom-left corner using the default credentials: + + - Username: `admin` + - Password: `infrahub` + +:::success Verification +You should see the Infrahub web interface with a navigation menu on the left side. +::: + +## Install Infrahub Sync + +1. Install the Python dependencies with uv: + +```bash +uv add "infrahub-sync==2.0.0" pynetbox +``` + +This installs the `infrahub-sync` command. `pynetbox` is required by the NetBox adapter. + +2. Verify infrahub-sync command is available: + +```bash +uv run infrahub-sync --help +``` + +For more about supported adapters and their Python requirements, see the [NetBox adapter documentation](../adapters/netbox.mdx). + +## Load a schema into Infrahub + +Infrahub stores data according to its schema. Before we can import any data, we need Infrahub to know the kinds of objects that the sync will create. + +We provide a collection of schemas that support basic DCIM and IPAM features. This is not an exact one-to-one mapping with NetBox, but that's not the goal either. + +:::info + +The example schema is a practical starting point. It does not try to reproduce NetBox one-to-one. Infrahub gives you a flexible graph model, so a real migration can preserve the parts of NetBox that matter while adapting the model to your future workflows. + +::: + +This collection of schemas is still a work in progress; eventually it will be available through the marketplace. For now, download the files directly: + +```bash +BRANCH="bgi-schema-library-v2" +DEST="schemas" +BASE_URL="https://raw.githubusercontent.com/opsmill/schema-library/${BRANCH}" + +FILES=( + "base/dcim.yml" + "base/location.yml" + "base/ipam.yml" + "base/organization.yml" + "extensions/aggregate/aggregate.yml" + "extensions/cable/cable.yml" + "extensions/circuit/circuit.yml" + "extensions/compute/compute.yml" + "extensions/cluster/cluster.yml" + "extensions/hosting_cluster/hosting_cluster.yml" + "extensions/lag/lag.yml" + "extensions/location_minimal/location_minimal.yml" + "extensions/vlan/vlan.yml" + "extensions/qinq/qinq.yml" + "extensions/rack/rack.yml" + "extensions/vrf/vrf.yml" +) + +for f in "${FILES[@]}"; do + curl -sSL --create-dirs -o "${DEST}/${f}" "${BASE_URL}/${f}" +done +``` + +Export the local Infrahub address and API token. Infrahub Sync will need them later to authenticate against your instance: + +```bash +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" +``` + +This is the default admin token that the project's Docker Compose stack seeds automatically, and it matches the default already configured in `infrahubctl.toml`. Do not reuse it for an internet-facing or shared Infrahub instance. + +Then load the schema into Infrahub using the project's `load-schema` task: + +```bash +uv run invoke load-schema +``` + +Refresh [the local Infrahub web interface](http://localhost:8000). You should see additional schema objects in the left navigation. You can also open the schema view in the UI to explore the kinds and relationships that were loaded. + +## Create a NetBox API token + +Create a token in the public NetBox demo instance so Infrahub Sync can read data from NetBox. + +1. Open the [public NetBox demo](https://demo.netbox.dev). +2. Log in with username `admin` and password `admin`. +3. Open your user profile. +4. Create an API token. +5. Copy the complete generated token value. + +Export the token locally: + +```bash +export NETBOX_URL="https://demo.netbox.dev" +export NETBOX_TOKEN="nbt_..." +``` + +Export the complete `nbt_...` token value. Do not include an authorization prefix such as `Bearer`, and do not copy only the short **Key** field. + +## Create the sync project + +A sync project is a directory containing a `config.yml` file. The configuration names the source and destination adapters, maps source fields to destination fields, and includes references that let Infrahub Sync compute write order. + +From your project directory (`infrahub-automation`), create a sync project directory and download the example NetBox to Infrahub configuration. + +```bash +mkdir -p sync-projects/netbox-demo +curl -L \ + https://raw.githubusercontent.com/opsmill/infrahub-sync/441c33a683c392c4c1d0190adee6542f9f5a1ec4/examples/netbox_to_infrahub/config.yml \ + -o sync-projects/netbox-demo/config.yml +``` + +The downloaded configuration is named `netbox-demo-to-infrahub`. It maps selected NetBox objects to the example Infrahub NetBox schema. + +Open `sync-projects/netbox-demo/config.yml` in an editor and scan the top-level keys: + +- `name` identifies the sync project. +- `source` configures the NetBox adapter. +- `destination` configures the Infrahub adapter. +- `schema_mapping` defines how NetBox API resources become Infrahub objects. + +The configuration includes default endpoint values, but the environment variables exported above take precedence for tokens and URLs. + +The example configuration comments also mention `apply`, `--run-id`, and `--parallel`. This tutorial uses the beginner path: preview with `diff`, then write with `sync --diff`. See [Run a sync](../running-a-sync.mdx) for cached plans and parallel execution. + +For a fuller explanation of this file, see [Create a sync project](../creating-a-sync-project.mdx) and the [schema mapping reference](../reference/schema-mapping.mdx). + +## Generate the sync code + +Generate the Python models and adapter code for this sync project. + +```bash +uv run infrahub-sync generate --name netbox-demo-to-infrahub --directory sync-projects +``` + +The `generate` command reads the sync configuration and the destination schema, then writes the Python code used at runtime. Run `generate` again any time you edit `config.yml`. + +For more about the command flow, see [Run a sync](../running-a-sync.mdx). + +## Preview the changes + +Run a dry-run diff before writing data to Infrahub. + +```bash +uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects +``` + +The `diff` command loads data from NetBox and Infrahub, compares both sides, and prints the planned changes. It also writes a cached plan under `.infrahub-sync-cache/netbox-demo-to-infrahub//plan.parquet`. + +Review the output before continuing. On a first run against an empty local Infrahub instance, most planned changes should be creates. Exact counts depend on the current public NetBox demo data. + +## Sync the data + +After reviewing the diff, run the sync. + +```bash +uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --diff +``` + +The `--diff` option prints the diff before applying the changes. The first sync can take a few minutes because it writes the imported objects and relationships into Infrahub. Because this configuration omits `order:`, Infrahub Sync derives the write order from the mapping references. + +## Verify the imported data + +Open [the local Infrahub web interface](http://localhost:8000) and check for imported objects from the NetBox demo. Depending on the current demo data and the example mapping, you may see objects such as: + +- tags and organizations (manufacturers, providers, RIRs) +- sites and racks +- devices and interfaces +- VRFs, VLANs, and VLAN groups +- prefixes, IP addresses, and aggregates +- circuits + +You can also verify from the CLI: + +```bash +uv run infrahubctl object get LocationSite --limit 5 +uv run infrahubctl object get IpamPrefix --limit 5 +``` + +If these commands return objects, the sync imported data successfully. + +## What happened + +You used Infrahub Sync to move data from NetBox into Infrahub in a controlled sequence: + +1. Infrahub provided the destination graph and schema. +2. NetBox provided the source data. +3. `config.yml` described the adapters, field mappings, references, filters, and Transformations. +4. `generate` converted the configuration into runnable sync code. +5. `diff` compared the source and destination without writing changes. +6. `sync` applied the reviewed changes to Infrahub. + +The same pattern applies to larger migrations: start with a clear schema, map a small set of objects, condition source data where needed, review the diff, and then synchronize. + +## Stop the local Infrahub instance + +When you are finished, stop the Docker Compose stack. + +```bash +uv run invoke stop +``` + +## Troubleshooting + +### `infrahub-sync` fails with "Both url and token must be specified" + +```text +ERROR | infrahub_sync.cli | Failed to initialize the Sync Instance: Error initializing InfrahubAdapter: Both url and token must be specified! +``` + +The NetBox and/or Infrahub adapter can't find credentials. `generate`, `diff`, `sync`, and `apply` all read them from the environment, not from `config.yml`, so make sure every variable is exported in the shell you're running the command from: + +```bash +export NETBOX_URL="https://demo.netbox.dev" +export NETBOX_TOKEN="" +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" +``` + +This error names whichever adapter (NetBox or Infrahub) is missing its variables — the same message is raised for either one. + +## Next steps + +Now that you have completed a first sync, you have covered some of the basic objects in NetBox. + +This is only the first step. You will likely want to bring in the parts that are unique to your own NetBox instance (e.g. roles, custom fields). + +Other guides will soon be available to cover: + +- How to sync locations/regions +- How to deal with VLAN/Prefix/Device roles +- How to cover custom attributes / relationships +- Migrate config context \ No newline at end of file From 5c0ee7fea4e9447b4f5cd900d098d48f626ae8d2 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Tue, 7 Jul 2026 14:56:23 +0200 Subject: [PATCH 06/16] fix: remove broken link --- docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx index af4bcc0..2bbfa75 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx @@ -17,10 +17,6 @@ By the end of this tutorial, you will know how to: **What you will build**: A running Infrahub instance with a NetBox-derived data model, populated with locations, devices, interfaces, IPAM, and organization data synchronized from the public NetBox demo. -:::note -Running Infrahub locally with Docker is the most convenient setup for development and experimentation, but it requires a machine with adequate resources. See [hardware requirements](../deploy-manage/install-configure/hardware-requirements) for details. If you prefer a lighter option, try the [Infrahub Sandbox](https://sandbox.infrahub.app/) or the [Getting Started Lab](https://opsmill.instruqt.com/pages/labs) instead. -::: - **Prerequisites**: - [Docker](https://docs.docker.com/get-docker/) installed and running (Docker Desktop or OrbStack) From 03c85b4be21e6f925e74d2d2a96208b656076c10 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Tue, 7 Jul 2026 17:23:51 +0200 Subject: [PATCH 07/16] fix: provide various improvement to tutorial --- .../tutorials/netbox-demo-to-infrahub-v2.mdx | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx index 2bbfa75..4684928 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx @@ -3,19 +3,23 @@ title: Sync NetBox to Infrahub description: Set up a local Infrahub instance, load a data model, and synchronize infrastructure data from the public NetBox demo. --- -In this tutorial we will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) into Infrahub. We will start from scratch, installing and configuring everything we need along the way. This tutorial covers the fundamental work needed to get started; more advanced topics will be covered in subsequent guides. +This tutorial is for people getting started with Infrahub: it starts from a blank page, with no existing Infrahub instance or schema required. We will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) into Infrahub, installing and configuring everything we need along the way. This tutorial covers the fundamental work needed to get started; more advanced topics will be covered in subsequent guides. + +:::tip Already running Infrahub? +If you already have an Infrahub instance running, you can skip straight to the [NetBox adapter documentation](../adapters/netbox.mdx). +::: By the end of this tutorial, you will know how to: -- install Infrahub Sync - run Infrahub in a Docker container -- load the example NetBox schema into Infrahub +- install Infrahub Sync +- load a schema into Infrahub - create a NetBox → Infrahub sync project - synchronize NetBox objects into Infrahub **Time**: ~30 minutes -**What you will build**: A running Infrahub instance with a NetBox-derived data model, populated with locations, devices, interfaces, IPAM, and organization data synchronized from the public NetBox demo. +**What you will build**: A running Infrahub instance with a production-grade schema covering the same core domains as NetBox (locations, devices, interfaces, IPAM, and organizations), populated with data synchronized from the public NetBox demo. **Prerequisites**: @@ -24,9 +28,7 @@ By the end of this tutorial, you will know how to: - [Python 3.10+](https://www.python.org/downloads/) :::warning Public demo data - -The NetBox demo instance is public and resets regularly. Object counts, names, and sample data may differ from the examples in this tutorial. - +The NetBox demo instance is public and resets regularly. Object counts, names, and sample data may differ from the examples in this tutorial. Because anyone can edit it, it can also contain malformed or unexpected data that breaks the sync — see [Troubleshooting](#troubleshooting) if you run into errors. ::: ## Create a project @@ -102,11 +104,11 @@ For more about supported adapters and their Python requirements, see the [NetBox Infrahub stores data according to its schema. Before we can import any data, we need Infrahub to know the kinds of objects that the sync will create. -We provide a collection of schemas that support basic DCIM and IPAM features. This is not an exact one-to-one mapping with NetBox, but that's not the goal either. +We provide a production-grade Infrahub schema covering DCIM and IPAM features similar to NetBox's. It is not a one-to-one port of NetBox's own data model, and that's intentional. :::info -The example schema is a practical starting point. It does not try to reproduce NetBox one-to-one. Infrahub gives you a flexible graph model, so a real migration can preserve the parts of NetBox that matter while adapting the model to your future workflows. +This is a production-grade Infrahub schema, not a copy of NetBox's data model. Infrahub gives you a flexible graph model, so a real migration can preserve the parts of NetBox that matter to you while adapting the model to your own workflows. ::: @@ -190,7 +192,7 @@ curl -L \ -o sync-projects/netbox-demo/config.yml ``` -The downloaded configuration is named `netbox-demo-to-infrahub`. It maps selected NetBox objects to the example Infrahub NetBox schema. +The downloaded configuration is named `netbox-demo-to-infrahub`. It maps selected NetBox objects onto the Infrahub schema. Open `sync-projects/netbox-demo/config.yml` in an editor and scan the top-level keys: From 1213a7de94fdb4e5708864c12e2797a829df0539 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Tue, 7 Jul 2026 17:28:02 +0200 Subject: [PATCH 08/16] docs: merge tutorial --- .../tutorials/netbox-demo-to-infrahub-v2.mdx | 315 ----------- .../tutorials/netbox-demo-to-infrahub.mdx | 534 +++++------------- 2 files changed, 152 insertions(+), 697 deletions(-) delete mode 100644 docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx deleted file mode 100644 index 4684928..0000000 --- a/docs/docs/tutorials/netbox-demo-to-infrahub-v2.mdx +++ /dev/null @@ -1,315 +0,0 @@ ---- -title: Sync NetBox to Infrahub -description: Set up a local Infrahub instance, load a data model, and synchronize infrastructure data from the public NetBox demo. ---- - -This tutorial is for people getting started with Infrahub: it starts from a blank page, with no existing Infrahub instance or schema required. We will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) into Infrahub, installing and configuring everything we need along the way. This tutorial covers the fundamental work needed to get started; more advanced topics will be covered in subsequent guides. - -:::tip Already running Infrahub? -If you already have an Infrahub instance running, you can skip straight to the [NetBox adapter documentation](../adapters/netbox.mdx). -::: - -By the end of this tutorial, you will know how to: - -- run Infrahub in a Docker container -- install Infrahub Sync -- load a schema into Infrahub -- create a NetBox → Infrahub sync project -- synchronize NetBox objects into Infrahub - -**Time**: ~30 minutes - -**What you will build**: A running Infrahub instance with a production-grade schema covering the same core domains as NetBox (locations, devices, interfaces, IPAM, and organizations), populated with data synchronized from the public NetBox demo. - -**Prerequisites**: - -- [Docker](https://docs.docker.com/get-docker/) installed and running (Docker Desktop or OrbStack) -- [uv](https://docs.astral.sh/uv/getting-started/installation/) (Python package manager) -- [Python 3.10+](https://www.python.org/downloads/) - -:::warning Public demo data -The NetBox demo instance is public and resets regularly. Object counts, names, and sample data may differ from the examples in this tutorial. Because anyone can edit it, it can also contain malformed or unexpected data that breaks the sync — see [Troubleshooting](#troubleshooting) if you run into errors. -::: - -## Create a project - -[Copier](https://copier.readthedocs.io/) is a project scaffolding tool. Use it to create a new Infrahub project from the official template, which includes the standard file structure, task definitions, and a `schemas/` folder: - -1. Run the following command to create a new project directory: - -```bash -uv tool run --from 'copier' copier copy https://github.com/opsmill/infrahub-template infrahub-automation -``` - -If you are unsure about the options, accept the default values for all prompts by pressing **Y**. - -2. Navigate to the project directory: - -```bash -cd infrahub-automation -``` - -3. Open the project in your IDE. If you have Visual Studio Code installed, you can run: - -```bash -code . -``` - -:::success Verification -Run `ls` in the project directory. You should see files including `pyproject.toml`, `tasks.py`, and a `schemas/` folder. -::: - -## Start Infrahub - -The project template includes [Invoke](https://www.pyinvoke.org/) tasks that wrap Docker Compose commands. - -1. Start all Infrahub services with a single command: - -```bash -uv run invoke start -``` - -The first run takes a few minutes while Docker downloads the container images. - -2. Open your browser and go to [http://localhost:8000](http://localhost:8000). - -3. Log in from the bottom-left corner using the default credentials: - - - Username: `admin` - - Password: `infrahub` - -:::success Verification -You should see the Infrahub web interface with a navigation menu on the left side. -::: - -## Install Infrahub Sync - -1. Install the Python dependencies with uv: - -```bash -uv add "infrahub-sync==2.0.0" pynetbox -``` - -This installs the `infrahub-sync` command. `pynetbox` is required by the NetBox adapter. - -2. Verify infrahub-sync command is available: - -```bash -uv run infrahub-sync --help -``` - -For more about supported adapters and their Python requirements, see the [NetBox adapter documentation](../adapters/netbox.mdx). - -## Load a schema into Infrahub - -Infrahub stores data according to its schema. Before we can import any data, we need Infrahub to know the kinds of objects that the sync will create. - -We provide a production-grade Infrahub schema covering DCIM and IPAM features similar to NetBox's. It is not a one-to-one port of NetBox's own data model, and that's intentional. - -:::info - -This is a production-grade Infrahub schema, not a copy of NetBox's data model. Infrahub gives you a flexible graph model, so a real migration can preserve the parts of NetBox that matter to you while adapting the model to your own workflows. - -::: - -This collection of schemas is still a work in progress; eventually it will be available through the marketplace. For now, download the files directly: - -```bash -BRANCH="bgi-schema-library-v2" -DEST="schemas" -BASE_URL="https://raw.githubusercontent.com/opsmill/schema-library/${BRANCH}" - -FILES=( - "base/dcim.yml" - "base/location.yml" - "base/ipam.yml" - "base/organization.yml" - "extensions/aggregate/aggregate.yml" - "extensions/cable/cable.yml" - "extensions/circuit/circuit.yml" - "extensions/compute/compute.yml" - "extensions/cluster/cluster.yml" - "extensions/hosting_cluster/hosting_cluster.yml" - "extensions/lag/lag.yml" - "extensions/location_minimal/location_minimal.yml" - "extensions/vlan/vlan.yml" - "extensions/qinq/qinq.yml" - "extensions/rack/rack.yml" - "extensions/vrf/vrf.yml" -) - -for f in "${FILES[@]}"; do - curl -sSL --create-dirs -o "${DEST}/${f}" "${BASE_URL}/${f}" -done -``` - -Export the local Infrahub address and API token. Infrahub Sync will need them later to authenticate against your instance: - -```bash -export INFRAHUB_ADDRESS="http://localhost:8000" -export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" -``` - -This is the default admin token that the project's Docker Compose stack seeds automatically, and it matches the default already configured in `infrahubctl.toml`. Do not reuse it for an internet-facing or shared Infrahub instance. - -Then load the schema into Infrahub using the project's `load-schema` task: - -```bash -uv run invoke load-schema -``` - -Refresh [the local Infrahub web interface](http://localhost:8000). You should see additional schema objects in the left navigation. You can also open the schema view in the UI to explore the kinds and relationships that were loaded. - -## Create a NetBox API token - -Create a token in the public NetBox demo instance so Infrahub Sync can read data from NetBox. - -1. Open the [public NetBox demo](https://demo.netbox.dev). -2. Log in with username `admin` and password `admin`. -3. Open your user profile. -4. Create an API token. -5. Copy the complete generated token value. - -Export the token locally: - -```bash -export NETBOX_URL="https://demo.netbox.dev" -export NETBOX_TOKEN="nbt_..." -``` - -Export the complete `nbt_...` token value. Do not include an authorization prefix such as `Bearer`, and do not copy only the short **Key** field. - -## Create the sync project - -A sync project is a directory containing a `config.yml` file. The configuration names the source and destination adapters, maps source fields to destination fields, and includes references that let Infrahub Sync compute write order. - -From your project directory (`infrahub-automation`), create a sync project directory and download the example NetBox to Infrahub configuration. - -```bash -mkdir -p sync-projects/netbox-demo -curl -L \ - https://raw.githubusercontent.com/opsmill/infrahub-sync/441c33a683c392c4c1d0190adee6542f9f5a1ec4/examples/netbox_to_infrahub/config.yml \ - -o sync-projects/netbox-demo/config.yml -``` - -The downloaded configuration is named `netbox-demo-to-infrahub`. It maps selected NetBox objects onto the Infrahub schema. - -Open `sync-projects/netbox-demo/config.yml` in an editor and scan the top-level keys: - -- `name` identifies the sync project. -- `source` configures the NetBox adapter. -- `destination` configures the Infrahub adapter. -- `schema_mapping` defines how NetBox API resources become Infrahub objects. - -The configuration includes default endpoint values, but the environment variables exported above take precedence for tokens and URLs. - -The example configuration comments also mention `apply`, `--run-id`, and `--parallel`. This tutorial uses the beginner path: preview with `diff`, then write with `sync --diff`. See [Run a sync](../running-a-sync.mdx) for cached plans and parallel execution. - -For a fuller explanation of this file, see [Create a sync project](../creating-a-sync-project.mdx) and the [schema mapping reference](../reference/schema-mapping.mdx). - -## Generate the sync code - -Generate the Python models and adapter code for this sync project. - -```bash -uv run infrahub-sync generate --name netbox-demo-to-infrahub --directory sync-projects -``` - -The `generate` command reads the sync configuration and the destination schema, then writes the Python code used at runtime. Run `generate` again any time you edit `config.yml`. - -For more about the command flow, see [Run a sync](../running-a-sync.mdx). - -## Preview the changes - -Run a dry-run diff before writing data to Infrahub. - -```bash -uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects -``` - -The `diff` command loads data from NetBox and Infrahub, compares both sides, and prints the planned changes. It also writes a cached plan under `.infrahub-sync-cache/netbox-demo-to-infrahub//plan.parquet`. - -Review the output before continuing. On a first run against an empty local Infrahub instance, most planned changes should be creates. Exact counts depend on the current public NetBox demo data. - -## Sync the data - -After reviewing the diff, run the sync. - -```bash -uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --diff -``` - -The `--diff` option prints the diff before applying the changes. The first sync can take a few minutes because it writes the imported objects and relationships into Infrahub. Because this configuration omits `order:`, Infrahub Sync derives the write order from the mapping references. - -## Verify the imported data - -Open [the local Infrahub web interface](http://localhost:8000) and check for imported objects from the NetBox demo. Depending on the current demo data and the example mapping, you may see objects such as: - -- tags and organizations (manufacturers, providers, RIRs) -- sites and racks -- devices and interfaces -- VRFs, VLANs, and VLAN groups -- prefixes, IP addresses, and aggregates -- circuits - -You can also verify from the CLI: - -```bash -uv run infrahubctl object get LocationSite --limit 5 -uv run infrahubctl object get IpamPrefix --limit 5 -``` - -If these commands return objects, the sync imported data successfully. - -## What happened - -You used Infrahub Sync to move data from NetBox into Infrahub in a controlled sequence: - -1. Infrahub provided the destination graph and schema. -2. NetBox provided the source data. -3. `config.yml` described the adapters, field mappings, references, filters, and Transformations. -4. `generate` converted the configuration into runnable sync code. -5. `diff` compared the source and destination without writing changes. -6. `sync` applied the reviewed changes to Infrahub. - -The same pattern applies to larger migrations: start with a clear schema, map a small set of objects, condition source data where needed, review the diff, and then synchronize. - -## Stop the local Infrahub instance - -When you are finished, stop the Docker Compose stack. - -```bash -uv run invoke stop -``` - -## Troubleshooting - -### `infrahub-sync` fails with "Both url and token must be specified" - -```text -ERROR | infrahub_sync.cli | Failed to initialize the Sync Instance: Error initializing InfrahubAdapter: Both url and token must be specified! -``` - -The NetBox and/or Infrahub adapter can't find credentials. `generate`, `diff`, `sync`, and `apply` all read them from the environment, not from `config.yml`, so make sure every variable is exported in the shell you're running the command from: - -```bash -export NETBOX_URL="https://demo.netbox.dev" -export NETBOX_TOKEN="" -export INFRAHUB_ADDRESS="http://localhost:8000" -export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" -``` - -This error names whichever adapter (NetBox or Infrahub) is missing its variables — the same message is raised for either one. - -## Next steps - -Now that you have completed a first sync, you have covered some of the basic objects in NetBox. - -This is only the first step. You will likely want to bring in the parts that are unique to your own NetBox instance (e.g. roles, custom fields). - -Other guides will soon be available to cover: - -- How to sync locations/regions -- How to deal with VLAN/Prefix/Device roles -- How to cover custom attributes / relationships -- Migrate config context \ No newline at end of file diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index 60a9208..4684928 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -1,163 +1,168 @@ --- title: Sync NetBox to Infrahub +description: Set up a local Infrahub instance, load a data model, and synchronize infrastructure data from the public NetBox demo. --- -In this tutorial we will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) -into Infrahub. We will start from scratch, installing and configuring everything we need along the way. +This tutorial is for people getting started with Infrahub: it starts from a blank page, with no existing Infrahub instance or schema required. We will use Infrahub Sync to copy data from the [public NetBox demo](https://demo.netbox.dev) into Infrahub, installing and configuring everything we need along the way. This tutorial covers the fundamental work needed to get started; more advanced topics will be covered in subsequent guides. + +:::tip Already running Infrahub? +If you already have an Infrahub instance running, you can skip straight to the [NetBox adapter documentation](../adapters/netbox.mdx). +::: By the end of this tutorial, you will know how to: +- run Infrahub in a Docker container - install Infrahub Sync -- run Infrahub in a Docker container; -- load the example NetBox schema into Infrahub; -- create a NetBox → Infrahub sync project; -- use Transformation rules to reshape source values before import; -- use `filters` to control which source objects are synchronized; -- review the planned changes with `infrahub-sync diff`; and -- synchronize NetBox objects into Infrahub. +- load a schema into Infrahub +- create a NetBox → Infrahub sync project +- synchronize NetBox objects into Infrahub -This tutorial is self-contained. When a related page provides more detail, we -link to it so you can continue exploring after the first successful sync. - -:::warning Public demo data - -The NetBox demo instance is public and resets regularly. Object counts, names, -and sample data may differ from the examples in this tutorial. - -::: +**Time**: ~30 minutes -## Prerequisites +**What you will build**: A running Infrahub instance with a production-grade schema covering the same core domains as NetBox (locations, devices, interfaces, IPAM, and organizations), populated with data synchronized from the public NetBox demo. -We walk through everything in this tutorial except Docker, Python, and basic command line -tooling like Git and curl. You also don't need to register for any accounts, we -will use open source software and the public NetBox demo to get the job done. +**Prerequisites**: -Specific requirements are: +- [Docker](https://docs.docker.com/get-docker/) installed and running (Docker Desktop or OrbStack) +- [uv](https://docs.astral.sh/uv/getting-started/installation/) (Python package manager) +- [Python 3.10+](https://www.python.org/downloads/) -- [Docker](https://www.docker.com/) installed and running -- [Python](https://www.python.org/) 3.10 or later -- [Git](https://git-scm.com/) -- [curl](https://curl.se/) -- A web browser +:::warning Public demo data +The NetBox demo instance is public and resets regularly. Object counts, names, and sample data may differ from the examples in this tutorial. Because anyone can edit it, it can also contain malformed or unexpected data that breaks the sync — see [Troubleshooting](#troubleshooting) if you run into errors. +::: -The local Infrahub stack runs several containers. Configure Docker with at least -8 GB of memory before starting the tutorial. +## Create a project -## Prepare a local workspace +[Copier](https://copier.readthedocs.io/) is a project scaffolding tool. Use it to create a new Infrahub project from the official template, which includes the standard file structure, task definitions, and a `schemas/` folder: -Create a directory for this tutorial. We will install Infrahub Sync and store the -sync project here. +1. Run the following command to create a new project directory: ```bash -mkdir -p ~/repos/demo-infrahub-sync -cd ~/repos/demo-infrahub-sync +uv tool run --from 'copier' copier copy https://github.com/opsmill/infrahub-template infrahub-automation ``` -## Install Infrahub Sync +If you are unsure about the options, accept the default values for all prompts by pressing **Y**. -Create a Python virtual environment and install Infrahub Sync from PyPI. This -tutorial pins Infrahub Sync and the example configuration to the same release so -the commands and configuration comments stay aligned. +2. Navigate to the project directory: ```bash -python3 -m venv .venv -source .venv/bin/activate -pip install "infrahub-sync==2.0.0" pynetbox +cd infrahub-automation ``` -`infrahub-sync` installs the Infrahub SDK, including the `infrahubctl` command. -`pynetbox` is required by the NetBox adapter. - -Verify both commands are available: +3. Open the project in your IDE. If you have Visual Studio Code installed, you can run: ```bash -infrahub-sync --help -infrahubctl --help +code . ``` -For more about supported adapters and their Python requirements, see the -[NetBox adapter documentation](../adapters/netbox.mdx). +:::success Verification +Run `ls` in the project directory. You should see files including `pyproject.toml`, `tasks.py`, and a `schemas/` folder. +::: + +## Start Infrahub -## Start Infrahub locally +The project template includes [Invoke](https://www.pyinvoke.org/) tasks that wrap Docker Compose commands. -Clone Infrahub and start it with Docker Compose. The token below is a well-known -local demo value. Do not use it for an internet-facing or shared Infrahub -instance. +1. Start all Infrahub services with a single command: ```bash -cd ~/repos -git clone https://github.com/opsmill/infrahub.git -cd infrahub -export INFRAHUB_INITIAL_ADMIN_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" -docker compose up -d +uv run invoke start ``` -Wait until the API answers: +The first run takes a few minutes while Docker downloads the container images. -```bash -until curl -s -f -o /dev/null http://localhost:8000/api/config; do sleep 2; done -``` +2. Open your browser and go to [http://localhost:8000](http://localhost:8000). + +3. Log in from the bottom-left corner using the default credentials: -Then open -[the local Infrahub web interface](http://localhost:8000). + - Username: `admin` + - Password: `infrahub` -Log in with the default local credentials: +:::success Verification +You should see the Infrahub web interface with a navigation menu on the left side. +::: -- Username: `admin` -- Password: `infrahub` +## Install Infrahub Sync -## Configure the Infrahub API token +1. Install the Python dependencies with uv: + +```bash +uv add "infrahub-sync==2.0.0" pynetbox +``` -Infrahub Sync needs an API token to write data into Infrahub. +This installs the `infrahub-sync` command. `pynetbox` is required by the NetBox adapter. -Export the local demo token in the shell where you will run `infrahub-sync`: +2. Verify infrahub-sync command is available: ```bash -export INFRAHUB_ADDRESS="http://localhost:8000" -export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" +uv run infrahub-sync --help ``` -`INFRAHUB_INITIAL_ADMIN_TOKEN` is read when Infrahub initializes a fresh -database. If you already started this local stack without that variable, create -an account token in the Infrahub web interface and export that token value -instead. +For more about supported adapters and their Python requirements, see the [NetBox adapter documentation](../adapters/netbox.mdx). + +## Load a schema into Infrahub -Do not print or commit API tokens. Keep them in your shell environment or secret -manager. +Infrahub stores data according to its schema. Before we can import any data, we need Infrahub to know the kinds of objects that the sync will create. + +We provide a production-grade Infrahub schema covering DCIM and IPAM features similar to NetBox's. It is not a one-to-one port of NetBox's own data model, and that's intentional. + +:::info -## Load the NetBox schema into Infrahub +This is a production-grade Infrahub schema, not a copy of NetBox's data model. Infrahub gives you a flexible graph model, so a real migration can preserve the parts of NetBox that matter to you while adapting the model to your own workflows. -Infrahub stores data according to its schema. Before we can import NetBox data, -we need Infrahub to know the kinds of objects that the sync will create. +::: -The Infrahub repository includes an example schema for data imported from -NetBox. Load that schema into your local Infrahub instance. +This collection of schemas is still a work in progress; eventually it will be available through the marketplace. For now, download the files directly: ```bash -cd ~/repos/demo-infrahub-sync -infrahubctl schema load ~/repos/infrahub/models/examples/netbox/netbox.yml +BRANCH="bgi-schema-library-v2" +DEST="schemas" +BASE_URL="https://raw.githubusercontent.com/opsmill/schema-library/${BRANCH}" + +FILES=( + "base/dcim.yml" + "base/location.yml" + "base/ipam.yml" + "base/organization.yml" + "extensions/aggregate/aggregate.yml" + "extensions/cable/cable.yml" + "extensions/circuit/circuit.yml" + "extensions/compute/compute.yml" + "extensions/cluster/cluster.yml" + "extensions/hosting_cluster/hosting_cluster.yml" + "extensions/lag/lag.yml" + "extensions/location_minimal/location_minimal.yml" + "extensions/vlan/vlan.yml" + "extensions/qinq/qinq.yml" + "extensions/rack/rack.yml" + "extensions/vrf/vrf.yml" +) + +for f in "${FILES[@]}"; do + curl -sSL --create-dirs -o "${DEST}/${f}" "${BASE_URL}/${f}" +done ``` -Refresh [the local Infrahub web interface](http://localhost:8000). You should -see additional schema objects in the left navigation. You can also open the -schema view in the UI to explore the kinds and relationships that were loaded. +Export the local Infrahub address and API token. Infrahub Sync will need them later to authenticate against your instance: -:::info +```bash +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" +``` -The example schema is a practical starting point. It does not try to reproduce -NetBox one-to-one. Infrahub gives you a flexible graph model, so a real migration -can preserve the parts of NetBox that matter while adapting the model to your -future workflows. +This is the default admin token that the project's Docker Compose stack seeds automatically, and it matches the default already configured in `infrahubctl.toml`. Do not reuse it for an internet-facing or shared Infrahub instance. -For more about the schema used by this adapter, see the -[NetBox adapter schema notes](../adapters/netbox.mdx#schema). +Then load the schema into Infrahub using the project's `load-schema` task: -::: +```bash +uv run invoke load-schema +``` -## Create a NetBox demo API token +Refresh [the local Infrahub web interface](http://localhost:8000). You should see additional schema objects in the left navigation. You can also open the schema view in the UI to explore the kinds and relationships that were loaded. -Create a token in the public NetBox demo instance so Infrahub Sync can read data -from NetBox. +## Create a NetBox API token + +Create a token in the public NetBox demo instance so Infrahub Sync can read data from NetBox. 1. Open the [public NetBox demo](https://demo.netbox.dev). 2. Log in with username `admin` and password `admin`. @@ -169,187 +174,48 @@ Export the token locally: ```bash export NETBOX_URL="https://demo.netbox.dev" -export NETBOX_TOKEN="" +export NETBOX_TOKEN="nbt_..." ``` -Export the complete `nbt_...` token value. Do not include an authorization -prefix such as `Bearer`, and do not copy only the short **Key** field. +Export the complete `nbt_...` token value. Do not include an authorization prefix such as `Bearer`, and do not copy only the short **Key** field. ## Create the sync project -A sync project is a directory containing a `config.yml` file. The configuration -names the source and destination adapters, maps source fields to destination -fields, and includes references that let Infrahub Sync compute write order. +A sync project is a directory containing a `config.yml` file. The configuration names the source and destination adapters, maps source fields to destination fields, and includes references that let Infrahub Sync compute write order. -Create a sync project directory and download the example NetBox to Infrahub -configuration. +From your project directory (`infrahub-automation`), create a sync project directory and download the example NetBox to Infrahub configuration. ```bash -cd ~/repos/demo-infrahub-sync mkdir -p sync-projects/netbox-demo curl -L \ - https://raw.githubusercontent.com/opsmill/infrahub-sync/refs/tags/2.0.0/examples/netbox_to_infrahub/config.yml \ + https://raw.githubusercontent.com/opsmill/infrahub-sync/441c33a683c392c4c1d0190adee6542f9f5a1ec4/examples/netbox_to_infrahub/config.yml \ -o sync-projects/netbox-demo/config.yml ``` -The downloaded configuration is named `from-netbox`. It maps selected NetBox -objects to the example Infrahub NetBox schema. +The downloaded configuration is named `netbox-demo-to-infrahub`. It maps selected NetBox objects onto the Infrahub schema. -Open `sync-projects/netbox-demo/config.yml` in an editor and scan the top-level -keys: +Open `sync-projects/netbox-demo/config.yml` in an editor and scan the top-level keys: - `name` identifies the sync project. - `source` configures the NetBox adapter. - `destination` configures the Infrahub adapter. - `schema_mapping` defines how NetBox API resources become Infrahub objects. -In Infrahub Sync 2.0 and later, `order:` is usually omitted. The engine derives -write order from `reference:` fields in `schema_mapping`. Use an explicit -`order:` list only when you need to override the computed order. - -The configuration includes default endpoint values, but the environment -variables exported above take precedence for tokens and URLs. - -The example configuration comments also mention `apply`, `--run-id`, and -`--parallel`. This tutorial uses the beginner path: preview with `diff`, then -write with `sync --diff`. See [Run a sync](../running-a-sync.mdx) for cached -plans and parallel execution. - -For a fuller explanation of this file, see [Create a sync project](../creating-a-sync-project.mdx) -and the [schema mapping reference](../reference/schema-mapping.mdx). - -## Understand the normalization step - -Public demo data is useful because it is real enough to exercise the sync, but -it can also be messy. The NetBox demo resets regularly, many people use it, and -some values that are valid in NetBox may collide with identifiers in the example -Infrahub schema. - -For this tutorial, we normalize a few fields before generating the sync code. -This shows an important Infrahub Sync pattern: a sync project is not only a field -mapping. It can also condition data as it moves between systems. - -We will use two configuration features: - -- Entries under `transforms` change a source value before Infrahub Sync uses it. - For example, a tenant called `Engineering` can become - `Engineering (engineering)` so it is unique and easier to trace back to the - NetBox slug. -- `filters` exclude source objects that should not be imported in this tutorial. - For example, we will ignore IP addresses already assigned to another NetBox - object so the first import path remains predictable. - -The example schema maps NetBox tenants, manufacturers, and circuit providers to -the same Infrahub kind, `OrganizationGeneric`. The normalization rules add source -context to names that might otherwise collide inside that shared destination -kind. - -These adjustments do not modify NetBox. They only change how this sync project -prepares data before writing it to Infrahub. - -## Normalize public demo identifiers - -Run the following script from the tutorial workspace. It edits -`sync-projects/netbox-demo/config.yml` and adds the normalization rules described -above. - -```bash -python - <<'PY' -from pathlib import Path - -import yaml - -path = Path("sync-projects/netbox-demo/config.yml") -config = yaml.safe_load(path.read_text()) - -for item in config["schema_mapping"]: - name = item["name"] - mapping = item.get("mapping") - - if name == "OrganizationGeneric" and mapping == "tenancy.tenants": - item["transforms"] = [ - {"field": "name", "expression": "{{ name }} ({{ slug }})"}, - ] - elif name == "OrganizationGeneric" and mapping == "dcim.manufacturers": - item["transforms"] = [ - {"field": "name", "expression": "Manufacturer: {{ name }}"}, - ] - elif name == "OrganizationGeneric" and mapping == "circuits.providers": - item["transforms"] = [ - {"field": "name", "expression": "Provider: {{ name }}"}, - ] - elif name == "InfraCircuit" and mapping == "circuits.circuits": - item["transforms"] = [ - {"field": "cid", "expression": "{{ cid }} (netbox-{{ id }})"}, - ] - elif name == "InfraIPAddress" and mapping == "ipam.ip-addresses": - item["filters"] = [ - {"field": "assigned_object", "operation": "is_empty"}, - *item.get("filters", []), - ] - -path.write_text("---\n" + yaml.safe_dump(config, sort_keys=False)) -PY -``` - -Run this normalization before `generate`, because `generate` reads `config.yml` -and creates Python code from it. +The configuration includes default endpoint values, but the environment variables exported above take precedence for tokens and URLs. -### Optional: inspect the normalization manually +The example configuration comments also mention `apply`, `--run-id`, and `--parallel`. This tutorial uses the beginner path: preview with `diff`, then write with `sync --diff`. See [Run a sync](../running-a-sync.mdx) for cached plans and parallel execution. -Open `sync-projects/netbox-demo/config.yml` again and search for -`tenancy.tenants`. You should see an entry like this: - -```yaml -transforms: - - field: name - expression: "{{ name }} ({{ slug }})" -``` - -This tells Infrahub Sync to build the destination `name` from two NetBox values. -If the NetBox demo contains duplicate tenant names, the slug helps distinguish -them in Infrahub. - -Now search for `ipam.ip-addresses`. You should see a filter like this: - -```yaml -filters: - - field: assigned_object - operation: is_empty -``` - -This tells Infrahub Sync to import only unassigned IP addresses for this tutorial -path. - -If a future run fails because the public demo contains another duplicate value, -you can use the same pattern: identify the destination field used as an -identifier, add enough source context with a Transformation expression to make it -unique, run `generate` again, and then rerun `diff`. - -## List the sync project - -Confirm that Infrahub Sync can find the project. - -```bash -infrahub-sync list --directory sync-projects -``` - -You should see a project named `from-netbox`. +For a fuller explanation of this file, see [Create a sync project](../creating-a-sync-project.mdx) and the [schema mapping reference](../reference/schema-mapping.mdx). ## Generate the sync code Generate the Python models and adapter code for this sync project. ```bash -infrahub-sync generate --name from-netbox --directory sync-projects +uv run infrahub-sync generate --name netbox-demo-to-infrahub --directory sync-projects ``` -The `generate` command reads the sync configuration and the destination schema, -then writes the Python code used at runtime. Run `generate` again any time you -edit `config.yml`. - -The command may print Pydantic warnings about `local_id` shadowing a parent -attribute. Those warnings are non-fatal. +The `generate` command reads the sync configuration and the destination schema, then writes the Python code used at runtime. Run `generate` again any time you edit `config.yml`. For more about the command flow, see [Run a sync](../running-a-sync.mdx). @@ -358,188 +224,92 @@ For more about the command flow, see [Run a sync](../running-a-sync.mdx). Run a dry-run diff before writing data to Infrahub. ```bash -infrahub-sync diff --name from-netbox --directory sync-projects +uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects ``` -The `diff` command loads data from NetBox and Infrahub, compares both sides, and -prints the planned changes. It also writes a cached plan under -`.infrahub-sync-cache/from-netbox//plan.parquet`. +The `diff` command loads data from NetBox and Infrahub, compares both sides, and prints the planned changes. It also writes a cached plan under `.infrahub-sync-cache/netbox-demo-to-infrahub//plan.parquet`. -Review the output before continuing. On a first run against an empty local -Infrahub instance, most planned changes should be creates. Exact counts depend -on the current public NetBox demo data. +Review the output before continuing. On a first run against an empty local Infrahub instance, most planned changes should be creates. Exact counts depend on the current public NetBox demo data. ## Sync the data After reviewing the diff, run the sync. ```bash -infrahub-sync sync --name from-netbox --directory sync-projects --diff +uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --diff ``` -The `--diff` option prints the diff before applying the changes. The first sync -can take a few minutes because it writes the imported objects and relationships -into Infrahub. Because this configuration omits `order:`, Infrahub Sync derives -the write order from the mapping references. +The `--diff` option prints the diff before applying the changes. The first sync can take a few minutes because it writes the imported objects and relationships into Infrahub. Because this configuration omits `order:`, Infrahub Sync derives the write order from the mapping references. ## Verify the imported data -Open [the local Infrahub web interface](http://localhost:8000) and check for -imported objects from the NetBox demo. Depending on the current demo data and -the example mapping, you may see objects such as: - -- tags -- organizations -- locations -- racks -- prefixes -- IP addresses -- VLANs +Open [the local Infrahub web interface](http://localhost:8000) and check for imported objects from the NetBox demo. Depending on the current demo data and the example mapping, you may see objects such as: -The example configuration also filters devices and interfaces to names that -contain `dmi01`. The public NetBox demo often has no matching devices. +- tags and organizations (manufacturers, providers, RIRs) +- sites and racks +- devices and interfaces +- VRFs, VLANs, and VLAN groups +- prefixes, IP addresses, and aggregates +- circuits You can also verify from the CLI: ```bash -infrahubctl object get OrganizationGeneric --limit 5 -infrahubctl object get InfraPrefix --limit 5 +uv run infrahubctl object get LocationSite --limit 5 +uv run infrahubctl object get IpamPrefix --limit 5 ``` If these commands return objects, the sync imported data successfully. ## What happened -You used Infrahub Sync to move data from NetBox into Infrahub in a controlled -sequence: +You used Infrahub Sync to move data from NetBox into Infrahub in a controlled sequence: 1. Infrahub provided the destination graph and schema. 2. NetBox provided the source data. -3. `config.yml` described the adapters, field mappings, references, filters, and - Transformations. +3. `config.yml` described the adapters, field mappings, references, filters, and Transformations. 4. `generate` converted the configuration into runnable sync code. 5. `diff` compared the source and destination without writing changes. 6. `sync` applied the reviewed changes to Infrahub. -The same pattern applies to larger migrations: start with a clear schema, map a -small set of objects, condition source data where needed, review the diff, and -then synchronize. +The same pattern applies to larger migrations: start with a clear schema, map a small set of objects, condition source data where needed, review the diff, and then synchronize. ## Stop the local Infrahub instance When you are finished, stop the Docker Compose stack. ```bash -cd ~/repos/infrahub -docker compose down +uv run invoke stop ``` ## Troubleshooting -### Infrahub does not become available on first startup - -The first startup can take several minutes. `infrahub-server` may restart while -it waits for Neo4j to become ready. - -If the API never answers after a few minutes, or if you reused an old local -Infrahub stack, remove the local volumes and start again: +### `infrahub-sync` fails with "Both url and token must be specified" -```bash -cd ~/repos/infrahub -docker compose down -v -export INFRAHUB_INITIAL_ADMIN_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" -docker compose up -d +```text +ERROR | infrahub_sync.cli | Failed to initialize the Sync Instance: Error initializing InfrahubAdapter: Both url and token must be specified! ``` -### Docker reports that port 8000 is already allocated - -Another container or local service is already using `localhost:8000`. Check the -running containers: +The NetBox and/or Infrahub adapter can't find credentials. `generate`, `diff`, `sync`, and `apply` all read them from the environment, not from `config.yml`, so make sure every variable is exported in the shell you're running the command from: ```bash -docker ps --format 'table {{.Names}}\t{{.Ports}}\t{{.Status}}' -``` - -If the port is held by an older Infrahub container from the same local compose -project, remove stale containers and retry: - -```bash -cd ~/repos/infrahub -docker compose down --remove-orphans -docker compose up -d -``` - -### NetBox authentication fails - -The public NetBox demo resets regularly. If authentication fails, create a new -NetBox API token and export `NETBOX_TOKEN` again. - -### Infrahub authentication fails - -Confirm that the token is exported in the shell where you run Infrahub Sync: - -```bash -test -n "$INFRAHUB_API_TOKEN" && echo "Token is set" || echo "Token is missing" -``` - -Do not print API tokens in shared terminals, logs, or support requests. - -### Schema errors occur during generate, diff, or sync - -Make sure the NetBox schema was loaded before running `generate`, `diff`, or -`sync`: - -```bash -infrahubctl schema load ~/repos/infrahub/models/examples/netbox/netbox.yml -``` - -The schema load may print `default_filter is deprecated` notices. Those notices -are non-fatal if the command says the schema loaded successfully. - -### Diff fails with `ObjectAlreadyExists` - -The public NetBox demo data is mutable and can contain duplicate tenant names, -circuit IDs, or IP addresses. Make sure you ran the normalization step before -`infrahub-sync generate`. Then rerun: - -```bash -infrahub-sync generate --name from-netbox --directory sync-projects -infrahub-sync diff --name from-netbox --directory sync-projects +export NETBOX_URL="https://demo.netbox.dev" +export NETBOX_TOKEN="" +export INFRAHUB_ADDRESS="http://localhost:8000" +export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" ``` -If the error identifies a different duplicate field, inspect the related mapping -in `config.yml`, add a targeted Transformation expression, regenerate, and diff -again. +This error names whichever adapter (NetBox or Infrahub) is missing its variables — the same message is raised for either one. -### A second diff after sync fails with `PeerIdentifierError` - -After data is synced, a follow-up `diff` may fail while reading back some -route-target relationships from Infrahub. The initial `sync` can still complete -successfully; verify imported data with `infrahubctl object get` or in the -Infrahub UI. - -### NetBox adapter import fails +## Next steps -Install the NetBox SDK in the same virtual environment: +Now that you have completed a first sync, you have covered some of the basic objects in NetBox. -```bash -pip install pynetbox -``` +This is only the first step. You will likely want to bring in the parts that are unique to your own NetBox instance (e.g. roles, custom fields). -## Next steps +Other guides will soon be available to cover: -Now that you have completed a first sync, continue with these pages: - -- Try syncing to an Infrahub branch with `--branch netbox-import`, then review - the imported data in Infrahub before merging it to `main`. See - [Infrahub version control](https://docs.infrahub.app/topics/version-control/). -- [Create a sync project](../creating-a-sync-project.mdx) to learn the structure - of `config.yml` in more detail. -- [Run a sync](../running-a-sync.mdx) to learn more about `list`, `generate`, - `diff`, and `sync`. -- [NetBox adapter](../adapters/netbox.mdx) to review adapter requirements, - direction, and configuration. -- [Schema mapping reference](../reference/schema-mapping.mdx) to learn more - about identifiers, references, filters, and Transformations. -- [Infrahub schema](https://docs.infrahub.app/topics/schema/) to learn how - Infrahub models infrastructure data. +- How to sync locations/regions +- How to deal with VLAN/Prefix/Device roles +- How to cover custom attributes / relationships +- Migrate config context \ No newline at end of file From 68175709549f174d557d7320a0e179f96eee0baf Mon Sep 17 00:00:00 2001 From: Baptiste Date: Tue, 7 Jul 2026 17:37:03 +0200 Subject: [PATCH 09/16] fix: fix linting issues --- .vale/styles/spelling-exceptions.txt | 1 + docs/docs/tutorials/netbox-demo-to-infrahub.mdx | 8 ++++---- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/.vale/styles/spelling-exceptions.txt b/.vale/styles/spelling-exceptions.txt index 11e2d78..712ab6a 100644 --- a/.vale/styles/spelling-exceptions.txt +++ b/.vale/styles/spelling-exceptions.txt @@ -113,6 +113,7 @@ Redoc repo ressources REST +RIRs rowcount Rowcount schema_mapping diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index 4684928..e8acee8 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -82,7 +82,7 @@ The first run takes a few minutes while Docker downloads the container images. You should see the Infrahub web interface with a navigation menu on the left side. ::: -## Install Infrahub Sync +## Install infrahub-sync 1. Install the Python dependencies with uv: @@ -284,7 +284,7 @@ uv run invoke stop ## Troubleshooting -### `infrahub-sync` fails with "Both url and token must be specified" +### `infrahub-sync` fails with `Both url and token must be specified` ```text ERROR | infrahub_sync.cli | Failed to initialize the Sync Instance: Error initializing InfrahubAdapter: Both url and token must be specified! @@ -305,11 +305,11 @@ This error names whichever adapter (NetBox or Infrahub) is missing its variables Now that you have completed a first sync, you have covered some of the basic objects in NetBox. -This is only the first step. You will likely want to bring in the parts that are unique to your own NetBox instance (e.g. roles, custom fields). +This is only the first step. You will likely want to bring in the parts that are unique to your own NetBox instance (for example, roles or custom fields). Other guides will soon be available to cover: - How to sync locations/regions - How to deal with VLAN/Prefix/Device roles - How to cover custom attributes / relationships -- Migrate config context \ No newline at end of file +- Migrate configuration context \ No newline at end of file From 1eb2136ee90e04628fb901b6b59d98d7602d6aa1 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Thu, 9 Jul 2026 11:17:33 +0200 Subject: [PATCH 10/16] fix: make sure float is translated to int and vlan are skipped --- examples/netbox_to_infrahub/config.yml | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/examples/netbox_to_infrahub/config.yml b/examples/netbox_to_infrahub/config.yml index a4ce94a..5ed7286 100644 --- a/examples/netbox_to_infrahub/config.yml +++ b/examples/netbox_to_infrahub/config.yml @@ -178,7 +178,7 @@ schema_mapping: - name: height mapping: integer_height - name: weight - mapping: weight + mapping: integer_weight # NOTE: Netbox tracks weight_unit separately (kg/lb/g/oz); schema # has no unit field, so mixed-unit sources will be inconsistent. - name: manufacturer @@ -187,7 +187,8 @@ schema_mapping: transforms: - field: integer_height expression: "{{ u_height|float|round(0, 'ceil') }}" - + - field: integer_weight + expression: "{{ weight|float|round(0, 'ceil') }}" # --------------------------------------------------------------------- # Netbox roles are user-defined/arbitrary, so there's no guarantee the # Netbox slug matches one of the fixed choices below — sync will fail for @@ -312,12 +313,13 @@ schema_mapping: - name: ip_addresses mapping: ip_addresses reference: IpamIPAddress - - name: untagged_vlan - mapping: untagged_vlan - reference: IpamVLAN - - name: tagged_vlan - mapping: tagged_vlans - reference: IpamVLAN + # TODO: Make sure no vlan is skipped due to missing group then enable this + # - name: untagged_vlan + # mapping: untagged_vlan + # reference: IpamVLAN + # - name: tagged_vlan + # mapping: tagged_vlans + # reference: IpamVLAN - name: device mapping: device reference: DcimDevice From 1d430b76f87b7ccdcd3ac94cdaf9a93e1d174a8b Mon Sep 17 00:00:00 2001 From: Baptiste <32564248+BaptisteGi@users.noreply.github.com> Date: Mon, 13 Jul 2026 10:54:44 +0200 Subject: [PATCH 11/16] Update docs/docs/tutorials/netbox-demo-to-infrahub.mdx Co-authored-by: lancamat1 <135596273+lancamat1@users.noreply.github.com> --- docs/docs/tutorials/netbox-demo-to-infrahub.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index e8acee8..a9948b2 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -41,7 +41,7 @@ The NetBox demo instance is public and resets regularly. Object counts, names, a uv tool run --from 'copier' copier copy https://github.com/opsmill/infrahub-template infrahub-automation ``` -If you are unsure about the options, accept the default values for all prompts by pressing **Y**. +When prompted, enter a project name (for example, `infrahub-automation`), then press **Enter** to accept the default for every remaining prompt (they all default to **No**). 2. Navigate to the project directory: From 2195542cd906f2489898b7859362c3b93ab263e1 Mon Sep 17 00:00:00 2001 From: Baptiste <32564248+BaptisteGi@users.noreply.github.com> Date: Mon, 13 Jul 2026 10:54:57 +0200 Subject: [PATCH 12/16] Update docs/docs/tutorials/netbox-demo-to-infrahub.mdx Co-authored-by: lancamat1 <135596273+lancamat1@users.noreply.github.com> --- docs/docs/tutorials/netbox-demo-to-infrahub.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index a9948b2..a8e5417 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -25,7 +25,7 @@ By the end of this tutorial, you will know how to: - [Docker](https://docs.docker.com/get-docker/) installed and running (Docker Desktop or OrbStack) - [uv](https://docs.astral.sh/uv/getting-started/installation/) (Python package manager) -- [Python 3.10+](https://www.python.org/downloads/) +- [Python 3.11+](https://www.python.org/downloads/) :::warning Public demo data The NetBox demo instance is public and resets regularly. Object counts, names, and sample data may differ from the examples in this tutorial. Because anyone can edit it, it can also contain malformed or unexpected data that breaks the sync — see [Troubleshooting](#troubleshooting) if you run into errors. From f82ca36c16ecc5adb7afd4f50aeaf3d196fa264f Mon Sep 17 00:00:00 2001 From: Baptiste <32564248+BaptisteGi@users.noreply.github.com> Date: Mon, 13 Jul 2026 11:01:59 +0200 Subject: [PATCH 13/16] Update docs/docs/tutorials/netbox-demo-to-infrahub.mdx Co-authored-by: lancamat1 <135596273+lancamat1@users.noreply.github.com> --- docs/docs/tutorials/netbox-demo-to-infrahub.mdx | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index a8e5417..57e4d1c 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -301,6 +301,14 @@ export INFRAHUB_API_TOKEN="06438eb2-8019-4776-878c-0941b1f1d1ec" This error names whichever adapter (NetBox or Infrahub) is missing its variables — the same message is raised for either one. +### `infrahub-sync` fails with `Object ... already present` + +```text +ValueError: An error occurred while loading Netbox: ('Object 172.16.0.2/24__Alpha already present', IpamIPAddress "172.16.0.2/24__Alpha") +``` + +Two source objects map to the same identifier — here, two IP addresses sharing an address and VRF. Delete one of the duplicates in NetBox (**IPAM > IP Addresses**, search the address from the message), then re-run the command. + ## Next steps Now that you have completed a first sync, you have covered some of the basic objects in NetBox. From c082f17d48b6f3eaf61f01d5a71d56592562ec86 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Mon, 13 Jul 2026 11:06:35 +0200 Subject: [PATCH 14/16] fix: adjust tutorial to work around that "parent is mandatory" for site --- docs/docs/tutorials/netbox-demo-to-infrahub.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index 57e4d1c..801b754 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -131,7 +131,7 @@ FILES=( "extensions/cluster/cluster.yml" "extensions/hosting_cluster/hosting_cluster.yml" "extensions/lag/lag.yml" - "extensions/location_minimal/location_minimal.yml" + "extensions/location_site/location_site.yml" "extensions/vlan/vlan.yml" "extensions/qinq/qinq.yml" "extensions/rack/rack.yml" From 0205d12daa13ae190a707ba106a4f466884291d9 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Mon, 13 Jul 2026 11:07:32 +0200 Subject: [PATCH 15/16] fix: pin the main version of the config --- docs/docs/tutorials/netbox-demo-to-infrahub.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index 801b754..3da66c4 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -188,7 +188,7 @@ From your project directory (`infrahub-automation`), create a sync project direc ```bash mkdir -p sync-projects/netbox-demo curl -L \ - https://raw.githubusercontent.com/opsmill/infrahub-sync/441c33a683c392c4c1d0190adee6542f9f5a1ec4/examples/netbox_to_infrahub/config.yml \ + https://raw.githubusercontent.com/opsmill/infrahub-sync/refs/heads/main/examples/netbox_to_infrahub/config.yml \ -o sync-projects/netbox-demo/config.yml ``` From a093275df0d36fe18d24bb42fd399b7cce3a6546 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Mon, 13 Jul 2026 11:47:08 +0200 Subject: [PATCH 16/16] feat: add branching in the process --- .../tutorials/netbox-demo-to-infrahub.mdx | 60 +++++++++++-------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx index 3da66c4..afeb222 100644 --- a/docs/docs/tutorials/netbox-demo-to-infrahub.mdx +++ b/docs/docs/tutorials/netbox-demo-to-infrahub.mdx @@ -16,10 +16,11 @@ By the end of this tutorial, you will know how to: - load a schema into Infrahub - create a NetBox → Infrahub sync project - synchronize NetBox objects into Infrahub +- review the import in a branch and open a proposed change **Time**: ~30 minutes -**What you will build**: A running Infrahub instance with a production-grade schema covering the same core domains as NetBox (locations, devices, interfaces, IPAM, and organizations), populated with data synchronized from the public NetBox demo. +**What you will build**: A running Infrahub instance with a production-grade schema covering the same core domains as NetBox (locations, devices, interfaces, IPAM, and organizations), populated with data synchronized from the public NetBox demo into a branch, ready to review as a proposed change. **Prerequisites**: @@ -219,47 +220,55 @@ The `generate` command reads the sync configuration and the destination schema, For more about the command flow, see [Run a sync](../running-a-sync.mdx). +## Create a branch + +Infrahub tracks changes through branches, so you can review a batch of imported data as a proposed change before it lands on `main`. Create a branch to hold this import: + +```bash +uv run infrahubctl branch create netbox-import +``` + +The rest of this tutorial runs `diff` and `sync` against this branch with `--branch netbox-import`, so the import can be reviewed before merging. For more about branches and proposed changes, see [Infrahub version control](https://docs.infrahub.app/topics/version-control/). + ## Preview the changes -Run a dry-run diff before writing data to Infrahub. +Run a dry-run diff against the `netbox-import` branch before writing any data to Infrahub. ```bash -uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects +uv run infrahub-sync diff --name netbox-demo-to-infrahub --directory sync-projects --branch netbox-import ``` The `diff` command loads data from NetBox and Infrahub, compares both sides, and prints the planned changes. It also writes a cached plan under `.infrahub-sync-cache/netbox-demo-to-infrahub//plan.parquet`. -Review the output before continuing. On a first run against an empty local Infrahub instance, most planned changes should be creates. Exact counts depend on the current public NetBox demo data. +Review the output before continuing. On a first run against an empty branch, most planned changes should be creates. Exact counts depend on the current public NetBox demo data. ## Sync the data -After reviewing the diff, run the sync. +After reviewing the diff, run the sync against the same branch. ```bash -uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --diff +uv run infrahub-sync sync --name netbox-demo-to-infrahub --directory sync-projects --branch netbox-import --diff ``` -The `--diff` option prints the diff before applying the changes. The first sync can take a few minutes because it writes the imported objects and relationships into Infrahub. Because this configuration omits `order:`, Infrahub Sync derives the write order from the mapping references. +The `--diff` option prints the diff before applying the changes. The first sync can take a few minutes because it writes the imported objects and relationships into the `netbox-import` branch — `main` is untouched until you merge the resulting proposed change. Because this configuration omits `order:`, Infrahub Sync derives the write order from the mapping references. ## Verify the imported data -Open [the local Infrahub web interface](http://localhost:8000) and check for imported objects from the NetBox demo. Depending on the current demo data and the example mapping, you may see objects such as: +The imported data lives on the `netbox-import` branch, not on `main`. Open [the local Infrahub web interface](http://localhost:8000): -- tags and organizations (manufacturers, providers, RIRs) -- sites and racks -- devices and interfaces -- VRFs, VLANs, and VLAN groups -- prefixes, IP addresses, and aggregates -- circuits +1. Use the branch selector in the top-left corner to switch from `main` to `netbox-import`. +2. Browse the left navigation for imported objects from the NetBox demo. Depending on the current demo data and the example mapping, you may see objects such as: -You can also verify from the CLI: + - tags and organizations (manufacturers, providers, RIRs) + - sites and racks + - devices and interfaces + - VRFs, VLANs, and VLAN groups + - prefixes, IP addresses, and aggregates + - circuits -```bash -uv run infrahubctl object get LocationSite --limit 5 -uv run infrahubctl object get IpamPrefix --limit 5 -``` +3. Once you're happy with the data, open a proposed change from `netbox-import` toward `main` so the import can be reviewed before it's merged. -If these commands return objects, the sync imported data successfully. +If you switch back to `main`, none of this data is there yet — that's expected, since it's still isolated in the branch. ## What happened @@ -267,12 +276,13 @@ You used Infrahub Sync to move data from NetBox into Infrahub in a controlled se 1. Infrahub provided the destination graph and schema. 2. NetBox provided the source data. -3. `config.yml` described the adapters, field mappings, references, filters, and Transformations. -4. `generate` converted the configuration into runnable sync code. -5. `diff` compared the source and destination without writing changes. -6. `sync` applied the reviewed changes to Infrahub. +3. A branch isolated the import from `main` so it could be reviewed first. +4. `config.yml` described the adapters, field mappings, references, filters, and Transformations. +5. `generate` converted the configuration into runnable sync code. +6. `diff` compared the source and destination without writing changes. +7. `sync` applied the reviewed changes to the branch. -The same pattern applies to larger migrations: start with a clear schema, map a small set of objects, condition source data where needed, review the diff, and then synchronize. +The same pattern applies to larger migrations: start with a clear schema, map a small set of objects, condition source data where needed, review the diff inside a branch, and then synchronize before opening a proposed change toward `main`. ## Stop the local Infrahub instance