diff --git a/AGENTS.md b/AGENTS.md index 14b8a9a..8deee1b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -183,8 +183,11 @@ Each should include the "Required Development Workflow" block and the "Approval ## Adding a New Adapter -1. Create `infrahub_sync/adapters/.py` following existing adapter patterns. -2. Add connection config schema and an example under `examples/`. -3. Provide `list` and `diff` pathways before enabling `sync`. -4. Document required environment variables and expected error cases. -5. Create a documentation page in `docs/docs/adapters/` (overview, config keys, env vars, example YAML, common errors), add it to the sidebar, and validate with `markdownlint-cli2 "docs/docs/adapters/**/*.{md,mdx}"`. +See [`dev/guides/adding-an-adapter.md`](dev/guides/adding-an-adapter.md) for the full +step-by-step procedure. Supporting developer reference lives under `dev/`: + +- [Adapter knowledge](dev/knowledge/README.md) — how the sync engine, the adapter contract, schema mapping, and the incremental cache work. +- [Adapter guidelines](dev/guidelines/README.md) — the rules for writing and testing an adapter. +- [Adapter guides](dev/guides/README.md) — adding and testing an adapter, step by step. + +Core rule unchanged: provide read-only `list` / `diff` pathways and validate them before enabling `sync`. diff --git a/dev/guidelines/README.md b/dev/guidelines/README.md index 382a7a1..a36b277 100644 --- a/dev/guidelines/README.md +++ b/dev/guidelines/README.md @@ -1,3 +1,18 @@ # Guidelines -Conventions and standards. +Prescriptive rules — how adapter code should be written. Guidelines say what you must and +must not do; for how the system works see [`dev/knowledge/`](../knowledge/README.md), and +for step-by-step procedures see [`dev/guides/`](../guides/README.md). + +## Adapters + +- [Writing an adapter](writing-an-adapter.md) — structure, typing, error handling, logging, + optional dependencies, and secret handling for a connector. +- [Testing adapters](testing-adapters.md) — the test coverage every adapter must ship and + the conventions those tests follow. + +## Related + +- [Adapter knowledge](../knowledge/README.md) — the architecture these rules apply to. +- [Adding an adapter](../guides/adding-an-adapter.md) — the procedure that applies them. +- [Constitution](../constitution.md) — the principles these rules serve. diff --git a/dev/guidelines/testing-adapters.md b/dev/guidelines/testing-adapters.md new file mode 100644 index 0000000..bb5e2ad --- /dev/null +++ b/dev/guidelines/testing-adapters.md @@ -0,0 +1,82 @@ +# Testing adapters + +> Part of: `dev/guidelines/` | Related: [Testing an adapter](../guides/testing-an-adapter.md), [Adapter anatomy](../knowledge/adapter-anatomy.md) + +Rules for what an adapter's tests must cover and how they are written. For the mechanics of +writing and running them — fixtures, mocking, commands — see +[Testing an adapter](../guides/testing-an-adapter.md). + +## Mock the upstream; never require a live server + +**Always stub the upstream client so the default suite runs offline:** + +```python +# ✅ Good — controlled responses, deterministic, fast +api = MagicMock() +api.dcim.devices.filter.return_value = [ ... ] + +# ❌ Bad — needs a real NetBox, flaky, slow, leaks credentials +api = pynetbox.api(url=os.environ["NETBOX_URL"], token=...) +``` + +The unit suite must pass with no network access and no secrets. Tests that need a live system +are integration tests (see below). + +## Cover the conversion path + +**Always test that records become correct DiffSync models:** + +- `model_loader` (or the `obj_to_diffsync` helper) maps fields to the right destination names. +- Filters keep and drop the right records. +- Transforms produce native-typed values. +- `identifiers` and `local_id` are populated, and `reference` fields resolve to peer + `unique_id`s — including the list-reference case. + +## Cover the incremental contract + +**Always test the cursor methods an adapter declares:** + +- `cursor_tier_for` returns the expected tier for mapped kinds and `CursorTier.NONE` for + unmapped ones. +- `list_changed_since` issues the correct change filter (for example `last_updated__gte`) and + yields records in `model_loader` shape. +- `list_existing_ids` yields the current `unique_id`s. +- An adapter that declares a non-`NONE` tier but omits `list_changed_since` must fail — assert + the `NotImplementedError`. `tests/test_diffsync_mixin_contract.py` covers the mixin defaults; + per-adapter tests cover the overrides. + +## Cover the error and edge cases + +**Always test the failure modes a connector actually hits:** + +- Empty result sets and pagination across multiple pages. +- Authentication failures (401 / 403) and timeouts surface as clear errors, not silent passes. +- Unknown model names raise rather than returning empty. + +## Skip cleanly when the optional dependency is absent + +**Always guard a module that hard-imports an optional SDK:** + +```python +import pytest + +pytest.importorskip("pynetbox") # module-level: skip, don't error, when the dep is missing +``` + +This keeps collection green in environments that did not install that adapter's extra. + +## Keep tests atomic and integration tests opt-in + +**Always isolate one behavior per test and mark live tests:** + +- One assertion target per test; parametrize config-parsing and mapping cases instead of + looping inside a test. +- Place unit tests under `tests/adapters/` named `test__*.py`. +- Mark anything that talks to a real system `@pytest.mark.integration` and keep it under + `tests/integration/` so the default `uv run pytest -q` stays offline. + +## See also + +- [Testing an adapter](../guides/testing-an-adapter.md) — fixtures, mocking, and commands. +- [Writing an adapter](writing-an-adapter.md) — the code these tests exercise. +- [Incremental sync and cache](../knowledge/incremental-and-cache.md) — the cursor behavior to test. diff --git a/dev/guidelines/writing-an-adapter.md b/dev/guidelines/writing-an-adapter.md new file mode 100644 index 0000000..4f62f33 --- /dev/null +++ b/dev/guidelines/writing-an-adapter.md @@ -0,0 +1,114 @@ +# Writing an adapter + +> Part of: `dev/guidelines/` | Related: [Adapter anatomy](../knowledge/adapter-anatomy.md), [Adding an adapter](../guides/adding-an-adapter.md) + +Rules for writing an adapter connector. They assume you understand the +[adapter anatomy](../knowledge/adapter-anatomy.md); this document is about *how* to write the +code, not what the pieces are. The repository-wide standards in `AGENTS.md` still apply — +this narrows them to adapters. + +## Inherit the mixins, in order + +**Always declare the mixin before the DiffSync base:** + +```python +# ✅ Good — mixin first, so its methods win +class MyAdapter(DiffSyncMixin, Adapter): ... +class MyModel(DiffSyncModelMixin, DiffSyncModel): ... + +# ❌ Bad — DiffSync base shadows the mixin contract +class MyAdapter(Adapter, DiffSyncMixin): ... +``` + +Set a `type` class attribute (for example `type = "NetBox"`); it is used in log lines and to +decide whether the adapter is acting as the source. + +## Reuse the loading helpers + +**Always filter and transform through the model mixin:** + +```python +# ✅ Good — honors the schema mapping consistently +filtered = model.filter_records(records=records, schema_mapping=element) +records = model.transform_records(records=filtered, schema_mapping=element) + +# ❌ Bad — re-implements filtering, diverges from every other adapter +records = [r for r in records if r["status"] == "active"] +``` + +Filtering and transforming are defined once on `DiffSyncModelMixin`. Re-implementing them in +an adapter means filters and transforms in `config.yml` behave differently per system. For a +REST source, subclass `GenericrestapiAdapter` instead of writing HTTP handling from scratch — +override only the settings defaults you need (see `infrahub_sync/adapters/peeringmanager.py`). + +## Type and document the public surface + +**Always type new code and give public classes and methods a concise docstring:** + +The codebase is clean under `ty` with no `[[tool.ty.overrides]]` blocks. Do not add overrides +to mask errors — fix the type, or use a targeted `# ty: ignore[]` with a short reason at +the call site. New code must be Ruff-clean and pass `uv run invoke lint`. + +## Raise specific exceptions + +**Always raise a specific exception with a clear message; never swallow errors broadly:** + +```python +# ✅ Good +if not schema_element: + msg = f"Schema mapping for model '{reference}' not found." + raise ValueError(msg) + +# ❌ Bad +try: + ... +except Exception: + pass +``` + +Handle the failure modes that matter for a connector explicitly: authentication (401 / 403), +timeouts, empty pages, and pagination. Let unexpected errors surface rather than hiding them. + +## Log with structlog, never secrets + +**Always use `structlog`; never `print`, and never log credentials:** + +Include useful context — endpoint, model name, object counts — but never tokens, passwords, +or full auth headers. The same rule applies to exception messages and tracebacks. + +> Some example adapters under `examples/` use `print` for illustration. Production adapters in +> `infrahub_sync/adapters/` use structured logging. + +## Handle optional dependencies and credentials + +**Always treat the upstream SDK as an optional dependency and read secrets from the +environment:** + +```python +import pynetbox # ty: ignore[unresolved-import] # optional dep, see pyproject extras + +url = os.environ.get("NETBOX_ADDRESS") or settings.get("url") +token = os.environ.get("NETBOX_TOKEN") or settings.get("token") +``` + +- The SDK (`pynetbox`, `pynautobot`, …) is not a core dependency. Import it at module top with + the `# ty: ignore[unresolved-import]` comment, and document the install extra. +- Resolve credentials from environment variables first, falling back to `settings`. Never + hardcode, print, commit, or log a secret. Keep example configs sanitized. + +## Anti-patterns + +| Anti-pattern | Do instead | +|--------------|------------| +| Inline filtering / transforming in the adapter | `model.filter_records` / `model.transform_records` | +| Hand-rolled HTTP for a REST source | Subclass `GenericrestapiAdapter` | +| `except Exception: pass` | Catch the specific error; surface the rest | +| `print()` for diagnostics | `structlog` with context | +| Hardcoded URL or token | Environment variable, then `settings` | +| Declaring `cursor_tier_for` without `list_changed_since` | Implement both, or leave the tier `NONE` | + +## See also + +- [Adapter anatomy](../knowledge/adapter-anatomy.md) — the contract these rules apply to. +- [Testing adapters](testing-adapters.md) — what to test once it is written. +- [Adding an adapter](../guides/adding-an-adapter.md) — the end-to-end procedure. diff --git a/dev/guides/README.md b/dev/guides/README.md index 8bd6e65..4128b82 100644 --- a/dev/guides/README.md +++ b/dev/guides/README.md @@ -1,3 +1,18 @@ # Guides -How-to documentation. +Step-by-step procedures for adapter tasks. Guides walk you through doing something; for the +rules that constrain the work see [`dev/guidelines/`](../guidelines/README.md), and for how +the system works see [`dev/knowledge/`](../knowledge/README.md). + +## Adapters + +- [Adding an adapter](adding-an-adapter.md) — the end-to-end procedure for connecting a new + source or destination system, from scaffolding to docs. +- [Testing an adapter](testing-an-adapter.md) — how to write and run an adapter's unit and + integration tests. + +## Related + +- [Adapter knowledge](../knowledge/README.md) — the architecture behind these steps. +- [Adapter guidelines](../guidelines/README.md) — the rules each step must satisfy. +- [Constitution](../constitution.md) — the principles these guides serve. diff --git a/dev/guides/adding-an-adapter.md b/dev/guides/adding-an-adapter.md new file mode 100644 index 0000000..d067687 --- /dev/null +++ b/dev/guides/adding-an-adapter.md @@ -0,0 +1,189 @@ +# Adding an adapter + +> Part of: `dev/guides/` | Related: [Adapter anatomy](../knowledge/adapter-anatomy.md), [Writing an adapter](../guidelines/writing-an-adapter.md) + +Step-by-step guide for connecting a new system to infrahub-sync as a source or destination. +This is the canonical procedure; `AGENTS.md` links here. + +## When to add an adapter + +Add an adapter when you need to read from, or write to, a system that has no connector yet. +The built-ins live in `infrahub_sync/adapters/` (`netbox`, `nautobot`, `infrahub`, `aci`, +`prometheus`, `peeringmanager`, `ipfabricsync`, `slurpitsync`, `genericrestapi`). + +Before writing one, check whether you can avoid it: + +- If the system has a REST API, subclass `GenericrestapiAdapter` and configure it rather than + writing a connector from scratch. +- If a built-in already covers the system, you may only need a new `config.yml`. + +## Prerequisites + +- A working dev environment (`uv sync`) — see `AGENTS.md`. +- An understanding of the [adapter anatomy](../knowledge/adapter-anatomy.md) and + [schema mapping](../knowledge/schema-mapping.md). +- Read access to the source system (URL, token) or write access to the destination. +- The destination schema (for an Infrahub destination, the node kinds you will map to). + +## Steps + +### Step 1: Choose the role and a starting point + +Decide whether the new system is a **source** (read-only) or a **destination** (written to), +and pick a base: + +| Situation | Start from | +|-----------|------------| +| REST/JSON API | Subclass `GenericrestapiAdapter` (`infrahub_sync/adapters/peeringmanager.py`) | +| Bespoke SDK or protocol | A fresh `DiffSyncMixin` adapter (`infrahub_sync/adapters/netbox.py`) | +| Learning the shape | Copy `examples/custom_adapter/custom_adapter_src/custom_adapter.py` | + +### Step 2: Create the adapter module + +Create `infrahub_sync/adapters/.py` (or a custom module outside the package). Define the +two classes and a client: + +```python +from diffsync import Adapter, DiffSyncModel +from infrahub_sync import DiffSyncMixin, DiffSyncModelMixin, SchemaMappingModel, SyncAdapter, SyncConfig + +class MysystemAdapter(DiffSyncMixin, Adapter): + type = "MySystem" + + def __init__(self, target, adapter, config, *args, **kwargs): + super().__init__(*args, **kwargs) + self.target = target + self.config = config + self.settings = adapter.settings or {} + self.client = self._create_client(self.settings) + + def model_loader(self, model_name, model): ... + +class MysystemModel(DiffSyncModelMixin, DiffSyncModel): + @classmethod + def create(cls, adapter, ids, attrs): ... + def update(self, attrs): ... +``` + +Follow [Writing an adapter](../guidelines/writing-an-adapter.md): mixin first, `structlog`, +optional-dependency import with `# ty: ignore[unresolved-import]`, credentials from the +environment. + +### Step 3: Implement `model_loader` + +For each model, find its schema-mapping entry, fetch the source records, filter and transform +them through the model mixin, convert each to the DiffSync shape, and add it: + +```python +def model_loader(self, model_name, model): + element = next(e for e in self.config.schema_mapping if e.name == model_name) + records = self.client.get(element.mapping) + if self.config.source.name.title() == self.type.title(): + records = model.filter_records(records=records, schema_mapping=element) + records = model.transform_records(records=records, schema_mapping=element) + for obj in records: + self.add(model(**self.obj_to_diffsync(obj=obj, mapping=element, model=model))) +``` + +Write the `obj_to_diffsync` helper to walk `element.fields` — `static`, plain `mapping`, and +`reference` (resolved to a peer `unique_id`) — and always set `local_id`. See +`examples/custom_adapter/custom_adapter_src/custom_adapter.py` for a complete version. + +### Step 4: Implement write methods (destination only) + +If the adapter can be a destination, implement `create`, `update`, and `delete` on the model +to mutate the target system. A source-only adapter can leave these deferring to the base. + +### Step 5: Write the schema mapping and `config.yml` + +Create an example project directory with a `config.yml` that selects the adapter and maps +resources to destination models: + +```yaml +--- +name: mysystem-example + +source: + name: mysystem + # built-in name above, OR a path/dotted-path to a custom class: + adapter: ./path/to/my_adapter.py:MysystemAdapter + settings: + url: "https://mysystem.example.com" + +destination: + name: infrahub + settings: + url: "http://localhost:8000" + +schema_mapping: + - name: InfraDevice + mapping: devices + identifiers: ["name"] + fields: + - name: name + mapping: name +``` + +Omit `order` — it is computed from `reference` edges. See +[schema mapping](../knowledge/schema-mapping.md) for fields, filters, and transforms. + +### Step 6: Add incremental support (optional) + +If the source can filter by change, override `cursor_tier_for` to return the right +`CursorTier` and implement `list_changed_since` (and optionally `list_existing_ids`). See +[incremental sync and cache](../knowledge/incremental-and-cache.md). Skip this and the adapter +simply does full extracts. + +### Step 7: Add an example and document env vars + +Add a directory under `examples/_to_infrahub/` (or `infrahub_to_/`) with the +`config.yml`, and document the required environment variables and the install extra for the +optional SDK. + +### Step 8: Add tests + +Write unit tests under `tests/adapters/` that mock the client. See +[Testing an adapter](testing-an-adapter.md) and the rules in +[Testing adapters](../guidelines/testing-adapters.md). + +### Step 9: Add a documentation page + +Create a page under `docs/docs/adapters/` (overview, config keys, env vars, example YAML, +common errors), add it to the sidebar, and lint it: + +```bash +markdownlint-cli2 "docs/docs/adapters/**/*.{md,mdx}" +``` + +## Verification + +Validate read-only paths before ever running `sync`: + +```bash +uv run invoke format +uv run invoke lint + +uv run infrahub-sync list --directory examples/ +uv run infrahub-sync generate --name mysystem-example --directory examples/mysystem_to_infrahub/ +uv run infrahub-sync diff --name mysystem-example --directory examples/mysystem_to_infrahub/ +``` + +`list` and `generate` need no live source; `diff` reads both sides but writes nothing. Run +`sync` only with explicit approval against a known-safe target. + +## Quality checklist + +- [ ] Adapter inherits `DiffSyncMixin` / `DiffSyncModelMixin`, mixin first, with a `type`. +- [ ] `model_loader` filters and transforms through the model mixin; `obj_to_diffsync` sets `local_id`. +- [ ] Optional SDK imported with `# ty: ignore[unresolved-import]`; credentials from env vars; no secrets logged or committed. +- [ ] `uv run invoke format` and `uv run invoke lint` are clean; `uv run ty check .` exits 0. +- [ ] `list` / `generate` / `diff` succeed for the example. +- [ ] Unit tests added under `tests/adapters/`; `uv run pytest -q` passes offline. +- [ ] Example added under `examples/`; env vars documented. +- [ ] Documentation page added under `docs/docs/adapters/` and in the sidebar. + +## Related resources + +- [Adapter anatomy](../knowledge/adapter-anatomy.md) — the classes and contract. +- [Writing an adapter](../guidelines/writing-an-adapter.md) — the rules. +- [Testing an adapter](testing-an-adapter.md) — the tests to add. diff --git a/dev/guides/testing-an-adapter.md b/dev/guides/testing-an-adapter.md new file mode 100644 index 0000000..1688581 --- /dev/null +++ b/dev/guides/testing-an-adapter.md @@ -0,0 +1,147 @@ +# Testing an adapter + +> Part of: `dev/guides/` | Related: [Testing adapters](../guidelines/testing-adapters.md), [Adding an adapter](adding-an-adapter.md) + +Step-by-step guide for writing and running an adapter's tests. It covers the mechanics — +mocking, fixtures, commands — while [Testing adapters](../guidelines/testing-adapters.md) +defines what coverage is required. + +## When to test + +Every new or changed adapter needs tests before it is merged. Write them alongside the +adapter (Step 8 of [Adding an adapter](adding-an-adapter.md)), not afterward. The default +suite must run offline, so tests mock the upstream system rather than calling it. + +## Prerequisites + +- A working dev environment (`uv sync`). +- The adapter under test, with its `model_loader` and any incremental methods. +- Familiarity with `unittest.mock` and pytest fixtures. +- The existing tests as templates: `tests/adapters/test_netbox_incremental.py` and + `tests/test_diffsync_mixin_contract.py`. + +## Steps + +### Step 1: Create the test module + +Add `tests/adapters/test__*.py` — for example `test_mysystem_incremental.py` for the +cursor methods and `test_mysystem_loader.py` for conversion. + +### Step 2: Guard the optional dependency + +If the adapter hard-imports an optional SDK, skip the whole module when it is absent so test +collection stays green: + +```python +import pytest + +pytest.importorskip("pynetbox") # at module top, before importing the adapter +``` + +### Step 3: Build a fake client + +Stub the upstream client with `MagicMock` and feed it sample records shaped like the real API +response. Construct the adapter, then replace its client with the mock: + +```python +from unittest.mock import MagicMock +from infrahub_sync.adapters.mysystem import MysystemAdapter + +def make_adapter(config): + adapter = MysystemAdapter(target="source", adapter=config.source, config=config) + adapter.client = MagicMock() + return adapter +``` + +Keep the sample records and a minimal `SyncConfig` (with a `schema_mapping`) in fixtures so +several tests share them. `tests/adapters/test_netbox_incremental.py` shows the concrete +construction for a real adapter — mirror it. + +### Step 4: Test the conversion path + +Drive `model_loader` (or `obj_to_diffsync`) with the fake records and assert the resulting +models. Check that fields map to the right names, that `local_id` and `identifiers` are set, +and that `reference` fields resolve to peer `unique_id`s: + +```python +def test_loader_maps_fields(adapter, device_model): + adapter.client.get.return_value = [{"name": "rtr1", "device_type": "qfx"}] + adapter.model_loader("InfraDevice", device_model) + obj = adapter.get(device_model, "rtr1") + assert obj.type == "qfx" +``` + +Add cases for filters (kept vs dropped) and transforms (native-typed output). + +### Step 5: Test the incremental contract + +If the adapter declares cursor support, assert each method: + +```python +from infrahub_sync.cache.cursors import CursorState, CursorTier + +def test_cursor_tier(adapter): + assert adapter.cursor_tier_for("InfraDevice") == CursorTier.TIMESTAMP + assert adapter.cursor_tier_for("Unmapped") == CursorTier.NONE + +def test_list_changed_since_uses_change_filter(adapter): + cursor = CursorState(tier=CursorTier.TIMESTAMP, value="2024-01-01T00:00:00Z") + list(adapter.list_changed_since("InfraDevice", cursor)) + # assert the client was queried with the change filter, e.g. last_updated__gte +``` + +Also assert that declaring a non-`NONE` tier without implementing `list_changed_since` raises +`NotImplementedError` — the mixin defaults are covered in `tests/test_diffsync_mixin_contract.py`. + +### Step 6: Test the edge cases + +Cover empty result sets, pagination, and that authentication failures (401 / 403), timeouts, +and unknown model names raise clear errors rather than passing silently. + +### Step 7: Run the suite + +```bash +uv run pytest -q tests/adapters/test_mysystem_loader.py +uv run pytest -q # full offline suite +``` + +## Integration tests + +Tests that talk to a live system go under `tests/integration/`, marked so they are opt-in: + +```python +import pytest + +@pytest.mark.integration +def test_live_load(): ... +``` + +Run them explicitly and only when credentials are available: + +```bash +uv run pytest -m integration +``` + +Keep them out of the default run — `uv run pytest -q` must pass with no network and no secrets. + +## Verification + +- `uv run pytest -q` passes offline, with no live system and no credentials. +- The module skips cleanly (not errors) when the optional SDK is not installed. +- `uv run invoke lint` is clean on the new test files. + +## Quality checklist + +- [ ] Upstream client mocked; no network in the default suite. +- [ ] Conversion covered: field mapping, `local_id`, identifiers, references (single and list). +- [ ] Filters and transforms covered. +- [ ] Cursor methods covered (tier, change filter, existing ids) if the adapter is incremental. +- [ ] Edge cases covered: empty, pagination, 401/403, timeout, unknown model. +- [ ] `pytest.importorskip` guards an optional-dependency import. +- [ ] Live tests marked `@pytest.mark.integration` under `tests/integration/`. + +## Related resources + +- [Testing adapters](../guidelines/testing-adapters.md) — the required coverage and conventions. +- [Adding an adapter](adding-an-adapter.md) — where testing fits in the full procedure. +- [Incremental sync and cache](../knowledge/incremental-and-cache.md) — the behavior to test. diff --git a/dev/knowledge/README.md b/dev/knowledge/README.md index 3cedd4e..67ee670 100644 --- a/dev/knowledge/README.md +++ b/dev/knowledge/README.md @@ -1,3 +1,24 @@ # Knowledge -Domain knowledge and glossary, loaded on demand. Houses `CONTEXT.md`. +How infrahub-sync works — descriptive reference, loaded on demand. These documents +explain the moving parts so you can reason about a change before making it. For +prescriptive rules see [`dev/guidelines/`](../guidelines/README.md); for step-by-step +procedures see [`dev/guides/`](../guides/README.md). + +## Adapters + +- [Sync architecture](sync-architecture.md) — how a sync runs end to end: the DiffSync + core, source and destination adapters, the Potenda engine, and the code-generation path. +- [Adapter anatomy](adapter-anatomy.md) — the two classes every adapter provides, the + `DiffSyncMixin` / `DiffSyncModelMixin` contract, and what you implement versus what you + get for free. +- [Schema mapping](schema-mapping.md) — how `config.yml` maps source resources to + destination models: fields, identifiers, references, filters, and transforms. +- [Incremental sync and cache](incremental-and-cache.md) — cursors, tiers, plans, and + row-count guardrails, and what an adapter implements to participate. + +## Related + +- [Adapter guidelines](../guidelines/README.md) — rules an adapter must follow. +- [Adapter guides](../guides/README.md) — adding and testing an adapter. +- [Constitution](../constitution.md) — project principles these documents serve. diff --git a/dev/knowledge/adapter-anatomy.md b/dev/knowledge/adapter-anatomy.md new file mode 100644 index 0000000..26f1d9a --- /dev/null +++ b/dev/knowledge/adapter-anatomy.md @@ -0,0 +1,105 @@ +# Adapter anatomy + +> Part of: `dev/knowledge/` | Related: [Sync architecture](sync-architecture.md), [Schema mapping](schema-mapping.md), [Adding an adapter](../guides/adding-an-adapter.md) + +An adapter is a single module under `infrahub_sync/adapters/.py` (or a custom module +outside the package) that defines two classes: an **adapter class** that loads and writes +objects, and a **model class** that the generated models inherit. `infrahub_sync/adapters/netbox.py` +is the reference example; `examples/custom_adapter/custom_adapter_src/custom_adapter.py` is a +minimal from-scratch one. + +## The two classes + +```python +from diffsync import Adapter, DiffSyncModel +from infrahub_sync import DiffSyncMixin, DiffSyncModelMixin + +class MyAdapter(DiffSyncMixin, Adapter): + type = "MySystem" + + def __init__(self, target, adapter, config, *args, **kwargs): + super().__init__(*args, **kwargs) + self.target = target # "source" or "destination" + self.config = config # the SyncConfig for this run + self.client = self._make_client(adapter.settings or {}) + + def model_loader(self, model_name, model): ... + +class MyModel(DiffSyncModelMixin, DiffSyncModel): + @classmethod + def create(cls, adapter, ids, attrs): ... + def update(self, attrs): ... +``` + +`DiffSyncMixin` and `DiffSyncModelMixin` live in `infrahub_sync/__init__.py`. The order of +base classes matters: the mixin comes first so its methods take precedence over the +DiffSync base. + +## The adapter contract (`DiffSyncMixin`) + +The mixin defines the surface Potenda calls. Each method is one of three kinds — provided +(use as-is), must-implement (raises `NotImplementedError` until you override), or optional. + +| Method | Kind | Purpose | +|--------|------|---------| +| `load()` | Provided | Iterates `top_level`; calls `load_()` if defined, otherwise `model_loader(name, model)`. Do not override. | +| `model_loader(model_name, model)` | Must implement | Fetch records from the system, filter and transform them, build each into the DiffSync shape, and `self.add(model(**data))`. | +| `cursor_tier_for(model_name)` | Optional | Strongest incremental tier the source supports for this model. Defaults to `CursorTier.NONE` (always full extract). | +| `list_changed_since(model_name, cursor)` | Conditional | Required only if `cursor_tier_for` returns a non-`NONE` tier. Yields records changed since the cursor, in the same shape `model_loader` produces. | +| `list_existing_ids(model_name)` | Optional | Yields current `unique_id` strings for delete detection between incremental runs. | + +A read-only-capable adapter that only ever does full extracts needs just `model_loader`. +Incremental support is additive — see [Incremental sync and cache](incremental-and-cache.md). + +## The model contract (`DiffSyncModelMixin`) + +The model mixin gives every model the helpers used during loading and the hooks used during +writing. + +Provided for you (used inside `model_loader`): + +- `filter_records(records, schema_mapping)` — drop records that fail the mapping's filters. +- `transform_records(records, schema_mapping)` — apply the mapping's Jinja2 transforms. +- `apply_filters` / `apply_transforms` / `is_list` / `get_resource_name` — the lower-level + building blocks the two above are built from. + +You implement on the model (used when it is the destination): + +- `create(cls, adapter, ids, attrs)` — create the object in the destination, return the + instance. +- `update(self, attrs)` — apply changed attributes. +- `delete(self)` — inherited from DiffSync; override only if deletion needs custom logic. + +If an adapter is only ever a source, its model's `create` / `update` / `delete` are never +called and can defer to the base implementation. + +## From upstream object to DiffSync model + +Inside `model_loader`, each raw record is converted to the field shape the generated model +expects. By convention this is a helper named `_obj_to_diffsync` (or `obj_to_diffsync` +on the REST base). It walks the mapping's `fields` and, for each: + +- copies a `static` literal, or +- reads `field.mapping` from the record (dot notation, via `get_value`), or +- resolves a `reference` to another model's `unique_id` (single or list) using the store. + +Every record also carries a `local_id` — the source-side primary key — so references can be +resolved across models. See [Schema mapping](schema-mapping.md) for the field semantics. + +## How the class is found + +`config.yml` selects the adapter: + +- `name: netbox` — a built-in under `infrahub_sync/adapters/`. +- `adapter: ./path/to/file.py:MyAdapter` — a filesystem path and class name. +- `adapter: my_pkg.adapters:MyAdapter` — a dotted import path. +- an installed package exposing an `infrahub_sync.adapters` entry point. + +`plugin_loader.py` resolves these in order. Custom adapters do not need to live inside the +package — point at them with `adapter` and, if needed, `adapters_path`. + +## See also + +- [Sync architecture](sync-architecture.md) — where the adapter sits in a run. +- [Writing an adapter](../guidelines/writing-an-adapter.md) — the rules to follow. +- [Adding an adapter](../guides/adding-an-adapter.md) — the step-by-step procedure. diff --git a/dev/knowledge/incremental-and-cache.md b/dev/knowledge/incremental-and-cache.md new file mode 100644 index 0000000..07c6710 --- /dev/null +++ b/dev/knowledge/incremental-and-cache.md @@ -0,0 +1,79 @@ +# Incremental sync and cache + +> Part of: `dev/knowledge/` | Related: [Adapter anatomy](adapter-anatomy.md), [Sync architecture](sync-architecture.md) + +A full sync extracts every object from both sides on every run. For large systems that is +slow and wasteful, so infrahub-sync can run *incrementally*: extract only what changed since +the last run, and reuse a cached diff plan. Incremental support is opt-in per adapter and +per model — an adapter that does nothing still works, it just always does a full extract. + +## Cursor tiers + +`cursor_tier_for(model_name)` declares the strongest extraction strategy the source supports +for a model. It returns a `CursorTier` (an `IntEnum`, defined in +`infrahub_sync/cache/cursors.py`): + +| Tier | Value | Meaning | +|------|-------|---------| +| `NONE` | 0 | The source cannot filter by change; always full extract. The default. | +| `ID` | 1 | The source exposes a stable id set; changes are detected by comparing ids. | +| `TIMESTAMP` | 2 | The source can filter by modification time (for example NetBox / Nautobot `last_updated__gte`); extract only changed-since records. | + +Higher tiers extract less data. NetBox returns `TIMESTAMP` for mapped kinds and `NONE` +otherwise; an adapter with no incremental support inherits the `NONE` default from the mixin. + +## What an adapter implements + +Three methods, layered on top of `model_loader`: + +- `cursor_tier_for(model_name)` — return the tier. This is the switch that turns incremental + on for a model. +- `list_changed_since(model_name, cursor)` — **required when the tier is not `NONE`.** Yield + the raw records changed since `cursor`, in the same shape `model_loader` feeds to + `self.add(...)`. The mixin raises `NotImplementedError` until you override it. +- `list_existing_ids(model_name)` — optional. Yield the current `unique_id` strings present + in the source so deletions can be detected between runs. Without it, a warm run cannot tell + that an object disappeared. + +`CursorState` (also in `cache/cursors.py`) carries the tier and the saved value (a timestamp +or id watermark) from the previous run. + +## Full resync cadence + +Incremental drift accumulates, so a periodic full extract re-establishes ground truth. +`IncrementalConfig.full_resync_every` (default `10`) forces a full resync every N incremental +runs. Configure it under `incremental` in `config.yml`: + +```yaml +incremental: + full_resync_every: 20 +``` + +## The diff plan and the cache + +Potenda can separate computing a diff from applying it: + +- `write_plan(diff)` serializes the diff to a Parquet **plan** on disk. +- `apply_plan()` reads that plan back and syncs without re-extracting either side. + +This lets you review a plan before applying it, or compute on one host and apply on another. +Cached side snapshots (also Parquet) and cursor state live alongside the plan. + +The cache root defaults to `/.infrahub-sync-cache//`, with each run under its +own `/`. Set `INFRAHUB_SYNC_CACHE_DIR` to relocate it (for example to a shared volume); +the path may not contain `..` traversal segments. Cursor state is written by +`persist_cursors_for_run()` at the end of a successful run and read at the start of the next. + +## The row-count guardrail + +A buggy source or an auth failure can return far fewer objects than reality, which would make +a sync delete most of the destination. `check_rowcount_guardrail()` compares the new run's +destination row counts against a persisted baseline and **fails the run** if the count drops +beyond the threshold. Override intentionally with the `--allow-rowcount-drop` flag (which maps +to `allow_rowcount_drop` on `sync_in_tiers`) when a large deletion is expected. + +## See also + +- [Adapter anatomy](adapter-anatomy.md) — where these methods sit in the contract. +- [Sync architecture](sync-architecture.md) — how Potenda drives load, diff, and sync. +- [Testing an adapter](../guides/testing-an-adapter.md) — testing the incremental methods. diff --git a/dev/knowledge/schema-mapping.md b/dev/knowledge/schema-mapping.md new file mode 100644 index 0000000..670b4a2 --- /dev/null +++ b/dev/knowledge/schema-mapping.md @@ -0,0 +1,105 @@ +# Schema mapping + +> Part of: `dev/knowledge/` | Related: [Adapter anatomy](adapter-anatomy.md), [Adding an adapter](../guides/adding-an-adapter.md) + +The `schema_mapping` block in `config.yml` is the declarative contract between a source +system and a destination schema. It says which source resources become which destination +models, which fields carry over, how objects are uniquely identified, and which records to +skip or rewrite. The generator turns each entry into a DiffSync model class, and adapters +read the same entries at load time. The pydantic models behind it live in +`infrahub_sync/__init__.py`. + +## A mapping entry + +Each list item is a `SchemaMappingModel`: + +```yaml +schema_mapping: + - name: InfraDevice # destination model / kind + mapping: dcim.devices # source resource path + identifiers: ["name"] # natural key + filters: # optional: which records to keep + - field: status.value + operation: "==" + value: active + transforms: # optional: rewrite fields + - field: name + expression: "{{ name | lower }}" + fields: # field-by-field mapping + - name: name + mapping: name +``` + +- `name` — the destination model/kind. Must match a node in the destination schema. +- `mapping` — the source resource. Its meaning is adapter-specific: a pynetbox endpoint + path like `dcim.devices`, a REST collection, a key in a JSON document. +- `identifiers` — the fields that uniquely identify the object. This is the DiffSync natural + key; it must be stable across runs and present on both sides. +- `filters` — records must match **all** filters to be loaded (see below). +- `transforms` — applied after filtering, in order. +- `fields` — the field mappings (see below). + +## Fields + +Each field is a `SchemaMappingField` with four levers, checked in this order: + +- `static` — assign a literal value, ignoring the source object entirely. +- `mapping` (without `reference`) — read a value from the source record by dot-notation + path (`get_value` walks dicts and objects, e.g. `status.value`). +- `mapping` + `reference` — treat the value as a foreign key into another mapped model. + The adapter resolves it to that peer's `unique_id`. If the destination field is a list, + every referenced id is resolved and collected. +- `name` — always the destination field name. + +References are how relationships are rebuilt on the destination. Because a reference points +at another model by its identifiers, the referenced model must also appear in +`schema_mapping`, and it must be written first — which is what `order` controls. + +## Identifiers, references, and write order + +`order` is the list of models in the sequence they are written to the destination. It is +**auto-computed** from the `reference` edges in `schema_mapping` (a model is written after +everything it references), so you normally omit it. Override it only when the computed order +is wrong — for example to break a cycle. The `local_id` carried on each loaded record is the +source primary key that reference resolution matches against. + +## Filters + +Filters drop records before they enter the store. A record is kept only if it passes every +filter. Each `SchemaMappingFilter` has a `field` (dot-notation path), an `operation`, and a +`value`. The supported operations are: + +| Category | Operations | +|----------|------------| +| Equality | `==`, `!=` | +| Ordering (numeric) | `>`, `<`, `>=`, `<=` | +| Membership | `in`, `not in`, `contains`, `not contains` | +| Presence | `is_empty`, `is_not_empty` | +| Pattern | `regex` | +| Network | `is_ip_within` | + +`is_empty` / `is_not_empty` ignore `value`; the ordering operations coerce both sides to +`int`. The full operator table is `FILTERS_OPERATIONS` in `infrahub_sync/__init__.py`. + +## Transforms + +Transforms rewrite a field with a Jinja2 expression after filtering. Each +`SchemaMappingTransform` names a `field` and an `expression`. Expressions run in a Jinja2 +`NativeEnvironment`, so the result keeps its native Python type (`list`, `dict`, `bool`, +`int`, `str`) instead of being stringified, and `StrictUndefined` makes a missing key fail +fast rather than silently producing an empty string. The whole record is available as +template context: + +```yaml +transforms: + - field: name + expression: "{{ name | upper }}" + - field: tags + expression: "{{ tags + ['synced'] }}" +``` + +## See also + +- [Adapter anatomy](adapter-anatomy.md) — how an adapter reads these entries at load time. +- [Adding an adapter](../guides/adding-an-adapter.md) — writing a mapping for a new system. +- User-facing configuration reference under `docs/docs/`. diff --git a/dev/knowledge/sync-architecture.md b/dev/knowledge/sync-architecture.md new file mode 100644 index 0000000..5701cc4 --- /dev/null +++ b/dev/knowledge/sync-architecture.md @@ -0,0 +1,76 @@ +# Sync architecture + +> Part of: `dev/knowledge/` | Related: [Adapter anatomy](adapter-anatomy.md), [Adding an adapter](../guides/adding-an-adapter.md) + +infrahub-sync moves data from a *source* system of record to a *destination* system of +record. It is built on [DiffSync](https://github.com/networktocode/diffsync): each side +loads its objects into an in-memory store, the two stores are diffed, and the destination +is reconciled to match the source. An adapter is the connector for one system; the same +adapter class can act as either source or destination depending on where it appears in +`config.yml`. + +## The DiffSync foundation + +DiffSync represents every object as a model instance with a stable identity. Two concepts +do the work: + +- **Adapter** — loads objects from one system into a store and knows how to `create`, + `update`, and `delete` them there. +- **Model** — a typed record with `_identifiers` (the natural key) and `_attributes` + (the comparable fields). Two objects with the same identifiers are "the same object" on + both sides; their attributes are what the diff compares. + +infrahub-sync layers two mixins on top of the DiffSync base classes: `DiffSyncMixin` for +adapters and `DiffSyncModelMixin` for models. See [Adapter anatomy](adapter-anatomy.md). + +## Source and destination + +A sync names exactly one `source` and one `destination` in `config.yml`. Each points at an +adapter by `name` (a built-in such as `netbox`, `nautobot`, or `infrahub`) or at a custom +class via `adapter`. The direction is fixed per run: data flows source → destination, and +only the destination is written to. The source is read-only. + +The same adapter is therefore used in both roles across the codebase — `infrahub` is the +destination in `netbox_to_infrahub` and the source in `infrahub_to_peering-manager`. An +adapter must be able to *load* (read) for its source role and to *create / update / delete* +for its destination role. + +## The Potenda engine + +`Potenda` (in `infrahub_sync/potenda/`) orchestrates a run in three stages: + +1. **Load** — `load_both_sides()` calls each adapter's `load()`, which iterates the + adapter's `top_level` model list and calls `model_loader()` per model. The source and + destination load concurrently by default. +2. **Diff** — `diff()` runs the DiffSync comparison and produces a structured set of + create / update / delete actions. +3. **Sync** — `sync()` (or `sync_in_tiers()`) applies those actions against the + destination in dependency order, calling the destination model's `create` / `update` / + `delete`. + +Potenda also owns the cross-cutting machinery — write order tiers, the incremental cursor +state, the Parquet diff plan, and the row-count guardrail. See +[Incremental sync and cache](incremental-and-cache.md). Adapters do not call Potenda; +Potenda calls adapters through the mixin contract. + +## The code-generation path + +Adapters do not hand-write a model class per object type. Instead: + +1. You write the adapter module (the connector logic) and a `config.yml` whose + `schema_mapping` describes which source resources map to which destination models. +2. `infrahub-sync generate` (in `infrahub_sync/generator/`) reads the config and the + destination schema and renders DiffSync model classes from Jinja2 templates. +3. `infrahub_sync/plugin_loader.py` resolves the adapter class — built-in by `name`, a + dotted import path, a filesystem path, or an installed entry point — and wires the + generated models onto the adapter instance at run time. + +So an adapter author supplies two things: the connector (how to talk to the system) and +the schema mapping (what to move). The model classes are generated. See +[Schema mapping](schema-mapping.md). + +## See also + +- [Adapter anatomy](adapter-anatomy.md) — the classes and methods you implement. +- [Adding an adapter](../guides/adding-an-adapter.md) — the end-to-end procedure. +- [Writing an adapter](../guidelines/writing-an-adapter.md) — the rules to follow.