Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,11 @@ Each should include the "Required Development Workflow" block and the "Approval

## Adding a New Adapter

1. Create `infrahub_sync/adapters/<name>.py` following existing adapter patterns.
2. Add connection config schema and an example under `examples/`.
3. Provide `list` and `diff` pathways before enabling `sync`.
4. Document required environment variables and expected error cases.
5. Create a documentation page in `docs/docs/adapters/` (overview, config keys, env vars, example YAML, common errors), add it to the sidebar, and validate with `markdownlint-cli2 "docs/docs/adapters/**/*.{md,mdx}"`.
See [`dev/guides/adding-an-adapter.md`](dev/guides/adding-an-adapter.md) for the full
step-by-step procedure. Supporting developer reference lives under `dev/`:

- [Adapter knowledge](dev/knowledge/README.md) — how the sync engine, the adapter contract, schema mapping, and the incremental cache work.
- [Adapter guidelines](dev/guidelines/README.md) — the rules for writing and testing an adapter.
- [Adapter guides](dev/guides/README.md) — adding and testing an adapter, step by step.

Core rule unchanged: provide read-only `list` / `diff` pathways and validate them before enabling `sync`.
17 changes: 16 additions & 1 deletion dev/guidelines/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
# Guidelines

Conventions and standards.
Prescriptive rules — how adapter code should be written. Guidelines say what you must and
must not do; for how the system works see [`dev/knowledge/`](../knowledge/README.md), and
for step-by-step procedures see [`dev/guides/`](../guides/README.md).

## Adapters

- [Writing an adapter](writing-an-adapter.md) — structure, typing, error handling, logging,
optional dependencies, and secret handling for a connector.
- [Testing adapters](testing-adapters.md) — the test coverage every adapter must ship and
the conventions those tests follow.

## Related

- [Adapter knowledge](../knowledge/README.md) — the architecture these rules apply to.
- [Adding an adapter](../guides/adding-an-adapter.md) — the procedure that applies them.
- [Constitution](../constitution.md) — the principles these rules serve.
82 changes: 82 additions & 0 deletions dev/guidelines/testing-adapters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Testing adapters

> Part of: `dev/guidelines/` | Related: [Testing an adapter](../guides/testing-an-adapter.md), [Adapter anatomy](../knowledge/adapter-anatomy.md)

Rules for what an adapter's tests must cover and how they are written. For the mechanics of
writing and running them — fixtures, mocking, commands — see
[Testing an adapter](../guides/testing-an-adapter.md).

## Mock the upstream; never require a live server

**Always stub the upstream client so the default suite runs offline:**

```python
# ✅ Good — controlled responses, deterministic, fast
api = MagicMock()
api.dcim.devices.filter.return_value = [ ... ]

# ❌ Bad — needs a real NetBox, flaky, slow, leaks credentials
api = pynetbox.api(url=os.environ["NETBOX_URL"], token=...)
```

The unit suite must pass with no network access and no secrets. Tests that need a live system
are integration tests (see below).

## Cover the conversion path

**Always test that records become correct DiffSync models:**

- `model_loader` (or the `obj_to_diffsync` helper) maps fields to the right destination names.
- Filters keep and drop the right records.
- Transforms produce native-typed values.
- `identifiers` and `local_id` are populated, and `reference` fields resolve to peer
`unique_id`s — including the list-reference case.

## Cover the incremental contract

**Always test the cursor methods an adapter declares:**

- `cursor_tier_for` returns the expected tier for mapped kinds and `CursorTier.NONE` for
unmapped ones.
- `list_changed_since` issues the correct change filter (for example `last_updated__gte`) and
yields records in `model_loader` shape.
- `list_existing_ids` yields the current `unique_id`s.
- An adapter that declares a non-`NONE` tier but omits `list_changed_since` must fail — assert
the `NotImplementedError`. `tests/test_diffsync_mixin_contract.py` covers the mixin defaults;
per-adapter tests cover the overrides.

## Cover the error and edge cases

**Always test the failure modes a connector actually hits:**

- Empty result sets and pagination across multiple pages.
- Authentication failures (401 / 403) and timeouts surface as clear errors, not silent passes.
- Unknown model names raise rather than returning empty.

## Skip cleanly when the optional dependency is absent

**Always guard a module that hard-imports an optional SDK:**

```python
import pytest

pytest.importorskip("pynetbox") # module-level: skip, don't error, when the dep is missing
```

This keeps collection green in environments that did not install that adapter's extra.

## Keep tests atomic and integration tests opt-in

**Always isolate one behavior per test and mark live tests:**

- One assertion target per test; parametrize config-parsing and mapping cases instead of
looping inside a test.
- Place unit tests under `tests/adapters/` named `test_<adapter>_*.py`.
- Mark anything that talks to a real system `@pytest.mark.integration` and keep it under
`tests/integration/` so the default `uv run pytest -q` stays offline.

## See also

- [Testing an adapter](../guides/testing-an-adapter.md) — fixtures, mocking, and commands.
- [Writing an adapter](writing-an-adapter.md) — the code these tests exercise.
- [Incremental sync and cache](../knowledge/incremental-and-cache.md) — the cursor behavior to test.
114 changes: 114 additions & 0 deletions dev/guidelines/writing-an-adapter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Writing an adapter

> Part of: `dev/guidelines/` | Related: [Adapter anatomy](../knowledge/adapter-anatomy.md), [Adding an adapter](../guides/adding-an-adapter.md)

Rules for writing an adapter connector. They assume you understand the
[adapter anatomy](../knowledge/adapter-anatomy.md); this document is about *how* to write the
code, not what the pieces are. The repository-wide standards in `AGENTS.md` still apply —
this narrows them to adapters.

## Inherit the mixins, in order

**Always declare the mixin before the DiffSync base:**

```python
# ✅ Good — mixin first, so its methods win
class MyAdapter(DiffSyncMixin, Adapter): ...
class MyModel(DiffSyncModelMixin, DiffSyncModel): ...

# ❌ Bad — DiffSync base shadows the mixin contract
class MyAdapter(Adapter, DiffSyncMixin): ...
```

Set a `type` class attribute (for example `type = "NetBox"`); it is used in log lines and to
decide whether the adapter is acting as the source.

## Reuse the loading helpers

**Always filter and transform through the model mixin:**

```python
# ✅ Good — honors the schema mapping consistently
filtered = model.filter_records(records=records, schema_mapping=element)
records = model.transform_records(records=filtered, schema_mapping=element)

# ❌ Bad — re-implements filtering, diverges from every other adapter
records = [r for r in records if r["status"] == "active"]
```

Filtering and transforming are defined once on `DiffSyncModelMixin`. Re-implementing them in
an adapter means filters and transforms in `config.yml` behave differently per system. For a
REST source, subclass `GenericrestapiAdapter` instead of writing HTTP handling from scratch —
override only the settings defaults you need (see `infrahub_sync/adapters/peeringmanager.py`).

## Type and document the public surface

**Always type new code and give public classes and methods a concise docstring:**

The codebase is clean under `ty` with no `[[tool.ty.overrides]]` blocks. Do not add overrides
to mask errors — fix the type, or use a targeted `# ty: ignore[<rule>]` with a short reason at
the call site. New code must be Ruff-clean and pass `uv run invoke lint`.

## Raise specific exceptions

**Always raise a specific exception with a clear message; never swallow errors broadly:**

```python
# ✅ Good
if not schema_element:
msg = f"Schema mapping for model '{reference}' not found."
raise ValueError(msg)

# ❌ Bad
try:
...
except Exception:
pass
```

Handle the failure modes that matter for a connector explicitly: authentication (401 / 403),
timeouts, empty pages, and pagination. Let unexpected errors surface rather than hiding them.

## Log with structlog, never secrets

**Always use `structlog`; never `print`, and never log credentials:**

Include useful context — endpoint, model name, object counts — but never tokens, passwords,
or full auth headers. The same rule applies to exception messages and tracebacks.

> Some example adapters under `examples/` use `print` for illustration. Production adapters in
> `infrahub_sync/adapters/` use structured logging.

## Handle optional dependencies and credentials

**Always treat the upstream SDK as an optional dependency and read secrets from the
environment:**

```python
import pynetbox # ty: ignore[unresolved-import] # optional dep, see pyproject extras

url = os.environ.get("NETBOX_ADDRESS") or settings.get("url")
token = os.environ.get("NETBOX_TOKEN") or settings.get("token")
```

- The SDK (`pynetbox`, `pynautobot`, …) is not a core dependency. Import it at module top with
the `# ty: ignore[unresolved-import]` comment, and document the install extra.
- Resolve credentials from environment variables first, falling back to `settings`. Never
hardcode, print, commit, or log a secret. Keep example configs sanitized.

## Anti-patterns

| Anti-pattern | Do instead |
|--------------|------------|
| Inline filtering / transforming in the adapter | `model.filter_records` / `model.transform_records` |
| Hand-rolled HTTP for a REST source | Subclass `GenericrestapiAdapter` |
| `except Exception: pass` | Catch the specific error; surface the rest |
| `print()` for diagnostics | `structlog` with context |
| Hardcoded URL or token | Environment variable, then `settings` |
| Declaring `cursor_tier_for` without `list_changed_since` | Implement both, or leave the tier `NONE` |

## See also

- [Adapter anatomy](../knowledge/adapter-anatomy.md) — the contract these rules apply to.
- [Testing adapters](testing-adapters.md) — what to test once it is written.
- [Adding an adapter](../guides/adding-an-adapter.md) — the end-to-end procedure.
17 changes: 16 additions & 1 deletion dev/guides/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
# Guides

How-to documentation.
Step-by-step procedures for adapter tasks. Guides walk you through doing something; for the
rules that constrain the work see [`dev/guidelines/`](../guidelines/README.md), and for how
the system works see [`dev/knowledge/`](../knowledge/README.md).

## Adapters

- [Adding an adapter](adding-an-adapter.md) — the end-to-end procedure for connecting a new
source or destination system, from scaffolding to docs.
- [Testing an adapter](testing-an-adapter.md) — how to write and run an adapter's unit and
integration tests.

## Related

- [Adapter knowledge](../knowledge/README.md) — the architecture behind these steps.
- [Adapter guidelines](../guidelines/README.md) — the rules each step must satisfy.
- [Constitution](../constitution.md) — the principles these guides serve.
Loading
Loading