Implement the repository extension mechanism (extends)

[ericof]

Summary

Implement the extends field in cookieplone-config.json, turning it from a reserved/no-op placeholder into a working extension mechanism.

A downstream repository (e.g. eea/cookieplone-templates, plonegovbr/cookieplone-templates) should be able to declare "extends": "gh:plone/cookieplone-templates" and:

• Inherit all templates and groups from the upstream repository by default.
• Override individual templates by re-declaring them under the same id in the downstream templates mapping.
• Add brand-new templates on top of the upstream set.
• Hide specific upstream templates it does not want to expose.

The current state is that the field is accepted by the schema validator at cookieplone/config/schemas/repository_config.schema.json but the generator never reads it, so every downstream repository has to copy-paste the upstream templates / groups blocks verbatim.

Why

Right now the only way to customize plone/cookieplone-templates is to fork it. That creates the usual fork problems:

• Downstream has to cherry-pick or merge upstream changes manually.
• Organizations that only want to add one in-house template still end up owning a full fork.
• It is hard to tell from the outside which templates in a fork are local overrides vs unchanged upstream copies.

An extends mechanism lets organizations keep a minimal repository with only their local templates and overrides, while transparently consuming upstream. This mirrors how extends works in GitLab CI, GitHub Actions composite workflows, and tsconfig / pyproject inheritance.

Scope

In scope

1. Load the upstream repository at runtime when an extends URL is present on the current repository's cookieplone-config.json.
   • Reuse the existing repository resolution machinery in cookieplone/repository.py (determine_repo_dir, clone-to-dir, abbreviation expansion, tag handling).
   • Cache the upstream clone alongside the downstream one for the duration of the run.
2. Merge templates and groups with downstream-wins semantics:
   • An id that appears in both maps resolves to the downstream definition.
   • An id that only appears upstream is visible as if it were local.
   • An id that only appears downstream is added as expected.
3. Merge config.versions as a two-layer stack: upstream base, downstream overrides on top.
4. Merge config.renderer: downstream wins when set, otherwise falls back to upstream.
5. Template path resolution for inherited templates must point back into the upstream checkout (not the downstream one), so that the actual template files, sub-templates, and post_gen_project.py hooks come from upstream.
6. Hide an upstream template by redeclaring it with "hidden": true in the downstream config.
7. Loop detection: if an upstream itself has extends, follow it (transitive inheritance), but refuse cycles with a clear error.
8. Schema validation must still run on both ends, with the merged result also satisfying the cross-referential checks (every template in a group exists, no template in two groups, etc.).

Out of scope for this issue (track separately)

• Offline / air-gapped inheritance (upstream unreachable → fall back to a pinned cache).
• Partial template overrides (e.g. override just a sub-template of an upstream template).
• Extending individual templates rather than whole repositories.
• A cookieplone repo show --resolved CLI command to display the merged config.

Proposed design

Resolution

1. get_repository() loads the downstream repository as it does today and parses its cookieplone-config.json.
2. If extends is present and non-empty, it calls a new helper _resolve_extends(upstream_url, tag, password, ...) that:
   • Expands abbreviations and delegates to determine_repo_dir() to clone or locate the upstream.
   • Recursively loads the upstream config (triggering any transitive extends).
   • Registers the upstream clone path in cleanup_paths so the tempdir is cleaned up after generation.
3. The downstream and upstream configs are merged via a new _merge_repo_configs(upstream, downstream) pure function. Merge rules are documented in the merged config dict itself (see next section) so that debugging the result is straightforward.

Merge semantics

Field	Rule
version	Must match on both sides; otherwise raise.
title / description	Downstream wins.
extends	Stripped from the merged result (it has been resolved).
templates	Keyed union; downstream entries replace upstream entries with the same id.
groups	Keyed union; downstream group entries replace upstream groups with the same id. For groups present in both, the templates list is replaced (not merged) so downstream can re-order or drop upstream entries explicitly.
config.versions	Shallow merge, downstream wins per key.
config.renderer	Downstream wins if set, otherwise upstream value.

RepositoryInfo changes

Add an upstream_repos: list[Path] field to cookieplone._types.RepositoryInfo recording each resolved upstream in order (closest-to-downstream first). This lets the generator resolve template paths correctly: when looking up a template id, first check the downstream clone, then each upstream in order.

Error handling

• Unreachable upstream → RepositoryException with a message pointing at the extends URL and the underlying determine_repo_dir error.
• Schema version mismatch → InvalidConfiguration listing both versions.
• Circular inheritance → InvalidConfiguration showing the cycle (e.g. A extends B extends A).
• Broken merged config (post-merge cross-referential validation fails) → InvalidConfiguration explaining which downstream declaration caused the break.

Acceptance criteria

• A downstream repository with only "extends": "gh:plone/cookieplone-templates" and no local templates successfully runs cookieplone and offers the full upstream template menu.
• Redeclaring an upstream template id under downstream templates replaces the upstream entry (path, title, description, hidden).
• config.versions and config.renderer merge with downstream-wins semantics.
• Transitive inheritance (A extends B extends C) resolves correctly.
• Circular inheritance is detected and reported with a clear error.
• Unreachable upstream produces a helpful error pointing at the URL.
• Schema validation runs on downstream, upstream, and the merged result.
• Temporary upstream clones are cleaned up after the run.
• Integration tests cover: basic inheritance, override, add, hide, versions merge, renderer merge, transitive chain, and cycle detection.
• Docs updated:
  • Remove the "reserved for a future version" note from docs/src/reference/repository-config.md.
  • Add a new how-to guide: "Extend an upstream template repository" under docs/src/how-to-guides/.
  • Add a conceptual page or extend docs/src/concepts/template-repositories.md with a section on inheritance and merge rules.
  • Ship a runnable example of an extending repository in the docs — either as a fully-worked walkthrough inside the how-to guide, or as a small reference repository committed under docs/src/_examples/ (or similar) that the guide links to. The example should show a cookieplone-config.json with extends, one overridden template, one brand-new template, and one upstream template hidden via "hidden": true, so readers can copy it as a starting point.
• News fragment under news/ (type: feature).

Notes

• Milestone: 2.0.0
• Blocks/related: any org-specific fork of plone/cookieplone-templates. Once this ships, those forks can be trimmed to just their local overrides.
• Worth checking how this interacts with the new cookieplone-config.json grouped template format (Support grouped template selection and repository configuration format #158) — the merge logic for groups must respect the grouped-selection UX.

@avoinea, @valentinab25, @sneridagh