-
Notifications
You must be signed in to change notification settings - Fork 180
docs: add ADR about the manifests directory #3383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
j-zimnowoda
wants to merge
9
commits into
main
Choose a base branch
from
adr
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
17d2a7d
docs: add ADR about the manifests directory
j-zimnowoda 5308597
docs: add ADR about the manifests directory
j-zimnowoda 432d51a
docs: add ADR about git-server
j-zimnowoda 5e7a0c9
Merge remote-tracking branch 'origin/main' into adr
j-zimnowoda b9cec77
Merge branch 'main' into adr
svcAPLBot dc118e6
Merge branch 'main' into adr
svcAPLBot a5078cb
Merge branch 'main' into adr
svcAPLBot 4a2d77f
Update adr/2026-06-25-git-server-as-default-values-repo.md
j-zimnowoda 0735e36
Update adr/2026-06-25-git-server-as-default-values-repo.md
j-zimnowoda File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| # git-server as the default values repository backend | ||
|
|
||
| - Status: accepted | ||
|
|
||
| ## Context and Problem Statement | ||
|
|
||
| APL stores its platform configuration (Helm values, team settings, secrets) in a Git repository — the values repo. Every GitOps reconciliation cycle reads from it; if it is unavailable, ArgoCD cannot pull and the platform cannot self-heal. | ||
|
|
||
| Historically, Gitea was the only internal option. Gitea is a full Git forge: it ships with a web UI, user management, pull-request workflows, and a PostgreSQL database. During platform upgrades, Gitea and its database are upgraded in sequence. Either component can fail to restart cleanly, leaving the values repo unreachable and breaking the reconciliation loop until the issue is resolved manually. | ||
|
|
||
| For the narrow use case of storing the values repo, APL only needs dumb HTTP git storage — nothing Gitea's forge features provide. | ||
|
|
||
| ## Considered Options | ||
|
|
||
| 1. **git-server** (default) — a minimal nginx + `git-http-backend` deployment with no database dependency | ||
| 2. **Gitea** (optional) — a self-hosted Git forge, enabled via `apps.gitea.enabled: true` | ||
| 3. **External git repository** (GitHub, GitLab, or any HTTP-accessible git host) — credentials supplied at install time via `otomi.git.*` Helm values | ||
|
|
||
| ## Decision Outcome | ||
|
|
||
| **git-server is the default values repository backend.** Gitea remains available as an optional app (`apps.gitea.enabled: true`) for teams that need a self-hosted Git forge — for example, to host application code repositories via `codeRepo.gitService: gitea`. External git repositories are supported for production environments where teams already operate GitHub, GitLab, or a similar service. | ||
|
|
||
| ### Why git-server instead of Gitea as the default | ||
|
|
||
| git-server has no database. A single stateless nginx process serves `git-http-backend` over HTTP. There is nothing to upgrade independently, no PostgreSQL connection pool to exhaust, and no init container sequence that can leave the pod in a broken state. If the pod is evicted or restarted it comes back immediately from its PVC without manual intervention. | ||
|
|
||
| Gitea's database dependency is the specific failure mode this change eliminates. A Gitea upgrade that leaves the database in an inconsistent state, or a database upgrade that Gitea cannot connect to, renders the values repo unreachable. Because ArgoCD pulls continuously, even a brief outage during upgrade propagates into a stalled reconciliation loop. | ||
|
|
||
| git-server is also sufficient for evaluation and bootstrap scenarios: a fresh install needs only a place to write the initial values commit; it does not need pull requests, webhooks, or user management. | ||
|
|
||
| ### Why Gitea is kept as an optional app | ||
|
|
||
| Gitea provides features that go beyond storing the values repo — repository browsing, access control, PR workflows, and integration with the `codeRepo` pipeline. Teams that use `codeRepo.gitService: gitea` to build application code depend on it. Making Gitea optional (rather than removing it) preserves those use cases without imposing Gitea's operational cost on installations that do not need it. | ||
|
|
||
| ### External git for production environments | ||
|
|
||
| Teams operating in production typically already have a managed Git service. Pointing APL at an existing GitHub or GitLab repository avoids running any in-cluster git storage. The `otomi.git.repoUrl`, `otomi.git.username`, and `otomi.git.password` (optionally also `branch` and `email`) Helm values configure the external repository at install time. These values are consumed once during bootstrap to populate `apl-git-config` in the `apl-secrets` namespace; they are not persisted in the values repo itself and do not need to be supplied on subsequent upgrades. | ||
|
|
||
| ### Positive consequences | ||
|
|
||
| - Platform upgrades no longer have a git-storage failure mode caused by a database restart | ||
| - Minimal installations (evaluation, CI) need fewer cluster resources | ||
| - Production teams can use a managed git service they already operate and back up | ||
|
|
||
| ### Negative consequences | ||
|
|
||
| - git-server exposes no web UI; operators cannot browse the values repo in-cluster without external tooling | ||
| - Gitea must be explicitly enabled for `codeRepo.gitService: gitea` pipelines; it is no longer available by default | ||
|
|
||
| ### Cloning the values repository from git-server | ||
|
|
||
| The values repo is publicly accessible at: | ||
|
|
||
| ``` | ||
| https://git.<domainSuffix>/otomi/values.git | ||
| ``` | ||
|
|
||
| Credentials are stored in the `apl-git-credentials` Secret in the operator namespace. Retrieve the hostname and credentials, then clone with: | ||
|
|
||
| ```sh | ||
| HOST=$(kubectl get httproute -n git-server git-server -o jsonpath='{.spec.hostnames[0]}') | ||
| PASSWORD=$(kubectl get secret -n apl-secrets apl-git-config -ojsonpath="{.data.password}" | jq -Rr '@base64d|@uri') | ||
| git clone https://otomi-admin:${PASSWORD}@${HOST}/otomi/values.git | ||
| ``` | ||
|
|
||
| ### Recreate update strategy | ||
|
|
||
| The git-server Deployment uses `strategy: type: Recreate` rather than the default `RollingUpdate`. The data volume is a `ReadWriteOnce` PVC, which can only be mounted by one node at a time. A rolling update would schedule the new pod before the old pod terminates, causing a multi-attach error that leaves the new pod stuck in `ContainerCreating` until the volume is released. Recreate terminates the old pod first, ensuring the volume is free before the new pod starts. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # Manifests directory in the values repo | ||
|
|
||
| - Status: accepted | ||
|
|
||
| ## Context and Problem Statement | ||
|
|
||
| The values repo (`ENV_DIR`) stores Helm values and platform configuration, reconciled by Helmfile. Some Kubernetes resources fall outside any Helm chart — SealedSecrets written during bootstrap, team custom manifests, and cluster-scoped operator configs such as CRDs and ClusterRoles. These resources need a GitOps-tracked home with clear ownership boundaries. | ||
|
|
||
| ## Decision Outcome | ||
|
|
||
| An `env/manifests/` directory in the values repo is the single GitOps source-of-truth for raw Kubernetes manifests. ArgoCD reconciles it directly. The structure: | ||
|
|
||
| ``` | ||
| env/manifests/ | ||
| namespaces/ | ||
| {namespace}/ ← one ArgoCD Application per directory, synced into that namespace. The corresponding ArgoCD Applicaiton creates the namespace if does not exist | ||
| sealedsecrets/ ← organisational subdirectory for SealedSecret resources | ||
| {resource-type}/ ← organisational subdirectory for a given resource kind | ||
| global/ ← one ArgoCD Application, no destination namespace (cluster-scoped resources) | ||
| crds/ | ||
| clusterroles/ | ||
| {resource-type}/ ← luster-scoped resource type | ||
| ``` | ||
|
|
||
| ### Namespace-scoped vs cluster-scoped split | ||
|
|
||
| `env/manifests/namespaces/{namespace}/` maps one-to-one to a Kubernetes namespace. `env/manifests/global/` targets no namespace and is intended for platform admins to manage cluster-scoped resources: CRDs, StorageClasses, ClusterRoles, and similar. | ||
|
|
||
| ### Operator-owned vs user-owned directories | ||
|
|
||
| Directories with an `apl-` prefix under `namespaces/` are operator-owned and must not be modified by users. The operator writes SealedSecrets into `namespaces/apl-secrets/` (platform secrets) and `namespaces/apl-users/` (user objects). All other namespace directories are user-owned. | ||
|
|
||
| ### SealedSecret bootstrap and ArgoCD handoff | ||
|
|
||
| During install, the operator writes SealedSecret manifests to disk and applies them directly to the cluster before ArgoCD is running. Once ArgoCD is deployed it picks up the same files from git and takes over continuous reconciliation. Persisting the manifests to the values repo is what enables this handoff — without it ArgoCD would have nothing to reconcile after bootstrap. | ||
|
|
||
| ### `sealedsecrets/` subdirectory | ||
|
|
||
| Within each namespace directory, SealedSecret YAMLs are grouped under a `sealedsecrets/` subdirectory for organisational clarity. There is no semantic significance; ArgoCD recurses the entire namespace directory tree regardless. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.