Various sysadmin files for KernelCI project

Deprecation Notice

Many files and directories in this repository are deprecated and will be removed in 1 month if no objections are raised. These files are not used anymore and are kept only for reference. Please check their respective sections for more details. Date when they will be (likely)removed: 2025-08-23

Root directory

Various files in root directory:

• ansible: Deprecated Ansible script from legacy KernelCI project, not used anymore. DEPRECATED: Will be removed in 1 month if no objections.
• chromeos.kernelci.org: Deprecated ChromeOS staging environment, not used anymore. DEPRECATED: Will be removed in 1 month if no objections.
• job.py: Deprecated Python script to run/control KernelCI jobs in legacy/jenkins, not used anymore. DEPRECATED: Will be removed in 1 month if no objections.
• kernel.py: Python script to update kernel mirror in KernelCI project.
• kernelci.org: Deprecated legacy production script, not used anymore. DEPRECATED: Will be removed in 1 month if no objections.
• pending.py: Script to handle pending PR and merge them into staging environment. Likely deprecated.
• staging.kernelci.org: Script to run staging environment for KernelCI project. Updated to use new workflows, mostly initiate github actions workflows, and then update local docker images
• update.py: Old script for staging environment, not used anymore. DEPRECATED: Will be removed in 1 month if no objections.

data/staging.ini

This file contains permit-list for users allowed to access the staging environment (their PRs are automatically deployed).

k8s/*

Mostly obsolete old recipes for legacy KernelCI Kubernetes cluster (builders) DEPRECATED: Will be removed in 1 month if no objections.

kernelci/*

Probably part of legacy scripts, some library. DEPRECATED: Will be removed in 1 month if no objections.

kubernetes/*

Various Kubernetes manifests and scripts for KernelCI project. Check README.md in this directory for more details.

deploy.cfg

This file contains the deployment configuration for the KernelCI project. Essential part of api-pipeline-deploy.sh script.

api-pipeline-deploy.sh

This script is used to deploy the KernelCI API and Pipeline services to a Kubernetes cluster. It sets up the necessary namespaces, configure IP, DNS name, and other parameters for the services. It might do complete deployment, or just update the existing deployment (secrets, configmaps, etc.).

api-production-update.sh

This script is used to update the KernelCI API and Pipeline services in a production environment with updates from the main branch. It also updates configuration configmap. This script is intended to be run as part of github actions workflow, but can be run manually as well.

create_kci_k8s_azure_build.sh

This is initial version (not complete yet) of script to create KernelCI Kubernetes cluster on Azure for builders.

extract_secret.py

Supplementary script to extract secrets encoded in base64 from Kubernetes cluster.

caching/*

This directory contains kubernetes manifests for caching services used by KernelCI builders, to reduce load on storage. Right now it is caching only linux-firmware downloads.

localinstall

This directory contains scripts and configuration files for local installation of KernelCI services. Please check included README.md for more details.

playbooks/*

This directory contains Ansible playbooks and roles for deploying and managing KernelCI services. Right now we have only complete playbook for production server, incomplete for monitoring server, and some roles for monitoring in all directory (node_exporter listening on port 2000)

playbooks/dashboard

Playbook for the web dashboard host, which runs the production dashboard (dashboard.kernelci.org, d.kernelci.org) and the staging one (staging.dashboard.kernelci.org) side by side:

cd playbooks/dashboard
ansible-playbook -i inventory.yaml main.yml

The compose files here are not templated: both stacks run the compose file that ships in the kernelci/dashboard checkout, so the playbook owns the checkouts, the environment files, the nginx front end and certificate renewal instead.

Roles:

• common: base packages, and Docker only when the host does not have it.
• nginx: the TLS front end and the anti-abuse rules (robots.txt, the user-agent blocklist and the fake_macos_blocked geo check). The blocklist file itself is treated as data: created if missing, never rewritten, so a run cannot wipe entries added by hand.
• dashboard-production: prebuilt ghcr.io images. Refresh them with -e dashboard_production_pull=true.
• dashboard-staging: builds its images on the host. Rebuild with -e dashboard_staging_build=true.
• uptime-kuma: the status page, bound to loopback.
• certbot: the renewal timer. Run once with -e certbot_verify_renewal=true to have the play prove renewal still works.

Both checkouts sit on a detached HEAD at a reviewed commit, and the playbook leaves them there. Set dashboard_production_version or dashboard_staging_version to move one deliberately; nothing fast-forwards a running dashboard as a side effect.

The .env files are never written from this repository: they hold the Django secret key, the database password, the Discord webhook and SMTP credentials. The play fails with instructions if one is missing rather than inventing it.

Note on certificates: this host renews with the pip certbot in /usr/local/bin, not the Debian package. The package ships only /etc/cron.d/certbot, which can never run here (cron is not installed, /usr/bin/certbot does not exist, and the entry skips itself under systemd), so the certbot role installs its own certbot-renew service and timer.

playbooks/kcidb-production

Playbook for the production KCIDB submission endpoint (db.kernelci.org), run with:

cd playbooks/kcidb-production
ansible-playbook -i inventory.yaml main.yml

It is not a copy of kcidb-staging, because the host is not built the same way: there is no caddy in front (kcidb-rest binds 80/443 itself and renews its own certificate through its built-in ACME client), no dashboard (that runs on a different host), the images are prebuilt from ghcr.io rather than built locally, and the database is Azure managed Postgres, so the self-hosted db/dbinit compose services stay behind their profile. The stack also keeps its deployed location in /home/azureuser/kcidb-ng, next to ~19G of spool and archive data, rather than the /srv path staging uses.

Unlike the kcidb-staging common role, this playbook does not move sshd to port 22022 and does not rewrite root's authorized_keys. This host answers on 22, which is what the Azure network security group publishes.

Roles:

• common: base packages, and Docker only when the host does not already have it.
• kcidb-ng: data directories, the compose file, and the stack itself. The .env file is only ever created, never rewritten: it holds the JWT secret, the storage token and the database password. Images are not refreshed by default; deploy current builds with -e kcidb_pull=always.
• archivarius: the submissions archiver built from tools/submissions_archivarius. Install or upgrade it with -e archivarius_deb=/path/to/*.deb.
• dozzle: the container log viewer, bound to loopback and reached over an ssh tunnel.

playbooks/production

Playbook for the production web/storage server (vm-production-2025, reachable as docs.kernelci.org:22022), run with:

cd playbooks/production
ansible-playbook -i inventory.yaml main.yml

Roles:

• webserver: nginx vhosts for docs, storage, chromeos storage (decommissioned, serves HTTP 410), files and the MCP endpoint, plus the shared Let's Encrypt certificate. Every vhost is rendered from templates/vhost.j2 out of the vhosts variable, so the recipe is the source of truth. Names that do not resolve yet are left out of the certificate request instead of failing the run.
• storage: the kernelci-storage container behind files.kernelci.org. The config file is created only if missing, since the live one holds credentials.
• mcp: public read-only KernelCI MCP server (kci-dev mcp, streamable HTTP) in a virtualenv under /srv/kci-mcp, running as the kci-mcp system user and bound to 127.0.0.1:8000. It is published by nginx. kci-dev's HTTP transport has no authentication, so the config deliberately carries no pipeline URL and no token: without them kci-dev never registers retry_job or trigger_checkout, and the public endpoint can only read.
• common: host bootstrap (packages, /data mount, ssh port). Excluded from the default run, it is only meant for provisioning a new host.

The MCP endpoint is temporarily served at https://storage.chromeos.kernelci.org/mcp, reusing a decommissioned vhost that already has a certificate. Once an A record for mcp.kernelci.org exists, set enabled: true on that vhost in roles/webserver/vars/main.yml, drop mcp_endpoint from the chromeos vhost and re-run the webserver role.

tools/*

This directory contains various tools and scripts used in the KernelCI project.

azure_blob_cleanup.py

Script to clean up old blobs in Azure Blob Storage, used for KernelCI artifacts.

azure_files_cleanup.py

Script to clean up old files in Azure File Storage, used for KernelCI artifacts. As we are not using Azure File Storage anymore, this script is going to be removed in the future. DEPRECATED: Will be removed in 1 month if no objections.

buildroot_checksum.sh

Script to calculate checksums for Buildroot images used in KernelCI.

docker_images_cleanup.py

Script to maintain Docker images in Docker hub, to clean up old images.

firmware-updater.py

Script to update linux-firmware tarball, stored on production storage, used by KernelCI builders.

kci-dockerwatch.py

Attempt to monitor and log Docker images in KernelCI project. Not working well, it is IMHO not useful. DEPRECATED: Will be removed in 1 month if no objections.

kci-k8swatch.py

Same for kubernetes cluster, not working well, not useful. DEPRECATED: Will be removed in 1 month if no objections.

legacy_watchdog.py

This is script to monitor legacy services. Not used anymore, as we are not running legacy services. DEPRECATED: Will be removed in 1 month if no objections.

managed_identity.sh

Script to manage Azure AD identities for KernelCI VM. So basically you can control Azure K8S cluster without installing credentials, VM by itself is credential. Unfortunately only for Azure K8S cluster, not for other cloud providers.

monitor-containers.py

One more legacy docker monitoring script, not used anymore. DEPRECATED: Will be removed in 1 month if no objections.

postgres_team_sync.py

Synchronizes read-only Postgres logins with the dashboard team file (kernelci/dashboard:.github/dashboard-team). Users in the list get a role with a generated password and read-only access to the public schema; managed roles no longer listed are dropped. Reserved accounts (kcidb*, superusers, pg_*, the connecting role) are never touched. Defaults to a dry run, --apply commits.

Connection settings come from the environment, loaded from a .env file so it can share the one the kcidb-ng stack uses:

./tools/postgres_team_sync.py --env-file /home/azureuser/kcidb-ng/.env

It reads DB_HOST, DB_USER, DB_PASSWORD, DB_NAME, optionally DB_PORT, GITHUB_TOKEN and TEAM_SYNC_EMAIL_TO. Real environment variables override the file. A legacy .dbauth file is still honoured when no DB_* values are set. Previously ran only on the production KCIDB host, outside version control.

ssh_key_sync.py

Rebuilds one local user's authorized_keys from the GitHub public keys of every user in the same dashboard team file, plus a static key list. Fully manages the file: it backs up the old content and replaces it atomically, and refuses to write when it produced suspiciously few keys. Defaults to a dry run, --apply writes, --install sets up a systemd service and hourly timer. It can also rotate Postgres role passwords, which is off by default.

Settings come from the environment or a .env file: GITHUB_TOKEN, DISCORD_WEBHOOK_URL, POSTGRES_HOST/PORT/USER/DATABASE/PASSWORD, POSTGRES_EMAIL_TO and SMTP_PASSWORD. Nothing secret lives in the script. Previously ran only on the production KCIDB host, outside version control.