diff --git a/README.md b/README.md index 9fb1d97..4af291b 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,10 @@ Documentation, including overviews and tutorials can be found in the `docs` fold - See the main StreamsHub site's [README](https://github.com/streamshub/streamshub-site?tab=readme-ov-file#pulling-dependant-sources) for more details on the docs generation process. - See the Flink SQL Examples [docs](https://github.com/streamshub/flink-sql-examples/tree/main/docs) for an example of how documentation can be laid out. +## Tutorials + +Hands-on GitOps tutorials can be found under the [lessons](./lessons) folder. Start with [lessons/00-setup](./lessons/00-setup) to spin up the shared local environment, then follow each numbered lesson. + ## Example Code -Example code can be found under the [examples](./examples) folder. +Production-oriented example code can be found under the [examples](./examples) folder. diff --git a/lessons/00-setup/README.md b/lessons/00-setup/README.md new file mode 100644 index 0000000..39aed80 --- /dev/null +++ b/lessons/00-setup/README.md @@ -0,0 +1,74 @@ +# Getting Started + +This directory sets up the shared infrastructure for the GitOps tutorial series. Run `setup.sh` once before starting any lesson. + +--- + +## Prerequisites + +| Tool | Purpose | Install | +|------|---------|---------| +| **Docker** or **Podman** | Container runtime | [docker](https://docs.docker.com/get-docker/)
[podman](https://podman.io/docs/installation) | +| **KinD** (v0.20+) | Local Kubernetes clusters | [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) | +| **kubectl** | Kubernetes CLI | [kubectl](https://kubernetes.io/docs/tasks/tools/) | +| **git** | Version control | [git-scm.com](https://git-scm.com/) | +| **curl** | HTTP requests | Usually pre-installed | + +**System requirements:** ~8 GB of available memory for Docker/Podman. + +--- + +## Setup + +Run the setup script from this directory: + +```bash +./setup.sh +``` + +This takes approximately 8 minutes and creates a fully self-contained local environment: + +1. A **KinD** Kubernetes cluster (`gitops-tutorial`) running on your machine +2. **ArgoCD** — the GitOps engine that watches Git and applies changes +3. **Strimzi** — the operator that manages Kafka resources on Kubernetes +4. **Gitea** — a lightweight Git server running inside the cluster, reachable at `http://localhost:3001` +5. A Git repository in Gitea containing the base Kafka configuration +6. An ArgoCD `Application` configured to watch that repository +7. A running Kafka cluster, already deployed via the GitOps workflow + +When the script finishes, it prints the ArgoCD admin password and tells you which lesson prep script to run next. + +--- + +## Starting a lesson + +After setup completes, follow the lesson guide of your choice: +* [Lesson 1: Your First GitOps Change](../01-lesson-1/README.md) + +--- + +## Teardown + +When you are done with all lessons, delete the cluster to remove everything: + +```bash +./teardown.sh +``` + +--- + +## Troubleshooting + +**Docker is not running** +Start Docker Desktop or your container runtime and run `./setup.sh` again. + +**Port 3001 is already in use** +Another application is using port 3001. Stop that application, or change the port in `kind-config.yaml` (update both `hostPort` and the `nodePort` in `gitea/deployment.yaml` to match). + +**Kafka cluster is not becoming ready** +Kafka takes a few minutes to start, especially on machines with limited resources. Check pod status: + +```bash +kubectl get pods -n kafka-tutorial +kubectl describe kafka my-cluster -n kafka-tutorial +``` diff --git a/lessons/00-setup/argocd/application.yaml b/lessons/00-setup/argocd/application.yaml new file mode 100644 index 0000000..4eb8ff5 --- /dev/null +++ b/lessons/00-setup/argocd/application.yaml @@ -0,0 +1,23 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: kafka-tutorial + namespace: argocd +spec: + project: default + source: + repoURL: http://gitea-http.gitea.svc:3000/tutorial-user/streamshub-gitops.git + targetRevision: main + path: manifests + destination: + server: https://kubernetes.default.svc + namespace: kafka-tutorial + syncPolicy: + automated: + prune: true + selfHeal: true + managedNamespaceMetadata: + labels: + argocd.argoproj.io/managed-by: argocd + syncOptions: + - CreateNamespace=true diff --git a/lessons/00-setup/argocd/repository-secret.yaml b/lessons/00-setup/argocd/repository-secret.yaml new file mode 100644 index 0000000..2e734c1 --- /dev/null +++ b/lessons/00-setup/argocd/repository-secret.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Secret +metadata: + name: gitea-repo + namespace: argocd + labels: + argocd.argoproj.io/secret-type: repository +type: Opaque +stringData: + type: git + url: http://gitea-http.gitea.svc:3000/tutorial-user/streamshub-gitops.git + username: tutorial-user + password: tutorial-password + insecure: "true" diff --git a/lessons/00-setup/base-manifests/combined-pool.yaml b/lessons/00-setup/base-manifests/combined-pool.yaml new file mode 100644 index 0000000..ce1ca8f --- /dev/null +++ b/lessons/00-setup/base-manifests/combined-pool.yaml @@ -0,0 +1,26 @@ +apiVersion: kafka.strimzi.io/v1 +kind: KafkaNodePool +metadata: + name: combined + namespace: kafka-tutorial + labels: + strimzi.io/cluster: my-cluster +spec: + replicas: 1 + roles: + - controller + - broker + storage: + type: jbod + volumes: + - id: 0 + type: persistent-claim + size: 2Gi + deleteClaim: true + resources: + requests: + memory: 512Mi + cpu: 250m + limits: + memory: 1Gi + cpu: 500m diff --git a/lessons/00-setup/base-manifests/kafka.yaml b/lessons/00-setup/base-manifests/kafka.yaml new file mode 100644 index 0000000..9e110ec --- /dev/null +++ b/lessons/00-setup/base-manifests/kafka.yaml @@ -0,0 +1,29 @@ +apiVersion: kafka.strimzi.io/v1 +kind: Kafka +metadata: + name: my-cluster + namespace: kafka-tutorial +spec: + kafka: + version: 4.2.0 + metadataVersion: "4.2-IV0" + listeners: + - name: plain + port: 9092 + type: internal + tls: false + config: + offsets.topic.replication.factor: 1 + transaction.state.log.replication.factor: 1 + transaction.state.log.min.isr: 1 + default.replication.factor: 1 + min.insync.replicas: 1 + entityOperator: + topicOperator: + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 250m + memory: 384Mi diff --git a/lessons/00-setup/base-manifests/kustomization.yaml b/lessons/00-setup/base-manifests/kustomization.yaml new file mode 100644 index 0000000..00e35ca --- /dev/null +++ b/lessons/00-setup/base-manifests/kustomization.yaml @@ -0,0 +1,6 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - combined-pool.yaml + - kafka.yaml diff --git a/lessons/00-setup/base-manifests/namespace.yaml b/lessons/00-setup/base-manifests/namespace.yaml new file mode 100644 index 0000000..a6dde5e --- /dev/null +++ b/lessons/00-setup/base-manifests/namespace.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: kafka-tutorial + labels: + name: kafka-tutorial diff --git a/lessons/00-setup/gitea/deployment.yaml b/lessons/00-setup/gitea/deployment.yaml new file mode 100644 index 0000000..d0c96fe --- /dev/null +++ b/lessons/00-setup/gitea/deployment.yaml @@ -0,0 +1,74 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gitea + namespace: gitea + labels: + app: gitea +spec: + replicas: 1 + selector: + matchLabels: + app: gitea + template: + metadata: + labels: + app: gitea + spec: + containers: + - name: gitea + image: gitea/gitea:1.23-rootless + ports: + - containerPort: 3000 + name: http + env: + - name: GITEA__database__DB_TYPE + value: sqlite3 + - name: GITEA__security__INSTALL_LOCK + value: "true" + - name: GITEA__server__ROOT_URL + value: http://localhost:3001 + - name: GITEA__server__HTTP_PORT + value: "3000" + - name: GITEA__service__DISABLE_REGISTRATION + value: "true" + volumeMounts: + - name: data + mountPath: /var/lib/gitea + - name: config + mountPath: /etc/gitea + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi + readinessProbe: + httpGet: + path: /api/v1/version + port: 3000 + initialDelaySeconds: 10 + periodSeconds: 5 + volumes: + - name: data + emptyDir: {} + - name: config + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: gitea-http + namespace: gitea + labels: + app: gitea +spec: + type: NodePort + selector: + app: gitea + ports: + - port: 3000 + targetPort: 3000 + nodePort: 30003 + name: http diff --git a/lessons/00-setup/gitea/kustomization.yaml b/lessons/00-setup/gitea/kustomization.yaml new file mode 100644 index 0000000..ad6b3f5 --- /dev/null +++ b/lessons/00-setup/gitea/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - deployment.yaml diff --git a/lessons/00-setup/gitea/namespace.yaml b/lessons/00-setup/gitea/namespace.yaml new file mode 100644 index 0000000..deb14ec --- /dev/null +++ b/lessons/00-setup/gitea/namespace.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: gitea + labels: + name: gitea diff --git a/lessons/00-setup/kind-config.yaml b/lessons/00-setup/kind-config.yaml new file mode 100644 index 0000000..60108de --- /dev/null +++ b/lessons/00-setup/kind-config.yaml @@ -0,0 +1,8 @@ +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: + - role: control-plane + extraPortMappings: + - containerPort: 30003 + hostPort: 3001 + protocol: TCP diff --git a/lessons/00-setup/setup.sh b/lessons/00-setup/setup.sh new file mode 100755 index 0000000..b630e55 --- /dev/null +++ b/lessons/00-setup/setup.sh @@ -0,0 +1,236 @@ +#!/usr/bin/env bash +set -euo pipefail + +CLUSTER_NAME="gitops-tutorial" +GITEA_USER="tutorial-user" +GITEA_PASSWORD="tutorial-password" +GITEA_REPO="streamshub-gitops" +GITEA_HOST_PORT=3001 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +info() { echo -e "\033[1;34m[INFO]\033[0m $*"; } +warn() { echo -e "\033[1;33m[WARN]\033[0m $*"; } +error() { echo -e "\033[1;31m[ERROR]\033[0m $*" >&2; } +b64decode() { echo "$1" | base64 -d 2>/dev/null || echo "$1" | base64 -D 2>/dev/null; } + +cleanup() { + if [[ -n "${WORK_DIR:-}" ]]; then + rm -rf "${WORK_DIR}" + fi +} + +# ─── Step 1: Check prerequisites ─────────────────────────────────────────────── + +info "Checking prerequisites..." +for cmd in kind kubectl git curl; do + if ! command -v "$cmd" &>/dev/null; then + error "'$cmd' is required but not found in PATH." + exit 1 + fi +done + +if ! docker info &>/dev/null 2>&1; then + error "Docker is not running. Please start Docker Desktop or your container runtime." + exit 1 +fi + +info "All prerequisites satisfied." + +# ─── Step 2: Create KinD cluster ─────────────────────────────────────────────── + +if kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$"; then + info "KinD cluster '${CLUSTER_NAME}' already exists, skipping creation." +else + info "Creating KinD cluster '${CLUSTER_NAME}'..." + kind create cluster --name "${CLUSTER_NAME}" --config "${SCRIPT_DIR}/kind-config.yaml" +fi + +kubectl cluster-info --context "kind-${CLUSTER_NAME}" >/dev/null 2>&1 +info "Cluster is ready." + +# ─── Step 3: Install ArgoCD ──────────────────────────────────────────────────── + +info "Installing ArgoCD..." +kubectl apply -k "${SCRIPT_DIR}/../../examples/argo-cd/overlays/kubernetes" --server-side 2>/dev/null || \ + kubectl apply -k "${SCRIPT_DIR}/../../examples/argo-cd/overlays/kubernetes" --server-side + +info "Waiting for ArgoCD to be ready (this may take a few minutes)..." +kubectl rollout status deployment/argocd-server -n argocd --timeout=300s +kubectl rollout status deployment/argocd-repo-server -n argocd --timeout=300s + +# ─── Step 4: Install Strimzi operator ────────────────────────────────────────── + +info "Installing Strimzi operator..." +kubectl create namespace strimzi-operator 2>/dev/null || true +kubectl apply -k "${SCRIPT_DIR}/../../examples/operators/strimzi/overlays/kubernetes" --server-side 2>/dev/null || \ + kubectl apply -k "${SCRIPT_DIR}/../../examples/operators/strimzi/overlays/kubernetes" --server-side + +# The upstream Strimzi YAML hardcodes 'myproject' as the ServiceAccount namespace in RoleBindings. +# Kustomize namespace override doesn't fix subject references, so patch them manually. +STRIMZI_NS_PATCH='[{"op":"replace","path":"/subjects/0/namespace","value":"strimzi-operator"}]' +for rb in strimzi-cluster-operator strimzi-cluster-operator-entity-operator-delegation strimzi-cluster-operator-leader-election strimzi-cluster-operator-watched; do + kubectl patch rolebinding "${rb}" -n strimzi-operator --type=json -p "${STRIMZI_NS_PATCH}" 2>/dev/null || true +done +for crb in strimzi-cluster-operator strimzi-cluster-operator-kafka-broker-delegation strimzi-cluster-operator-kafka-client-delegation; do + kubectl patch clusterrolebinding "${crb}" --type=json -p "${STRIMZI_NS_PATCH}" 2>/dev/null || true +done + +# Tell the operator to watch the kafka-tutorial namespace and create the RoleBindings it needs there. +kubectl create namespace kafka-tutorial 2>/dev/null || true +kubectl set env deployment/strimzi-cluster-operator -n strimzi-operator STRIMZI_NAMESPACE='kafka-tutorial' + +for rb_name in strimzi-cluster-operator strimzi-cluster-operator-entity-operator-delegation strimzi-cluster-operator-watched; do + ROLE_REF=$(kubectl get rolebinding "${rb_name}" -n strimzi-operator -o jsonpath='{.roleRef.name}') + kubectl create rolebinding "${rb_name}" \ + --namespace kafka-tutorial \ + --clusterrole="${ROLE_REF}" \ + --serviceaccount=strimzi-operator:strimzi-cluster-operator 2>/dev/null || true +done + +info "Waiting for Strimzi operator to be ready..." +kubectl rollout status deployment/strimzi-cluster-operator -n strimzi-operator --timeout=300s + +# ─── Step 5: Install Gitea ───────────────────────────────────────────────────── + +info "Installing Gitea (local Git server)..." +kubectl apply -k "${SCRIPT_DIR}/gitea" + +info "Waiting for Gitea to be ready..." +kubectl rollout status deployment/gitea -n gitea --timeout=120s + +GITEA_POD=$(kubectl get pods -n gitea -l app=gitea -o jsonpath='{.items[0].metadata.name}') +info "Waiting for Gitea to finish initialising..." +for i in $(seq 1 30); do + if kubectl exec -n gitea "${GITEA_POD}" -- curl -sf http://localhost:3000/api/v1/version >/dev/null 2>&1; then + break + fi + sleep 2 +done + +# ─── Step 6: Configure Gitea ─────────────────────────────────────────────────── + +info "Configuring Gitea user and repository..." + +kubectl exec -n gitea "${GITEA_POD}" -- gitea admin user create \ + --username "${GITEA_USER}" \ + --password "${GITEA_PASSWORD}" \ + --email "tutorial@example.com" \ + --must-change-password=false 2>/dev/null || true + +info "Waiting for Gitea to be reachable on localhost:${GITEA_HOST_PORT}..." +GITEA_READY=false +for i in $(seq 1 60); do + if curl -sf "http://localhost:${GITEA_HOST_PORT}/api/v1/version" >/dev/null 2>&1; then + GITEA_READY=true + break + fi + sleep 3 +done + +if [[ "${GITEA_READY}" != "true" ]]; then + error "Gitea is not reachable on localhost:${GITEA_HOST_PORT} after 3 minutes." + error "Check pod status: kubectl get pods -n gitea" + exit 1 +fi + +TOKEN_RESPONSE=$(curl -sf -X POST \ + "http://localhost:${GITEA_HOST_PORT}/api/v1/users/${GITEA_USER}/tokens" \ + -u "${GITEA_USER}:${GITEA_PASSWORD}" \ + -H "Content-Type: application/json" \ + -d '{"name":"setup-token","scopes":["all"]}' 2>/dev/null || echo "{}") + +TOKEN=$(echo "${TOKEN_RESPONSE}" | grep -o '"sha1":"[^"]*"' | cut -d'"' -f4) +[[ -z "${TOKEN}" ]] && TOKEN=$(echo "${TOKEN_RESPONSE}" | grep -o '"token":"[^"]*"' | cut -d'"' -f4) + +if [[ -z "${TOKEN}" ]]; then + error "Failed to create Gitea access token. Response: ${TOKEN_RESPONSE}" + exit 1 +fi + +HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" -X POST \ + "http://localhost:${GITEA_HOST_PORT}/api/v1/user/repos" \ + -H "Authorization: token ${TOKEN}" \ + -H "Content-Type: application/json" \ + -d "{\"name\":\"${GITEA_REPO}\",\"auto_init\":true,\"default_branch\":\"main\"}" 2>/dev/null || echo "000") + +if [[ "${HTTP_CODE}" == "201" ]]; then + info "Repository '${GITEA_REPO}' created." +elif [[ "${HTTP_CODE}" == "409" ]]; then + info "Repository '${GITEA_REPO}' already exists." +else + warn "Repository creation returned HTTP ${HTTP_CODE} (may already exist)." +fi + +# ─── Step 7: Seed the Gitea repository with base manifests ──────────────────── + +info "Seeding Gitea repository with base manifests..." + +WORK_DIR=$(mktemp -d) +trap cleanup EXIT + +git clone "http://${GITEA_USER}:${GITEA_PASSWORD}@localhost:${GITEA_HOST_PORT}/${GITEA_USER}/${GITEA_REPO}.git" "${WORK_DIR}/repo" 2>/dev/null + +mkdir -p "${WORK_DIR}/repo/manifests" +cp "${SCRIPT_DIR}/base-manifests/"* "${WORK_DIR}/repo/manifests/" + +pushd "${WORK_DIR}/repo" >/dev/null +git add . +if git diff --cached --quiet; then + info "Manifests already present in Gitea repo, skipping commit." +else + git -c user.name="Tutorial Setup" -c user.email="setup@tutorial.local" commit -m "Initial tutorial manifests" + git push + info "Manifests pushed to Gitea." +fi +popd >/dev/null + +# ─── Step 8: Configure ArgoCD to access Gitea ───────────────────────────────── + +info "Configuring ArgoCD to watch the Gitea repository..." +kubectl apply -f "${SCRIPT_DIR}/argocd/repository-secret.yaml" +kubectl apply -f "${SCRIPT_DIR}/argocd/application.yaml" + +# ─── Step 9: Wait for initial sync ──────────────────────────────────────────── + +info "Waiting for ArgoCD to sync the application (this may take a minute)..." +for i in $(seq 1 60); do + SYNC_STATUS=$(kubectl get application kafka-tutorial -n argocd -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "Unknown") + if [[ "${SYNC_STATUS}" == "Synced" ]]; then + break + fi + sleep 5 +done + +if [[ "${SYNC_STATUS}" != "Synced" ]]; then + warn "ArgoCD has not synced yet (status: ${SYNC_STATUS}). It may still be processing." + warn "Check status with: kubectl get application kafka-tutorial -n argocd" +else + info "ArgoCD application is synced." +fi + +info "Waiting for Kafka cluster to be ready (this may take several minutes)..." +kubectl wait kafka/my-cluster --for=condition=Ready -n kafka-tutorial --timeout=600s 2>/dev/null || \ + warn "Kafka cluster is not yet ready. It may still be starting — check with: kubectl get kafka -n kafka-tutorial" + +# ─── Step 10: Print instructions ─────────────────────────────────────────────── + +ARGOCD_PASSWORD=$(b64decode "$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}')") + +echo "" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "" +info "Setup complete! Your tutorial environment is ready." +echo "" +echo " Next step: run the prep script for the lesson you want to start:" +echo " cd ../01-lesson-1 && ./prep.sh" +echo "" +echo " ArgoCD Dashboard (open in a separate terminal):" +echo " kubectl port-forward svc/argocd-server -n argocd 8080:443" +echo " URL: https://localhost:8080" +echo " Username: admin" +echo " Password: ${ARGOCD_PASSWORD}" +echo "" +echo " Cleanup when done with all lessons:" +echo " ./teardown.sh" +echo "" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" diff --git a/lessons/00-setup/teardown.sh b/lessons/00-setup/teardown.sh new file mode 100755 index 0000000..71d7e01 --- /dev/null +++ b/lessons/00-setup/teardown.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail + +CLUSTER_NAME="gitops-tutorial" + +echo "Deleting KinD cluster '${CLUSTER_NAME}'..." +kind delete cluster --name "${CLUSTER_NAME}" +echo "Done. All tutorial resources have been removed." diff --git a/lessons/01-lesson-1/README.md b/lessons/01-lesson-1/README.md new file mode 100644 index 0000000..d933057 --- /dev/null +++ b/lessons/01-lesson-1/README.md @@ -0,0 +1,321 @@ +# Lesson 1: Your First GitOps Change + +**Series:** GitOps with StreamsHub — 3-part series +**Time:** ~20 minutes (plus ~8 minutes for first-time setup) + +--- + +## What you will learn + +By the end of this lesson you will understand: + +- What the GitOps workflow looks like in practice +- How ArgoCD watches a Git repository and automatically applies changes to a Kubernetes cluster +- How Strimzi manages Kafka resources declaratively + +You will do this by making a real change — adding a Kafka topic — and watching it flow automatically from Git to a running cluster, without ever running `kubectl apply` yourself. + +--- + +## Prerequisites + +If you haven't done this yet, run through the [Getting Started](../00-setup/README.md) guide. You only need to do this once. (takes ~8 minutes) + +--- + +## Background: The GitOps idea in one paragraph + +In traditional operations you make changes to a running system by running commands directly against it — `kubectl apply`, a config panel, an API call. GitOps flips this around: a Git repository is the single source of truth for what the system should look like. A tool (in this case ArgoCD) watches the repository and continuously reconciles the live system to match. If Git says a topic should exist, the topic exists. If you remove it from Git, it disappears from the cluster. You never touch the system directly; you only change Git. + +--- + +## Setup + +Run the prep script from this directory: + +```bash +./prep.sh +``` + +This takes under a minute. It resets the Gitea repository to the lesson-1 starting state and confirms that ArgoCD has synced. When it finishes it prints the Gitea and ArgoCD credentials. + +You can re-run `./prep.sh` at any time to reset back to the lesson starting state — useful if you make a mistake and want to start over without re-running the full setup. + +--- + +## Part 1: Look at what's already running + +Before you make any changes, take a moment to explore the environment. This is where the lesson starts: everything you are about to see was deployed by ArgoCD from Git. + +### Clone the repository + +The Gitea server is running inside the cluster and is exposed on port 3001. Clone the repository it holds: + +```bash +git clone http://tutorial-user:tutorial-password@localhost:3001/tutorial-user/streamshub-gitops.git /tmp/gitops-lesson-1 +cd /tmp/gitops-lesson-1 +``` + +This is the repository ArgoCD is watching. Any change you push here will be picked up and applied to the cluster. + +### Check the running Kafka cluster + +```bash +kubectl get kafka -n kafka-tutorial +``` + +Expected output: + +``` +NAME DESIRED KAFKA REPLICAS DESIRED ZK REPLICAS READY METADATA STATE WARNINGS +my-cluster 1 True KRaft +``` + +`READY: True` means Kafka is up. Now look at *how* this cluster got here — open `manifests/kafka.yaml`: + +```bash +cat manifests/kafka.yaml +``` + +That YAML file, committed to the Git repository, is the complete description of this Kafka cluster. ArgoCD read it from Git and the Strimzi operator created the cluster from it. You didn't run any `kubectl apply` commands — the setup script pushed the file to Git and ArgoCD took it from there. + +### Check for Kafka topics + +```bash +kubectl get kafkatopic -n kafka-tutorial +``` + +You should see no topics listed (or only internal Strimzi housekeeping topics). There is no application topic yet. + +### Understand the kustomization file + +ArgoCD uses [Kustomize](https://kustomize.io/) to decide which YAML files to deploy. The entry point is `manifests/kustomization.yaml`: + +```bash +cat manifests/kustomization.yaml +``` + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - combined-pool.yaml + - kafka.yaml +``` + +This tells ArgoCD: "deploy these three files." Notice that `topic.yaml` is not listed, even though the file exists in the repository: + +```bash +ls manifests/ +``` + +`topic.yaml` is there — but because it is not in `kustomization.yaml`, ArgoCD ignores it. The cluster's state is determined entirely by what Kustomize includes, not by what files happen to exist in the folder. + +--- + +## Part 2: Make your first GitOps change + +Your application team needs a Kafka topic to send and receive messages. Your job is to add it to the cluster — the GitOps way. + +### Look at the topic definition + +Open `manifests/topic.yaml`: + +```bash +cat manifests/topic.yaml +``` + +```yaml +apiVersion: kafka.strimzi.io/v1 +kind: KafkaTopic +metadata: + name: my-first-topic + namespace: kafka-tutorial + labels: + strimzi.io/cluster: my-cluster +spec: + partitions: 3 + replicas: 1 + config: + retention.ms: "86400000" + segment.bytes: "1073741824" +``` + +This defines a topic called `my-first-topic` with 3 partitions. The `strimzi.io/cluster: my-cluster` label tells Strimzi which cluster this topic belongs to. Messages will be retained for 24 hours (`86400000` ms). + +The file is ready — you just need to tell Kustomize to include it. + +### Edit kustomization.yaml + +Open `manifests/kustomization.yaml` in your editor and add `- topic.yaml` as the last entry in the resources list: + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - combined-pool.yaml + - kafka.yaml + - topic.yaml +``` + +Save the file. + +### Commit and push + +```bash +git add manifests/kustomization.yaml +git commit -m "Add my-first-topic Kafka topic" +git push +``` + +That's it. You've made your GitOps change. The commit is now in the repository that ArgoCD is watching. + +--- + +## Part 3: Watch the GitOps loop + +ArgoCD polls the repository every 3 minutes by default. You can watch it detect and apply the change: + +```bash +kubectl get application kafka-tutorial -n argocd -w +``` + +Watch the `SYNC STATUS` column. It will move from `Synced` → `OutOfSync` (when ArgoCD detects your push) → `Synced` again (when it has applied the change). Press `Ctrl+C` once you see it settle back to `Synced`. + +> **Don't want to wait?** You can trigger an immediate sync — see the [Force an immediate sync](#optional-force-an-immediate-sync) section below. + +### Verify the topic was created + +Once ArgoCD shows `Synced`, check that the topic now exists: + +```bash +kubectl get kafkatopic my-first-topic -n kafka-tutorial +``` + +Expected output: + +``` +NAME CLUSTER PARTITIONS REPLICATION FACTOR READY +my-first-topic my-cluster 3 1 True +``` + +`READY: True` confirms that Strimzi's Topic Operator received the `KafkaTopic` resource from ArgoCD and created the topic inside the Kafka broker. + +**You just deployed a Kafka topic using GitOps.** The change went from your editor, through Git, through ArgoCD, and into the cluster — automatically. + +--- + +## How it worked + +Here is the full sequence of what happened after you ran `git push`: + +``` +git push + └─▶ Gitea (Git server inside the cluster) receives the commit + +ArgoCD polls Gitea every 3 minutes + └─▶ ArgoCD detects that kustomization.yaml now includes topic.yaml + └─▶ ArgoCD renders the Kustomize manifests (now four resources instead of three) + └─▶ ArgoCD compares the rendered state to what is live in the cluster + └─▶ ArgoCD applies the diff — creating the KafkaTopic resource + +Strimzi Topic Operator watches for KafkaTopic resources + └─▶ Strimzi sees the new KafkaTopic and creates the topic inside the Kafka broker +``` + +The key point: **you never ran `kubectl apply`**. You changed Git, and the system reconciled itself to match. This is what GitOps means in practice. + +--- + +## Optional: View the ArgoCD dashboard + +ArgoCD has a web UI where you can see the application's resource tree, sync history, and current state. In a separate terminal: + +```bash +kubectl port-forward svc/argocd-server -n argocd 8080:443 +``` + +Open [https://localhost:8080](https://localhost:8080) in your browser (accept the self-signed certificate warning). + +Retrieve the admin password: + +```bash +kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d; echo +``` + +Log in with username `admin` and the password above. Click the `kafka-tutorial` application to see the full resource tree — Namespace, KafkaNodePool, Kafka, and now KafkaTopic, all managed by ArgoCD from a single Git repository. + +--- + +## Optional: Force an immediate sync + +If you don't want to wait up to 3 minutes for the next poll, you can trigger an immediate sync using `kubectl`: + +```bash +kubectl annotate application kafka-tutorial -n argocd \ + argocd.argoproj.io/refresh=normal --overwrite +``` + +This tells ArgoCD to poll Gitea right now. The annotation is cleared automatically once the sync completes. + +--- + +## Cleanup + +When you are done with all lessons, delete the cluster to remove everything: + +```bash +../00-setup/teardown.sh +``` + +This deletes the KinD cluster and all resources within it. Clean up the cloned repo too: + +```bash +rm -rf /tmp/gitops-lesson-1 +``` + +--- + +## What's next + +In **Lesson 2: Promotion from Staging to Production**, you will build on this environment by creating separate staging and production configurations and walk through the process of promoting a change through environments — the same Git-as-source-of-truth principle, applied to multi-environment workflows. + +--- + +## Troubleshooting + +**Infrastructure is not running** +If `./prep.sh` reports that the cluster or Kafka is not found, you need to run the setup script first: `../00-setup/setup.sh`. See [Getting Started](../00-setup/README.md) for setup troubleshooting. + +**Kafka cluster is not becoming ready** +Kafka takes a few minutes to start, especially on machines with limited resources. Check pod status and events: + +```bash +kubectl get pods -n kafka-tutorial +kubectl describe kafka my-cluster -n kafka-tutorial +``` + +**ArgoCD is not syncing** +Check the application for error messages: + +```bash +kubectl get application kafka-tutorial -n argocd -o yaml +``` + +If Gitea is unreachable from inside the cluster, verify the Gitea pod is running: + +```bash +kubectl get pods -n gitea +``` + +**Topic is not appearing after sync** +Check that the `kustomization.yaml` edit was saved and committed correctly: + +```bash +git log --oneline -3 +git show HEAD:manifests/kustomization.yaml +``` + +Confirm `- topic.yaml` appears in the resources list. If it does not, re-edit, commit, and push. diff --git a/lessons/01-lesson-1/lesson-manifests/combined-pool.yaml b/lessons/01-lesson-1/lesson-manifests/combined-pool.yaml new file mode 100644 index 0000000..ce1ca8f --- /dev/null +++ b/lessons/01-lesson-1/lesson-manifests/combined-pool.yaml @@ -0,0 +1,26 @@ +apiVersion: kafka.strimzi.io/v1 +kind: KafkaNodePool +metadata: + name: combined + namespace: kafka-tutorial + labels: + strimzi.io/cluster: my-cluster +spec: + replicas: 1 + roles: + - controller + - broker + storage: + type: jbod + volumes: + - id: 0 + type: persistent-claim + size: 2Gi + deleteClaim: true + resources: + requests: + memory: 512Mi + cpu: 250m + limits: + memory: 1Gi + cpu: 500m diff --git a/lessons/01-lesson-1/lesson-manifests/kafka.yaml b/lessons/01-lesson-1/lesson-manifests/kafka.yaml new file mode 100644 index 0000000..9e110ec --- /dev/null +++ b/lessons/01-lesson-1/lesson-manifests/kafka.yaml @@ -0,0 +1,29 @@ +apiVersion: kafka.strimzi.io/v1 +kind: Kafka +metadata: + name: my-cluster + namespace: kafka-tutorial +spec: + kafka: + version: 4.2.0 + metadataVersion: "4.2-IV0" + listeners: + - name: plain + port: 9092 + type: internal + tls: false + config: + offsets.topic.replication.factor: 1 + transaction.state.log.replication.factor: 1 + transaction.state.log.min.isr: 1 + default.replication.factor: 1 + min.insync.replicas: 1 + entityOperator: + topicOperator: + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 250m + memory: 384Mi diff --git a/lessons/01-lesson-1/lesson-manifests/kustomization.yaml b/lessons/01-lesson-1/lesson-manifests/kustomization.yaml new file mode 100644 index 0000000..00e35ca --- /dev/null +++ b/lessons/01-lesson-1/lesson-manifests/kustomization.yaml @@ -0,0 +1,6 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - combined-pool.yaml + - kafka.yaml diff --git a/lessons/01-lesson-1/lesson-manifests/namespace.yaml b/lessons/01-lesson-1/lesson-manifests/namespace.yaml new file mode 100644 index 0000000..a6dde5e --- /dev/null +++ b/lessons/01-lesson-1/lesson-manifests/namespace.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: kafka-tutorial + labels: + name: kafka-tutorial diff --git a/lessons/01-lesson-1/lesson-manifests/topic.yaml b/lessons/01-lesson-1/lesson-manifests/topic.yaml new file mode 100644 index 0000000..c95c6fa --- /dev/null +++ b/lessons/01-lesson-1/lesson-manifests/topic.yaml @@ -0,0 +1,13 @@ +apiVersion: kafka.strimzi.io/v1 +kind: KafkaTopic +metadata: + name: my-first-topic + namespace: kafka-tutorial + labels: + strimzi.io/cluster: my-cluster +spec: + partitions: 3 + replicas: 1 + config: + retention.ms: "86400000" + segment.bytes: "1073741824" diff --git a/lessons/01-lesson-1/prep.sh b/lessons/01-lesson-1/prep.sh new file mode 100755 index 0000000..10dfdf3 --- /dev/null +++ b/lessons/01-lesson-1/prep.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +set -euo pipefail + +CLUSTER_NAME="gitops-tutorial" +GITEA_USER="tutorial-user" +GITEA_PASSWORD="tutorial-password" +GITEA_REPO="streamshub-gitops" +GITEA_HOST_PORT=3001 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +info() { echo -e "\033[1;34m[INFO]\033[0m $*"; } +warn() { echo -e "\033[1;33m[WARN]\033[0m $*"; } +error() { echo -e "\033[1;31m[ERROR]\033[0m $*" >&2; } +b64decode() { echo "$1" | base64 -d 2>/dev/null || echo "$1" | base64 -D 2>/dev/null; } + +cleanup() { + if [[ -n "${WORK_DIR:-}" ]]; then + rm -rf "${WORK_DIR}" + fi +} + +# ─── Step 1: Validate infrastructure ────────────────────────────────────────── + +info "Validating tutorial infrastructure..." + +if ! kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$"; then + error "KinD cluster '${CLUSTER_NAME}' is not running." + error "Please run the setup script first: ../00-setup/setup.sh" + exit 1 +fi + +if ! kubectl get kafka my-cluster -n kafka-tutorial &>/dev/null; then + error "Kafka cluster 'my-cluster' not found in namespace 'kafka-tutorial'." + error "Please run the setup script first: ../00-setup/setup.sh" + exit 1 +fi + +if ! curl -sf "http://localhost:${GITEA_HOST_PORT}/api/v1/version" >/dev/null 2>&1; then + error "Gitea is not reachable on localhost:${GITEA_HOST_PORT}." + error "Please run the setup script first: ../00-setup/setup.sh" + exit 1 +fi + +info "Infrastructure checks passed." + +# ─── Step 2: Reset Gitea repo to lesson-1 starting state ────────────────────── + +info "Resetting Gitea repository to lesson-1 starting state..." + +WORK_DIR=$(mktemp -d) +trap cleanup EXIT + +git clone "http://${GITEA_USER}:${GITEA_PASSWORD}@localhost:${GITEA_HOST_PORT}/${GITEA_USER}/${GITEA_REPO}.git" "${WORK_DIR}/repo" 2>/dev/null + +rm -rf "${WORK_DIR}/repo/manifests" +mkdir -p "${WORK_DIR}/repo/manifests" +cp "${SCRIPT_DIR}/lesson-manifests/"* "${WORK_DIR}/repo/manifests/" + +pushd "${WORK_DIR}/repo" >/dev/null +git add . +if git diff --cached --quiet; then + info "Gitea repo is already in lesson-1 starting state, skipping commit." +else + git -c user.name="Tutorial Setup" -c user.email="setup@tutorial.local" commit -m "Reset to lesson-1 starting state" + git push + info "Lesson-1 starting state pushed to Gitea." +fi +TARGET_REVISION=$(git rev-parse HEAD) +popd >/dev/null + +# ─── Step 3: Wait for ArgoCD sync ───────────────────────────────────────────── + +info "Waiting for ArgoCD to sync..." + +# Trigger an immediate refresh so ArgoCD picks up the new commit without waiting up to 3 minutes. +kubectl annotate application kafka-tutorial -n argocd argocd.argoproj.io/refresh=normal --overwrite >/dev/null 2>&1 || true + +SYNC_STATUS="Unknown" +for i in $(seq 1 24); do + CURRENT_REVISION=$(kubectl get application kafka-tutorial -n argocd -o jsonpath='{.status.sync.revision}' 2>/dev/null || echo "") + SYNC_STATUS=$(kubectl get application kafka-tutorial -n argocd -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "Unknown") + if [[ "${CURRENT_REVISION}" == "${TARGET_REVISION}" && "${SYNC_STATUS}" == "Synced" ]]; then + break + fi + sleep 5 +done + +if [[ "${SYNC_STATUS}" != "Synced" ]]; then + warn "ArgoCD has not synced yet (status: ${SYNC_STATUS})." + warn "Check status with: kubectl get application kafka-tutorial -n argocd" +else + info "ArgoCD application is synced." +fi + +# ─── Step 4: Print starting instructions ────────────────────────────────────── + +ARGOCD_PASSWORD=$(b64decode "$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}')") + +echo "" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "" +info "Lesson 1 is ready. Open README.md and follow the lesson steps." +echo "" +echo " Gitea (your Git server): http://localhost:${GITEA_HOST_PORT}" +echo " Username: ${GITEA_USER} Password: ${GITEA_PASSWORD}" +echo "" +echo " ArgoCD Dashboard (open in a separate terminal):" +echo " kubectl port-forward svc/argocd-server -n argocd 8080:443" +echo " URL: https://localhost:8080" +echo " Username: admin" +echo " Password: ${ARGOCD_PASSWORD}" +echo "" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" diff --git a/lessons/README.md b/lessons/README.md new file mode 100644 index 0000000..028ce98 --- /dev/null +++ b/lessons/README.md @@ -0,0 +1,38 @@ +# GitOps with StreamsHub + +A hands-on tutorial series for learning GitOps with Apache Kafka on Kubernetes. Each lesson builds on the last, using real tools — ArgoCD, Strimzi, and a local Git server — running entirely on your machine. + +--- + +## How the series works + +A shared cluster runs for the whole series. You set it up once, then run a short prep script before each lesson to put the environment in the right starting state. Lessons take 20–30 minutes each and involve making real changes to a Git repository and watching the effects propagate to the cluster automatically. + +--- + +## Lessons + +| # | Title | What you will do | +|---|-------|-----------------| +| [Getting Started](00-setup/README.md) | Environment setup | Install the shared cluster, ArgoCD, Strimzi, and a local Git server. Run this once before any lesson. | +| [Lesson 1](01-lesson-1/README.md) | Your First GitOps Change | Add a Kafka topic by editing a file in Git and watch ArgoCD deploy it automatically — without running `kubectl apply`. | + +--- + +## Start here + +**1. Complete the one-time setup** (~8 minutes): + +```bash +cd 00-setup +./setup.sh +``` + +**2. Run the prep script for Lesson 1**, then open its README: + +```bash +cd 01-lesson-1 +./prep.sh +``` + +See [Getting Started](00-setup/README.md) for prerequisites and troubleshooting.