diff --git a/.gitignore b/.gitignore index 8614146e8..ded6fc6fb 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,9 @@ MANIFEST # to change depending on the environment. *.tfvars *.tfvars.json +# exclude the provider.tf file on arc-enabled k8s microhack (contains subscription id) +03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/**/provider.tf + # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. @@ -202,4 +205,6 @@ id_rsa.pub *.private.pub # Ignore ARM Parameter files -*.parameters.json \ No newline at end of file +*.parameters.json +03-Azure/01-05-SAP/01_MicroHack-SAP-Cashflow-Prediction/SAP-Data-MicroHack.pptx +03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/templates/parameters-my.json diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/Readme.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/Readme.md index 82b2fa481..f5aca9894 100644 --- a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/Readme.md +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/Readme.md @@ -1 +1,106 @@ -Coming soon stay tuned \ No newline at end of file +![image](img/banner.png) + +# MicroHack Azure Arc-enabled Kubernetes + +* [**MicroHack Introduction**](#microhack-introduction) + * [What is Azure Arc for Kubernetes?](#what-is-azure-arc-for-kubernetes) +* [**MicroHack Context**](#microhack-context) +* [**Objectives**](#objectives) +* [**General Prerequisites**](#general-prerequisites) +* [**MicroHack Challenges**](#microhack-challenges) + * [Challenge 01 - Onboarding your Kubernetes Cluster](challenges/challenge-01.md)) + * [Challenge 02 - Enable Azure Monitor for Containers](challenges/challenge-02.md) + * [Challenge 03 - Deploy CPU based Large & Small Language Models (LLM/SLM) on Azure Arc-enabled Kubernetes](challenges/challenge-03.md) + * [Challenge 04 - Deploy SQL Managed Instance](challenges/challenge-04.md) + * [Challenge 05 - Configure GitOps for Cluster Management](challenges/challenge-05.md) +* [**Contributors**](#contributors) + + +## MicroHack Introduction + +### What is Azure Arc for Kubernetes? + +Azure Arc-enabled Kubernetes allows you to attach Kubernetes clusters running anywhere so that you can manage and configure them in Azure. By managing all of your Kubernetes resources in a single control plane, you can enable a more consistent development and operation experience, helping you run cloud-native apps anywhere and on any Kubernetes platform. + +![image](./img/architectural-overview.png) + +Once your Kubernetes clusters are connected to Azure, you can: + +- View all connected Kubernetes clusters for inventory, grouping, and tagging, along with your Azure Kubernetes Service (AKS) clusters. + +- Configure clusters and deploy applications using GitOps-based configuration management. + +- View and monitor your clusters using Azure Monitor for containers. + +- Enforce threat protection using Microsoft Defender for Kubernetes. + +- Ensure governance through applying policies with Azure Policy for Kubernetes. + +- Grant access and connect to your Kubernetes clusters from anywhere, and manage access by using Azure role-based access control (RBAC) on your cluster. + +- Deploy machine learning workloads using Azure Machine Learning for Kubernetes clusters. + +- Deploy and manage Kubernetes applications from Azure Marketplace. + +- Deploy Azure PaaS services that allow you to take advantage of specific hardware, comply with data residency requirements, or enable new scenarios. Examples of services include: + + - Azure Arc-enabled data services + - Azure Machine Learning for Kubernetes clusters + - Workload Orchestration + - Event Grid on Kubernetes + - App Services on Azure Arc + - Open Service Mesh + +## MicroHack Context + +This MicroHack is a challenge-based experience which will walk you through the onboarding process and step by step enabling additional use cases. + +💡 *Optional*: Have a look at the following resources after completing this lab to deepen your learning: + +* [Azure Arc-enabled Kubernetes documentation](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/) +* [Azure Arc Jumpstart - Arc-enabled Kubernetes](https://jumpstart.azure.com/azure_arc_jumpstart/azure_arc_k8s) +* [Azure Arc Jumpstart - Data Services](https://jumpstart.azure.com/azure_arc_jumpstart/azure_arc_data) +* [Azure Arc - Workload Orchestration](https://learn.microsoft.com/en-us/azure/azure-arc/workload-orchestration/overview) +* [Azure Arc Jumpstart - Machine Learning](https://jumpstart.azure.com/azure_arc_jumpstart/azure_arc_ml) +* [Azure Arc Jumpstart - Iot Operations](https://jumpstart.azure.com/azure_arc_jumpstart/azure_edge_iot_ops) +* [Speed Innovation with Arc-enabled Kubernetes Applications](https://techcommunity.microsoft.com/blog/azurearcblog/speed-innovation-with-arc-enabled-kubernetes-applications/4298658) +* [Azure Arc-Enabled Kubernetes now available on Azure Marketplace](https://techcommunity.microsoft.com/blog/azurearcblog/azure-arc-enabled-kubernetes-now-available-on-azure-marketplace/4034060) +* [Introduction to Azure Arc landing zone accelerator for hybrid and multicloud](https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/hybrid/enterprise-scale-landing-zone) + +## Objectives + +After completing this MicroHack you will be familiar with: + +* How to connect your Kubernetes cluster running anywhere to Azure Arc +* Understand how you can streamline your operations and development processes for your Kubernetes clusters running anywhere +* Deploying Azure PaaS services such as SQL Managed Instance in your Kubernetes cluster running anywhere + +## MicroHack Challenges + +In order to play through the challenges, your microhack coach prepared a k8s cluster for you, which you will use as your onprem environment. In the case of this microhack, we are using an K3s cluster. + +For each user there are two resource groups pre-created by your coach. +| Name | Description | +|-----------------|---------------------------------------------------------------------------------------------| +| xy-k8s-onprem | In this resource group you can find the k8s cluster which simulates your onprem environment | +| xy-k8s-arc | Into this resource group your arc resources will be stored | +(xy is a placeholder for your LabUser number you will receive from your coach to access the environment) +### General Prerequisites + +In order to successfully work through the challenges in this MicroHack, you will need the following prerequisites: + +* [An Azure account with owner permissions on an active subscription](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) +* [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) (Hint: Make sure to use the lastest version) +* [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management) +* [Helm](https://helm.sh/docs/intro/install/) + +💡*Hint*: +* The solution has been verified using [Visual Studio Code](https://code.visualstudio.com/) with integrated Linux Bash Shell ([WSL](https://learn.microsoft.com/en-us/windows/wsl/install)). +* In order to clone this repository to your local system, use either git or the github plugin for VSC. + +## Contributors +* Simon Schwingel [GitHub](https://github.com/skiddder); [LinkedIn](https://www.linkedin.com/in/simon-schwingel-b602869a/) +* Lars Fischer [GitHub](https://github.com/MSFT-LarsFisch); [LinkedIn](https://www.linkedin.com/in/lars-fischer-5464a5175/) + +## Get Started +[Challenge-01](challenges/challenge-01.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-01.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-01.md new file mode 100644 index 000000000..13c086b33 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-01.md @@ -0,0 +1,30 @@ +# Challenge 1 - Onboarding your Kubernetes Cluster + +## Goal +In challenge 1 you will connect/onboard your existing K8s cluster to Azure Arc. + +## Actions +* Verify all prerequisites are in place + * Resource Providers + * Azure CLI extensions + * Resource group (Name: mh-arc-k8s-) + * Connectivity to required Azure endpoints +* Deploy the Azure Arc agent pods to your k8s cluster +* Assign permissions to view k8s resources in the Azure portal + +## Success Criteria +* Your k8s cluster appears in the Azure portal under Azure Arc > Infrastructure > Kubernetes clusters and is in status "Connected". Which arc agent version is running? +* In the Azure portal below Kubernetes resources > Workloads you can see all deployments and pods running on your cluster. What arc-specific namespaces were deployed during onboarding? + +## Learning Resources +* (https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/overview) +* (https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli) +* (https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/cluster-connect) +* (https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/azure-rbac) +* (https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/kubernetes-resource-view) +* (https://learn.microsoft.com/en-us/cli/azure/connectedk8s?view=azure-cli-latest) + +## Solution - Spoilerwarning +[Solution Steps](../walkthroughs/challenge-01/solution.md) + +[Next challenge](challenge-02.md) | [Back](../Readme.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-02.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-02.md new file mode 100644 index 000000000..c623a54a1 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-02.md @@ -0,0 +1,37 @@ +# Challenge 2 - Enable Azure Monitor for Containers + +In this challenge, you’ll configure the core monitoring and governance capabilities that turn an Arc‑enabled k8s cluster into an enterprise‑ready platform. +* Azure Monitor Container Insights provides real‑time visibility into cluster health, performance, and workload behavior, while +* Microsoft Defender for Kubernetes adds runtime threat detection and security hardening to protect your applications and infrastructure. +* Azure Policy for Kubernetes ensures consistent governance by enforcing configuration and compliance standards across the cluster. + +All telemetry, logs, and security signals generated by these services flow into Log Analytics, which serves as the central, scalable persistence layer for querying, alerting, and correlating operational and security data. + +💡*Hint*: There is a [Monitoring microhack](../07_Azure_Monitor/README.md) which guides you on how to create alerts, dashboards and workbooks to operationalize your monitoring experience. + +💡*Hint*: In this microhack we are focusing on infrastructure monitoring. But you easily can add application monitoring using either [Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) or [Azure Managed Prometheus](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/prometheus-metrics-overview) for workloads running in your k8s cluster. + +## Goal +* Establish foundational monitoring, security, and governance for an Arc‑enabled Kubernetes cluster + +## Actions +* Create a Log Analytics workspace as centralized storage for all logs and metrics. +* Enable Azure Monitor – Container Insights via the Arc extension to collect cluster, node, pod, and container telemetry. +* Onboard the cluster to Microsoft Defender for Kubernetes to activate runtime threat detection and security posture management. +* Assign Azure Policy for Kubernetes to enforce governance rules and ensure consistent configuration and compliance across the cluster. + +## Success Criteria +* In the Azure portal navigate to your arc-enabled k8s cluster. Under Monitoring > Insights you can see node performance etc, workload status and container logs +* Defender for Kubernetes displays active security assessments, no onboarding errors, and visible recommendations and alerts. +* Azure Policy shows evaluated policy results with compliant/non‑compliant resources and enforcement functioning as expected. +* Telemetry from all components is visible and queryable in Log Analytics, confirming correct data ingestion and workspace linkage. + +## Learning Resources +* https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-overview +* https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable-arc +* https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-containers-arc-enable-programmatically + +## Solution - Spoilerwarning +[Solution Steps](../walkthroughs/challenge-02/solution.md) + +[Next challenge](challenge-03.md) | [Back](../Readme.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-03.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-03.md new file mode 100644 index 000000000..5e0af10bd --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-03.md @@ -0,0 +1,25 @@ +## Challenge 3 - Deploy CPU based Large & Small Language Models (LLM/SLM) + +### Goal +In challenge 3 you will deploy a LLM/SLM to your Azure Arc-enabled Kubernetes Cluster to run an AI prompt. + +### Actions +* Login to your Kubernetes via Azure CLI Kubernetes Proxy and use kubectl +* Create a new namespace and deploy a AIMH for CPU based compute resources +* Deploy openwebUI, access it and link it to the local ollama API + +### Success Criteria +* In the Azure portal navigate to your arc-enabled k8s cluster. Under Namespaces > you can see a new namespace called AIMH +* In the AIMH namespace you see your Ollama and OpenWebUI deployment +* you entered your first prompt "What is a Microsoft MicroHack?" and you see a result. + +### Learning Resources +* (https://learn.microsoft.com/en-us/azure/aks/aksarc/deploy-ai-model?tabs=portal) +* (https://github.com/otwld/ollama-helm) CPU based LLM/SLM +* (https://docs.openwebui.com/getting-started/quick-start/) +* (https://github.com/kaito-project/kaito) GPU Based LLM/SLM + +### Solution - Spoilerwarning +[Solution Steps](../walkthroughs/challenge-03/solution.md) + +[Next challenge](challenge-04.md) | [Back](../Readme.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-04.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-04.md new file mode 100644 index 000000000..6cd12e6c1 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-04.md @@ -0,0 +1,30 @@ +# Challenge 4 - Deploy SQL Managed Instance to your cluster + +## Goal +In challenge 4 you will deploy Azure Arc-enabled Data Services to your cluster and create a SQL Managed Instance. This enables you to run Azure SQL services on-premises or at the edge while maintaining cloud management and control through Azure Arc. + +## Actions +* Register the Microsoft.AzureArcData resource provider in your subscription +* Deploy an Azure Arc Data Controller to your Arc-enabled K8s cluster +* Create a custom location that represents your on-premises K8s cluster as a deployment target in Azure +* Deploy a SQL Managed Instance to your cluster through the Arc Data Controller +* Connect to the SQL Managed Instance and query the database version + +## Success Criteria +* The Azure Arc Data Controller is successfully deployed and visible in the Azure portal under your resource group +* A SQL Managed Instance resource appears in Azure portal with status "Ready" +* In your Kubernetes cluster, you can see the SQL MI pods running in the custom location namespace +* You can successfully connect to the SQL Managed Instance using the master node's public IP and the assigned NodePort +* You can execute a test query (e.g., `SELECT @@VERSION`) and see the query result + +## Learning Resources +* [Deploy a SQL Managed Instance enabled by Azure Arc](https://learn.microsoft.com/en-us/azure/azure-arc/data/create-sql-managed-instance) +* [Azure Arc-enabled data services overview](https://learn.microsoft.com/en-us/azure/azure-arc/data/overview) +* [What is Azure Arc-enabled SQL Managed Instance?](https://learn.microsoft.com/en-us/azure/azure-arc/data/managed-instance-overview) +* [Custom locations on Azure Arc-enabled Kubernetes](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/custom-locations) +* [Azure Arc Data Controller deployment](https://learn.microsoft.com/en-us/azure/azure-arc/data/create-data-controller) + +## Solution - Spoilerwarning +[Solution Steps](../walkthroughs/challenge-04/solution.md) + +[Next challenge](challenge-05.md) | [Back](../Readme.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-05.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-05.md new file mode 100644 index 000000000..fe0fa78d8 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/challenges/challenge-05.md @@ -0,0 +1,26 @@ +# Challenge 5 - Configure GitOps for cluster management + +## Goal +Configure GitOps with Flux for cluster management. In this microhack we chose to manage Kubernetes namespaces from a Git repository as an example how GitOps can be used to centralize configuration of multiple clusters from a single source of truth. + +## Actions +1. Ensure the Flux extension is installed on your Arc-enabled Kubernetes cluster. +2. Fork the MicroHack repository (public for ease of use) and clone only the required `namespaces` folder via sparse checkout. +3. Create a Flux configuration that points to the `namespaces` folder in your fork. +4. Verify the initial namespace from the repo is created automatically. +5. Add a new namespace manifest (team1) and push it to your fork. + +## Success Criteria +* A Flux configuration exists and is in a healthy state for your Arc-enabled cluster. +* The initial namespace from the repository is created in the cluster. +* A second namespace (team1) appears after you push the new manifest. + +## Learning Resources +* [GitOps for Azure Kubernetes Service](https://learn.microsoft.com/en-us/azure/architecture/example-scenario/gitops-aks/gitops-blueprint-aks) +* [GitOps with Flux on Azure Arc-enabled Kubernetes clusters](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-flux2) +* [Flux documentation - Get started](https://fluxcd.io/docs/get-started/) + +## Solution - Spoilerwarning +[Solution Steps](../walkthroughs/challenge-05/solution.md) + +[Next challenge](challenge-06.md) | [Back](../Readme.md) diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/img/architectural-overview.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/img/architectural-overview.png new file mode 100644 index 000000000..9b467bf85 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/img/architectural-overview.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/img/banner.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/img/banner.png new file mode 100644 index 000000000..2790346d0 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/img/banner.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/container-registry.tf b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/container-registry.tf new file mode 100644 index 000000000..e094deb78 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/container-registry.tf @@ -0,0 +1,17 @@ +# will be used in challenge 04-gitops + +resource "azurerm_container_registry" "this" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}${var.acr_name}" + resource_group_name = azurerm_resource_group.mh_k8s_arc[count.index].name + location = azurerm_resource_group.mh_k8s_arc[count.index].location + sku = var.container_registry_sku + admin_enabled = var.container_registry_admin_enabled +} + +output "acr_names" { + value = { + for i, acr in azurerm_container_registry.this : + local.indices[i] => acr.name + } +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/img/image.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/img/image.png new file mode 100644 index 000000000..1f4872aac Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/img/image.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k3s-master-setup.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k3s-master-setup.sh new file mode 100644 index 000000000..d86f2eed0 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k3s-master-setup.sh @@ -0,0 +1,77 @@ +#!/bin/bash +set -euo pipefail + +# Update system +apt-get update +apt-get upgrade -y + +# Install required packages +apt-get install -y curl wget apt-transport-https ca-certificates gnupg lsb-release + +# Install Docker (optional for some workloads) +# Note: k3s includes its own container runtime, so Docker is optional +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg +echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null +apt-get update + +# Install Docker without pinning containerd.io version, continue on error +apt-get install -y docker-ce docker-ce-cli containerd.io || echo "Docker installation failed, continuing with k3s (which includes its own container runtime)" + +# Add admin user to docker group if docker was installed successfully +if command -v docker &> /dev/null; then + usermod -aG docker ${admin_user} +fi + +# Install K3s server (master node) +export INSTALL_K3S_VERSION=${k3s_version} +export K3S_TOKEN=${cluster_token} + +# Get external IP reliably +EXTERNAL_IP=$(curl -s --max-time 10 https://ipinfo.io/ip 2>/dev/null || curl -s --max-time 10 http://checkip.amazonaws.com 2>/dev/null || echo "") + +# Install K3s with embedded etcd and disable traefik (we might want to use nginx-ingress) +if [ -n "$EXTERNAL_IP" ] && [[ "$EXTERNAL_IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + curl -sfL https://get.k3s.io | sh -s - server \ + --cluster-init \ + --disable traefik \ + --disable servicelb \ + --write-kubeconfig-mode 644 \ + --node-external-ip "$EXTERNAL_IP" \ + --token $${K3S_TOKEN} +else + # Fallback without external IP if detection fails + curl -sfL https://get.k3s.io | sh -s - server \ + --cluster-init \ + --disable traefik \ + --disable servicelb \ + --write-kubeconfig-mode 644 \ + --token $${K3S_TOKEN} +fi + +# Wait for K3s to be ready +while ! kubectl get nodes > /dev/null 2>&1; do + echo "Waiting for K3s to be ready..." + sleep 10 +done + +# Install Helm +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash + +# Make kubectl accessible for admin user +mkdir -p /home/${admin_user}/.kube +cp /etc/rancher/k3s/k3s.yaml /home/${admin_user}/.kube/config +chown ${admin_user}:${admin_user} /home/${admin_user}/.kube/config + +# Create a script to get the node token for workers +cat > /home/${admin_user}/get-node-token.sh << 'EOF' +#!/bin/bash +sudo cat /var/lib/rancher/k3s/server/node-token +EOF +chmod +x /home/${admin_user}/get-node-token.sh +chown ${admin_user}:${admin_user} /home/${admin_user}/get-node-token.sh + +# Enable and start K3s +systemctl enable k3s +systemctl start k3s + +echo "K3s master node setup completed!" \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k3s-worker-setup.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k3s-worker-setup.sh new file mode 100644 index 000000000..d326915b1 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k3s-worker-setup.sh @@ -0,0 +1,60 @@ +#!/bin/bash +set -euo pipefail + +# Wait for master to be ready (give it some time to start up) +sleep 60 + +# Update system +apt-get update +apt-get upgrade -y + +# Install required packages +apt-get install -y curl wget apt-transport-https ca-certificates gnupg lsb-release + +# Install Docker (optional for some workloads) +# Note: k3s includes its own container runtime, so Docker is optional +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg +echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null +apt-get update + +# Install Docker without pinning containerd.io version, continue on error +apt-get install -y docker-ce docker-ce-cli containerd.io || echo "Docker installation failed, continuing with k3s (which includes its own container runtime)" + +# Add admin user to docker group if docker was installed successfully +if command -v docker &> /dev/null; then + usermod -aG docker ${admin_user} +fi + +# Install K3s agent (worker node) +export INSTALL_K3S_VERSION=${k3s_version} +export K3S_TOKEN=${cluster_token} +export K3S_URL=https://${master_ip}:6443 + +# Wait for master to be accessible +echo "Waiting for K3s master at $${K3S_URL}..." +while ! curl -k -s $${K3S_URL}/ping > /dev/null 2>&1; do + echo "Master not ready yet, waiting..." + sleep 10 +done + +# Get external IP reliably +EXTERNAL_IP=$(curl -s --max-time 10 https://ipinfo.io/ip 2>/dev/null || curl -s --max-time 10 http://checkip.amazonaws.com 2>/dev/null || echo "") + +# Install K3s agent +if [ -n "$EXTERNAL_IP" ] && [[ "$EXTERNAL_IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + curl -sfL https://get.k3s.io | sh -s - agent \ + --server $${K3S_URL} \ + --token $${K3S_TOKEN} \ + --node-external-ip "$EXTERNAL_IP" +else + # Fallback without external IP if detection fails + curl -sfL https://get.k3s.io | sh -s - agent \ + --server $${K3S_URL} \ + --token $${K3S_TOKEN} +fi + +# Enable and start K3s agent +systemctl enable k3s-agent +systemctl start k3s-agent + +echo "K3s worker node setup completed!" \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k8s-cluster.tf b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k8s-cluster.tf new file mode 100644 index 000000000..4c514b4ef --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/k8s-cluster.tf @@ -0,0 +1,375 @@ +# Create Virtual Network for K3s cluster +resource "azurerm_virtual_network" "onprem" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-vnet" + address_space = ["10.${100 + local.indices[count.index]}.0.0/16"] + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +# Create subnet for K3s VMs +resource "azurerm_subnet" "onprem" { + count = length(local.indices) + name = "k3s-subnet" + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + virtual_network_name = azurerm_virtual_network.onprem[count.index].name + address_prefixes = ["10.${100 + local.indices[count.index]}.1.0/24"] +} + +# Create Network Security Group for K3s VMs +resource "azurerm_network_security_group" "onprem" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-nsg" + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + + security_rule { + name = "SSH" + priority = 1001 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "22" + source_address_prefix = "*" + destination_address_prefix = "*" + } + + security_rule { + name = "K3s-API" + priority = 1002 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "6443" + source_address_prefix = "*" + destination_address_prefix = "*" + } + + security_rule { + name = "K3s-NodePort" + priority = 1003 + direction = "Inbound" + access = "Allow" + protocol = "Tcp" + source_port_range = "*" + destination_port_range = "30000-32767" + source_address_prefix = "*" + destination_address_prefix = "*" + } + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +# Associate Network Security Group to the subnet +resource "azurerm_subnet_network_security_group_association" "onprem" { + count = length(local.indices) + subnet_id = azurerm_subnet.onprem[count.index].id + network_security_group_id = azurerm_network_security_group.onprem[count.index].id +} + +# Create public IPs for K3s VMs +resource "azurerm_public_ip" "onprem_master" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-master-ip" + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + allocation_method = "Static" + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +resource "azurerm_public_ip" "onprem_worker" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-worker1-ip" + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + allocation_method = "Static" + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +resource "azurerm_public_ip" "onprem_worker2" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-worker2-ip" + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + allocation_method = "Static" + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +# Create Network Interfaces +resource "azurerm_network_interface" "onprem_master" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-master-nic" + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + + ip_configuration { + name = "internal" + subnet_id = azurerm_subnet.onprem[count.index].id + private_ip_address_allocation = "Static" + private_ip_address = "10.${100 + local.indices[count.index]}.1.10" + public_ip_address_id = azurerm_public_ip.onprem_master[count.index].id + } + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +resource "azurerm_network_interface" "onprem_worker" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-worker1-nic" + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + + ip_configuration { + name = "internal" + subnet_id = azurerm_subnet.onprem[count.index].id + private_ip_address_allocation = "Static" + private_ip_address = "10.${100 + local.indices[count.index]}.1.11" + public_ip_address_id = azurerm_public_ip.onprem_worker[count.index].id + } + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +resource "azurerm_network_interface" "onprem_worker2" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-worker2-nic" + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + + ip_configuration { + name = "internal" + subnet_id = azurerm_subnet.onprem[count.index].id + private_ip_address_allocation = "Static" + private_ip_address = "10.${100 + local.indices[count.index]}.1.12" + public_ip_address_id = azurerm_public_ip.onprem_worker2[count.index].id + } + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +# Create K3s Master VM +resource "azurerm_linux_virtual_machine" "onprem_master" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-master" + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + size = var.vm_size + disable_password_authentication = false + admin_username = var.admin_user + admin_password = var.admin_password + + network_interface_ids = [ + azurerm_network_interface.onprem_master[count.index].id, + ] + + identity { + type = "SystemAssigned" + } + + os_disk { + caching = "ReadWrite" + storage_account_type = "Premium_LRS" + } + + source_image_reference { + publisher = "Canonical" + offer = "0001-com-ubuntu-server-jammy" + sku = "22_04-lts-gen2" + version = "latest" + } + + custom_data = base64encode(templatefile("${path.module}/k3s-master-setup.sh", { + k3s_version = var.k3s_version + cluster_token = var.cluster_token + admin_user = var.admin_user + })) + + tags = { + Project = "simulated onprem k8s cluster for microhack" + Role = "master" + } +} + +# Create K3s Worker VM 1 +resource "azurerm_linux_virtual_machine" "onprem_worker" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-worker1" + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + size = var.vm_size + disable_password_authentication = false + admin_username = var.admin_user + admin_password = var.admin_password + + network_interface_ids = [ + azurerm_network_interface.onprem_worker[count.index].id, + ] + + identity { + type = "SystemAssigned" + } + + os_disk { + caching = "ReadWrite" + storage_account_type = "Premium_LRS" + } + + source_image_reference { + publisher = "Canonical" + offer = "0001-com-ubuntu-server-jammy" + sku = "22_04-lts-gen2" + version = "latest" + } + + custom_data = base64encode(templatefile("${path.module}/k3s-worker-setup.sh", { + k3s_version = var.k3s_version + cluster_token = var.cluster_token + master_ip = "10.${100 + local.indices[count.index]}.1.10" + admin_user = var.admin_user + })) + + depends_on = [azurerm_linux_virtual_machine.onprem_master] + + tags = { + Project = "simulated onprem k8s cluster for microhack" + Role = "worker" + } +} + +# Create K3s Worker VM 2 +resource "azurerm_linux_virtual_machine" "onprem_worker2" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-worker2" + resource_group_name = azurerm_resource_group.mh_k8s_onprem[count.index].name + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + size = var.vm_size + disable_password_authentication = false + admin_username = var.admin_user + admin_password = var.admin_password + + network_interface_ids = [ + azurerm_network_interface.onprem_worker2[count.index].id, + ] + + identity { + type = "SystemAssigned" + } + + os_disk { + caching = "ReadWrite" + storage_account_type = "Premium_LRS" + } + + source_image_reference { + publisher = "Canonical" + offer = "0001-com-ubuntu-server-jammy" + sku = "22_04-lts-gen2" + version = "latest" + } + + custom_data = base64encode(templatefile("${path.module}/k3s-worker-setup.sh", { + k3s_version = var.k3s_version + cluster_token = var.cluster_token + master_ip = "10.${100 + local.indices[count.index]}.1.10" + admin_user = var.admin_user + })) + + depends_on = [azurerm_linux_virtual_machine.onprem_master] + + tags = { + Project = "simulated onprem k8s cluster for microhack" + Role = "worker" + } +} + +# Auto-shutdown schedule for K3s Master VMs +resource "azurerm_dev_test_global_vm_shutdown_schedule" "master" { + count = length(local.indices) + virtual_machine_id = azurerm_linux_virtual_machine.onprem_master[count.index].id + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + enabled = true + + daily_recurrence_time = "1800" + timezone = "W. Europe Standard Time" + + notification_settings { + enabled = false + } + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +# Auto-shutdown schedule for K3s Worker1 VMs +resource "azurerm_dev_test_global_vm_shutdown_schedule" "worker" { + count = length(local.indices) + virtual_machine_id = azurerm_linux_virtual_machine.onprem_worker[count.index].id + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + enabled = true + + daily_recurrence_time = "1800" + timezone = "W. Europe Standard Time" + + notification_settings { + enabled = false + } + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +# Auto-shutdown schedule for K3s Worker2 VMs +resource "azurerm_dev_test_global_vm_shutdown_schedule" "worker2" { + count = length(local.indices) + virtual_machine_id = azurerm_linux_virtual_machine.onprem_worker2[count.index].id + location = azurerm_resource_group.mh_k8s_onprem[count.index].location + enabled = true + + daily_recurrence_time = "1800" + timezone = "W. Europe Standard Time" + + notification_settings { + enabled = false + } + + tags = { + Project = "simulated onprem k8s cluster for microhack" + } +} + +output "k3s_cluster_info" { + value = { + for i in range(length(local.indices)) : + format("%02d", local.indices[i]) => { + master_ssh = "ssh ${var.admin_user}@${azurerm_public_ip.onprem_master[i].ip_address}" + worker1_ssh = "ssh ${var.admin_user}@${azurerm_public_ip.onprem_worker[i].ip_address}" + worker2_ssh = "ssh ${var.admin_user}@${azurerm_public_ip.onprem_worker2[i].ip_address}" + kubeconfig_setup = "mkdir -p ~/.kube && scp ${var.admin_user}@${azurerm_public_ip.onprem_master[i].ip_address}:/home/${var.admin_user}/.kube/config ~/.kube/config && sed -i 's/127.0.0.1/${azurerm_public_ip.onprem_master[i].ip_address}/g' ~/.kube/config" + } + } +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/locals.tf b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/locals.tf new file mode 100644 index 000000000..abd5e71c0 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/locals.tf @@ -0,0 +1,5 @@ +locals { + # Create a list of indices from start to end + indices = range(var.start_index, var.end_index) +} + diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/log-analytics.tf b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/log-analytics.tf new file mode 100644 index 000000000..c8df2b5d9 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/log-analytics.tf @@ -0,0 +1,18 @@ + +resource "azurerm_log_analytics_workspace" "law" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-law" + + resource_group_name = azurerm_resource_group.mh_k8s_arc[count.index].name + location = azurerm_resource_group.mh_k8s_arc[count.index].location + + sku = "PerGB2018" + retention_in_days = 30 +} + +output "law" { + value = { + for i, item in azurerm_log_analytics_workspace.law : + local.indices[i] => item.id + } +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/main.tf b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/main.tf new file mode 100644 index 000000000..2046d4c32 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/main.tf @@ -0,0 +1,28 @@ +resource "azurerm_resource_group" "mh_k8s_onprem" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-onprem" + location = var.onprem_resources[count.index % length(var.onprem_resources)] + tags = { "SecurityControl" = "Ignore" } +} + +resource "azurerm_resource_group" "mh_k8s_arc" { + count = length(local.indices) + name = "${format("%02d", local.indices[count.index])}-${var.resource_group_base_name}-arc" + location = var.arc_location +} + +output "rg_names_onprem" { + #value = azurerm_resource_group.mh_k8s_onprem.name + value = { + for i, rg in azurerm_resource_group.mh_k8s_onprem : + local.indices[i] => rg.name + } +} + +output "rg_names_arc" { + #value = azurerm_resource_group.mh_k8s_onprem.name + value = { + for i, rg in azurerm_resource_group.mh_k8s_arc : + local.indices[i] => rg.name + } +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/provider-template.txt b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/provider-template.txt new file mode 100644 index 000000000..da37e755b --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/provider-template.txt @@ -0,0 +1,23 @@ +# +# Providers Configuration +# + +terraform { + required_version = ">= 1.1.9" + required_providers { + azurerm = { + source = "hashicorp/azurerm" + version = "~> 4.22.0" + } + } +} + +# Configure the Azure Provider +provider "azurerm" { + features { + resource_group { + prevent_deletion_if_contains_resources = false + } + } + subscription_id = "REPLACE-ME" +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/readme.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/readme.md new file mode 100644 index 000000000..30860dd22 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/readme.md @@ -0,0 +1,258 @@ +# Environment Setup +When working through the challenges of this microhack, it's assumed that you have an onprem k8s cluster available which you can use to arc-enable it. Also, it's assumed that you have a container registry, which you can use for the gitops challenge. + +In this folder you find terraform code to deploy a **K3s cluster** and container registry in Azure for each participant of the microhack. It's intended that coaches create these resources for their participants before the microhack starts, so the participants can directly start with challenge 1 (onboarding/arc-enabling their cluster). + +## K3s Architecture + +This Terraform configuration deploys a **3-node K3s cluster** in Azure VMs, simulating an on-premises Kubernetes environment for Azure Arc enablement. + +Each deployment creates: +- **1 Master Node**: K3s server with embedded etcd (10.{100+index}.1.10) +- **2 Worker Nodes**: K3s agents (10.{100+index}.1.11, 10.{100+index}.1.12) +- **Virtual Network**: Isolated network per cluster (10.{100+index}.0.0/16) +- **Network Security Groups**: Allow SSH and Kubernetes traffic +- **Public IPs**: For SSH access to all nodes + +## Resources to be deployed +2 resource groups, 1 K3s cluster (3 VMs), 1 container registry per participant where xy represents the participant number: +``` +subscription +| +├── -k8s-arc (resource group) +| | +│ └── mhacr (container registry) +| | +| └── -law (log analytics workspace) +| +└── -k8s-onprem (resource group) + | + ├── -k8s-master (VM - K3s server) + ├── -k8s-worker1 (VM - K3s agent) + ├── -k8s-worker2 (VM - K3s agent) + ├── -k8s-vnet (Virtual Network) + └── Associated NICs, NSGs, and Public IPs +``` + +## Prerequisites +* bash shell (tested with Ubuntu 22.04) +* Azure CLI +* terraform +* clone this repo locally, so you can adjust the deployment files according to your needs +* Azure subscription +* User account with subscription owner permissions +* Sufficient quota limits to support creation of K3s VMs per participant + +## K3s Default Configuration +- **VM Size**: Standard_D4ds_v6 (sufficient for K3s, smaller than AKS requirements) +- **OS**: Ubuntu 22.04 LTS +- **K3s Version**: v1.33.6+k3s1 +- **Admin User**: Set via admin_user in fixtures.tfvars +- **VMs per cluster**: 3 VMs (1 master + 2 workers) +- **Password**: Must be set in fixtures.tfvars (no default value) + +If you don't change the default value of parameter "vm_size" in variables.tf, three Standard_D4ds_v6 VMs per cluster are used (1 master + 2 workers). If you have many participants you need to ensure that the quota limit in your subscription is sufficient to support the required cores. The terraform code will distribute the K3s clusters to 10 different regions. This setting can be adjusted via the parameter "onprem_resources" (variables.tf) value. + +You can check this limit via Azure Portal (subscription > settings > Usage & Quotas): + +![alt text](img/image.png) + + + +## Installation instructions +As a microhack coach, you will be given a subscription in the central microhack tenant. Terraform expects the subscription id within the azurerm provider. Therefore, you need to to create the provider.tf file in this folder. To achieve this + +* Copy the provider-template.txt and rename the copy to 'provider.tf'. +* Login to Azure CLI and run the "start_here.sh" script located in this folder +```bash +az logout # only if you were logged in with a different user already +az login # in the browser popup, provide the user credentials you got from your microhack coach + +./start_here.sh # sets the subscription_id in the provider.tf file +``` + +The terraform code deploys **K3s clusters** on Azure VMs which will be used as onprem k8s clusters. We chose K3s as it provides a true "on-premises" experience compared to AKS, making Arc enablement more realistic and meaningful for learning purposes. + +## How K3s Setup Works + +The K3s installation is **fully automated** during VM provisioning using cloud-init: + +1. **k3s-master-setup.sh**: Automatically runs on the master VM during boot + - Installs Docker and required packages + - Downloads and installs K3s server with embedded etcd + - Configures kubeconfig for the configured `admin_user` + - Creates a script to retrieve the node token for workers + +2. **k3s-worker-setup.sh**: Automatically runs on worker VMs during boot + - Installs Docker and required packages + - Waits for the master to be ready + - Downloads and installs K3s agent + - Connects to the master using the shared cluster token + +The scripts are executed via Terraform's `custom_data` parameter, so **no manual intervention is required**. The cluster will be ready approximately 5-10 minutes after VM deployment completes. + +The K3s deployment uses VM managed identities and doesn't require service principals like AKS deployments. + +* Create a file called fixtures.tfvars and set the admin password for the VMs: + +* All resources which are created by this terraform code will get a two-digit numeric prefix. It's intended that each user easily finds "his" resources. If a user i.e. got assigned the account "LabUser-37" he should work with the resources with the prefix "37". The central microhack team precreates the user accounts and assigns them to the different microhacks (which ususally run in parallel on the same day). So the users probably do not start with "01". Depending on what user accounts you got provided, you can use the start_index and end_index in the fixtures.tfvars file to adjust the prefixes to match your user numbers. Example: You receive the users LabUser-50 to LabUser-59, set the start_index value to 50 and the end_index value to 59. Make sure you saved your changes. + +* your fixtures.tfvars file should now look like this: +```terraform +# Deployment range +start_index = 37 +end_index = 39 + +# Security - REQUIRED +admin_user = "" +admin_password = "" +cluster_token = "" # Simple string for K3s +``` + +```bash +terraform init # download terraform providers + +terraform plan -var-file=fixtures.tfvars -out=tfplan + +# have a look at the resources which will be created. There should be resource groups per participant, K3s VMs, and Azure container registry. +# after validation: + +terraform apply tfplan +``` + +### What Happens After Deployment + +1. **VMs are created** with Ubuntu 22.04 LTS +2. **K3s setup scripts run automatically** via cloud-init: + - Master node: Installs K3s server, configures networking, sets up kubeconfig + - Worker nodes: Wait for master, then join the cluster as K3s agents +3. **Cluster becomes ready** in ~5-10 minutes after VM deployment +4. **SSH access** is available immediately with the admin_user and your password + +The expected output looks approximately like this depending on the start_index and end_index parameters: +```bash +Outputs: + +acr_names = { + "37" = "37mhacr" + "38" = "38mhacr" +} +k3s_cluster_info = { + "37" = { + "kubeconfig_setup" = "mkdir -p ~/.kube && scp @x.x.x.x:/home//.kube/config ~/.kube/config && sed -i 's/127.0.0.1/x.x.x.x/g' ~/.kube/config" + "master_ssh" = "ssh @x.x.x.x" + "worker1_ssh" = "ssh @y.y.y.y" + "worker2_ssh" = "ssh @z.z.z.z" + } + "38" = { + "kubeconfig_setup" = "mkdir -p ~/.kube && scp @20.19.166.105:/home//.kube/config ~/.kube/config && sed -i 's/127.0.0.1/a.a.a.a/g' ~/.kube/config" + "master_ssh" = "ssh @a.a.a.a" + "worker1_ssh" = "ssh @b.b.b.b" + "worker2_ssh" = "ssh @c.c.c.c" + } +} +law = { + "37" = "/subscriptions/.../resourceGroups/37-k8s-arc/providers/Microsoft.OperationalInsights/workspaces/37-law" + "38" = "/subscriptions/.../resourceGroups/38-k8s-arc/providers/Microsoft.OperationalInsights/workspaces/38-law" +} +rg_names_arc = { + "37" = "37-k8s-arc" + "38" = "38-k8s-arc" +} +rg_names_onprem = { + "37" = "37-k8s-onprem" + "38" = "38-k8s-onprem" +} +``` + +**Important**: Wait 5-10 minutes after terraform completes before trying to access the K3s cluster to allow the setup scripts to finish. + +## Post-Deployment - Accessing Your K3s Cluster + +### 1. Access your cluster +```bash +# Set admin username (must match the admin_user value in fixtures.tfvars) +admin_user="" + +# Extract user number from Azure username before '@' (e.g., LabUser-37@... -> 37) +azure_user=$(az account show --query user.name --output tsv) +user_number=$(echo "${azure_user%@*}" | grep -oE '[0-9]+' | tail -n1 | sed 's/^0*//; s/^$/0/') + +# Get public ip of master node via Azure CLI according to user-number +master_pip=$(az vm list-ip-addresses --resource-group "${user_number}-k8s-onprem" --name "${user_number}-k8s-master" --query "[0].virtualMachine.network.publicIpAddresses[0].ipAddress" --output tsv) + +echo "Connecting to master node: $master_pip with user: $admin_user" + +# Create .kube directory if it doesn't exist +mkdir -p ~/.kube + +# Copy the kubeconfig to standard location +scp $admin_user@$master_pip:/home/$admin_user/.kube/config ~/.kube/config + +# Replace localhost address with the public ip of master node +sed -i "s/127.0.0.1/$master_pip/g" ~/.kube/config + +# Now kubectl works directly +kubectl get nodes +``` + +### 2. Verify K3s Installation +```bash +# On master node +kubectl get nodes +systemctl status k3s + +# On worker nodes +systemctl status k3s-agent +``` + +## Troubleshooting + +### Check K3s Logs +```bash +# On master node +kubectl get nodes + +systemctl status k3s + +sudo journalctl -u k3s -f + +# On worker nodes +systemctl status k3s-agent + +sudo journalctl -u k3s-agent -f +``` + +### Verify Network Connectivity +```bash +# From worker to master (port 6443 should be open) +telnet 6443 +``` + +### Reset K3s Installation (if needed) +```bash +# On master +sudo /usr/local/bin/k3s-uninstall.sh + +# On worker nodes +sudo /usr/local/bin/k3s-agent-uninstall.sh +``` + +## Security Notes +- VMs use password authentication (consider using SSH keys for production) +- Network security groups restrict access to necessary ports only +- K3s runs without Traefik (disabled for flexibility) +- Docker is installed for container runtime and additional workloads + +## Clean Up - After Microhack + +When done with the microhack, call terraform destroy to clean up. + +```bash +terraform destroy -var-file=fixtures.tfvars +``` + +This will remove all created resources including VMs, networks, and public IPs. + +[To challenge 01](../challenges/challenge-01.md) + diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-client-connectivity-check.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-client-connectivity-check.sh new file mode 100644 index 000000000..0444e4763 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-client-connectivity-check.sh @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +set -euo pipefail + +# ------------------------------------------------------------------------------ +# Synopsis: Azure Arc CLIENT Onboarding Connectivity Validation Script +# +# This script validates the network connectivity required on a *client machine* +# (e.g., admin workstation, jump host, DevOps runner) to perform Azure Arc– +# enabled Kubernetes **onboarding** and Arc management operations using: +# - az login +# - az connectedk8s connect +# - GitOps (KubernetesConfiguration) authoring commands +# - kubectl (local execution only; no Cluster Connect checks) +# +# The script verifies only the endpoints needed for: +# ✔ Azure Resource Manager (ARM) +# ✔ Azure Active Directory authentication flows +# ✔ GitOps management-plane interactions from the CLI +# ✔ Optional Azure RBAC (graph.microsoft.com) +# ✔ kubectl binary download (dl.k8s.io) +# ✔ Basic Microsoft Container Registry access (MCR) +# +# It intentionally excludes: +# ✘ Cluster Connect +# ✘ Service Bus WebSockets +# ✘ Guest Notification Service regional endpoints +# ✘ Any Arc Portal UI endpoints +# +# Purpose: +# - Ensure your client has correct outbound access to authenticate, onboard, +# register, and manage Azure Arc–enabled Kubernetes resources. +# - Detect issues such as blocked outbound HTTPS, DNS failures, TLS inspection, +# or restricted access to Microsoft endpoints required for Arc onboarding. +# +# Validated Functional Areas: +# - ARM API: management.azure.com +# - AAD authentication: login.microsoftonline.com, .login.microsoft.com, +# login.windows.net +# - GitOps management-plane: .dp.kubernetesconfiguration.azure.com +# - Optional RBAC: graph.microsoft.com +# - kubectl download: dl.k8s.io +# - Optional: mcr.microsoft.com registry availability +# +# How to Run on Your Client: +# +# 1. Download the script: +# curl -sSL -o arc-client-onboarding-check.sh +# +# 2. Make it executable: +# chmod +x arc-client-onboarding-check.sh +# +# 3. Execute it with your Azure region: +# ./arc-client-onboarding-check.sh westeurope +# +# Output: +# - DNS, TLS, and HTTPS validation results per endpoint +# - PASS/FAIL summary +# - Remediation hints for network/proxy/TLS inspection issues +# +# Exit Codes: +# - 0 = All critical connectivity checks passed +# - 1 = One or more required endpoints unreachable +# +# Intended Audience: +# - Operators onboarding Arc-enabled Kubernetes clusters +# - Network and security engineers validating client egress paths +# - Architects preparing hybrid network readiness for Arc deployments +# +# ------------------------------------------------------------------------------ + +REGION="${1:-westeurope}" +CURL_TIMEOUT=7 +OPENSSL_TIMEOUT=7 +COLOR_OK="\033[32m"; COLOR_ERR="\033[31m"; COLOR_WARN="\033[33m"; COLOR_DIM="\033[90m"; COLOR_RESET="\033[0m" + +h1(){ echo -e "\n\033[1m$1\033[0m"; } +ok(){ echo -e "${COLOR_OK}✔${COLOR_RESET} $1"; } +err(){ echo -e "${COLOR_ERR}✖${COLOR_RESET} $1"; } +warn(){ echo -e "${COLOR_WARN}⚠${COLOR_RESET} $1"; } + +# Resolver selection +RESOLVER_CMD="" +if command -v dig >/dev/null 2>&1; then RESOLVER_CMD="dig +short" +elif command -v nslookup >/dev/null 2>&1; then RESOLVER_CMD="nslookup" +elif command -v getent >/dev/null 2>&1; then RESOLVER_CMD="getent hosts" +else echo "Need dig/nslookup/getent"; exit 2; fi + +# Client-relevant endpoints (minimal) +CLIENT_CORE=( + "management.azure.com" # ARM + "login.microsoftonline.com" # AAD + "${REGION}.login.microsoft.com" # regional AAD + "login.windows.net" # legacy AAD + "${REGION}.dp.kubernetesconfiguration.azure.com" # GitOps mgmt-plane + "graph.microsoft.com" # Azure RBAC (optional but checked) + "dl.k8s.io" # kubectl download +) + +CLIENT_MCR=("mcr.microsoft.com") + +PASSED=(); FAILED=() + +check_dns(){ local h="$1" + if [[ "$RESOLVER_CMD" == "dig +short" ]]; then + dig +short "$h" | grep -E '^[0-9a-fA-F:.]+$' >/dev/null && ok "DNS resolves: $h" || { err "DNS failed: $h"; return 1; } + elif [[ "$RESOLVER_CMD" == "nslookup" ]]; then + nslookup "$h" >/dev/null 2>&1 && ok "DNS resolves: $h" || { err "DNS failed: $h"; return 1; } + else + getent hosts "$h" >/dev/null 2>&1 && ok "DNS resolves: $h" || { err "DNS failed: $h"; return 1; } + fi +} + +check_tls(){ local h="$1" + timeout "${OPENSSL_TIMEOUT}" bash -c "echo | openssl s_client -servername ${h} -connect ${h}:443 >/dev/null 2>&1" \ + && ok "TLS handshake OK: ${h}:443" || { err "TLS handshake failed: ${h}:443"; return 1; } +} + +check_https(){ local h="$1" + curl -sS -I --connect-timeout "${CURL_TIMEOUT}" "https://${h}" >/dev/null 2>&1 \ + && ok "HTTPS reachable: https://${h}" || { err "HTTPS blocked/unreachable: https://${h}"; return 1; } +} + +mcr_probe(){ + local okflag=1 + curl -sS -I --connect-timeout "${CURL_TIMEOUT}" "https://mcr.microsoft.com/v2/" >/dev/null 2>&1 \ + && ok "MCR registry reachable: mcr.microsoft.com" || { err "MCR blocked/unreachable: mcr.microsoft.com"; okflag=0; } + curl -sS -I --connect-timeout "${CURL_TIMEOUT}" "https://mcr.microsoft.com/v2/azurearck8s/agent/tags/list" >/dev/null 2>&1 \ + && ok "MCR path probe OK: /v2/azurearck8s/agent" || warn "MCR path probe failed (CDN filtering possible)." + return $([[ $okflag -eq 1 ]]) +} + +h1 "Azure Arc CLIENT onboarding checks (region: ${REGION})" +[[ -n "${HTTPS_PROXY:-}" || -n "${HTTP_PROXY:-}" ]] && echo -e "${COLOR_DIM}Proxy: HTTPS_PROXY=${HTTPS_PROXY:-} HTTP_PROXY=${HTTP_PROXY:-}${COLOR_RESET}" + +h1 "1) DNS" +for h in "${CLIENT_CORE[@]}" "${CLIENT_MCR[@]}"; do + check_dns "$h" && PASSED+=("DNS:$h") || FAILED+=("DNS:$h") +done + +h1 "2) TLS handshakes (443)" +for h in "${CLIENT_CORE[@]}" "${CLIENT_MCR[@]}"; do + check_tls "$h" && PASSED+=("TLS:$h") || FAILED+=("TLS:$h") +done + +h1 "3) HTTPS reachability (443)" +for h in "${CLIENT_CORE[@]}" "${CLIENT_MCR[@]}"; do + check_https "$h" && PASSED+=("HTTPS:$h") || FAILED+=("HTTPS:$h") +done + +h1 "4) Optional MCR probes" +mcr_probe && PASSED+=("MCR:mcr.microsoft.com") || FAILED+=("MCR:mcr.microsoft.com") + +h1 "Summary (CLIENT)" +echo -e "Passed: ${COLOR_OK}${#PASSED[@]}${COLOR_RESET} | Failed: ${COLOR_ERR}${#FAILED[@]}${COLOR_RESET}" +if (( ${#FAILED[@]} > 0 )); then + echo -e "${COLOR_ERR}Failures:${COLOR_RESET}"; for f in "${FAILED[@]}"; do echo " - $f"; done + echo -e "\nHints:\n - Disable TLS inspection for these FQDNs.\n - If a proxy is in use, verify NO_PROXY excludes cluster private ranges." + exit 1 +else + echo -e "${COLOR_OK}All critical client onboarding paths look good.${COLOR_RESET}"; exit 0 +fi diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-connectivity-check.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-connectivity-check.sh new file mode 100644 index 000000000..3555a4b18 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-connectivity-check.sh @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +set -euo pipefail + + +# ------------------------------------------------------------------------------ +# Synopsis: Azure Arc Connectivity Validation Script (In‑Cluster Execution) +# +# This script validates the network connectivity required for Azure Arc– +# enabled Kubernetes onboarding and steady‑state operation. It focuses +# exclusively on the endpoints required by the Arc agents (no Cluster Connect +# or Service Bus checks). +# +# Purpose: +# - Verify outbound connectivity from inside the Kubernetes cluster. +# - Ensure Arc agents can reach all mandatory Azure control‑plane and +# data‑plane endpoints. +# - Validate DNS resolution, TLS handshake, and HTTPS accessibility. +# - Detect firewall, proxy, TLS inspection, or egress restrictions that may +# break Arc onboarding, GitOps, extensions, policy, or identity flows. +# +# Validated Functional Areas: +# - Azure Resource Manager (management.azure.com) +# - KubernetesConfiguration data-plane (GitOps status/config delivery) +# - Azure Active Directory token retrieval for Arc agents +# - Managed Identity certificate retrieval (his.arc endpoints) +# - Microsoft Container Registry (MCR) image pullability for Arc agents +# - Optional: graph.microsoft.com if Azure RBAC for Kubernetes is enabled +# +# How to Run Inside the Cluster: +# +# 1. Start an Ubuntu diagnostic pod with resource limits on master node: +# kubectl run arccheck --image=ubuntu:22.04 -it --restart=Never -- bash +# +# If you already have a suitable pod, you can exec into it instead: +# kubectl exec -it -- bash +# i.e. kubectl exec -it arccheck -- bash +# +# 2. Install required tools inside the pod (optimized for speed): +# apt update --quiet && apt install -y --no-install-recommends \ +# curl dnsutils openssl ca-certificates +# +# 3. Download the script into the pod: +# curl -sSL -o arc-node-onboarding-check.sh +# +# 4. Make it executable: +# chmod +x arc-node-onboarding-check.sh +# +# 5. Run the script: +# ./arc-node-onboarding-check.sh westeurope +# +# Output: +# - Per-endpoint DNS / TLS / HTTPS validation results. +# - Summary indicating PASS/FAIL. +# - Remediation hints for proxies, TLS inspection, or blocked egress. +# +# Exit Codes: +# - 0 = All critical checks successful. +# - 1 = One or more required endpoints unreachable. +# +# Intended Audience: +# - Kubernetes administrators validating Arc readiness. +# - Network/security teams reviewing outbound requirements. +# - Architects designing hybrid or on-prem Arc deployments. +# +# ------------------------------------------------------------------------------ + + +REGION="${1:-westeurope}" +CURL_TIMEOUT=7 +OPENSSL_TIMEOUT=7 +COLOR_OK="\033[32m"; COLOR_ERR="\033[31m"; COLOR_WARN="\033[33m"; COLOR_DIM="\033[90m"; COLOR_RESET="\033[0m" + +h1(){ echo -e "\n\033[1m$1\033[0m"; } +ok(){ echo -e "${COLOR_OK}✔${COLOR_RESET} $1"; } +err(){ echo -e "${COLOR_ERR}✖${COLOR_RESET} $1"; } +warn(){ echo -e "${COLOR_WARN}⚠${COLOR_RESET} $1"; } + +RESOLVER_CMD="" +if command -v dig >/dev/null 2>&1; then RESOLVER_CMD="dig +short" +elif command -v nslookup >/dev/null 2>&1; then RESOLVER_CMD="nslookup" +elif command -v getent >/dev/null 2>&1; then RESOLVER_CMD="getent hosts" +else echo "Need dig/nslookup/getent"; exit 2; fi + +# Node-required endpoints (continuous) +NODE_CORE=( + "management.azure.com" # ARM heartbeat/registration + "${REGION}.dp.kubernetesconfiguration.azure.com" # GitOps data-plane + "login.microsoftonline.com" # AAD tokens + "${REGION}.login.microsoft.com" # regional AAD + "login.windows.net" # legacy AAD + "mcr.microsoft.com" # agent/ext images + "gbl.his.arc.azure.com" # MSI certs + "graph.microsoft.com" # RBAC (optional but checked) + "linuxgeneva-microsoft.azurecr.io" # some extensions payloads + "${REGION}.obo.arc.azure.com:8084" # Arc agent communication +) + +PASSED=(); FAILED=() + +check_dns(){ local h="$1" + # Strip port if present for DNS lookup + local host_only="${h%:*}" + if [[ "$RESOLVER_CMD" == "dig +short" ]]; then + dig +short "$host_only" | grep -E '^[0-9a-fA-F:.]+$' >/dev/null && ok "DNS resolves: $host_only" || { err "DNS failed: $host_only"; return 1; } + elif [[ "$RESOLVER_CMD" == "nslookup" ]]; then + nslookup "$host_only" >/dev/null 2>&1 && ok "DNS resolves: $host_only" || { err "DNS failed: $host_only"; return 1; } + else + getent hosts "$host_only" >/dev/null 2>&1 && ok "DNS resolves: $host_only" || { err "DNS failed: $host_only"; return 1; } + fi +} + +check_tls(){ local h="$1" + local host_only="${h%:*}" + local port="${h##*:}" + # If no port specified, default to 443 + [[ "$port" == "$host_only" ]] && port="443" + timeout "${OPENSSL_TIMEOUT}" bash -c "echo | openssl s_client -servername ${host_only} -connect ${host_only}:${port} >/dev/null 2>&1" \ + && ok "TLS handshake OK: ${host_only}:${port}" || { err "TLS handshake failed: ${host_only}:${port}"; return 1; } +} + +check_https(){ local h="$1" + local host_only="${h%:*}" + local port="${h##*:}" + # If no port specified, default to 443 + [[ "$port" == "$host_only" ]] && port="443" + # For non-443 ports, use HTTP instead of HTTPS and just test connectivity + if [[ "$port" == "443" ]]; then + curl -sS -I --connect-timeout "${CURL_TIMEOUT}" "https://${h}" >/dev/null 2>&1 \ + && ok "HTTPS reachable: https://${h}" || { err "HTTPS blocked/unreachable: https://${h}"; return 1; } + else + # For custom ports, test TCP connectivity instead of HTTPS + timeout "${CURL_TIMEOUT}" bash -c "echo >/dev/tcp/${host_only}/${port}" 2>/dev/null \ + && ok "TCP reachable: ${host_only}:${port}" || { err "TCP blocked/unreachable: ${host_only}:${port}"; return 1; } + fi +} + +mcr_probe(){ + local okflag=1 + curl -sS -I --connect-timeout "${CURL_TIMEOUT}" "https://mcr.microsoft.com/v2/" >/dev/null 2>&1 \ + && ok "MCR registry reachable: mcr.microsoft.com" || { err "MCR blocked/unreachable: mcr.microsoft.com"; okflag=0; } + curl -sS -I --connect-timeout "${CURL_TIMEOUT}" "https://mcr.microsoft.com/v2/azurearck8s/agent/tags/list" >/dev/null 2>&1 \ + && ok "MCR path probe OK: /v2/azurearck8s/agent" || warn "MCR path probe failed (CDN/edge filtering?)." + return $([[ $okflag -eq 1 ]]) +} + +h1 "Azure Arc NODE onboarding/steady-state checks (region: ${REGION})" +[[ -n "${HTTPS_PROXY:-}" || -n "${HTTP_PROXY:-}" ]] && echo -e "${COLOR_DIM}Proxy: HTTPS_PROXY=${HTTPS_PROXY:-} HTTP_PROXY=${HTTP_PROXY:-}${COLOR_RESET}" + +h1 "1) DNS" +for h in "${NODE_CORE[@]}"; do + check_dns "$h" && PASSED+=("DNS:$h") || FAILED+=("DNS:$h") +done + +h1 "2) TLS handshakes (443)" +for h in "${NODE_CORE[@]}"; do + check_tls "$h" && PASSED+=("TLS:$h") || FAILED+=("TLS:$h") +done + +h1 "3) HTTPS reachability (443)" +for h in "${NODE_CORE[@]}"; do + check_https "$h" && PASSED+=("HTTPS:$h") || FAILED+=("HTTPS:$h") +done + +h1 "4) MCR probes (agent/ext images)" +mcr_probe && PASSED+=("MCR:mcr.microsoft.com") || FAILED+=("MCR:mcr.microsoft.com") + +h1 "Summary (NODE)" +echo -e "Passed: ${COLOR_OK}${#PASSED[@]}${COLOR_RESET} | Failed: ${COLOR_ERR}${#FAILED[@]}${COLOR_RESET}" +if (( ${#FAILED[@]} > 0 )); then + echo -e "${COLOR_ERR}Failures:${COLOR_RESET}"; for f in "${FAILED[@]}"; do echo " - $f"; done + echo -e "\nHints:\n - Disable TLS inspection for Arc endpoints and MCR/CDN.\n - If pods use a different egress (egress gateway), run this from inside a Pod for parity." + exit 1 +else + echo -e "${COLOR_OK}All critical node onboarding/steady-state paths look good.${COLOR_RESET}"; exit 0 +fi diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-defender-check.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-defender-check.sh new file mode 100644 index 000000000..93028a84a --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-defender-check.sh @@ -0,0 +1,203 @@ +#!/usr/bin/env bash +set -euo pipefail + +# ------------------------------------------------------------------------------ +# Synopsis: Azure Arc NODE Connectivity Check for Defender for Containers +# (Public Mode — Programmatic Enablement Requirements) +# +# This script validates *only* the additional network connectivity required +# to onboard and operate **Defender for Containers** on **Arc-enabled +# Kubernetes clusters** when enabling the extension programmatically. +# +# The checks below are based on the official Defender for Cloud documentation: +# https://learn.microsoft.com/azure/defender-for-cloud/defender-for-containers-arc-enable-programmatically +# +# Purpose: +# Validate the outbound DNS/TLS/HTTPS connectivity from inside a Kubernetes +# cluster to the endpoints used by Defender for Containers for: +# - Microsoft Defender for Cloud backplane services +# - Vulnerability assessment ingestion APIs +# - Microsoft Container Registry for extension images +# - Azure Storage for artifact downloads +# +# What this script checks: +# ✔ Defender for Cloud backend: +# - *.securitycenter.azure.com +# - management.azure.com (ARM calls for onboarding) +# ✔ Defender agent + Arc extension image registry: +# - mcr.microsoft.com +# - *.data.mcr.microsoft.com +# ✔ Vulnerability assessment ingestion: +# - *.prod.securitycenter.windows.com +# - *.vulnerability.assessment.azure.com +# ✔ Azure Storage for Defender artifacts: +# - *.blob.core.windows.net +# +# What this script does NOT check: +# ✘ Azure Monitor / Container Insights endpoints +# ✘ Arc core agent / GitOps / Policy endpoints +# ✘ Cluster Connect (Service Bus) +# ✘ AMPLS / Private Link endpoints +# +# How to run from inside your Kubernetes cluster: +# +# 1. Start a diagnostic pod: +# kubectl run arccheck \ +# --image=ubuntu:22.04 -it --restart=Never -- bash +# +# 2. Install required tools: +# apt update +# apt install -y curl dnsutils openssl ca-certificates +# +# 3. Copy or curl the script into the pod: +# curl -sSL -o arc-node-defender-check.sh +# +# 4. Make it executable: +# chmod +x arc-node-defender-check.sh +# +# 5. Run: +# ./arc-node-defender-check.sh +# +# Output: +# - DNS, TLS, and HTTPS validation per endpoint +# - Summary with PASS / FAIL status +# - Troubleshooting hints for network teams +# +# Exit Codes: +# - 0 = all required Defender endpoints reachable +# - 1 = one or more endpoints unreachable +# +# ------------------------------------------------------------------------------ + +CURL_TIMEOUT=7 +OPENSSL_TIMEOUT=7 + +COLOR_OK="\033[32m" +COLOR_ERR="\033[31m" +COLOR_WARN="\033[33m" +COLOR_DIM="\033[90m" +COLOR_RESET="\033[0m" + +h1(){ echo -e "\n\033[1m$1\033[0m"; } +ok(){ echo -e "${COLOR_OK}✔${COLOR_RESET} $1"; } +err(){ echo -e "${COLOR_ERR}✖${COLOR_RESET} $1"; } +warn(){ echo -e "${COLOR_WARN}⚠${COLOR_RESET} $1"; } + +# Determine DNS resolver +RESOLVER_CMD="" +if command -v dig >/dev/null 2>&1; then RESOLVER_CMD="dig +short" +elif command -v nslookup >/dev/null 2>&1; then RESOLVER_CMD="nslookup" +elif command -v getent >/dev/null 2>&1; then RESOLVER_CMD="getent hosts" +else echo "ERROR: Install dig/nslookup/getent"; exit 2; fi + +# ------------------------------------------------------------------------------ +# Defender for Containers Required Endpoints (Public Cloud) +# ------------------------------------------------------------------------------ + +DEFENDER_ENDPOINTS=( + # ARM API for onboarding Defender extension + "management.azure.com" + + # Defender for Cloud backend + "securitycenter.azure.com" + "*.securitycenter.azure.com" + "*.prod.securitycenter.windows.com" + "*.vulnerability.assessment.azure.com" + + # MCR (image pulls for Defender extension) + "mcr.microsoft.com" + "*.data.mcr.microsoft.com" + + # Azure Storage (artifact downloads) + "*.blob.core.windows.net" +) + +PASSED=() +FAILED=() + +# ------------------------------------------------------------------------------ +# Helper functions +# ------------------------------------------------------------------------------ + +dns_check(){ + local host="$1" + [[ "$host" == *"*"* ]] && host="${host#*.}" # strip wildcard for DNS test + + if [[ "$RESOLVER_CMD" == "dig +short" ]]; then + dig +short "$host" | grep -E '^[0-9a-fA-F:.]+$' >/dev/null \ + && ok "DNS resolves: $host" \ + || { err "DNS failed: $host"; return 1; } + elif [[ "$RESOLVER_CMD" == "nslookup" ]]; then + nslookup "$host" >/dev/null 2>&1 \ + && ok "DNS resolves: $host" \ + || { err "DNS failed: $host"; return 1; } + else + getent hosts "$host" >/dev/null 2>&1 \ + && ok "DNS resolves: $host" \ + || { err "DNS failed: $host"; return 1; } + fi +} + +tls_check(){ + local host="$1" + [[ "$host" == *"*"* ]] && host="${host#*.}" + + timeout "$OPENSSL_TIMEOUT" bash -c \ + "echo | openssl s_client -servername ${host} -connect ${host}:443 >/dev/null 2>&1" \ + && ok "TLS OK: ${host}:443" \ + || { err "TLS FAILED: ${host}:443"; return 1; } +} + +https_check(){ + local host="$1" + [[ "$host" == *"*"* ]] && host="${host#*.}" + + curl -sS -I --connect-timeout "$CURL_TIMEOUT" "https://${host}" >/dev/null 2>&1 \ + && ok "HTTPS OK: https://${host}" \ + || { err "HTTPS FAILED: https://${host}"; return 1; } +} + +# ------------------------------------------------------------------------------ +# Execution +# ------------------------------------------------------------------------------ + +h1 "Defender for Containers NODE Connectivity Check (Public Mode)" + +[[ -n "${HTTPS_PROXY:-}" || -n "${HTTP_PROXY:-}" ]] \ + && echo -e "${COLOR_DIM}Proxy detected → HTTPS_PROXY=${HTTPS_PROXY:-}${COLOR_RESET}" + +# DNS +h1 "1) DNS Checks" +for host in "${DEFENDER_ENDPOINTS[@]}"; do + dns_check "$host" && PASSED+=("DNS:$host") || FAILED+=("DNS:$host") +done + +# TLS +h1 "2) TLS Handshake Checks (443)" +for host in "${DEFENDER_ENDPOINTS[@]}"; do + tls_check "$host" && PASSED+=("TLS:$host") || FAILED+=("TLS:$host") +done + +# HTTPS Reachability +h1 "3) HTTPS Reachability Checks (443)" +for host in "${DEFENDER_ENDPOINTS[@]}"; do + https_check "$host" && PASSED+=("HTTPS:$host") || FAILED+=("HTTPS:$host") +done + +h1 "Summary" +echo -e "Passed: ${COLOR_OK}${#PASSED[@]}${COLOR_RESET} | Failed: ${COLOR_ERR}${#FAILED[@]}${COLOR_RESET}" + +if (( ${#FAILED[@]} > 0 )); then + echo -e "${COLOR_ERR}Failures:${COLOR_RESET}" + for f in "${FAILED[@]}"; do echo " - $f"; done + + echo -e "\nTroubleshooting:" + echo " - Ensure outbound HTTPS to *.securitycenter.azure.com and *.vulnerability.assessment.azure.com" + echo " - Ensure mcr.microsoft.com and *.data.mcr.microsoft.com reachable for Defender images" + echo " - Ensure *.blob.core.windows.net reachable for Defender artifacts" + echo " - Disable TLS inspection for all Defender ingestion domains" + exit 1 +else + echo -e "${COLOR_OK}All Defender for Containers connectivity checks passed.${COLOR_RESET}" + exit 0 +fi \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-monitor-connectivity-check.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-monitor-connectivity-check.sh new file mode 100644 index 000000000..e304ea038 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/scripts/arc-node-monitor-connectivity-check.sh @@ -0,0 +1,231 @@ +#!/usr/bin/env bash +set -euo pipefail + +# ------------------------------------------------------------------------------ +# Synopsis: Azure Arc NODE Monitoring Connectivity Validation Script +# (Public Mode, v3 — region-aware, instance-aware) +# +# Purpose: +# Validate outbound DNS/TLS/HTTPS from inside the Kubernetes cluster to +# the Azure Monitor endpoints used by: +# • Container Insights (logs via Data Collection Endpoints / workspace) +# • Managed Prometheus (metrics via DCE) +# in PUBLIC (non–Private Link / non–AMPLS) mode. +# +# Why v3: +# Earlier versions probed base/wildcard hosts that do not exist in DNS +# (e.g., "ingest.monitor.azure.com", "ods.opinsights.azure.com", +# "blob.core.windows.net") and produced false failures. v3 only checks +# globally resolvable endpoints by default and lets you specify your real +# ingestion/workspace/blob FQDNs explicitly. +# +# What is always checked (public mode): +# ✔ Global + regional control handlers: +# - global.handler.control.monitor.azure.com +# - .handler.control.monitor.azure.com +# ✔ Authentication & telemetry: +# - login.microsoftonline.com +# - dc.services.visualstudio.com +# ✔ Extension container images: +# - mcr.microsoft.com +# +# What you can additionally validate (your real FQDNs): +# • Logs DCE: dce-..ingest.monitor.azure.com +# • Metrics DCE: dce-..metrics.ingest.monitor.azure.com +# • Workspace ingestion: +# .ods.opinsights.azure.com +# (optional legacy) +# .oms.opinsights.azure.com +# • Blob storage account used by agents/extensions: +# .blob.core.windows.net +# +# Flags (all optional — supply the actual hostnames for your environment): +# --dce-logs e.g., dce-1234...abcd.westeurope.ingest.monitor.azure.com +# --dce-metrics e.g., dce-5678...ef90.westeurope.metrics.ingest.monitor.azure.com +# --ods-host e.g., 1234...abcd.ods.opinsights.azure.com +# --oms-host e.g., 1234...abcd.oms.opinsights.azure.com +# --blob-account e.g., mystorageacct.blob.core.windows.net +# +# Usage: +# # Minimal (global/region control, AAD, telemetry, MCR): +# ./arc-node-monitoring-check_v3.sh westeurope +# +# # Full run with real endpoints (replace with your FQDNs): +# ./arc-node-monitoring-check_v3.sh westeurope \ +# --dce-logs dce-12345678-90ab-cdef-1234-567890abcdef.westeurope.ingest.monitor.azure.com \ +# --dce-metrics dce-22345678-90ab-cdef-1234-567890abcdef.westeurope.metrics.ingest.monitor.azure.com \ +# --ods-host 12345678-90ab-cdef-1234-567890abcdef.ods.opinsights.azure.com \ +# --blob-account mystorageacct.blob.core.windows.net +# +# How to discover your FQDNs (from a client with Azure CLI): +# az monitor data-collection endpoint list -g -o table +# az monitor data-collection rule list -g -o table +# # Copy the ingestion URLs and workspace hostnames into the flags above. +# +# Output & Exit Codes: +# • Prints DNS/TLS/HTTPS results per endpoint and a summary. +# • exit 0 → all tested endpoints OK +# • exit 1 → one or more tested endpoints failed +# +# Requirements inside the pod: +# curl, dnsutils (or nslookup/getent), openssl, ca-certificates. +# +# References (public mode firewall guidance): +# - Network firewall requirements for monitoring Kubernetes clusters +# - Outbound network & FQDN rules for AKS (Azure Monitor section) +# ------------------------------------------------------------------------------ + +REGION_RAW="${1:-westeurope}" +shift || true + +# Optional flags: pass actual endpoint FQDNs +DCE_LOGS_FQDN="" +DCE_METRICS_FQDN="" +ODS_FQDN="" +OMS_FQDN="" +BLOB_ACCOUNT_FQDN="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --dce-logs) DCE_LOGS_FQDN="${2:-}"; shift 2 ;; + --dce-metrics) DCE_METRICS_FQDN="${2:-}"; shift 2 ;; + --ods-host) ODS_FQDN="${2:-}"; shift 2 ;; + --oms-host) OMS_FQDN="${2:-}"; shift 2 ;; + --blob-account) BLOB_ACCOUNT_FQDN="${2:-}"; shift 2 ;; + *) echo "Unknown flag: $1"; exit 2 ;; + esac +done + +# Normalize region to Azure FQDN segment +REGION="$(echo "$REGION_RAW" | tr '[:upper:]' '[:lower:]' | tr -d ' ')" + +CURL_TIMEOUT=7 +OPENSSL_TIMEOUT=7 + +COLOR_OK="\033[32m" +COLOR_ERR="\033[31m" +COLOR_WARN="\033[33m" +COLOR_DIM="\033[90m" +COLOR_RESET="\033[0m" + +h1(){ echo -e "\n\033[1m$1\033[0m"; } +ok(){ echo -e "${COLOR_OK}✔${COLOR_RESET} $1"; } +err(){ echo -e "${COLOR_ERR}✖${COLOR_RESET} $1"; } +warn(){ echo -e "${COLOR_WARN}⚠${COLOR_RESET} $1"; } + +# DNS resolver selection +RESOLVER_CMD="" +if command -v dig >/dev/null 2>&1; then + RESOLVER_CMD="dig +short" +elif command -v nslookup >/dev/null 2>&1; then + RESOLVER_CMD="nslookup" +elif command -v getent >/dev/null 2>&1; then + RESOLVER_CMD="getent hosts" +else + echo "Need dig/nslookup/getent installed"; exit 2 +fi + +# Always-available endpoints in public mode +CONTROL_ENDPOINTS=( + "global.handler.control.monitor.azure.com" + "${REGION}.handler.control.monitor.azure.com" +) +AUX_ENDPOINTS=( + "login.microsoftonline.com" + "dc.services.visualstudio.com" +) +ARTIFACT_ENDPOINTS=( + "mcr.microsoft.com" +) + +# Optional environment-specific endpoints (only if provided) +OPTIONAL_ENDPOINTS=() +[[ -n "$DCE_LOGS_FQDN" ]] && OPTIONAL_ENDPOINTS+=("$DCE_LOGS_FQDN") +[[ -n "$DCE_METRICS_FQDN" ]] && OPTIONAL_ENDPOINTS+=("$DCE_METRICS_FQDN") +[[ -n "$ODS_FQDN" ]] && OPTIONAL_ENDPOINTS+=("$ODS_FQDN") +[[ -n "$OMS_FQDN" ]] && OPTIONAL_ENDPOINTS+=("$OMS_FQDN") +[[ -n "$BLOB_ACCOUNT_FQDN" ]] && OPTIONAL_ENDPOINTS+=("$BLOB_ACCOUNT_FQDN") + +ALL_ENDPOINTS=( + "${CONTROL_ENDPOINTS[@]}" + "${AUX_ENDPOINTS[@]}" + "${ARTIFACT_ENDPOINTS[@]}" + "${OPTIONAL_ENDPOINTS[@]}" +) + +PASSED=() +FAILED=() +SKIPPED=() + +dns_check(){ + local host="$1" + if [[ "$RESOLVER_CMD" == "dig +short" ]]; then + dig +short "$host" | grep -E '^[0-9a-fA-F:.]+$' >/dev/null \ + && ok "DNS resolves: $host" || { err "DNS failed: $host"; return 1; } + elif [[ "$RESOLVER_CMD" == "nslookup" ]]; then + nslookup "$host" >/dev/null 2>&1 \ + && ok "DNS resolves: $host" || { err "DNS failed: $host"; return 1; } + else + getent hosts "$host" >/dev/null 2>&1 \ + && ok "DNS resolves: $host" || { err "DNS failed: $host"; return 1; } + fi +} + +tls_check(){ + local host="$1" + timeout "$OPENSSL_TIMEOUT" bash -c "echo | openssl s_client -servername ${host} -connect ${host}:443 >/dev/null 2>&1" \ + && ok "TLS OK: ${host}:443" || { err "TLS FAILED: ${host}:443"; return 1; } +} + +https_check(){ + local host="$1" + curl -sS -I --connect-timeout "$CURL_TIMEOUT" "https://${host}" >/dev/null 2>&1 \ + && ok "HTTPS OK: https://${host}" || { err "HTTPS FAILED: https://${host}"; return 1; } +} + +h1 "Azure Arc NODE Monitoring Connectivity Check (Public Mode, v3) — region: ${REGION}" +[[ -n "${HTTPS_PROXY:-}" || -n "${HTTP_PROXY:-}" ]] && echo -e "${COLOR_DIM}Proxy: HTTPS_PROXY=${HTTPS_PROXY:-} HTTP_PROXY=${HTTP_PROXY:-}${COLOR_RESET}" + +# Inform about skipped environment-specific checks +if [[ -z "$DCE_LOGS_FQDN" || -z "$DCE_METRICS_FQDN" || -z "$ODS_FQDN" || -z "$BLOB_ACCOUNT_FQDN" ]]; then + h1 "Note on environment-specific endpoints (skipped unless provided)" + [[ -z "$DCE_LOGS_FQDN" ]] && { warn "Skipping Logs DCE (use --dce-logs )"; SKIPPED+=("DCE_LOGS"); } + [[ -z "$DCE_METRICS_FQDN" ]] && { warn "Skipping Metrics DCE (use --dce-metrics )"; SKIPPED+=("DCE_METRICS"); } + [[ -z "$ODS_FQDN" ]] && { warn "Skipping Workspace ingestion (use --ods-host / --oms-host )"; SKIPPED+=("ODS/OMS"); } + [[ -z "$BLOB_ACCOUNT_FQDN" ]] && { warn "Skipping Blob account check (use --blob-account .blob.core.windows.net)"; SKIPPED+=("BLOB"); } +fi + +h1 "1) DNS" +for host in "${ALL_ENDPOINTS[@]}"; do + dns_check "$host" && PASSED+=("DNS:$host") || FAILED+=("DNS:$host") +done + +h1 "2) TLS Handshake (443)" +for host in "${ALL_ENDPOINTS[@]}"; do + tls_check "$host" && PASSED+=("TLS:$host") || FAILED+=("TLS:$host") +done + +h1 "3) HTTPS Reachability (443)" +for host in "${ALL_ENDPOINTS[@]}"; do + https_check "$host" && PASSED+=("HTTPS:$host") || FAILED+=("HTTPS:$host") +done + +h1 "Summary" +echo -e "Passed: ${COLOR_OK}${#PASSED[@]}${COLOR_RESET} | Failed: ${COLOR_ERR}${#FAILED[@]}${COLOR_RESET} | Skipped: ${COLOR_WARN}${#SKIPPED[@]}${COLOR_RESET}" + +if (( ${#FAILED[@]} > 0 )); then + echo -e "${COLOR_ERR}Failures:${COLOR_RESET}" + for f in "${FAILED[@]}"; do echo " - $f"; done + echo -e "\nTroubleshooting Hints:" + echo " - Allow outbound HTTPS (443) to your DCE hosts (logs & metrics) and workspace ingestion FQDNs." + echo " - Allow global and regional handler.control endpoints." + echo " - Disable TLS inspection; agents expect end-to-end TLS." + echo " - Ensure mcr.microsoft.com and your Blob account are reachable for extension artifacts." + exit 1 +else + echo -e "${COLOR_OK}All required checks passed for the endpoints tested.${COLOR_RESET}" + if (( ${#SKIPPED[@]} > 0 )); then + echo -e "${COLOR_WARN}Note: ${#SKIPPED[@]} environment-specific checks were skipped. Supply flags to validate them.${COLOR_RESET}" + fi + exit 0 +fi \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/start_here.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/start_here.sh new file mode 100644 index 000000000..ae3624dad --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/start_here.sh @@ -0,0 +1,20 @@ +subscription_id=$(az account show --query id --output tsv) +echo "Using subcription ID: $subscription_id" + +# Create a service principal with the name "mh-arc-aks-onprem" and assign it the "Contributor" role +# service_principal=$(az ad sp create-for-rbac -n "mh-arc-aks-onprem" --role "Contributor" --scopes /subscriptions/$subscription_id) +# echo "created service principal: $service_principal" + +# Extract client_id and client_secret from the service_principal JSON output +# client_id=$(echo $service_principal | jq -r .appId) +# client_secret=$(echo $service_principal | jq -r .password) + +# echo "replacing client_id and client_secret in fixtures.tfvars..." +# # Replace client_id and client_secret in fixtures.tfvars +# sed -i "s|client_id=\".*\"|client_id=\"$client_id\"|" fixtures.tfvars +# sed -i "s|client_secret=\".*\"|client_secret=\"$client_secret\"|" fixtures.tfvars + +echo "replacing subscription_id in provider.tf..." +# replace the subscription id in provider.tf +sed -i "s|subscription_id = \".*\"|subscription_id = \"$subscription_id\"|" provider.tf + diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/variables.tf b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/variables.tf new file mode 100644 index 000000000..a88008a40 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/lab/variables.tf @@ -0,0 +1,80 @@ +variable "start_index" { + description = "Starting index for resource naming" + type = number + default = 37 +} + +variable "end_index" { + description = "Ending index for resource naming" + type = number + default = 39 +} + +variable "arc_location" { + description = "The Azure Region in which all resources for Azure Arc should be provisioned" + default = "westeurope" +} + +variable "onprem_resources" { + description = "The Azure Regions in which K3s cluster VMs should be provisioned" + default = ["francecentral", "swedencentral", "germanywestcentral", "northeurope", "uksouth"] +} + +variable "resource_group_base_name" { + description = "Base name for resource groups (will be prefixed with index)" + default = "k8s" +} + +variable "k3s_cluster_base_name" { + description = "Base name for K3s cluster resources" + default = "k3s-onprem" +} + +variable "prefix" { + description = "A prefix used for all K3s cluster resources" + default = "k3s" +} + +variable "k3s_version" { + description = "K3s version to install" + default = "v1.33.6+k3s1" +} + +variable "cluster_token" { + description = "Token for K3s cluster authentication" + type = string + sensitive = true +} + +variable "admin_user" { + description = "Admin username for VMs" + type = string +} + +variable "admin_password" { + description = "Admin password for VMs" + type = string + sensitive = true +} + +variable "vm_size" { + description = "The Azure VM size for K3s nodes" + default = "Standard_D4ds_v6" # For arc-enabled Managed SQL Instances, ARM cores not supported +} + +# container reguistry variables for gitops challenge +variable "acr_name" { + description = "The name of the Azure Container Registry" + default = "mhacr" +} + +variable "container_registry_sku" { + description = "The SKU of the Azure Container Registry" + default = "Basic" +} + +variable "container_registry_admin_enabled" { + description = "Specifies whether the admin user is enabled. Defaults to false." + type = bool + default = true +} diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/az_connect_k8s.sh b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/az_connect_k8s.sh new file mode 100644 index 000000000..5c83bdd8b --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/az_connect_k8s.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# exit on first error +set -e + +# This script connects an existing K3s cluster to Azure Arc with Azure RBAC enabled +echo "Exporting environment variables" + +# Extract user number from Azure username (e.g., LabUser-37 -> 37) +azure_user=$(az account show --query user.name --output tsv) +user_number=$(echo "$azure_user" | sed -E -n 's/.*[^0-9]([0-9]+)$/\1/p' | sed 's/^0*//; s/^$/0/') + +if [ -z "$user_number" ]; then + echo "Error: Could not extract user number from Azure username: $azure_user" + echo "Please make sure you're logged in as LabUser-XX" + exit 1 +fi + +echo "Detected user number: $user_number" + +# Set variables based on detected user number +export onprem_resource_group="${user_number}-k8s-onprem" +export arc_resource_group="${user_number}-k8s-arc" +export arc_cluster_name="${user_number}-k8s-arc-enabled" +export location="westeurope" + +echo "Using resource groups: $onprem_resource_group (onprem) and $arc_resource_group (arc)" + +# Registering Azure Arc providers +echo "Registering Azure Arc providers" +az provider register --namespace Microsoft.Kubernetes --wait +az provider register --namespace Microsoft.KubernetesConfiguration --wait +az provider register --namespace Microsoft.ExtendedLocation --wait + +az provider show -n Microsoft.Kubernetes -o table +az provider show -n Microsoft.KubernetesConfiguration -o table +az provider show -n Microsoft.ExtendedLocation -o table + +echo "Clear cached helm Azure Arc Helm Charts" +rm -rf ~/.azure/AzureArcCharts + +echo "Checking if you have up-to-date Azure Arc AZ CLI 'connectedk8s' extension..." +if ! az extension show --name connectedk8s > /dev/null 2>&1; then + az extension add --name connectedk8s +else + az extension update --name connectedk8s +fi +echo "" + +echo "Checking if you have up-to-date Azure Arc AZ CLI 'k8s-configuration' extension..." +if ! az extension show --name k8s-configuration > /dev/null 2>&1; then + az extension add --name k8s-configuration +else + az extension update --name k8s-configuration +fi +echo "" + +echo "Connecting the cluster to Azure Arc" +az connectedk8s connect --name $arc_cluster_name \ + --resource-group $arc_resource_group \ + --location $location \ + --infrastructure 'generic' \ + --distribution 'k3s' + +echo "Waiting for Arc connection to be established..." +sleep 30 + +echo "" +echo "Summary:" +echo " - Resource Group: $arc_resource_group" +echo " - Status:" +az connectedk8s show --resource-group $arc_resource_group --name $arc_cluster_name --query "{name:name, connectivityStatus:connectivityStatus}" + diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/access-token.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/access-token.png new file mode 100644 index 000000000..2be8241d5 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/access-token.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/namespaces.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/namespaces.png new file mode 100644 index 000000000..bb4d76b5b Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/namespaces.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/vm-start.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/vm-start.png new file mode 100644 index 000000000..8abd8411c Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/img/vm-start.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/solution.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/solution.md new file mode 100644 index 000000000..333d69846 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-01/solution.md @@ -0,0 +1,202 @@ +# Walkthrough Challenge 1 - Onboarding your Kubernetes Cluster +Duration: 15-30 minutes + +## Prerequisites +Please ensure that you successfully verified +* the [general prerequisites](../../Readme.md#general-prerequisites) before starting this challenge. +* that you can see your two resource groups in the [Azure portal](https://portal.azure.com) depending on your LabUser number. I.e. if you are LabUser-37, you should see the resource groups "37-k8s-arc" and "37-k8s-onprem". +* that you can successfully connect to all [required Azure endpoints](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements?tabs=azure-cloud) + +💡*Hint*: There are several connectivity-check scripts available [here](../../lab/scripts/). + +## Task 1 - Login to Azure +In your shell environment, login to Azure using the account you got assigned during the microhack. +```bash +az logout # only required if you are logged in with another user from a previous session + +az login # browser popup opens with credential prompt. Provide the user credentials you got from your microhack coach +``` +In case you are prompted to select a subscription, please do so. In the microhack environment you just can hit enter as you only have one subscription available. + +Validate that you can see your two resource groups in the [Azure portal](https://portal.azure.com) depending on your LabUser number. I.e. if you are LabUser-37, you should see the resource groups "37-k8s-arc" and "37-k8s-onprem". +Click on your onprem resource group's name (i.e. 37-k8s-onprem). +There should be 3 VMs in this resource group. Make sure that all VMs are in state 'running'. +![img-start-vm](img/vm-start.png) + +To connect to your k8s cluster, we first need to merge the cluster credentials into your local ~/.kube/config file. You can use the following bash script for this: +```bash +# Set admin username (use the admin_user value provided by your coach) +admin_user="" + +# Extract trailing number from Azure username before '@' (e.g., LabUser-37@... or hackuser-067@... -> 37 or 67) +azure_user=$(az account show --query user.name --output tsv) +user_number=$(echo "${azure_user%@*}" | grep -oE '[0-9]+' | tail -n1 | sed 's/^0*//; s/^$/0/') + +# Get public ip of master node via Azure cli according to user-number +master_pip=$(az vm list-ip-addresses --resource-group "${user_number}-k8s-onprem" --name "${user_number}-k8s-master" --query "[0].virtualMachine.network.publicIpAddresses[0].ipAddress" --output tsv) + +# Create .kube directory if it doesn't exist +mkdir -p ~/.kube + +# Copy the kubeconfig to standard location +scp $admin_user@$master_pip:~/.kube/config ~/.kube/config + +# replace localhost address with the public ip of master node +sed -i "s/127.0.0.1/$master_pip/g" ~/.kube/config + +# Now kubectl works directly on your local client - no need to ssh into the master node anymore +kubectl get nodes +``` + +## Task 2 - Connect K8s cluster using script + +* In your shell go to the folder where you cloned the microhack repository +* Change to the sub-folder '03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthroughs/challenge-01' +* Execute the script to + * register required resource providers in your subscription (this step may take several minutes if the resource providers have not been registered before): + * Microsoft.Kubernetes + * Microsoft.KubernetesConfiguration + * Microsoft.ExtendedLocation + * remove Azure Arc helm charts which might exist from previous connection attempts + * install required Azure CLI extensions or update them to latest version: + * connectedk8s + * k8s-configuration + * connecting the simulated onprem cluster to Azure Arc using the Azure CLI approach + +💡 *Important*: Make sure that your kubectl works and is pointing to the k8s cluster you want to onboard before executing the script! + +```bash +# Clone the repository (or copy the script content) +git clone https://github.com/microsoft/MicroHack.git +cd MicroHack/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthroughs/challenge-01 + +# Make the script executable and run it +chmod +x az_connect_k8s.sh +./az_connect_k8s.sh +``` + +Wait until the script terminates. Expected result should look comparable to this output: +```bash +Exporting environment variables +Registering Azure Arc providers +Namespace RegistrationPolicy RegistrationState +-------------------- -------------------- ------------------- +Microsoft.Kubernetes RegistrationRequired Registered +Namespace RegistrationPolicy RegistrationState +--------------------------------- -------------------- ------------------- +Microsoft.KubernetesConfiguration RegistrationRequired Registered +Namespace RegistrationPolicy RegistrationState +-------------------------- -------------------- ------------------- +Microsoft.ExtendedLocation RegistrationRequired Registered +Getting AKS credentials (kubeconfig) +Merged "37-k8s-onprem" as current context in /home/simon/.kube/config +Clear cached helm Azure Arc Helm Charts +Checking if you have up-to-date Azure Arc AZ CLI 'connectedk8s' extension... +Latest version of 'connectedk8s' is already installed. + +Use --debug for more information + +Checking if you have up-to-date Azure Arc AZ CLI 'k8s-configuration' extension... +Latest version of 'k8s-configuration' is already installed. + +Use --debug for more information + +Connecting the cluster to Azure Arc +This operation might take a while... + +Step: 2025-12-12T14-32-41Z: Validating custom access token +Step: 2025-12-12T14-32-41Z: Checking Provider Registrations +Step: 2025-12-12T14-32-42Z: Setting KubeConfig +Step: 2025-12-12T14-32-42Z: Escape Proxy Settings, if passed in +Step: 2025-12-12T14-32-42Z: Checking Connectivity to Cluster +Step: 2025-12-12T14-32-43Z: Do node validations +Step: 2025-12-12T14-32-43Z: Install Kubectl client if it does not exist +Step: 2025-12-12T14-32-43Z: Install Helm client if it does not exist +Step: 2025-12-12T14-32-43Z: Starting Pre-onboarding-check +Step: 2025-12-12T14-32-43Z: Creating folder for Cluster Diagnostic Checks Logs +Step: 2025-12-12T14-32-43Z: Get namespace of release: cluster-diagnostic-checks +Step: 2025-12-12T14-32-44Z: Determine Helmchart Export Path +Step: 2025-12-12T14-32-44Z: Pulling HelmChart: mcr.microsoft.com/azurearck8s/helmchart/stable/clusterdiagnosticchecks, Version: 1.31.2 +Step: 2025-12-12T14-32-46Z: Chart path for Cluster Diagnostic Checks Job: /home/simon/.azure/PreOnboardingChecksCharts/clusterdiagnosticchecks +Step: 2025-12-12T14-32-46Z: Creating Cluster Diagnostic Checks job +Step: 2025-12-12T14-32-59Z: The required pre-checks for onboarding have succeeded. +Step: 2025-12-12T14-32-59Z: Checking if user can create ClusterRoleBindings +Step: 2025-12-12T14-32-59Z: Determining Cluster Distribution and Infrastructure +Connecting an Azure Kubernetes Service (AKS) cluster to Azure Arc is only required for running Arc enabled services like App Services and Data Services on the cluster. Other features like Azure Monitor and Azure Defender are natively available on AKS. Learn more at https://go.microsoft.com/fwlink/?linkid=2144200. +Step: 2025-12-12T14-32-59Z: Checking Connect RP is available in the Location passed in. +Step: 2025-12-12T14-32-59Z: Check if an earlier azure-arc release exists +Step: 2025-12-12T14-32-59Z: Get namespace of release: azure-arc +Step: 2025-12-12T14-33-01Z: Deleting Arc CRDs +Step: 2025-12-12T14-33-09Z: Check if ResourceGroup exists. Try to create if it doesn't +Step: 2025-12-12T14-33-09Z: Generating Public-Private Key pair +Step: 2025-12-12T14-33-14Z: Generating ARM Request Payload +Step: 2025-12-12T14-33-14Z: Azure resource provisioning has begun. +Step: 2025-12-12T14-34-49Z: Checking Custom Location(Microsoft.ExtendedLocation) RP Registration state for this Subscription, and attempt to get the Custom Location Object ID (OID),if registered +Step: 2025-12-12T14-34-52Z: Azure resource provisioning has finished. +Step: 2025-12-12T14-34-53Z: Determine Helmchart Export Path +Step: 2025-12-12T14-34-53Z: Pulling HelmChart: mcr.microsoft.com/azurearck8s/batch1/stable/v2/azure-arc-k8sagents, Version: 1.31.3 +Step: 2025-12-12T14-34-55Z: Starting to install Azure arc agents on the Kubernetes cluster. +{ + "aadProfile": { + "adminGroupObjectIDs": null, + "tenantId": null + }, + [...] + "arcAgentProfile": { + "agentAutoUpgrade": "Enabled", + "agentErrors": null, + "agentState": null, + "desiredAgentVersion": null, + "systemComponents": null + }, + "arcAgentryConfigurations": null, + "azureHybridBenefit": "NotApplicable", + "connectivityStatus": "Connecting", + "distribution": "k3s", + "distributionVersion": null, + "gateway": null, + [...] + "location": "westeurope", + "managedIdentityCertificateExpirationTime": null, + "miscellaneousProperties": null, + "name": "37-k8s-arc-enabled", + "offering": null, + "oidcIssuerProfile": null, + "privateLinkScopeResourceId": null, + "privateLinkState": "Disabled", + "provisioningState": "Succeeded", + "resourceGroup": "37-k8s-arc", + "securityProfile": null, + [...] + "tags": {}, + "totalCoreCount": null, + "totalNodeCount": null, + "type": "microsoft.kubernetes/connectedclusters" +} +``` +In the [Azure portal](https://portal.azure.com) type 'Azure Arc' into the search bar at the top of the page. In the results in section 'Services' click 'Azure Arc'. +In the Azure Arc page in the left navigation pane, open the 'infrastucture' section and click 'Kubernetes clusters'. +You should see your resource of type 'Kubernetes - Azure Arc'. (i.e. if you are LabUser-37 you should see a resource named '37-k8s-arc-enabled'). Click on the name of the resource matching your user number. + +Notice the Arc Agent version in the overview page. + +In the navigation pane in section 'Kubernetes resources' click on 'Namespaces'. You will see a prompt to provide an access token: + +![access-token](img/access-token.png) + +In order to get access to the k8s resources from the Azure portal assign your entra user a clusterRoleBinding with appropriate permissions: +```bash +# get the user principal from entra +azure_user=$(az ad signed-in-user show --query userPrincipalName -o tsv) + +# create a clusterRoleBinding for the user +kubectl create clusterrolebinding demo-user-binding --clusterrole cluster-admin --user=$azure_user + +``` +Now, reload the resources page in the Azure portl. You should see at least the following namespaces: + +![img-namespaces](img/namespaces.png) + +You successfully completed challenge 1! 🚀🚀🚀 + +[Next challenge](../challenge-02/solution.md) - [Next Challenge's Solution](../../walkthroughs/challenge-02/solution.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/01_env_settings.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/01_env_settings.png new file mode 100644 index 000000000..f2c74c1dd Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/01_env_settings.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/02_container_plan_settings.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/02_container_plan_settings.png new file mode 100644 index 000000000..d4852b8f9 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/02_container_plan_settings.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/03_settings_n_monitoring.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/03_settings_n_monitoring.png new file mode 100644 index 000000000..daf0de72a Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/03_settings_n_monitoring.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/04_save.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/04_save.png new file mode 100644 index 000000000..35db6b927 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/04_save.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/05_defender.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/05_defender.png new file mode 100644 index 000000000..bec0d5d3c Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/img/05_defender.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/solution.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/solution.md new file mode 100644 index 000000000..368886f88 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-02/solution.md @@ -0,0 +1,368 @@ +# Walkthrough Challenge 2 - Enable Azure Monitor for Containers + +Duration: 30-45 min + +## Prerequisites +* You have an arc-connected k8s cluster/finisched challenge 01. +* You require at least Contributor access to the cluster for onboarding. +* You require Monitoring Reader or Monitoring Contributor to view data after monitoring is enabled. +* Verify the firewall requirements in addition to the Azure Arc-enabled Kubernetes network requirements. +* A Log Analytics workspace (law). (If you used the terraform to deploy the microhack environment, each participant already has a law in his arc resource group.) +* You must be logged in to az cli (az login) + +## Task 1 - Enable Azure Monitor for k8s +Execute the following commands in your bash shell to install the container log extension with default settings: +```bash +# Extract user number from Azure username before '@' (e.g., LabUser-37@... -> 37) +azure_user=$(az account show --query user.name --output tsv) +user_number=$(echo "${azure_user%@*}" | grep -oE '[0-9]+' | tail -n1 | sed 's/^0*//; s/^$/0/') +echo $user_number + +# if you are running this in a non-microhack env, adjust the values to match your env +export arc_resource_group="${user_number}-k8s-arc" +export arc_cluster_name="${user_number}-k8s-arc-enabled" +export law_resource_id=$(az monitor log-analytics workspace show --resource-group $arc_resource_group --workspace-name "${user_number}-law" --query 'id' -o tsv) + +az k8s-extension create \ + --name azuremonitor-containers \ + --cluster-name $arc_cluster_name \ + --resource-group $arc_resource_group \ + --cluster-type connectedClusters \ + --extension-type Microsoft.AzureMonitor.Containers \ + --configuration-settings azure-monitor-workspace-resource-id=$law_resource_id + +``` +The output should look roughly like this: +```bash +Ignoring name, release-namespace and scope parameters since microsoft.azuremonitor.containers only supports cluster scope and single instance of this extension. +Defaulting to extension name 'azuremonitor-containers' and release-namespace 'azuremonitor-containers' +{ + [...] + "isSystemExtension": false, + "name": "azuremonitor-containers", + "packageUri": null, + "plan": null, + "provisioningState": "Succeeded", + "releaseTrain": "Stable", + "resourceGroup": "37-k8s-arc", + "scope": { + "cluster": { + "releaseNamespace": "azuremonitor-containers" + }, + "namespace": null + }, + [...] + "type": "Microsoft.KubernetesConfiguration/extensions", + "version": null +} +``` + +To verify the installation, navigate to your arc-enabled k8s cluster in the Azure portal. +* In the left navigation pane in section Monitoring select Insights. Then in the main windows check the tabs Cluster, Reports, Nodes, Controllers and Containers. You should see a dashboard in each tab. Please note that it takes a few minutes after activation until the first values are displayed. +* In tab Containers find "clusterconnectservice-operator" and click the title. This opens an Overview pane on the right hand side. Click on "View in Log Analytics" to see the stdout logs of this container. + +## Task 2 - Enable Defender for Containers Plan +To enable the Defender for Containers plan on your subscription, +* Open the [Defender for Cloud | Environment settings](https://portal.azure.com/#view/Microsoft_Azure_Security/SecurityMenuBlade/~/EnvironmentSettings) - if prompted for credentials, use the LabUser you were provided for the microhack. +* At the bottom of your page find your subscription and click the elipses on the right hand side, then in the popup click "Edit settings". + +![environment-settings](img/01_env_settings.png) +* In section Cloud Workload Protection (CWPP) find the line for the Containers Plan and click the "Settings" links in that line. + +![container-plan-setting](img/02_container_plan_settings.png) +* In the Settings & monitoring page ensure the following settings are turned on (please note that you are working with several participants in the same subscription. Someone might already turned on the recommended settings.): + * Defender sensor - Required because Arc clusters do not have agentless telemetry collection like AKS. Installs the Defender sensor DaemonSet on every node for runtime threat detection. + * Azure Policy - Installs Gatekeeper/OPA in your Arc cluster. Required for Kubernetes posture management, admission control, and workload hardening. + * Kubernetes API access - sets permissions to allow API-based discovery of your Kubernetes clusters. For Arc, this enables Defender to read Kubernetes API objects for configuration assessment. + * Registry access - Vulnerability assessment scanning for images stored in ACR registries. It does NOT scan non‑Azure registries. +* When finished click on the Continue link at the top of the page: + +![settings_n_monitoring](img/03_settings_n_monitoring.png) +* If you made changes, they are not yet saved. Make sure to click the Save link at the top of the page: + +![save](img/04_save.png) + + +## Task 3 - Deploy Defender for Container + +For Arc‑enabled Kubernetes, Defender for Containers cannot operate agentlessly the way it does on AKS. Arc clusters require the Defender sensor DaemonSet to be deployed. + +If auto‑provisioning is enabled in the portal (the toggle “Defender sensor” → ON”), then Defender for Cloud will attempt to deploy the sensor automatically using the Arc extension mechanism. You can check whether the defender extension is already present with the following command: + +```bash +# Check current extensions +echo "Checking current Arc extensions:" +az k8s-extension list \ + --cluster-name $arc_cluster_name \ + --resource-group $arc_resource_group \ + --cluster-type connectedClusters \ + --query '[].{Name:name, Type:extensionType, State:provisioningState}' \ + -o table +``` + +If you see the following extension, the defender extension already got deployed automatically: + +```bash +Name Type State +---------------------------------- ---------------------------------- --------- +microsoft.azuredefender.kubernetes microsoft.azuredefender.kubernetes Succeeded +``` + +If it's not yet present or you want to force it (recommended), use: + +```bash +az k8s-extension create \ + --name microsoft.azuredefender.kubernetes \ + --cluster-type connectedClusters \ + --cluster-name $arc_cluster_name \ + --resource-group $arc_resource_group \ + --extension-type microsoft.azuredefender.kubernetes \ + --configuration-settings logAnalyticsWorkspaceResourceID=$law_resource_id +``` + +If you run into an error telling you "Helm installation failed : Resource already existing in your cluster" this means that the Defender for Cloud policy already installed the extension successfully. + +Optionally, check again for the existence of the defender extension as described above (az k8s-extensions list command). + +Let's see whether the defender related pods are running in our k8s cluster: + +```bash +# Check Defender pods including example output +kubectl get pods -n mdc +NAME READY STATUS RESTARTS AGE +microsoft-defender-collectors-bct8t 2/2 Running 0 10m +microsoft-defender-collectors-h7q6l 2/2 Running 0 10m +microsoft-defender-collectors-htqq5 2/2 Running 0 10m +microsoft-defender-pod-collector-misc-65c56849bd-zq6fk 1/1 Running 0 10m +microsoft-defender-publisher-k9lt9 1/1 Running 0 10m +microsoft-defender-publisher-mcq7g 1/1 Running 0 10m +microsoft-defender-publisher-q4bwq 1/1 Running 0 10m + +# Check Azure Policy (Gatekeeper) pods including example output +kubectl get pods -n gatekeeper-system +NAME READY STATUS RESTARTS AGE +gatekeeper-audit-7df994876b-jrfqt 1/1 Running 0 17m +gatekeeper-controller-manager-7655d54c66-4d95b 1/1 Running 0 17m +gatekeeper-controller-manager-7655d54c66-pcp26 1/1 Running 0 17m + +``` + +Check Policy recommendations in Defender for Cloud: +* In the Azure Portal > Defender for Cloud > General > Inventory +* In the search bar filter for your arc-enabled k8s cluster's name - i.e. 37-k8s-arc-enabled +![alt text](img/05_defender.png) +* Click on the name of your cluster and view the recommendations and alerts + +## Task 4 - Assign Azure Policy for Kubernetes +For Arc-enabled Kubernetes clusters, Azure Policy requires installing the Azure Policy extension (which includes Gatekeeper/OPA). Let's set this up: + +```bash +# Installing Azure Policy extension on Arc cluster... +az k8s-extension create \ + --cluster-name $arc_cluster_name \ + --resource-group $arc_resource_group \ + --cluster-type connectedClusters \ + --extension-type Microsoft.PolicyInsights \ + --name azurepolicy + +{ + "aksAssignedIdentity": null, + "autoUpgradeMinorVersion": true, + "configurationProtectedSettings": {}, + "configurationSettings": {}, + "currentVersion": "1.15.0", + "customLocationSettings": null, + "errorInfo": null, + "extensionType": "microsoft.policyinsights", + [...] + "isSystemExtension": false, + "name": "azurepolicy", + "packageUri": null, + "plan": null, + "provisioningState": "Succeeded", + "releaseTrain": "Stable", + "resourceGroup": "37-k8s-arc", + "scope": { + "cluster": { + "releaseNamespace": "kube-system" + }, + "namespace": null + }, + "statuses": [], + [...] + "type": "Microsoft.KubernetesConfiguration/extensions", + "version": null +} +``` + +Verify the Gatekeeper pods are running: + +```bash +# Checking Gatekeeper (Azure Policy) pods +kubectl get pods -n gatekeeper-system + +NAME READY STATUS RESTARTS AGE +gatekeeper-audit-79d8755674-7rvqs 1/1 Running 0 3m38s +gatekeeper-controller-manager-666b666854-mg5lr 1/1 Running 0 3m38s +gatekeeper-controller-manager-666b666854-rr66f 1/1 Running 0 3m38s + +# Checking all policy-related pods +kubectl get pods -A | grep -E "(policy|gate)" + +gatekeeper-system gatekeeper-audit-79d8755674-7rvqs 1/1 Running 0 4m44s +gatekeeper-system gatekeeper-controller-manager-666b666854-mg5lr 1/1 Running 0 4m44s +gatekeeper-system gatekeeper-controller-manager-666b666854-rr66f 1/1 Running 0 4m44s +kube-system azure-policy-665f9645d-577xx 2/2 Running 0 4m44s +kube-system azure-policy-webhook-685f6f584b-glbrh 1/1 Running 1 (4m16s ago) 4m44s +``` + +Now, let's check what Azure policies are available out-of-the-box for k8s: + +```bash +# Looking for Kubernetes policy initiatives +az policy set-definition list --query "[?contains(displayName, 'Kubernetes')].{DisplayName:displayName, ResourceId:id}" -o table + +Name DisplayName +------------------------------------ ------------------------------------------------------------------------------------------------------------------------------------------- +42b8ef37-b724-4e24-bbc8-7a7708edfe00 Kubernetes cluster pod security restricted standards for Linux-based workloads +4fd005fd-51be-478f-a8fb-149d48b20d48 [Preview]: Kubernetes cluster should follow the security control recommendations of Center for Internet Security (CIS) Kubernetes benchmark +a8640138-9b0a-4a28-b8cb-1666c838647d Kubernetes cluster pod security baseline standards for Linux-based workloads +``` + +Let's assign the pod security baseline policy: + +```bash +# validate the required az cli extension "connected8s" is installed +if ! az extension show --name connectedk8s > /dev/null 2>&1; then + az extension add --name connectedk8s +else + az extension update --name connectedk8s +fi + +cluster_resource_id=$(az connectedk8s show --name $arc_cluster_name --resource-group $arc_resource_group --query id -o tsv) +echo "Cluster Resource ID: $cluster_resource_id" + +policy_id="/providers/Microsoft.Authorization/policySetDefinitions/a8640138-9b0a-4a28-b8cb-1666c838647d" + +# Assign the baseline pod security standards policy +az policy assignment create \ + --name "k8s-pod-security-baseline-${user_number}" \ + --policy-set-definition $policy_id \ + --scope "$cluster_resource_id" \ + --display-name "Kubernetes Pod Security Baseline for cluster ${arc_cluster_name}" \ + --description "Enforces pod security baseline standards on Arc-enabled Kubernetes cluster" + +{ + "definitionVersion": "1.*.*", + "description": "Enforces pod security baseline standards on Arc-enabled Kubernetes cluster", + "displayName": "Kubernetes Pod Security Baseline for cluster 37-k8s-arc-enabled", + "enforcementMode": "Default", + [...] + "name": "k8s-pod-security-baseline-37", + "policyDefinitionId": "/providers/Microsoft.Authorization/policySetDefinitions/a8640138-9b0a-4a28-b8cb-1666c838647d", + "resourceGroup": "37-k8s-arc", + "scope": "/subscriptions/a271110f-4e09-47aa-9d1c-743b520ccbca/resourceGroups/37-k8s-arc/providers/Microsoft.Kubernetes/connectedClusters/37-k8s-arc-enabled", + [...] + "type": "Microsoft.Authorization/policyAssignments" +} +``` +*PLEASE NOTE*: At the time of writing a bug in the az cli prevents the above command from execution. If you run in error for the az policy assignment create command, here is a workaround: +```bash +assignment_name="k8s-pod-security-baseline-${user_number}" +policy_set_id="/providers/Microsoft.Authorization/policySetDefinitions/a8640138-9b0a-4a28-b8cb-1666c838647d" + +az rest \ + --method PUT \ + --url "https://management.azure.com${cluster_resource_id}/providers/Microsoft.Authorization/policyAssignments/${assignment_name}?api-version=2023-04-01" \ + --body "{ + \"properties\": { + \"displayName\": \"Kubernetes Pod Security Baseline for cluster ${arc_cluster_name}\", + \"description\": \"Enforces pod security baseline standards on Arc-enabled Kubernetes cluster\", + \"policyDefinitionId\": \"${policy_set_id}\" + } + }" +``` + +As a result there should appear several contrainttemplates in your cluster: +```bash +kubectl get constrainttemplates +NAME AGE +k8sazurev1blockdefault 11m +k8sazurev1ingresshttpsonly 11m +k8sazurev1serviceallowedports 11m +k8sazurev2blockautomounttoken 11m +k8sazurev2blockhostnamespace 11m +k8sazurev2containerallowedimages 11m +k8sazurev2noprivilege 11m +k8sazurev3allowedcapabilities 11m +k8sazurev3allowedusersgroups 11m +k8sazurev3containerlimits 11m +k8sazurev3disallowedcapabilities 11m +k8sazurev3enforceapparmor 11m +k8sazurev3hostnetworkingports 11m +k8sazurev3noprivilegeescalation 11m +k8sazurev3readonlyrootfilesystem 11m +k8sazurev4hostfilesystem 11m +``` + +⏳ Constraint templates will appear within 15-30 minutes + +⏳ Policy compliance data will populate in Azure Portal within 15-30 minutes + +Let's check the compliance status via az cli: + +```bash +# Policy assignment details +az policy assignment show \ + --name "k8s-pod-security-baseline-${user_number}" \ + --scope "$cluster_resource_id" \ + --query '{Name:name, DisplayName:displayName, EnforcementMode:enforcementMode, PolicyDefinitionId:policyDefinitionId}' \ + -o table + +#expected output: +Name DisplayName EnforcementMode PolicyDefinitionId +---------------------------- --------------------------------------------------------------- ----------------- -------------------------------------------------------------------------------------------- +k8s-pod-security-baseline-37 Kubernetes Pod Security Baseline for cluster 37-k8s-arc-enabled Default /providers/Microsoft.Authorization/policySetDefinitions/a8640138-9b0a-4a28-b8cb-1666c838647d + +# Policy compliance state +az policy state list \ + --resource "$cluster_resource_id" \ + --query '[].{PolicyDefinitionName:policyDefinitionName, PolicyAssignmentName:policyAssignmentName, ComplianceState:complianceState}' \ + -o table + +# expected output (there are many more but these are the ones corresponding to the pod security baseline policy we applied earlier): +PolicyDefinitionName PolicyAssignmentName ComplianceState +------------------------------------ --------------------------------------------------------------- ----------------- +[...] +098fc59e-46c7-4d99-9b16-64990e543d75 k8s-pod-security-baseline-37 NonCompliant +47a1ee2f-2a2a-4576-bf2a-e0e36709c2b8 k8s-pod-security-baseline-37 Compliant +82985f06-dc18-4a48-bc1c-b9f4f0098cfe k8s-pod-security-baseline-37 NonCompliant +95edb821-ddaf-4404-9732-666045e056b4 k8s-pod-security-baseline-37 Compliant +c26596ff-4d70-4e6a-9a30-c2506bd2f80c k8s-pod-security-baseline-37 NonCompliant +``` +Some policies should show as 'Compliant' and some as 'NonCompliant'. + +Optionally, check on your k8s cluster how the policies are implemented there. +```bash +# in the list of all constraints injected via Azure Policy, identify constraints with violations +kubectl get constraints + +# expected output (shortened): +NAME ENFORCEMENT-ACTION TOTAL-VIOLATIONS +k8sazurev3allowedcapabilities.constraints.gatekeeper.sh/azurepolicy-k8sazurev3allowedcapabilities-69e35e5e70785d0cd7c5 dryrun 0 +k8sazurev3allowedcapabilities.constraints.gatekeeper.sh/azurepolicy-k8sazurev3allowedcapabilities-e5d0b70be34e42817a95 dryrun 0 +k8sazurev3allowedcapabilities.constraints.gatekeeper.sh/azurepolicy-k8sazurev3allowedcapabilities-f9e7a5d4539f18fafe48 dryrun 3 + +# identify the k8sazurev3allowedcapabilities.constraint with violations and store it in a variable +constraint_name="k8sazurev3allowedcapabilities.constraints.gatekeeper.sh/azurepolicy-k8sazurev3allowedcapabilities-f9e7a5d4539f18fafe48" + +# In the constraint details you can see a message for each violation +kubectl describe $constraint_name +``` + +💡**Note:** Many violations are originating from the defender pods itself. This is (at time of writing) a known issue. A simple workaround is to exclude the **mdc, gatekeeper-system and azure-arc** namespaces from policy evaluation (not in scope of the microhack). + + +You successfully completed challenge 2! 🚀🚀🚀 + +[Back to challenge 01](../challenge-01/solution.md) - [Next challenge](../challenge-03/solution.md) - [Next Challenge's Solution](../../walkthroughs/challenge-03/solution.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/01_add_connection.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/01_add_connection.png new file mode 100644 index 000000000..475b452de Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/01_add_connection.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/02_add_connection_ollama.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/02_add_connection_ollama.png new file mode 100644 index 000000000..6664c0696 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/02_add_connection_ollama.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/03_add_connection_ip_port.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/03_add_connection_ip_port.png new file mode 100644 index 000000000..4bfa50255 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/img/03_add_connection_ip_port.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/solution.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/solution.md new file mode 100644 index 000000000..5c09c1de4 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-03/solution.md @@ -0,0 +1,88 @@ +# Walkthrough Challenge 3 - Deploy CPU based Large & Small Language Models (LLM/SLM) + +[Back to challenge](../../challenges/challenge-03.md) - [Next Challenge's Solution](../challenge-04/solution.md) + +## Prerequisites +* You have an arc-connected k8s cluster/finisched challenge 01. +* Verify the firewall requirements in addition to the Azure Arc-enabled Kubernetes network requirements. +* You must be logged in to az cli (az login) +* You need kubectl and helm + +## Task 1 - Check the Kubernetes cluster, check nodes and size +Execute the following script in your bash shell to access the kuberentes cluster: +```bash +kubectl get nodes +kubectl get all --all-namespaces +``` + +## Task 2 - Create a new namespace and prepare the helm repository + +```bash +helm repo add otwld https://helm.otwld.com/ +helm repo add open-webui https://helm.openwebui.com/ +helm repo update +kubectl create namespace aimh +``` + +## Task 3 - install ollama and openwebui as an AI framework running in the Azure Arc enabled kubernetes Cluster + + +### Step 1: Install ollama with phi4-mini + +```bash +helm install ollama otwld/ollama \ + --namespace aimh \ + --set service.type=ClusterIP \ + --set ollama.port=15000 \ + --set persistentVolume.enabled=true \ + --set persistentVolume.size=20Gi \ + --set ollama.models.pull[0]=phi4-mini:latest \ + --set ollama.models.run[0]=phi4-mini:latest \ + --set resources.requests.cpu="2" \ + --set resources.requests.memory="2Gi" \ + --set resources.limits.cpu="4" \ + --set resources.limits.memory="4Gi" +``` + +### Step 2: Install openwebUI + +```bash +helm install openwebui open-webui/open-webui \ + --namespace aimh \ + --set service.type=NodePort \ + --set service.nodePort=30080 \ + --set ollama.enabled=false \ + --set ollamaUrls[0]="http://ollama.slm.svc.cluster.local:15000" \ + --set persistence.enabled=true \ + --set persistence.size=5Gi +``` + +Get the external IP of the openwebui +```bash +kubectl get svc -n aimh + +azure_user=$(az account show --query user.name -o tsv) +user_number=$(echo "${azure_user%@*}" | grep -oE '[0-9]+' | tail -n1 | sed 's/^0*//; s/^$/0/') +node_pip=$(az vm list-ip-addresses \ + --resource-group "${user_number}-k8s-onprem" \ + --name "${user_number}-k8s-master" \ + --query "[0].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) + +echo "Open: http://${node_pip}:30080" +``` + +## Task 5 - Access the openwebUI and run a prompt +open webbrowser to access the external IP, create user login to the openwebUI portal +![add-connection](img/01_add_connection.png) + +add an ollama connection +![add-connection-ollama](img/02_add_connection_ollama.png) + +add IP Address and port and click in save(2x) +![add-connection-IPandPort](img/03_add_connection_ip_port.png) +and try out your first prompt +If your connection works, you should see in the upper left corner the deployed model "phi4-mini". + +You successfully completed challenge 3! 🚀🚀🚀 + +[Next challenge](../../challenges/challenge-04.md) - [Next Challenge's Solution](../challenge-04/solution.md) diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/01-sql-plugin-install.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/01-sql-plugin-install.png new file mode 100644 index 000000000..d9dfe1dbe Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/01-sql-plugin-install.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/02-connect.sqlmi.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/02-connect.sqlmi.png new file mode 100644 index 000000000..98cfdef59 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/02-connect.sqlmi.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/03-create-query.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/03-create-query.png new file mode 100644 index 000000000..4670543c3 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/03-create-query.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/04-query-version.png b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/04-query-version.png new file mode 100644 index 000000000..94ea753d1 Binary files /dev/null and b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/img/04-query-version.png differ diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/solution.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/solution.md new file mode 100644 index 000000000..6879859d2 --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/solution.md @@ -0,0 +1,198 @@ +# Walkthrough Challenge 4 - Deploy SQL Managed Instance to your cluster + +[Back to challenge](../../challenges/challenge-04.md) - [Next Challenge's Solution](../challenge-05/solution.md) + +## prerequisites +- [client tools](https://learn.microsoft.com/en-us/azure/azure-arc/data/install-client-tools) +- Provider reqistration +```shell +az provider register --namespace Microsoft.AzureArcData +``` + +## Read about the prerequisites and concepts +1. Create Azure Arc [data services cluster extension](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-extensions) +2. Create a [custom location] on your arc-enabled k8s(https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/custom-locations#create-custom-location) +3. create the Arc data controller + +## Task 1 - Create arc data services controller + +The deployment uses ARM templates as this is the most robust way for deployment at the time of writing this microhack. Before deploying, you need to prepare the parameter file with your specific values. + +### Step 1: Gather required values + +```bash +# Get current LabUser's number +export azure_user=$(az account show --query user.name --output tsv) +export user_number=$(echo "${azure_user%@*}" | grep -oE '[0-9]+' | tail -n1 | sed 's/^0*//; s/^$/0/') +export subscription_id=$(az account show --query id -o tsv) +export resource_group="$user_number-k8s-arc" + +# Get Log Analytics workspace details +export log_analytics_workspace_id=$(az monitor log-analytics workspace show -g $resource_group -n "$user_number-law" --query customerId -o tsv) +export log_analytics_primary_key=$(az monitor log-analytics workspace get-shared-keys -g $resource_group -n "$user_number-law" --query primarySharedKey -o tsv) + +# Generate new GUIDs for role assignments +export guid1=$(uuidgen) +export guid2=$(uuidgen) + +echo "Your values:" +echo "USER_NUMBER: $user_number" +echo "SUBSCRIPTION_ID: $subscription_id" +echo "LOG_ANALYTICS_WORKSPACE_ID: $log_analytics_workspace_id" +echo "GUID 1: $guid1" +echo "GUID 2: $guid2" +``` + +### Step 2: Update parameters.json + +Copy the template file and replace placeholders: + +```bash +cd walkthroughs/challenge-04/templates + +# Remove the old one and start fresh +rm parameters-my.json +# Create a working copy +cp parameters.json parameters-my.json + +# Replace placeholders (Linux/WSL) +# Note: Escape special sed characters in the primary key to handle any potential delimiters +escaped_key=$(printf '%s\n' "$log_analytics_primary_key" | sed 's/[\/&]/\\&/g') + +sed -i "s//$user_number/g" parameters-my.json +sed -i "s//$subscription_id/g" parameters-my.json +sed -i "s//$log_analytics_workspace_id/g" parameters-my.json +sed -i "s//$escaped_key/g" parameters-my.json +sed -i "s//$guid1/g" parameters-my.json +sed -i "s//$guid2/g" parameters-my.json + +# Alternative: manual replacement +# Edit parameters-my.json and replace: +# with your user number +# with your subscription ID +# with workspace ID +# with workspace key +# with first new GUID (use 'uuidgen' command) +# with second new GUID (use 'uuidgen' command again) +``` + +### Step 3: Deploy the Arc Data Controller + +```bash +read -s -p "Enter password for metrics/logs dashboard: " DC_PASSWORD && echo + +# Validate the deployment first +az deployment group validate \ + --resource-group $resource_group \ + --template-file template.json \ + --parameters parameters-my.json \ + --parameters dataController_metricsAndLogsDashboardPassword="$DC_PASSWORD" + +# Deploy (you'll be prompted for the dashboard password) +az deployment group create \ + --resource-group $resource_group \ + --template-file template.json \ + --parameters parameters-my.json \ + --parameters dataController_metricsAndLogsDashboardPassword="$DC_PASSWORD" \ + --verbose +``` +*NOTE*: At the time of writing there is a bug in the az cli command to create the arc data controller. This is why we are using a workaround based on an ARM template deployment. + +The deployment takes 10-15 minutes. You can monitor progress with: + +```bash +# Watch deployment status +az deployment group show \ + --resource-group $resource_group \ + --name template \ + --query properties.provisioningState + +# Monitor operations +az deployment operation group list \ + --resource-group $resource_group \ + --name template \ + --query "[].{Resource:properties.targetResource.resourceName, State:properties.provisioningState}" -o table +``` + +## Task 2 - Create SQL Managed Instance in connected cluster + +**IMPORTANT: 'sa' is not allowed as sql admin at managed instance level!** + +```bash +# Set SQL MI name (must start with a letter for Kubernetes naming rules) +export sqlmi_name="sqlmi-${user_number}-1" +export custom_location="${user_number}-onprem" + +# Create SQL MI (you'll be prompted for sql admin user and password) +az sql mi-arc create \ + --name $sqlmi_name \ + --resource-group $resource_group \ + --custom-location $custom_location \ + --cores-request 1 \ + --memory-request 3Gi +``` + +The creation takes 5-10 minutes. Monitor progress: + +```bash +# Check deployment status +az sql mi-arc show --name $sqlmi_name --resource-group $resource_group --query "properties.k8sRaw.status.state" -o tsv + +# Check pods in the namespace +kubectl get pods -n ${user_number}-onprem -l app.kubernetes.io/name=$sqlmi_name +``` + +## Task 3 - Connect to your SQL Managed Instance + +### Prerequisites + +**Network Security Group (NSG)**: The Terraform deployment has already configured the NSG to allow external access to the NodePort range (30000-32767) required for SQL MI connectivity. No additional NSG configuration is needed. + +### Get connection details + +```bash +# Get public ip of master node via Azure cli according to user-number +master_pip=$(az vm list-ip-addresses --resource-group "${user_number}-k8s-onprem" --name "${user_number}-k8s-master" --query "[0].virtualMachine.network.publicIpAddresses[0].ipAddress" --output tsv) + +# Get the NodePort assigned to SQL MI +node_port=$(kubectl get svc ${sqlmi_name}-external-svc -n ${user_number}-onprem -o jsonpath='{.spec.ports[0].nodePort}') + +echo "Connection details:" +echo "Server: $master_pip,$node_port" +echo "Username: " +echo "Password: " +echo "" +echo "Connection string:" +echo "Server=$master_pip,$node_port;Database=master;User Id=sa;Password=;TrustServerCertificate=true;" +``` + +### Connect using VS Code SQL Server extension + +#### 1. Install the **SQL Server (mssql)** extension in VS Code: + +![SQL PlugIn installation](img/01-sql-plugin-install.png) + +#### 2. After installation completed, in the SQL PlutIn click **Add Connection**: + +![Connect to SQL MI](img/02-connect.sqlmi.png) + +#### 3. Enter connection details: +**💡Note: In this lab we are using the public ip to connect to the service.** + - **Input type**: Parameters + - **Server:** `,` (e.g., `20.123.45.67,31433`, optionally, use the following command: ```echo "$master_pip,$(kubectl get svc ${sqlmi_name}-external-svc -n ${custom_location} -o jsonpath='{.spec.ports[0].nodePort}')"``` + - **Trust Server Certificate:** Yes + - **Authentication Type:** SQL Login + - **User name:** (the admin account you entered during SQL MI creation) + - **Password:** (the password you entered during SQL MI creation) + - **Database:** `master` (or leave empty) + +#### 4. Click **Connect** +#### 5. Expand Databases, then expand System Databases and right click on **master**. In the context menu select **New Query**. +![alt text](img/03-create-query.png) + +#### 6. Query database version using ```SELECT @@VERSION;```: +![alt text](img/04-query-version.png) + +You successfully completed challenge 4! 🚀🚀🚀 + +[Next challenge](../../challenges/challenge-05.md) - [Next Challenge's Solution](../challenge-05/solution.md) \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/templates/parameters.json b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/templates/parameters.json new file mode 100644 index 000000000..07a78606a --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/templates/parameters.json @@ -0,0 +1,68 @@ +{ + "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", + "contentVersion": "1.0.0.0", + "parameters": { + "parentResource": { + "value": "Microsoft.Kubernetes/connectedClusters/-k8s-arc-enabled" + }, + "k8sExtensionName": { + "value": "-onprem-ext" + }, + "namespace": { + "value": "-onprem" + }, + "location": { + "value": "westeurope" + }, + "roleDefinitionId_Contributor": { + "value": "/subscriptions//resourceGroups/-k8s-arc/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c" + }, + "roleAssignment_Contributor": { + "value": "" + }, + "roleDefinitionId_MonitoringMetricsPublisher": { + "value": "/subscriptions//resourceGroups/-k8s-arc/providers/Microsoft.Authorization/roleDefinitions/3913510d-42f4-4e42-8a64-420c390055eb" + }, + "roleAssignment_MonitoringMetricsPublisher": { + "value": "" + }, + "customLocationName": { + "value": "-onprem" + }, + "customLocationId": { + "value": "/subscriptions//resourceGroups/-k8s-arc/providers/Microsoft.ExtendedLocation/customLocations/-onprem" + }, + "customLocation_hostResourceId": { + "value": "/subscriptions//resourceGroups/-k8s-arc/providers/Microsoft.Kubernetes/connectedClusters/-k8s-arc-enabled" + }, + "customLocation_clusterExtensionIds": { + "value": [ + "/subscriptions//resourceGroups/-k8s-arc/providers/Microsoft.Kubernetes/connectedClusters/-k8s-arc-enabled/providers/Microsoft.KubernetesConfiguration/extensions/-onprem-ext" + ] + }, + "resourceSyncRuleName": { + "value": "-onprem/defaultResourceSyncRule" + }, + "dataController_metricsAndLogsDashboardPassword": { + "value": null + }, + "dataController_logAnalyticsWorkspaceId": { + "value": "" + }, + "dataController_logAnalyticsPrimaryKey": { + "value": "" + }, + "storageClass": { + "value": "local-path" + }, + "dataController_subscription": { + "value": "" + }, + "dataController_resourceGroup": { + "value": "-k8s-arc" + }, + "dataControllerName": { + "value": "-dc" + } + } +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/templates/template.json b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/templates/template.json new file mode 100644 index 000000000..58d89efda --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-04/templates/template.json @@ -0,0 +1,325 @@ +{ + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "parameters": { + "parentResource": { + "type": "String", + "metadata": { + "description": "The resource ID of the arc-enabled K8s cluster where the extension will be installed." + } + }, + "k8sExtensionName": { + "type": "String", + "metadata": { + "description": "The name of the Kubernetes extension to create." + } + }, + "namespace": { + "type": "String", + "metadata": { + "description": "The namespace in the Kubernetes cluster where the extension will be installed." + } + }, + "location": { + "type": "String", + "defaultValue": "westeurope" + }, + "roleDefinitionId_Contributor": { + "type": "String", + "metadata": { + "description": "The role definition ID for the Contributor role." + } + }, + "roleAssignment_Contributor": { + "type": "String", + "metadata": { + "description": "The role assignment name (GUID) for the Contributor role." + } + }, + "roleDefinitionId_MonitoringMetricsPublisher": { + "type": "String", + "metadata": { + "description": "The role definition ID for the Monitoring Metrics Publisher role." + } + }, + "roleAssignment_MonitoringMetricsPublisher": { + "type": "String", + "metadata": { + "description": "The role assignment name (GUID) for the Monitoring Metrics Publisher role." + } + }, + "customLocationName": { + "type": "String", + "metadata": { + "description": "The name of the custom location." + } + }, + "customLocationId": { + "type": "String", + "metadata": { + "description": "The ID of the custom location." + } + }, + "customLocation_hostResourceId": { + "type": "String", + "metadata": { + "description": "The host resource ID of the custom location." + } + }, + "customLocation_clusterExtensionIds": { + "type": "Array", + "metadata": { + "description": "The cluster extension IDs of the custom location." + } + }, + "resourceSyncRuleName": { + "type": "String", + "metadata": { + "description": "The name of the resource sync rule." + } + }, + "dataController_metricsAndLogsDashboardPassword": { + "type": "SecureString" + }, + "dataController_logAnalyticsWorkspaceId": { + "type": "String", + "metadata": { + "description": "The Log Analytics Workspace ID for the data controller." + } + }, + "dataController_logAnalyticsPrimaryKey": { + "type": "String", + "metadata": { + "description": "The primary key for the Log Analytics Workspace." + } + }, + "storageClass": { + "type": "String", + "defaultValue": "local-path", + "metadata": { + "description": "The storage class for data and logs storage." + } + }, + "dataController_subscription": { + "type": "String" + }, + "dataController_resourceGroup": { + "type": "String" + }, + "dataControllerName": { + "type": "String", + "metadata": { + "description": "The name of the data controller." + } + } + }, + "variables": {}, + "functions": [], + "resources": [ + { + "type": "Microsoft.KubernetesConfiguration/extensions", + "apiVersion": "2021-09-01", + "name": "[parameters('k8sExtensionName')]", + "identity": { + "type": "SystemAssigned" + }, + "properties": { + "extensionType": "microsoft.arcdataservices", + "autoUpgradeMinorVersion": false, + "releaseTrain": "stable", + "version": "1.42.0", + "scope": { + "cluster": { + "releaseNamespace": "[parameters('namespace')]" + } + }, + "configurationSettings": { + "Microsoft.CustomLocation.ServiceAccount": "sa-arc-bootstrapper", + "imageCredentials.registry": "mcr.microsoft.com", + "systemDefaultValues.image": "mcr.microsoft.com/arcdata/arc-bootstrapper:v1.42.0_2025-10-14", + "systemDefaultValues.imagePullPolicy": "Always" + }, + "configurationProtectedSettings": {} + }, + "scope": "[parameters('parentResource')]" + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2018-09-01-preview", + "name": "[parameters('roleAssignment_Contributor')]", + "dependsOn": [ + "[parameters('k8sExtensionName')]" + ], + "properties": { + "roleDefinitionId": "[parameters('roleDefinitionId_Contributor')]", + "principalId": "[reference(parameters('k8sExtensionName'), '2021-09-01', 'Full').identity.principalId]" + } + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2018-09-01-preview", + "name": "[parameters('roleAssignment_MonitoringMetricsPublisher')]", + "dependsOn": [ + "[parameters('roleAssignment_Contributor')]" + ], + "properties": { + "roleDefinitionId": "[parameters('roleDefinitionId_MonitoringMetricsPublisher')]", + "principalId": "[reference(parameters('k8sExtensionName'), '2021-09-01', 'Full').identity.principalId]" + } + }, + { + "type": "Microsoft.ExtendedLocation/customLocations", + "apiVersion": "2021-08-31-preview", + "name": "[parameters('customLocationName')]", + "location": "[parameters('location')]", + "dependsOn": [ + "[parameters('roleAssignment_MonitoringMetricsPublisher')]" + ], + "properties": { + "hostType": "Kubernetes", + "hostResourceId": "[parameters('customLocation_hostResourceId')]", + "namespace": "[parameters('namespace')]", + "displayName": "", + "clusterExtensionIds": "[parameters('customLocation_clusterExtensionIds')]", + "authentication": {} + } + }, + { + "type": "Microsoft.ExtendedLocation/customLocations/resourceSyncRules", + "apiVersion": "2021-08-31-preview", + "name": "[parameters('resourceSyncRuleName')]", + "location": "[parameters('location')]", + "dependsOn": [ + "[parameters('customLocationName')]" + ], + "properties": { + "targetResourceGroup": "[concat('/subscriptions/', parameters('dataController_subscription'), '/resourceGroups/', parameters('dataController_resourceGroup'))]", + "priority": 100, + "selector": { + "matchLabels": { + "management.azure.com/provider-name": "Microsoft.AzureArcData" + } + } + } + }, + { + "type": "Microsoft.AzureArcData/dataControllers", + "apiVersion": "2023-01-15-preview", + "name": "[parameters('dataControllerName')]", + "location": "[parameters('location')]", + "extendedLocation": { + "name": "[parameters('customLocationId')]", + "type": "CustomLocation" + }, + "dependsOn": [ + "[last(split(parameters('resourceSyncRuleName'), '/'))]" + ], + "tags": {}, + "properties": { + "metricsDashboardCredential": { + "username": "mhadmin", + "password": "[parameters('dataController_metricsAndLogsDashboardPassword')]" + }, + "logsDashboardCredential": { + "username": "mhadmin", + "password": "[parameters('dataController_metricsAndLogsDashboardPassword')]" + }, + "logAnalyticsWorkspaceConfig": { + "workspaceId": "[parameters('dataController_logAnalyticsWorkspaceId')]", + "primaryKey": "[parameters('dataController_logAnalyticsPrimaryKey')]" + }, + "infrastructure": "onpremises", + "k8sRaw": { + "kind": "DataController", + "spec": { + "credentials": { + "dockerRegistry": "arc-private-registry", + "domainServiceAccount": "domain-service-account-secret", + "serviceAccount": "sa-arc-controller" + }, + "docker": { + "registry": "mcr.microsoft.com", + "repository": "arcdata", + "imageTag": "v1.42.0_2025-10-14", + "imagePullPolicy": "Always" + }, + "security": { + "allowDumps": true, + "allowNodeMetricsCollection": true, + "allowPodMetricsCollection": true + }, + "services": [ + { + "name": "controller", + "port": 30080, + "serviceType": "NodePort" + } + ], + "settings": { + "ElasticSearch": { + "vm.max_map_count": "-1" + }, + "azure": { + "autoUploadMetrics": "true", + "autoUploadLogs": "true", + "subscription": "[parameters('dataController_subscription')]", + "resourceGroup": "[parameters('dataController_resourceGroup')]", + "location": "[parameters('location')]", + "connectionMode": "direct" + }, + "controller": { + "logs.rotation.days": "7", + "logs.rotation.size": "5000", + "displayName": "[parameters('dataControllerName')]" + }, + "controldb": { + "backupCount": "5" + } + }, + "storage": { + "data": { + "accessMode": "ReadWriteOnce", + "className": "[parameters('storageClass')]", + "size": "15Gi" + }, + "logs": { + "accessMode": "ReadWriteOnce", + "className": "[parameters('storageClass')]", + "size": "10Gi" + } + }, + "infrastructure": "onpremises", + "resources": { + "controller": { + "requests": { + "cpu": "400m", + "memory": "2Gi" + }, + "limits": { + "cpu": "1800m", + "memory": "2Gi" + } + }, + "controllerDb": { + "requests": { + "cpu": "200m", + "memory": "4Gi" + }, + "limits": { + "cpu": "800m", + "memory": "6Gi" + } + } + } + }, + "metadata": { + "namespace": "[parameters('namespace')]", + "name": "datacontroller" + }, + "apiVersion": "arcdata.microsoft.com/v5" + } + } + } + ], + "outputs": {} +} \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-05/namespaces/itops.yaml b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-05/namespaces/itops.yaml new file mode 100644 index 000000000..8f58c1c1f --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-05/namespaces/itops.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: Namespace +metadata: + labels: + name: itops + annotations: + contoso.com/owner: "ops@contoso.com" + name: itops \ No newline at end of file diff --git a/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-05/solution.md b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-05/solution.md new file mode 100644 index 000000000..c520d103e --- /dev/null +++ b/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthrough/challenge-05/solution.md @@ -0,0 +1,102 @@ +# Walkthrough Challenge 5 - Configure Gitops for cluster management + +[Back to challenge](../../challenges/challenge-05.md) - [Next Challenge's Solution](../challenge-06/solution.md) + +### Prerequisites +* [helm](https://helm.sh/docs/intro/install/) +* Read and write permissions on the resource types + * Microsoft.Kubernetes/connectedClusters + * Microsoft.ContainerService/managedClusters + * Microsoft.KubernetesConfiguration/extensions + * Microsoft.KubernetesConfiguration/fluxConfigurations +* Registration of the following Azure resource providers: +```bash +az provider register --namespace Microsoft.Kubernetes +az provider register --namespace Microsoft.ContainerService +az provider register --namespace Microsoft.KubernetesConfiguration +``` +* Required cli extensions +```bash +az extension add -n k8s-configuration +az extension add -n k8s-extension +``` +* Flux CLI installed (optional; not required for this challenge) +* Extension microsoft.flux installed on your kubernetes cluster +```bash +# Extract user number from Azure username before '@' (e.g., LabUser-37@... -> 37) +azure_user=$(az account show --query user.name --output tsv) +user_number=$(echo "${azure_user%@*}" | grep -oE '[0-9]+' | tail -n1 | sed 's/^0*//; s/^$/0/') + +export arc_resource_group="${user_number}-k8s-arc" +export arc_cluster_name="${user_number}-k8s-arc-enabled" + +az k8s-extension create \ + --name fluxExtension \ + --cluster-name $arc_cluster_name \ + --resource-group $arc_resource_group \ + --cluster-type connectedClusters \ + --extension-type microsoft.flux +``` +* **Your own fork of the MicroHack repository on GitHub** + + To push changes to the Flux configuration, you need your own fork of the MicroHack repository. Here's how to set it up: + + 1. **Fork the repository** on GitHub: + - Go to https://github.com/microsoft/MicroHack + - Click the "Fork" button in the top-right corner + - Complete the fork process + - Ensure the fork is **public** for ease of use (no authentication needed for Flux) + + 2. **Clone only the required folder** (sparse checkout) to keep the repository size small: + ```bash + # Create a directory for the repository + mkdir microhack-gitops + cd microhack-gitops + + # Initialize a git repository with sparse checkout + git init + git remote add origin https://github.com//MicroHack.git + + # Enable sparse checkout + git config core.sparseCheckout true + + # Specify the folder to checkout (the namespaces folder for this challenge) + echo "03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthroughs/challenge-05/namespaces" >> .git/info/sparse-checkout + + # Pull the content + git pull origin main + ``` + + After this, you'll have only the `namespaces` folder with the configuration files you need to modify and push. + +### Solution - Manage cluster configuration using GitOps +In order to manage a namespace via flux, you need a repository. In this microhack we're using a public github repository. If using a private repo make sure to add credentials so flux is able to access your repository. The following command creates a flux configuration which watches the namespaces folder within this repository. All namespace definitions found in this folder will be applied to the cluster. +```bash +repository="https://github.com//MicroHack" #Change to your own fork of the Microhack repository +path="/03-Azure/01-03-Infrastructure/03_Hybrid_Azure_Arc_Kubernetes/walkthroughs/challenge-05/namespaces" + +az k8s-configuration flux create \ + --resource-group $arc_resource_group \ + --cluster-name $arc_cluster_name \ + --cluster-type connectedClusters \ + --name flux-config-namespace \ + --namespace flux-system \ + --scope cluster \ + --url $repository \ + --branch main \ + --interval 1m \ + --kustomization name=namespaces path=$path prune=true interval=1m +``` + +💡 **Note**: The `--interval 1m` parameter sets how often Flux checks your Git repository for changes, while the kustomization `interval=1m` controls how often the configuration is applied. Both are set to 1 minute for quick feedback during this challenge. In production this should be set in accordance to the --sync-timeout setting. + +The first namespace from the existing YAML in the `namespaces` folder is created automatically after Flux picks up the configuration. This usually takes 1-2 minutes (the sync interval is set to 1 minute). + +Now create an additional namespace for team1: +Copy itops.yaml and name it team1.yaml. Open it in your editor and change the labels.name and name values to "team1". Save the file and commit and push it. The flux configuration will pull the change on the next sync (typically within 1-2 minutes), and the new namespace will appear in your cluster. + + +### Resources +* [GitOps for Azure Kubernetes Service](https://learn.microsoft.com/en-us/azure/architecture/example-scenario/gitops-aks/gitops-blueprint-aks) + +You successfully completed challenge 5! 🚀🚀🚀