Terraform AKS with FluxCD

This project sets up an Azure Kubernetes Service (AKS) cluster with FluxCD for GitOps-based deployment management and Azure Managed Grafana for monitoring and observability.

Architecture

This solution follows a two-phased approach:

1. Infrastructure Provisioning: Using Terraform to create the AKS cluster, container registry, Azure Managed Grafana, and base FluxCD installation.
2. GitOps Configuration: Using FluxCD CLI in a CI/CD pipeline to configure the GitRepository and Kustomization resources.

IAC global architecture

Gitops global architecture

Prerequisites

• Terraform (v1.0.0+)
• Azure CLI (latest version)
• kubectl (latest version)
• FluxCD CLI (latest version)
• An Azure subscription
• A Git repository for your kubernetes manifests

Project Structure

.
├── .github/
│   └── workflows/
│       └── infra.yml     # GitHub Actions workflow for the Infra & Kubernetes configuration - 2 separates jobs
│       └── flux.yml      # GitHub Actions workflow for FluxCD configuration
│
├── gitops/
│   ├── kustomization.yaml          # Kustomization file to be applied
│   └── git-repository.yaml         # GitRepo file to be applied
│
├── k8s/
│   ├── main.tf          # main.tf for the k8s config
│   └── backend.tf         # backend corresponding to the k8s
│   └── variables.tf         # variables k8s
│
├── infra/
│   ├── main.tf          # main.tf for the infra config
│   └── backend.tf         # backend corresponding to the infra
│   └── variables.tf         # variables infra 
│   └── outputs.tf         # pass kubeconfig values to k8s
│
└── README.md                       # This file

Variables

The Terraform state is stored in a Azure Blob Storage for collaboration, state locking, and versioning. Also it is in a dedicated rg to make sure there are not conflicts. Currently, most variables are stored as GitHub secrets. A more secure and scalable approach would be to store them in Azure Key Vault or Hashicorp Vault in the future.

resource_group_name = "aks-demo-rg"
location            = "eastus"
aks_cluster_name    = "aks-cluster"
kubernetes_version  = "1.30"
node_count          = 2
node_size           = "Standard_DS2_v2"
acr_name            = "myacrregistry"
sku                 = "Basic"
gitops_repo_url     = "ssh://git@github.com/yourusername/your-gitops-repo.git"
fluxcd_key          = "your-private-key"
fluxcd_key_pub      = "your-public-key"
known_hosts         = "known_hosts-content"

Deployment Process

1. Infrastructure Deployment

To deploy the infrastructure:

# Initialize Terraform
terraform init

# Plan the deployment
terraform plan -out=tfplan

# Apply the deployment
terraform apply tfplan

This will create:

• Azure Resource Group
• AKS Cluster
• Azure Container Registry
• FluxCD installation in the AKS cluster

2. Configure FluxCD

After the infrastructure is deployed, configure FluxCD using the GitHub Actions workflow:

1. Store the following secrets in your GitHub repository secrets:

   • KUBE_CONFIG: The kubeconfig output from Terraform
   • GIT_REPOSITORY_URL: Your GitOps repository URL

2. Trigger the workflow manually from the GitHub Actions tab or set it up to run automatically after infrastructure deployment.

GitOps Repository Structure

Your GitOps repository should follow a standard structure:

.
├── 

Troubleshooting

Common issues and their solutions:

FluxCD not detecting changes

Ensure that:

1. The SSH key has proper permissions to the repository
2. The branch name in the GitRepository resource matches your repository's default branch
3. The path in the Kustomization resource points to the correct directory

AKS cluster connection issues

If you can't connect to the AKS cluster:

1. Ensure the kubeconfig is properly set up
2. Check if the AKS cluster has public network access enabled
3. Verify your Azure credentials are still valid

Cleanup

To destroy the infrastructure:

terraform destroy

Security Considerations & Potential Upgrades

• Use Azure Key Vault for secure storage of secrets and manage access via Azure RBAC.
• System-assigned Managed Identity for AKS enhances security by avoiding credentials management. Consider using Azure AD integration for tighter access control.
• Enable Network Policies and use Private Endpoints for services to restrict access to private networks.
• Automate Grafana deployment with Terraform and assign appropriate Azure RBAC roles for secure access.
• Set up a full landing zone with VNET, IAM, Key Vault, NSGs, and Firewalls for enhanced security.
• Follow least privilege access with MFA for privileged accounts and regularly audit permissions.
• Enable Azure Monitor and Security Center for threat detection and use Azure Sentinel for security incident management.
• Implement backup and disaster recovery plans to ensure business continuity.

Links

• iac repo
• gitops repo
• json-server repo