From aa6b28821d0c69f0461443c652add12930618a89 Mon Sep 17 00:00:00 2001
From: bhumi46 <111699703+bhumi46@users.noreply.github.com>
Date: Mon, 6 Oct 2025 17:52:58 +0530
Subject: [PATCH 1/3] Merge pull request #81 from
bhumi46/docs-improvements-bug-fixes
[MOSIP-43149]Docs improvements bug fixes
---
.github/scripts/README.md | 64 +-
.github/scripts/WORKFLOW_TESTING_GUIDE.md | 76 +-
.github/workflows/README.md | 272 +--
.github/workflows/terraform-destroy.yml | 2 +-
Helmsman/dsf/README.md | 254 +--
Helmsman/helmsman-workflow-guide.md | 32 +-
Helmsman/utils/istio-gateway/README.md | 4 +-
README.md | 1515 +++++++++++------
docs/DSF_CONFIGURATION_GUIDE.md | 814 +++++++++
docs/ENVIRONMENT_DESTRUCTION_GUIDE.md | 981 +++++++++++
docs/GLOSSARY.md | 363 ++++
docs/README.md | 306 ++++
docs/SECRET_GENERATION_GUIDE.md | 666 ++++++++
docs/WORKFLOW_GUIDE.md | 763 +++++++++
docs/_images/ARCHITECTURE_DIAGRAMS.md | 562 +++---
terraform/README.md | 582 +++----
terraform/base-infra/WIREGUARD_SETUP.md | 54 +-
terraform/modules/aws/README.md | 204 +--
.../aws/aws-resource-creation/README.md | 66 +-
terraform/modules/aws/nfs-setup/README.md | 46 +-
terraform/modules/aws/nginx-setup/README.md | 50 +-
...L_CLEAN_RANCHER_KEYCLOAK_IMPLEMENTATION.md | 84 +-
terraform/modules/aws/rke2-cluster/README.md | 38 +-
.../aws/rke2-cluster/ansible/README.md | 44 +-
terraform/modules/azure/README.md | 4 +-
terraform/modules/gcp/README.md | 4 +-
26 files changed, 6091 insertions(+), 1759 deletions(-)
create mode 100644 docs/DSF_CONFIGURATION_GUIDE.md
create mode 100644 docs/ENVIRONMENT_DESTRUCTION_GUIDE.md
create mode 100644 docs/GLOSSARY.md
create mode 100644 docs/README.md
create mode 100644 docs/SECRET_GENERATION_GUIDE.md
create mode 100644 docs/WORKFLOW_GUIDE.md
diff --git a/.github/scripts/README.md b/.github/scripts/README.md
index 2b3ccb59..d05f6e10 100644
--- a/.github/scripts/README.md
+++ b/.github/scripts/README.md
@@ -127,8 +127,8 @@ These scripts handle complex operations that would otherwise make workflow files
### generate-pg-secrets.sh
-**Status**: Legacy - No longer used
-**Reason**: PostgreSQL configuration now handled via Terraform variables (`enable_postgresql_setup`)
+**Status**: Legacy - No longer used
+**Reason**: PostgreSQL configuration now handled via Terraform variables (`enable_postgresql_setup`)
**Replacement**: Configure PostgreSQL in `terraform/implementations/{cloud}/{component}/{cloud}.tfvars`
## Placeholder Scripts
@@ -157,36 +157,36 @@ Some scripts are empty placeholders for future functionality:
```mermaid
graph TD
- A[Workflow Start] --> B[setup-gpg.sh]
- B --> C{Backend Type?}
- C -->|Local| D[decrypt-state.sh]
- C -->|Remote| E[setup-cloud-storage.sh]
- D --> F[configure-backend.sh]
- E --> F
- F --> G[Terraform Operations]
- G --> H[encrypt-state.sh]
- G --> I[cleanup-state-locking.sh]
- H --> J[Workflow End]
- I --> J
+ A[Workflow Start] --> B[setup-gpg.sh]
+ B --> C{Backend Type?}
+ C -->|Local| D[decrypt-state.sh]
+ C -->|Remote| E[setup-cloud-storage.sh]
+ D --> F[configure-backend.sh]
+ E --> F
+ F --> G[Terraform Operations]
+ G --> H[encrypt-state.sh]
+ G --> I[cleanup-state-locking.sh]
+ H --> J[Workflow End]
+ I --> J
```
## Directory Structure
```
.github/scripts/
-├── README.md # This file - scripts documentation
-├── configure-backend.sh # Backend configuration generation
-├── setup-cloud-storage.sh # Remote storage setup
-├── encrypt-state.sh # GPG state encryption
-├── decrypt-state.sh # GPG state decryption
-├── setup-gpg.sh # GPG environment setup
-├── cleanup-state-locking.sh # State lock cleanup
-├── generate-pg-secrets.sh # Legacy PostgreSQL secrets
-├── test-infrastructure.sh # Comprehensive testing
-├── validate-workflow-integration.sh # Workflow validation
-├── test-*.sh # Various test scripts
-├── setup-s3-backend.sh # Empty placeholder
-└── setup-remote-storage.sh # Empty placeholder
+├── README.md # This file - scripts documentation
+├── configure-backend.sh # Backend configuration generation
+├── setup-cloud-storage.sh # Remote storage setup
+├── encrypt-state.sh # GPG state encryption
+├── decrypt-state.sh # GPG state decryption
+├── setup-gpg.sh # GPG environment setup
+├── cleanup-state-locking.sh # State lock cleanup
+├── generate-pg-secrets.sh # Legacy PostgreSQL secrets
+├── test-infrastructure.sh # Comprehensive testing
+├── validate-workflow-integration.sh # Workflow validation
+├── test-*.sh # Various test scripts
+├── setup-s3-backend.sh # Empty placeholder
+└── setup-remote-storage.sh # Empty placeholder
```
## Usage from Workflows
@@ -195,14 +195,14 @@ Scripts are called from GitHub Actions workflows with proper error handling:
```yaml
- name: Setup GPG
- run: |
- chmod +x .github/scripts/setup-gpg.sh
- .github/scripts/setup-gpg.sh --passphrase "${{ secrets.GPG_PASSPHRASE }}"
+ run: |
+ chmod +x .github/scripts/setup-gpg.sh
+ .github/scripts/setup-gpg.sh --passphrase "${{ secrets.GPG_PASSPHRASE }}"
- name: Decrypt State
- run: |
- chmod +x .github/scripts/decrypt-state.sh
- .github/scripts/decrypt-state.sh --backend-type local --passphrase "${{ secrets.GPG_PASSPHRASE }}"
+ run: |
+ chmod +x .github/scripts/decrypt-state.sh
+ .github/scripts/decrypt-state.sh --backend-type local --passphrase "${{ secrets.GPG_PASSPHRASE }}"
```
## Script Development Guidelines
diff --git a/.github/scripts/WORKFLOW_TESTING_GUIDE.md b/.github/scripts/WORKFLOW_TESTING_GUIDE.md
index 41db52a7..067c634d 100644
--- a/.github/scripts/WORKFLOW_TESTING_GUIDE.md
+++ b/.github/scripts/WORKFLOW_TESTING_GUIDE.md
@@ -15,7 +15,7 @@ This guide demonstrates how to test and use the modernized MOSIP Terraform infra
### Test Coverage
- **Providers**: AWS, Azure, GCP (all 3)
-- **Components**: base-infra, infra, observ-infra (all 3)
+- **Components**: base-infra, infra, observ-infra (all 3)
- **Backends**: local, remote (both)
- **Workflows**: terraform.yml, terraform-destroy.yml (both)
@@ -32,9 +32,9 @@ This guide demonstrates how to test and use the modernized MOSIP Terraform infra
### Notable Differences
1. **Cloud Storage Setup**:
- - `terraform.yml` uses `setup-cloud-storage.sh` (creates storage)
- - `terraform-destroy.yml` has custom storage checking logic (reads existing storage)
- - **This is correct behavior** - destroy doesn't need to create storage
+ - `terraform.yml` uses `setup-cloud-storage.sh` (creates storage)
+ - `terraform-destroy.yml` has custom storage checking logic (reads existing storage)
+ - **This is correct behavior** - destroy doesn't need to create storage
### Test Environment Limitations
@@ -50,7 +50,7 @@ This guide demonstrates how to test and use the modernized MOSIP Terraform infra
# Run complete infrastructure validation
.github/scripts/test-infrastructure.sh
-# Run workflow integration validation
+# Run workflow integration validation
.github/scripts/validate-workflow-integration.sh
# Run end-to-end workflow simulation
@@ -65,24 +65,24 @@ This guide demonstrates how to test and use the modernized MOSIP Terraform infra
```yaml
Workflow: terraform plan / apply
Inputs:
- CLOUD_PROVIDER: aws
- TERRAFORM_COMPONENT: infra
- BACKEND_TYPE: local
- REMOTE_BACKEND_CONFIG: (empty)
- SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
- TERRAFORM_APPLY: false # Plan only first
+ CLOUD_PROVIDER: aws
+ TERRAFORM_COMPONENT: infra
+ BACKEND_TYPE: local
+ REMOTE_BACKEND_CONFIG: (empty)
+ SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
+ TERRAFORM_APPLY: false # Plan only first
```
**Step 2: Remote Backend Test**
```yaml
-Workflow: terraform plan / apply
+Workflow: terraform plan / apply
Inputs:
- CLOUD_PROVIDER: aws
- TERRAFORM_COMPONENT: infra
- BACKEND_TYPE: remote
- REMOTE_BACKEND_CONFIG: aws:your-bucket-name:us-east-1
- SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
- TERRAFORM_APPLY: false # Plan only first
+ CLOUD_PROVIDER: aws
+ TERRAFORM_COMPONENT: infra
+ BACKEND_TYPE: remote
+ REMOTE_BACKEND_CONFIG: aws:your-bucket-name:us-east-1
+ SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
+ TERRAFORM_APPLY: false # Plan only first
```
#### Testing terraform-destroy.yml Workflow
@@ -91,24 +91,24 @@ Inputs:
```yaml
Workflow: terraform destroy
Inputs:
- CLOUD_PROVIDER: aws
- TERRAFORM_COMPONENT: infra
- BACKEND_TYPE: local
- REMOTE_BACKEND_CONFIG: (empty)
- SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
- TERRAFORM_DESTROY: false # Dry run first
+ CLOUD_PROVIDER: aws
+ TERRAFORM_COMPONENT: infra
+ BACKEND_TYPE: local
+ REMOTE_BACKEND_CONFIG: (empty)
+ SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
+ TERRAFORM_DESTROY: false # Dry run first
```
**Step 2: Actual Destroy**
```yaml
Workflow: terraform destroy
Inputs:
- CLOUD_PROVIDER: aws
- TERRAFORM_COMPONENT: infra
- BACKEND_TYPE: local
- REMOTE_BACKEND_CONFIG: (empty)
- SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
- TERRAFORM_DESTROY: true # Confirm destruction
+ CLOUD_PROVIDER: aws
+ TERRAFORM_COMPONENT: infra
+ BACKEND_TYPE: local
+ REMOTE_BACKEND_CONFIG: (empty)
+ SSH_PRIVATE_KEY: YOUR_SSH_SECRET_NAME
+ TERRAFORM_DESTROY: true # Confirm destruction
```
### 3. Multi-Cloud Testing Matrix
@@ -150,7 +150,7 @@ gcp:mosip-base-infra-state:us-central1
**Application Infrastructure (Medium Security)**
```
aws:mosip-infra-state:us-east-1
-azure:mosip-infra-rg:mosipinfra:terraform-state
+azure:mosip-infra-rg:mosipinfra:terraform-state
gcp:mosip-infra-state:us-central1
```
@@ -183,7 +183,7 @@ aws:mosip-dev-infra:us-east-1
### 1. Testing Order
1. **Start with Local Backends** - No cloud dependencies
-2. **Test Each Component Separately** - base-infra → infra → observ-infra
+2. **Test Each Component Separately** - base-infra → infra → observ-infra
3. **Test Remote Backends** - After local testing succeeds
4. **Test Destroy Workflows** - After successful deployments
@@ -198,18 +198,18 @@ aws:mosip-dev-infra:us-east-1
**Deployment Order:**
```
base-infra (VPC, networking)
- ↓
+ ↓
infra (applications, databases)
- ↓
+ ↓
observ-infra (monitoring, logging)
```
**Destruction Order (reverse):**
```
observ-infra (safe to destroy anytime)
- ↓
+ ↓
infra (can be destroyed/recreated)
- ↓
+ ↓
base-infra (typically permanent)
```
@@ -228,7 +228,7 @@ Solution: Add AWS credentials to GitHub secrets
Error: Not logged into Azure
Solution: Add Azure service principal to GitHub secrets
- AZURE_CLIENT_ID
-- AZURE_CLIENT_SECRET
+- AZURE_CLIENT_SECRET
- AZURE_TENANT_ID
```
@@ -259,7 +259,7 @@ BACKEND_TYPE: remote
REMOTE_BACKEND_CONFIG: aws:mosip-prod-base:us-east-1
TERRAFORM_APPLY: true
-# Step 2: Deploy application infrastructure
+# Step 2: Deploy application infrastructure
Workflow: terraform plan / apply
CLOUD_PROVIDER: aws
TERRAFORM_COMPONENT: infra
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 54550735..571e02d0 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -6,7 +6,7 @@
This directory contains GitHub Actions workflows for automated MOSIP deployment:
- **Terraform workflows** for infrastructure provisioning (AWS complete, Azure/GCP placeholders)
-- **Helmsman workflows** for application deployment on provisioned infrastructure
+- **Helmsman workflows** for application deployment on provisioned infrastructure
- **GPG encrypted state management** with branch isolation
- **Sequential workflow execution** with parallel optimization within Helmsman phase
@@ -34,20 +34,20 @@ This directory contains GitHub Actions workflows for automated MOSIP deployment:
1. **Navigate**: Actions → "terraform plan / apply"
2. **Configure Parameters**:
- ```yaml
- CLOUD_PROVIDER: aws # Currently only AWS fully supported
- TERRAFORM_COMPONENT: base-infra # base-infra | infra | observ-infra
- BACKEND_TYPE: local # local with GPG encryption (recommended)
- REMOTE_BACKEND_CONFIG: "" # Not used with local backend
- SSH_PRIVATE_KEY: SSH_PRIVATE_KEY # GitHub secret name
- TERRAFORM_APPLY: true # false = plan only
- ENABLE_STATE_LOCKING: false # Optional DynamoDB state locking
- ```
+ ```yaml
+ CLOUD_PROVIDER: aws # Currently only AWS fully supported
+ TERRAFORM_COMPONENT: base-infra # base-infra | infra | observ-infra
+ BACKEND_TYPE: local # local with GPG encryption (recommended)
+ REMOTE_BACKEND_CONFIG: "" # Not used with local backend
+ SSH_PRIVATE_KEY: SSH_PRIVATE_KEY # GitHub secret name
+ TERRAFORM_APPLY: true # false = plan only
+ ENABLE_STATE_LOCKING: false # Optional DynamoDB state locking
+ ```
3. **Execute**: Click "Run workflow"
4. **PostgreSQL Configuration**: Set in `terraform/implementations/aws/infra/aws.tfvars`:
- ```hcl
- enable_postgresql_setup = true # External PostgreSQL via Terraform + Ansible
- ```
+ ```hcl
+ enable_postgresql_setup = true # External PostgreSQL via Terraform + Ansible
+ ```
### Step 2: Deploy Prerequisites & External Dependencies (After Infrastructure Ready)
@@ -60,7 +60,7 @@ This directory contains GitHub Actions workflows for automated MOSIP deployment:
### Step 3: Deploy MOSIP Services (After Prerequisites Ready)
-1. **Navigate**: Actions → "helmsman mosip services"
+1. **Navigate**: Actions → "helmsman mosip services"
2. **PostgreSQL Integration**: Automatically uses deployed PostgreSQL
3. **Duration**: ~25-35 minutes
@@ -79,10 +79,10 @@ PostgreSQL is configured in Terraform variables, not as a workflow input:
```hcl
# terraform/implementations/aws/infra/aws.tfvars
-enable_postgresql_setup = true # Enable external PostgreSQL
-nginx_node_ebs_volume_size_2 = 200 # EBS volume size for PostgreSQL data
-postgresql_version = "15" # PostgreSQL version
-postgresql_port = "5433" # PostgreSQL port
+enable_postgresql_setup = true # Enable external PostgreSQL
+nginx_node_ebs_volume_size_2 = 200 # EBS volume size for PostgreSQL data
+postgresql_version = "15" # PostgreSQL version
+postgresql_port = "5433" # PostgreSQL port
```
### PostgreSQL Deployment Options
@@ -115,15 +115,15 @@ Step 3: Application (Repeatable)
```mermaid
graph LR
- A[base-infra
Foundation] --> B[observ-infra
Management
Optional]
- A --> C[infra
MOSIP Clusters]
- B -.->|Import Clusters| C
- C --> D[Multiple MOSIP
Deployments]
-
- style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
- style B fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
- style C fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
- style D fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
+ A[base-infra
Foundation] --> B[observ-infra
Management
Optional]
+ A --> C[infra
MOSIP Clusters]
+ B -.->|Import Clusters| C
+ C --> D[Multiple MOSIP
Deployments]
+
+ style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
+ style B fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
+ style C fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
+ style D fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
```
| Component | Purpose | Deployment Order | Dependencies | Lifecycle |
@@ -136,29 +136,29 @@ graph LR
```mermaid
graph TD
- START[GitHub Actions Trigger] --> VALIDATE[Validate Parameters]
- VALIDATE --> SETUP[Setup Terraform & Cloud Credentials]
- SETUP --> BACKEND[Configure Local Backend with GPG]
-
- BACKEND --> DECRYPT{Encrypted State Exists?}
- DECRYPT -->|Yes| DECRYPTSTATE[Decrypt State with GPG]
- DECRYPT -->|No| INIT[terraform init]
- DECRYPTSTATE --> INIT
-
- INIT --> PLAN[terraform plan]
- PLAN --> DECISION{Apply or Plan Only?}
-
- DECISION -->|Plan Only| OUTPUT[Show Plan Output]
- DECISION -->|Apply| APPLY[terraform apply]
-
- APPLY --> ENCRYPT[Encrypt State with GPG]
- ENCRYPT --> SUCCESS[Deployment Complete]
- OUTPUT --> COMPLETE[Workflow Complete]
- SUCCESS --> COMPLETE
-
- style START fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
- style APPLY fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
- style SUCCESS fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
+ START[GitHub Actions Trigger] --> VALIDATE[Validate Parameters]
+ VALIDATE --> SETUP[Setup Terraform & Cloud Credentials]
+ SETUP --> BACKEND[Configure Local Backend with GPG]
+
+ BACKEND --> DECRYPT{Encrypted State Exists?}
+ DECRYPT -->|Yes| DECRYPTSTATE[Decrypt State with GPG]
+ DECRYPT -->|No| INIT[terraform init]
+ DECRYPTSTATE --> INIT
+
+ INIT --> PLAN[terraform plan]
+ PLAN --> DECISION{Apply or Plan Only?}
+
+ DECISION -->|Plan Only| OUTPUT[Show Plan Output]
+ DECISION -->|Apply| APPLY[terraform apply]
+
+ APPLY --> ENCRYPT[Encrypt State with GPG]
+ ENCRYPT --> SUCCESS[Deployment Complete]
+ OUTPUT --> COMPLETE[Workflow Complete]
+ SUCCESS --> COMPLETE
+
+ style START fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
+ style APPLY fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
+ style SUCCESS fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
```
**Note**: PostgreSQL setup is handled by Terraform modules and Ansible during the `terraform apply` step based on `enable_postgresql_setup` configuration in `.tfvars` files, not as a separate workflow step.
@@ -190,9 +190,9 @@ terraform/observ-infra/aws-observ-infra-testgrid-terraform.tfstate
```bash
# Required GitHub Secret
GPG_PRIVATE_KEY: |
- -----BEGIN PGP PRIVATE KEY BLOCK-----
-
- -----END PGP PRIVATE KEY BLOCK-----
+ -----BEGIN PGP PRIVATE KEY BLOCK-----
+
+ -----END PGP PRIVATE KEY BLOCK-----
```
## Parameter Reference
@@ -216,7 +216,7 @@ BACKEND_TYPE: local
# Custom naming: aws-infra-testgrid-terraform.tfstate
```
-#### Remote Backend (Legacy Support)
+#### Remote Backend (Legacy Support)
```yaml
BACKEND_TYPE: remote
REMOTE_BACKEND_CONFIG: aws:bucket-name:region
@@ -240,15 +240,15 @@ REMOTE_BACKEND_CONFIG: aws:bucket-name:region
```yaml
# GPG Encryption for State Files
GPG_PRIVATE_KEY: |
- -----BEGIN PGP PRIVATE KEY BLOCK-----
-
- -----END PGP PRIVATE KEY BLOCK-----
+ -----BEGIN PGP PRIVATE KEY BLOCK-----
+
+ -----END PGP PRIVATE KEY BLOCK-----
# SSH Access for jumpserver
SSH_PRIVATE_KEY: |
- -----BEGIN OPENSSH PRIVATE KEY-----
-
- -----END OPENSSH PRIVATE KEY-----
+ -----BEGIN OPENSSH PRIVATE KEY-----
+
+ -----END OPENSSH PRIVATE KEY-----
# AWS Credentials (Complete Implementation)
AWS_ACCESS_KEY_ID: AKIA...
@@ -262,11 +262,11 @@ AZURE_TENANT_ID: 12345678-1234-1234-1234-123456789012
# GCP Credentials (Placeholder Implementation)
GOOGLE_CREDENTIALS: |
- {
- "type": "service_account",
- "project_id": "your-project",
- ...
- }
+ {
+ "type": "service_account",
+ "project_id": "your-project",
+ ...
+ }
# Optional: Slack notifications
SLACK_WEBHOOK_URL: https://hooks.slack.com/services/...
@@ -304,7 +304,7 @@ TERRAFORM_COMPONENT: infra
BACKEND_TYPE: local
# State: aws-infra-main-terraform.tfstate.gpg
-# Staging deployment (staging branch)
+# Staging deployment (staging branch)
CLOUD_PROVIDER: aws
TERRAFORM_COMPONENT: infra
BACKEND_TYPE: local
@@ -315,14 +315,14 @@ BACKEND_TYPE: local
```yaml
# Step 1: Deploy Terraform infrastructure (must complete first)
Workflow: terraform.yml
- Component: base-infra → Deploy foundational infrastructure
- Component: observ-infra → Deploy monitoring cluster (optional)
- Component: infra → Deploy MOSIP cluster + PostgreSQL
+ Component: base-infra → Deploy foundational infrastructure
+ Component: observ-infra → Deploy monitoring cluster (optional)
+ Component: infra → Deploy MOSIP cluster + PostgreSQL
# Step 2: Deploy Helmsman components (after Terraform complete)
Workflow: helmsman_external.yml → Parallel deployment:
- - Prerequisites: Monitoring, Istio, Logging
- - External Dependencies: PostgreSQL connection, MinIO, Keycloak, Kafka
+ - Prerequisites: Monitoring, Istio, Logging
+ - External Dependencies: PostgreSQL connection, MinIO, Keycloak, Kafka
# Step 3: Deploy MOSIP services (after external dependencies ready)
Workflow: helmsman_mosip.yml → Deploy MOSIP applications
@@ -369,93 +369,93 @@ Workflow: helmsman_testrigs.yml → Deploy testing infrastructure
### GitHub Actions Workflow Sequence
```mermaid
sequenceDiagram
- participant User
- participant Terraform Workflows
- participant Helmsman Workflows
- participant AWS Infrastructure
- participant PostgreSQL
-
- User->>Terraform Workflows: 1. Deploy base-infra
- Terraform Workflows->>AWS Infrastructure: Create VPC, WireGuard
- AWS Infrastructure-->>User: Base infrastructure ready
-
- User->>Terraform Workflows: 2. Deploy infra + PostgreSQL
- Terraform Workflows->>AWS Infrastructure: Create RKE2 cluster
- Terraform Workflows->>PostgreSQL: Setup PostgreSQL 15 via Ansible
- AWS Infrastructure-->>User: MOSIP cluster + PostgreSQL ready
-
- User->>Helmsman Workflows: 3. Deploy Prerequisites (parallel)
- User->>Helmsman Workflows: 3. Deploy External Dependencies (parallel)
- Helmsman Workflows-->>AWS Infrastructure: Install Prerequisites & Dependencies
- AWS Infrastructure-->>User: Prerequisites & Dependencies ready (70-110 min)
-
- User->>Helmsman Workflows: 4. Deploy MOSIP Services
- Helmsman Workflows->>PostgreSQL: Connect to external database
- AWS Infrastructure-->>User: MOSIP platform operational
+ participant User
+ participant Terraform Workflows
+ participant Helmsman Workflows
+ participant AWS Infrastructure
+ participant PostgreSQL
+
+ User->>Terraform Workflows: 1. Deploy base-infra
+ Terraform Workflows->>AWS Infrastructure: Create VPC, WireGuard
+ AWS Infrastructure-->>User: Base infrastructure ready
+
+ User->>Terraform Workflows: 2. Deploy infra + PostgreSQL
+ Terraform Workflows->>AWS Infrastructure: Create RKE2 cluster
+ Terraform Workflows->>PostgreSQL: Setup PostgreSQL 15 via Ansible
+ AWS Infrastructure-->>User: MOSIP cluster + PostgreSQL ready
+
+ User->>Helmsman Workflows: 3. Deploy Prerequisites (parallel)
+ User->>Helmsman Workflows: 3. Deploy External Dependencies (parallel)
+ Helmsman Workflows-->>AWS Infrastructure: Install Prerequisites & Dependencies
+ AWS Infrastructure-->>User: Prerequisites & Dependencies ready (70-110 min)
+
+ User->>Helmsman Workflows: 4. Deploy MOSIP Services
+ Helmsman Workflows->>PostgreSQL: Connect to external database
+ AWS Infrastructure-->>User: MOSIP platform operational
```
### Workflow Execution Order
1. **Terraform Workflows** (Sequential - Infrastructure Setup):
- - `terraform.yml` → Deploy base-infra
- - `terraform.yml` → Deploy observ-infra (optional)
- - `terraform.yml` → Deploy infra + PostgreSQL
+ - `terraform.yml` → Deploy base-infra
+ - `terraform.yml` → Deploy observ-infra (optional)
+ - `terraform.yml` → Deploy infra + PostgreSQL
2. **Helmsman Workflows** (Can run in parallel after Terraform complete):
- - `helmsman_external.yml` → Prerequisites + External Dependencies (simultaneous)
- - `helmsman_mosip.yml` → MOSIP Services (after external dependencies ready)
- - `helmsman_testrigs.yml` → Test components (optional)
+ - `helmsman_external.yml` → Prerequisites + External Dependencies (simultaneous)
+ - `helmsman_mosip.yml` → MOSIP Services (after external dependencies ready)
+ - `helmsman_testrigs.yml` → Test components (optional)
### Parallel Deployment Architecture
```mermaid
graph TD
- A[Terraform Infrastructure Complete] --> B[Helmsman Workflows Triggered]
-
- B --> C[Prerequisites Workflow]
- B --> D[External Dependencies Workflow]
-
- C -->|Parallel| E[Monitoring Stack]
- C -->|Parallel| F[Istio Service Mesh]
- C -->|Parallel| G[Logging Infrastructure]
-
- D -->|Parallel| H[PostgreSQL Connection]
- D -->|Parallel| I[MinIO Storage]
- D -->|Parallel| J[Keycloak IAM]
- D -->|Parallel| K[Kafka Messaging]
-
- E --> L[MOSIP Services Deployment]
- F --> L
- G --> L
- H --> L
- I --> L
- J --> L
- K --> L
-
- style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
- style B fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
- style C fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
- style D fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
- style L fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
+ A[Terraform Infrastructure Complete] --> B[Helmsman Workflows Triggered]
+
+ B --> C[Prerequisites Workflow]
+ B --> D[External Dependencies Workflow]
+
+ C -->|Parallel| E[Monitoring Stack]
+ C -->|Parallel| F[Istio Service Mesh]
+ C -->|Parallel| G[Logging Infrastructure]
+
+ D -->|Parallel| H[PostgreSQL Connection]
+ D -->|Parallel| I[MinIO Storage]
+ D -->|Parallel| J[Keycloak IAM]
+ D -->|Parallel| K[Kafka Messaging]
+
+ E --> L[MOSIP Services Deployment]
+ F --> L
+ G --> L
+ H --> L
+ I --> L
+ J --> L
+ K --> L
+
+ style A fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
+ style B fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
+ style C fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
+ style D fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
+ style L fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
```
-**Sequential Dependency**: Terraform workflows must complete before Helmsman workflows
+**Sequential Dependency**: Terraform workflows must complete before Helmsman workflows
**Parallel Optimization**: Prerequisites and External Dependencies run simultaneously via separate Helmsman workflows
---
## Support & Best Practices
-**Workflow Maintenance**: Keep workflows updated with latest Terraform versions
-**State Management**: GPG encrypted state with branch-based isolation
-**Security Reviews**: Regular rotation of GPG keys and cloud credentials
-**PostgreSQL Management**: Automated setup via Terraform + Ansible integration
-**Performance Optimization**: Use parallel deployment for 20% faster setup times
+**Workflow Maintenance**: Keep workflows updated with latest Terraform versions
+**State Management**: GPG encrypted state with branch-based isolation
+**Security Reviews**: Regular rotation of GPG keys and cloud credentials
+**PostgreSQL Management**: Automated setup via Terraform + Ansible integration
+**Performance Optimization**: Use parallel deployment for 20% faster setup times
## Cloud Provider Contribution Guide
-**AWS** - Production ready with full feature set
-**Azure** - [Placeholder implementation](../terraform/base-infra/azure/main.tf) - contributions welcome
-**GCP** - [Placeholder implementation](../terraform/base-infra/gcp/main.tf) - contributions welcome
+**AWS** - Production ready with full feature set
+**Azure** - [Placeholder implementation](../terraform/base-infra/azure/main.tf) - contributions welcome
+**GCP** - [Placeholder implementation](../terraform/base-infra/gcp/main.tf) - contributions welcome
**Community contributions needed for Azure and GCP implementations**
diff --git a/.github/workflows/terraform-destroy.yml b/.github/workflows/terraform-destroy.yml
index dadba0dc..edfb0392 100644
--- a/.github/workflows/terraform-destroy.yml
+++ b/.github/workflows/terraform-destroy.yml
@@ -52,7 +52,7 @@ on:
description: 'Enable state locking cleanup (must match the setting used during creation)'
required: false
type: boolean
- default: true
+ default: false
SSH_PRIVATE_KEY:
description: 'The GitHub secret containing the private key of the SSH key named in the preceding input variable `SSH_PRIVATE_KEY` is used for SSH login purposes on nginx node.'
required: true
diff --git a/Helmsman/dsf/README.md b/Helmsman/dsf/README.md
index 617a61e0..63099938 100644
--- a/Helmsman/dsf/README.md
+++ b/Helmsman/dsf/README.md
@@ -24,7 +24,7 @@ The Instructions that include within DSF:
To deploy Pre-requisites and External services of mosip we have two dsf files i,e
-* `prereq-dsf.yaml`: Installs Pre-requisites such as monitoring, logging, alerting, istio, httpbin and global_configmap.
+* `prereq-dsf.yaml`: Installs Pre-requisites such as monitoring, logging, alerting, istio, httpbin and global_configmap.
* `external-dsf.yaml`: Intsalls all the External services of mosip.
Make sure to update the above two dsf files with the required configuration changes as per the environement and also update the `global_configmap.yaml` file with the required domain's then initiate the `helmsman_external.yml` workflow file.
@@ -39,192 +39,192 @@ To deploy Mosip corde service we have below dsf file i,e
## Key Customization Areas In mosip-dsf To Deploy Mosip Core Service
### 1. Helm Configuration
-**Default Configuration**
-- `helmDefaults`
- - `tillerNamespace`: kube-system
- - `tillerless`: true
- - `install`: true
+**Default Configuration**
+- `helmDefaults`
+ - `tillerNamespace`: kube-system
+ - `tillerless`: true
+ - `install`: true
> *Kept unchanged in the customized version.*
---
### 2. Helm Repositories
-**Default Configuration**
-- `helmRepos`
- - `bitnami`: https://charts.bitnami.com/bitnami
- - `mosip`: https://mosip.github.io/mosip-helm
+**Default Configuration**
+- `helmRepos`
+ - `bitnami`: https://charts.bitnami.com/bitnami
+ - `mosip`: https://mosip.github.io/mosip-helm
> *Kept unchanged in the customized version.*
---
### 3. Namespace Configuration
-**Modifications**
-- Created multiple namespaces for different MOSIP components
-- Set `protected: false` for all namespaces
-- Key namespaces include:
- - conf‑secrets
- - config‑server
- - artifactory
- - captcha
- - keymanager
- - …and 17 more (22 in total)
+**Modifications**
+- Created multiple namespaces for different MOSIP components
+- Set `protected: false` for all namespaces
+- Key namespaces include:
+ - conf‑secrets
+ - config‑server
+ - artifactory
+ - captcha
+ - keymanager
+ - …and 17 more (22 in total)
---
### 4. Applications Configuration
#### 4.1 Config Server
-- Custom configuration with multiple Git repositories:
- - Type: git, URI: https://github.com/mosip/inji-config, Version: develop
- - Type: git, URI: https://github.com/mosip/mosip-config, Version: soil‑helmsman
- - Example:
+- Custom configuration with multiple Git repositories:
+ - Type: git, URI: https://github.com/mosip/inji-config, Version: develop
+ - Type: git, URI: https://github.com/mosip/mosip-config, Version: soil‑helmsman
+ - Example:
```
- set:
- image.repository: "mosipqa/kernel-config-server"
- image.tag: "develop"
- gitRepo.uri: "https://github.com/mosip/mosip-config"
- gitRepo.version: "soil-helmsman"
- gitRepo.searchFolders: ""
- gitRepo.private: "false"
- gitRepo.username: ""
- gitRepo.token: ""
- resources.limits.cpu: "500m"
- resources.limits.memory: "2000Mi"
- resources.requests.cpu: "200m"
- resources.requests.memory: "1000Mi"
- spring_profiles.enabled: true
- spring_profiles.spring_profiles_active: "composite"
- spring_profiles.spring_compositeRepos[0].type: "git"
- spring_profiles.spring_compositeRepos[0].uri: "https://github.com/mosip/inji-config"
- spring_profiles.spring_compositeRepos[0].version: "develop"
- spring_profiles.spring_compositeRepos[0].spring_cloud_config_server_git_cloneOnStart: true
- spring_profiles.spring_compositeRepos[0].spring_cloud_config_server_git_force_pull: true
- spring_profiles.spring_compositeRepos[0].spring_cloud_config_server_git_refreshRate: 5
- spring_profiles.spring_compositeRepos[1].type: "git"
- spring_profiles.spring_compositeRepos[1].uri: "https://github.com/mosip/mosip-config"
- spring_profiles.spring_compositeRepos[1].version: "soil-helmsman"
- spring_profiles.spring_compositeRepos[1].spring_cloud_config_server_git_cloneOnStart: true
- spring_profiles.spring_compositeRepos[1].spring_cloud_config_server_git_force_pull: true
- spring_profiles.spring_compositeRepos[1].spring_cloud_config_server_git_refreshRate: 5
- spring_profiles.spring_fail_on_composite_error: false
- localRepo.enabled: false
- localRepo.spring_profiles_active: "native"
- localRepo.spring_cloud_config_server_native_search_locations: "file:///var/lib/config_repo"
- localRepo.spring_cloud_config_server_accept_empty: true
- localRepo.spring_cloud_config_server_git_force_pull: false
- localRepo.spring_cloud_config_server_git_refreshRate: 0
- localRepo.spring_cloud_config_server_git_cloneOnStart: false
- volume.name: "config-server"
- volume.storageClass: "nfs-client"
- volume.accessModes[0]: "ReadWriteMany"
- volume.size: "10Mi"
- volume.mountDir: "/var/lib/config_repo"
- volume.nfs.path: ""
- volume.nfs.server: ""
+ set:
+ image.repository: "mosipqa/kernel-config-server"
+ image.tag: "develop"
+ gitRepo.uri: "https://github.com/mosip/mosip-config"
+ gitRepo.version: "soil-helmsman"
+ gitRepo.searchFolders: ""
+ gitRepo.private: "false"
+ gitRepo.username: ""
+ gitRepo.token: ""
+ resources.limits.cpu: "500m"
+ resources.limits.memory: "2000Mi"
+ resources.requests.cpu: "200m"
+ resources.requests.memory: "1000Mi"
+ spring_profiles.enabled: true
+ spring_profiles.spring_profiles_active: "composite"
+ spring_profiles.spring_compositeRepos[0].type: "git"
+ spring_profiles.spring_compositeRepos[0].uri: "https://github.com/mosip/inji-config"
+ spring_profiles.spring_compositeRepos[0].version: "develop"
+ spring_profiles.spring_compositeRepos[0].spring_cloud_config_server_git_cloneOnStart: true
+ spring_profiles.spring_compositeRepos[0].spring_cloud_config_server_git_force_pull: true
+ spring_profiles.spring_compositeRepos[0].spring_cloud_config_server_git_refreshRate: 5
+ spring_profiles.spring_compositeRepos[1].type: "git"
+ spring_profiles.spring_compositeRepos[1].uri: "https://github.com/mosip/mosip-config"
+ spring_profiles.spring_compositeRepos[1].version: "soil-helmsman"
+ spring_profiles.spring_compositeRepos[1].spring_cloud_config_server_git_cloneOnStart: true
+ spring_profiles.spring_compositeRepos[1].spring_cloud_config_server_git_force_pull: true
+ spring_profiles.spring_compositeRepos[1].spring_cloud_config_server_git_refreshRate: 5
+ spring_profiles.spring_fail_on_composite_error: false
+ localRepo.enabled: false
+ localRepo.spring_profiles_active: "native"
+ localRepo.spring_cloud_config_server_native_search_locations: "file:///var/lib/config_repo"
+ localRepo.spring_cloud_config_server_accept_empty: true
+ localRepo.spring_cloud_config_server_git_force_pull: false
+ localRepo.spring_cloud_config_server_git_refreshRate: 0
+ localRepo.spring_cloud_config_server_git_cloneOnStart: false
+ volume.name: "config-server"
+ volume.storageClass: "nfs-client"
+ volume.accessModes[0]: "ReadWriteMany"
+ volume.size: "10Mi"
+ volume.mountDir: "/var/lib/config_repo"
+ volume.nfs.path: ""
+ volume.nfs.server: ""
```
#### 4.2 Image Repositories
-- Registries used:
- - mosipdev
- - mosipid
- - mosipqa
-- Tags in use:
- - develop
- - release‑1.3.x
- - specific version tags
- - Example:
+- Registries used:
+ - mosipdev
+ - mosipid
+ - mosipqa
+- Tags in use:
+ - develop
+ - release‑1.3.x
+ - specific version tags
+ - Example:
```
- set:
- image.repository: "mosipdev/artifactory-server"
- image.tag: "release-1.3.x"
+ set:
+ image.repository: "mosipdev/artifactory-server"
+ image.tag: "release-1.3.x"
```
#### 4.3 Resource Configurations
-- Resource limits and requests defined for several services:
- - Limits: CPU 500m, Memory 2000Mi
- - Requests: CPU 200m, Memory 1000Mi
- - Example:
+- Resource limits and requests defined for several services:
+ - Limits: CPU 500m, Memory 2000Mi
+ - Requests: CPU 200m, Memory 1000Mi
+ - Example:
```
- set:
- image.repository: ""
- image.tag: ""
- resources.limits.cpu: 500m
- resources.limits.memory: 3500Mi
- resources.requests.cpu: 500m
- resources.requests.memory: 3500Mi
- startupProbe.failureThreshold: 100
+ set:
+ image.repository: ""
+ image.tag: ""
+ resources.limits.cpu: 500m
+ resources.limits.memory: 3500Mi
+ resources.requests.cpu: 500m
+ resources.requests.memory: 3500Mi
+ startupProbe.failureThreshold: 100
```
#### 4.4 Istio Configuration
-- Host configurations and CORS policy:
- - Hosts: prereg.soil.mosip.net
- - CORS Allow Origins: https://admin.soil.mosip.net
- - Example:
+- Host configurations and CORS policy:
+ - Hosts: prereg.soil.mosip.net
+ - CORS Allow Origins: https://admin.soil.mosip.net
+ - Example:
```
- set:
- image.repository: "mosipdev/partner-management-service"
- image.tag: "develop"
- istio.corsPolicy.allowOrigins[0].prefix: "https://pmp.soil.mosip.net"
- istio.corsPolicy.allowOrigins[1].prefix: "https://pmp-revam.soil.mosip.net"
+ set:
+ image.repository: "mosipdev/partner-management-service"
+ image.tag: "develop"
+ istio.corsPolicy.allowOrigins[0].prefix: "https://pmp.soil.mosip.net"
+ istio.corsPolicy.allowOrigins[1].prefix: "https://pmp-revam.soil.mosip.net"
```
#### 4.5 Persistent Volume Configurations
-- PVC and storage class settings:
- - Storage Class: nfs‑client
- - Access Modes: ReadWriteMany
- - Size: 10Mi
- - Example:
+- PVC and storage class settings:
+ - Storage Class: nfs‑client
+ - Access Modes: ReadWriteMany
+ - Size: 10Mi
+ - Example:
```
- set:
- persistence.enabled: false
- volumePermissions.enabled: false
- persistence.size:
- persistence.mountDir: ""
- persistence.pvc_claim_name: ""
+ set:
+ persistence.enabled: false
+ volumePermissions.enabled: false
+ persistence.size:
+ persistence.mountDir: ""
+ persistence.pvc_claim_name: ""
```
---
### 5. Priority‑Based Deployment
-- Implemented a priority‑based deployment strategy
-- Priorities range from –15 (highest) to –1 (lowest)
+- Implemented a priority‑based deployment strategy
+- Priorities range from –15 (highest) to –1 (lowest)
- Ensures critical services are deployed first
- Example:
```
- priority: -8
+ priority: -8
```
---
### 6. Hooks Configuration
-- Added post‑install and pre-install hooks for various services, including:
- - Setting up namespaces
- - Copying configurations
- - Preparing environments
- - Example:
+- Added post‑install and pre-install hooks for various services, including:
+ - Setting up namespaces
+ - Copying configurations
+ - Preparing environments
+ - Example:
```
- hooks:
- preInstall: "kubectl label ns artifactory istio-injection=enabled --overwrite"
- postInstall: "../hooks/common-cm-and-secret-sharing.sh"
+ hooks:
+ preInstall: "kubectl label ns artifactory istio-injection=enabled --overwrite"
+ postInstall: "../hooks/common-cm-and-secret-sharing.sh"
```
## Deployment Considerations
-- Ensure all referenced Git repositories and image tags are accessible
-- Verify network policies and storage configurations
+- Ensure all referenced Git repositories and image tags are accessible
+- Verify network policies and storage configurations
- Confirm Istio mesh settings and namespace protections align with your infrastructure
---
## Recommended Steps
-1. **Review & Validate** each service configuration
-2. **Test** component interactions in a staging environment
-3. **Verify** persistent volume claims and data persistence
+1. **Review & Validate** each service configuration
+2. **Test** component interactions in a staging environment
+3. **Verify** persistent volume claims and data persistence
4. **Validate** Istio ingress, virtual services, and gateways
---
-> **Note:** This guide provides a high‑level overview. Always refer to the actual DSF Helm charts and values files for precise configuration details.
+> **Note:** This guide provides a high‑level overview. Always refer to the actual DSF Helm charts and values files for precise configuration details.
### Note:
diff --git a/Helmsman/helmsman-workflow-guide.md b/Helmsman/helmsman-workflow-guide.md
index 1f9d77db..d85b537e 100644
--- a/Helmsman/helmsman-workflow-guide.md
+++ b/Helmsman/helmsman-workflow-guide.md
@@ -2,7 +2,7 @@
This repository contains a GitHub Actions workflow to deploy external services and mosip services of MOSIP using Helmsman. we have two workflow's i,e `helmsman_external.yml` and `helmsman_mosip.yml`.
-> **⚠️ Important**: Always use `apply` mode for MOSIP deployments. The `dry-run` mode will fail because MOSIP services depend on shared configmaps and secrets from other namespaces that are not available during dry-run validation.
+> **Important**: Always use `apply` mode for MOSIP deployments. The `dry-run` mode will fail because MOSIP services depend on shared configmaps and secrets from other namespaces that are not available during dry-run validation.
## helmsman_external.yml
@@ -30,8 +30,8 @@ The deployment is done in a matrix strategy to handle multiple configuration fil
- **Default**: `dry-run`
- **Recommended**: `apply` (dry-run will fail due to namespace dependencies)
- **Options**:
- - `dry-run`: Simulates the deployment without making changes (will fail for MOSIP).
- - `apply`: Applies the deployment changes.
+ - `dry-run`: Simulates the deployment without making changes (will fail for MOSIP).
+ - `apply`: Applies the deployment changes.
### Secrets
@@ -89,9 +89,9 @@ The following secrets are required to run this workflow:
- This ensures proper sequencing and reduces manual intervention
**Error Recovery:**
-- If automatic trigger fails: Manually run "Helmsman MOSIP Deployment" from Actions tab
+- If automatic trigger fails: Manually run "Deploy MOSIP services using Helmsman" from Actions tab
- If onboarding processes fail: Manual re-onboarding is required (see limitations in main README)
-- Monitor deployment logs for any failures requiring intervention
+- Monitor deployment logs for any failures requiring intervention
### Triggering the Workflow Manually
1. Navigate to the "Actions" tab in your repository.
@@ -108,16 +108,16 @@ The following secrets are required to run this workflow:
### Phase 1: Prerequisites & External Dependencies (Parallel)
1. **Prerequisites Deployment** (`prereq-dsf.yaml`):
- - Monitoring stack (Rancher monitoring, Grafana, AlertManager)
- - Logging infrastructure (Cattle logging system)
- - Service mesh (Istio) and networking components
+ - Monitoring stack (Rancher monitoring, Grafana, AlertManager)
+ - Logging infrastructure (Cattle logging system)
+ - Service mesh (Istio) and networking components
2. **External Dependencies Deployment** (`external-dsf.yaml`):
- - Databases (PostgreSQL with initialization)
- - Identity & Access (Keycloak)
- - Security (SoftHSM, ClamAV antivirus)
- - Object Storage (MinIO)
- - Message Queues (ActiveMQ, Kafka with UI)
+ - Databases (PostgreSQL with initialization)
+ - Identity & Access (Keycloak)
+ - Security (SoftHSM, ClamAV antivirus)
+ - Object Storage (MinIO)
+ - Message Queues (ActiveMQ, Kafka with UI)
### Phase 2: MOSIP Services (Automatic Trigger)
- **Trigger Condition**: Both prerequisites and external dependencies complete successfully
@@ -131,7 +131,7 @@ Before deploying test rigs, verify:
kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
# Verify specific namespaces
-kubectl get pods -n mosip
+kubectl get pods -A
kubectl get pods -n keycloak
kubectl get pods -n postgres
```
@@ -169,8 +169,8 @@ The workflow is triggered by:
- **Default**: `dry-run`
- **Recommended**: `apply` (dry-run will fail due to namespace dependencies)
- **Options**:
- - `dry-run`: Simulates the deployment without making changes (will fail for MOSIP).
- - `apply`: Applies the deployment changes.
+ - `dry-run`: Simulates the deployment without making changes (will fail for MOSIP).
+ - `apply`: Applies the deployment changes.
### Secrets
diff --git a/Helmsman/utils/istio-gateway/README.md b/Helmsman/utils/istio-gateway/README.md
index b3bd5083..eed9680e 100644
--- a/Helmsman/utils/istio-gateway/README.md
+++ b/Helmsman/utils/istio-gateway/README.md
@@ -2,8 +2,8 @@
## Overview
* Gateways are the medium to access the k8 services outside the cluster.
* In MOSIP we create two Istio gateways by default for making services accessible.
- * Internal: to make sure services are accessible using domain `api-internal.sandbox.xyz.net` over wireguard restriting private access.
- * Public: to make sure services are accessible using domain `api.sandbox.xyz.net`
+ * Internal: to make sure services are accessible using domain `api-internal.sandbox.xyz.net` over wireguard restriting private access.
+ * Public: to make sure services are accessible using domain `api.sandbox.xyz.net`
* Script here picks up domain related values from confligmap `global` created in `default` namespace.
## Install
```
diff --git a/README.md b/README.md
index dba1c4b5..130289ed 100644
--- a/README.md
+++ b/README.md
@@ -1,53 +1,69 @@
-

+
# MOSIP Rapid Deployment
This repository provides a **3-step rapid deployment model** for MOSIP (Modular Open Source Identity Platform) with enhanced security features including GPG (GNU Privacy Guard) encryption for local backends and integrated PostgreSQL setup via Terraform modules.
+**Complete Documentation Index:** [View All Documentation](docs/README.md)
+
+**First Time Deploying? Start Here!**
+
+We've created comprehensive beginner-friendly guides to help you succeed:
+
+| Guide | What You'll Learn | When to Read |
+| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------- |
+| **[Glossary](docs/GLOSSARY.md)** | Plain-language explanations of all technical terms (AWS, Kubernetes, Terraform, VPN, etc.) | Before you start - understand the terminology |
+| **[Secret Generation Guide](docs/SECRET_GENERATION_GUIDE.md)** | Step-by-step instructions to generate SSH keys, AWS credentials, GPG passwords, and more | Before deployment - setup required secrets |
+| **[Workflow Guide](docs/WORKFLOW_GUIDE.md)** | Visual walkthrough of GitHub Actions workflows with screenshots and navigation help | During deployment - run workflows correctly |
+| **[DSF Configuration Guide](docs/DSF_CONFIGURATION_GUIDE.md)** | How to configure Helmsman files including clusterid and domain settings | Before Helmsman deployment - configure applications |
+| **[Environment Destruction Guide](docs/ENVIRONMENT_DESTRUCTION_GUIDE.md)** | Safe teardown procedures, backup steps, and cost monitoring | After deployment - clean up resources |
+
+---
+
## Complete Deployment Flow
```mermaid
graph TB
- %% Prerequisites
- A[Fork Repository] --> B[Configure Secrets]
- B --> C[Select Cloud Provider
AWS/Azure/GCP/etc.]
-
- %% Infrastructure Phase
- C --> D[Run Terraform: base-infra
VPC, Networking, Jumpserver, WireGuard
One-time deployment]
- D --> E{Deploy Observability?}
- E -->|Yes| F[Run Terraform: observ-infra
Rancher UI, Keycloak, Monitoring
Can be destroyed/recreated]
- E -->|No| G[Run Terraform: infra
Complete MOSIP Infrastructure Setup
Can be destroyed/recreated]
- F --> G
-
- %% Helmsman Deployment Phase
- G --> H[Deploy Helmsman: Prerequisites
Monitoring, Istio, Logging]
- H --> I[Deploy Helmsman: External Dependencies
PostgreSQL containers, Keycloak, MinIO, Kafka]
-
- %% MOSIP Services
- I --> J[Deploy Helmsman: MOSIP Core Services]
- J --> K{Deploy Test Rigs?}
- K -->|Yes| L[Deploy Helmsman: Test Rigs
API Testing, UI Testing, DSL Testing]
- K -->|No| M[Verify Deployment]
- L --> M
-
- %% Final Verification
- M --> N[Access MOSIP Platform
Web UI, APIs, Admin Console]
- N --> O[Complete MOSIP Platform]
-
- %% Styling
- classDef prereq fill:#fff3e0,stroke:#ff8f00,stroke-width:2px
- classDef terraform fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
- classDef helmsman fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
- classDef success fill:#e8f5e8,stroke:#388e3c,stroke-width:2px
- classDef decision fill:#fce4ec,stroke:#c2185b,stroke-width:2px
-
- class A,B,C prereq
- class D,F,G terraform
- class H,I,J,L helmsman
- class M,N,O success
- class E,K decision
+ %% Prerequisites
+ A[Fork Repository] --> B[Configure Secrets]
+ B --> C[Select Cloud Provider
AWS/Azure/GCP/etc.]
+
+ %% Infrastructure Phase
+ C --> D[Run Terraform: base-infra
VPC, Networking, Jumpserver, WireGuard
One-time deployment]
+ D --> E{Deploy Observability?}
+ E -->|Yes| F[Run Terraform: observ-infra
Rancher UI, Keycloak, Monitoring
Can be destroyed/recreated]
+ E -->|No| G[Run Terraform: infra
Complete MOSIP Infrastructure Setup
Can be destroyed/recreated]
+ F --> G
+
+ %% Helmsman Deployment Phase
+ G --> H[Deploy Helmsman: Prerequisites
Monitoring, Istio, Logging]
+ H --> I[Deploy Helmsman: External Dependencies
PostgreSQL containers, Keycloak, MinIO, Kafka]
+
+ %% MOSIP Services
+ I --> J[Deploy Helmsman: MOSIP Core Services]
+ J --> K{Deploy Test Rigs?}
+ K -->|Yes| L[Deploy Helmsman: Test Rigs
API Testing, UI Testing, DSL Testing]
+ K -->|No| M[Verify Deployment]
+ L --> M
+
+ %% Final Verification
+ M --> N[Access MOSIP Platform
Web UI, APIs, Admin Console]
+ N --> O[Complete MOSIP Platform]
+
+ %% Styling
+ classDef prereq fill:#fff3e0,stroke:#ff8f00,stroke-width:2px
+ classDef terraform fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
+ classDef helmsman fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
+ classDef success fill:#e8f5e8,stroke:#388e3c,stroke-width:2px
+ classDef decision fill:#fce4ec,stroke:#c2185b,stroke-width:2px
+
+ class A,B,C prereq
+ class D,F,G terraform
+ class H,I,J,L helmsman
+ class M,N,O success
+ class E,K decision
```
> **Note:** Complete Terraform scripts are available only for **AWS**. For **Azure and GCP**, only placeholder structures are configured - community contributions are welcome to implement full functionality.
@@ -58,59 +74,67 @@ graph TB
```
terraform/
-├── base-infra/ # Foundation infrastructure (VPC, networking, security)
-├── observ-infra/ # Management cluster with Rancher UI (Optional)
-├── infra/ # MOSIP Kubernetes clusters
-├── modules/ # Reusable Terraform modules
-│ ├── aws/ # AWS-specific modules
-│ │ ├── aws-resource-creation/ # VPC, subnets, security groups, EC2 instances
-│ │ ├── nginx-setup/ # Load balancer and reverse proxy configuration
-│ │ ├── postgresql-setup/ # PostgreSQL database setup and configuration
-│ │ ├── rke2-cluster/ # RKE2 Kubernetes cluster provisioning
-│ │ ├── rancher-keycloak-setup/ # Identity management and SSO setup
-│ │ └── nfs-setup/ # Network File System configuration
-│ ├── azure/ # Azure-specific modules (placeholder - community contributions needed)
-│ └── gcp/ # GCP-specific modules (placeholder - community contributions needed)
-└── implementations/ # Cloud-specific implementations
- ├── aws/ # AWS deployment configurations
- ├── azure/ # Azure deployment configurations
- └── gcp/ # GCP deployment configurations
+├── base-infra/ # Foundation infrastructure (VPC, networking, security)
+├── observ-infra/ # Management cluster with Rancher UI (Optional)
+├── infra/ # MOSIP Kubernetes clusters
+├── modules/ # Reusable Terraform modules
+│ ├── aws/ # AWS-specific modules
+│ │ ├── aws-resource-creation/ # VPC, subnets, security groups, EC2 instances
+│ │ ├── nginx-setup/ # Load balancer and reverse proxy configuration
+│ │ ├── postgresql-setup/ # PostgreSQL database setup and configuration
+│ │ ├── rke2-cluster/ # RKE2 Kubernetes cluster provisioning
+│ │ ├── rancher-keycloak-setup/ # Identity management and SSO setup
+│ │ └── nfs-setup/ # Network File System configuration
+│ ├── azure/ # Azure-specific modules (placeholder - community contributions needed)
+│ └── gcp/ # GCP-specific modules (placeholder - community contributions needed)
+└── implementations/ # Cloud-specific implementations
+ ├── aws/ # AWS deployment configurations
+ ├── azure/ # Azure deployment configurations
+ └── gcp/ # GCP deployment configurations
```
### Application Layer (Helmsman)
```
Helmsman/
-├── dsf/ # Desired State Files for deployments
-│ ├── prereq-dsf.yaml # Prerequisites (monitoring, Istio, logging)
-│ ├── external-dsf.yaml # External dependencies (PostgreSQL, Keycloak, MinIO, ActiveMQ, Kafka, etc.)
-│ ├── mosip-dsf.yaml # MOSIP core services
-│ └── testrigs-dsf.yaml # Testing suite (API, DSL, UI test rigs)
-├── hooks/ # Scripts needed for automated deployment
-└── utils/ # Utilities and configurations
- ├── istio-addons/ # Service mesh components
- ├── logging/ # Logging stack configurations (optional)
- └── monitoring/ # Monitoring and alerting setup (optional)
+├── dsf/ # Desired State Files for deployments
+│ ├── prereq-dsf.yaml # Prerequisites (monitoring, Istio, logging)
+│ ├── external-dsf.yaml # External dependencies (PostgreSQL, Keycloak, MinIO, ActiveMQ, Kafka, etc.)
+│ ├── mosip-dsf.yaml # MOSIP core services
+│ └── testrigs-dsf.yaml # Testing suite (API, DSL, UI test rigs)
+├── hooks/ # Scripts needed for automated deployment
+└── utils/ # Utilities and configurations
+ ├── istio-addons/ # Service mesh components
+ ├── logging/ # Logging stack configurations (optional)
+ └── monitoring/ # Monitoring and alerting setup (optional)
```
### Automation Layer (GitHub Actions)
```
.github/workflows/
-├── terraform.yml # Infrastructure provisioning workflow
+├── terraform.yml # Infrastructure provisioning workflow
├── terraform-destroy.yml # Infrastructure cleanup workflow
├── helmsman_external.yml # External dependencies deployment
-├── helmsman_mosip.yml # MOSIP core services deployment
+├── helmsman_mosip.yml # MOSIP core services deployment
└── helmsman_testrigs.yml # Testing infrastructure deployment
```
## Prerequisites
+> **New to Cloud Deployment?** Check out our [Glossary](docs/GLOSSARY.md) for beginner-friendly explanations of all technical terms used in this guide!
+
> **Note:** As of now we support AWS based automated deployment. We are looking for community contribution around terraform modules and changes for other cloud service providers.
+> **Important for Beginners**: Start with AWS deployment only. Azure and GCP implementations are not yet complete. You'll need:
+>
+> - An AWS account ([Create one here](https://aws.amazon.com/free/))
+> - Basic understanding of cloud concepts ([See our Glossary](docs/GLOSSARY.md))
+> - GitHub account for running automated workflows
+
### Required Cloud Provider Account
-- **AWS account** with appropriate permissions (fully supported)
+- **AWS account** with appropriate permissions (fully supported) - [How to create AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/)
- Azure or GCP account (placeholder implementations - community contributions needed)
- Service account/access keys with infrastructure creation rights
@@ -129,20 +153,20 @@ Helmsman/
```json
{
- "Version": "2012-10-17",
- "Statement": [
- {
- "Effect": "Allow",
- "Action": [
- "ec2:*",
- "vpc:*",
- "route53:*",
- "iam:*",
- "s3:*"
- ],
- "Resource": "*"
- }
- ]
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "ec2:*",
+ "vpc:*",
+ "route53:*",
+ "iam:*",
+ "s3:*"
+ ],
+ "Resource": "*"
+ }
+ ]
}
```
@@ -176,79 +200,137 @@ Helmsman/
### Required Secrets for Rapid Deployment
+> **Need help generating secrets?** See our comprehensive [Secret Generation Guide](docs/SECRET_GENERATION_GUIDE.md) for step-by-step instructions with screenshots and examples!
+
> **Secret Configuration Types:**
>
> - **Repository Secrets**: Global secrets shared across all environments (set once in GitHub repo settings)
+> - Think of these as "master keys" that work everywhere
+> - Examples: AWS credentials, SSH keys
> - **Environment Secrets**: Environment-specific secrets (configured per deployment environment)
+> - Think of these as "room keys" for specific environments
+> - Examples: KUBECONFIG, WireGuard configs (different for each environment)
+>
+> **Still confused?** Read the [Secret Generation Guide](docs/SECRET_GENERATION_GUIDE.md) - it explains everything in plain language!
#### Terraform Secrets
+> **How to generate each secret**: See [Secret Generation Guide](docs/SECRET_GENERATION_GUIDE.md) for detailed instructions
+
**Repository Secrets** (configured in GitHub repository settings):
```yaml
# GPG Encryption (for local backend)
-GPG_PASSPHRASE: "your-gpg-passphrase" # Required for GPG encryption
+GPG_PASSPHRASE: "your-gpg-passphrase"
+# What it's for: Encrypts Terraform state files to keep them secure
+# How to generate: Create a strong 16+ character password
+# Details: https://docs.github.com/en/actions/security-guides/encrypted-secrets
+# Guide: See "GPG Passphrase" section in Secret Generation Guide
# Cloud Provider Credentials
-AWS_ACCESS_KEY_ID: "AKIA..." # AWS Access Key ID
-AWS_SECRET_ACCESS_KEY: "..." # AWS Secret Access Key
+AWS_ACCESS_KEY_ID: "AKIA..."
+# What it's for: Allows Terraform to create AWS resources
+# How to get: AWS Console → IAM → Users → Security credentials → Create access key
+# Details: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
+# Guide: See "AWS Credentials" section in Secret Generation Guide
+
+AWS_SECRET_ACCESS_KEY: "..."
+# What it's for: Secret key that pairs with access key ID (like a password)
+# IMPORTANT: Keep this SECRET! Never commit to Git or share publicly
# SSH Private Key (must match ssh_key_name in tfvars)
-YOUR_SSH_KEY_NAME: | # Replace YOUR_SSH_KEY_NAME with actual ssh_key_name value
- -----BEGIN RSA PRIVATE KEY-----
- your-ssh-private-key-content
- -----END RSA PRIVATE KEY-----
+YOUR_SSH_KEY_NAME: |
+# Replace YOUR_SSH_KEY_NAME with actual ssh_key_name value from your tfvars
+# What it's for: Allows secure access to EC2 instances
+# How to generate: ssh-keygen -t rsa -b 4096 -C "your-email@example.com"
+# Details: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
+# Guide: See "SSH Keys" section in Secret Generation Guide
+ -----BEGIN RSA PRIVATE KEY-----
+ your-ssh-private-key-content
+ -----END RSA PRIVATE KEY-----
```
+**Quick Secret Generation Checklist:**
+
+- [ ] GPG Passphrase created (16+ characters)
+- [ ] AWS Access Key ID obtained from IAM
+- [ ] AWS Secret Access Key saved securely
+- [ ] SSH key pair generated (public + private)
+- [ ] SSH public key uploaded to AWS EC2 Key Pairs
+- [ ] SSH private key added to GitHub secrets
+- [ ] All secret names match exactly (case-sensitive!)
+
+**Need step-by-step help?** [Secret Generation Guide](docs/SECRET_GENERATION_GUIDE.md)
+
**Environment Secrets** (configured per deployment environment):
```yaml
# WireGuard VPN (optional - for infrastructure access)
TF_WG_CONFIG: |
- [Interface]
- PrivateKey = terraform-private-key
- Address = 10.0.1.2/24
-
- [Peer]
- PublicKey = server-public-key
- Endpoint = your-server:51820
- AllowedIPs = 10.0.0.0/16
+ [Interface]
+ PrivateKey = terraform-private-key
+ Address = 10.0.1.2/24
+
+ [Peer]
+ PublicKey = server-public-key
+ Endpoint = your-server:51820
+ AllowedIPs = 10.0.0.0/16
# Notifications (optional)
-SLACK_WEBHOOK_URL: "https://hooks.slack.com/services/..." # Slack notifications
+SLACK_WEBHOOK_URL: "https://hooks.slack.com/services/..." # Slack notifications
```
#### Helmsman Secrets
**Environment Secrets** (configured per deployment environment):
+> **Important**: These are generated AFTER infrastructure deployment, not before!
+
```yaml
# Kubernetes Access
-KUBECONFIG: "apiVersion: v1..." # Complete kubeconfig file content
+KUBECONFIG: "apiVersion: v1..."
+# What it's for: Allows Helmsman to deploy applications to your Kubernetes cluster
+# When available: After Terraform infra deployment completes
+# Where to find: terraform/implementations/aws/infra/kubeconfig_
+# Guide: See "Kubernetes Config" section in Secret Generation Guide
# WireGuard VPN Access (for cluster access)
CLUSTER_WIREGUARD_WG0: |
- [Interface]
- PrivateKey = helmsman-wg0-private-key
- Address = 10.0.0.2/24
-
- [Peer]
- PublicKey = cluster-public-key
- Endpoint = cluster-server:51820
- AllowedIPs = 10.0.0.0/16
+# What it's for: Secure VPN connection to access private Kubernetes cluster
+# When available: After base-infra deployment and WireGuard setup
+# How to get: Follow WireGuard setup guide
+# Details: See terraform/base-infra/WIREGUARD_SETUP.md
+# Guide: See "WireGuard VPN" section in Secret Generation Guide
+ [Interface]
+ PrivateKey = helmsman-wg0-private-key
+ Address = 10.0.0.2/24
+
+ [Peer]
+ PublicKey = cluster-public-key
+ Endpoint = cluster-server:51820
+ AllowedIPs = 10.0.0.0/16
# Secondary WireGuard Config (optional)
CLUSTER_WIREGUARD_WG1: |
- [Interface]
- PrivateKey = helmsman-wg1-private-key
- Address = 10.0.2.2/24
-
- [Peer]
- PublicKey = cluster-public-key-2
- Endpoint = cluster-server-2:51820
- AllowedIPs = 10.0.0.0/16
+# Optional: Additional WireGuard peer for redundancy
+ [Interface]
+ PrivateKey = helmsman-wg1-private-key
+ Address = 10.0.2.2/24
+
+ [Peer]
+ PublicKey = cluster-public-key-2
+ Endpoint = cluster-server-2:51820
+ AllowedIPs = 10.0.0.0/16
```
+**Deployment Order for Secrets:**
+
+1. **Before starting**: Add Repository Secrets (GPG, AWS, SSH)
+2. **After base-infra**: Add TF_WG_CONFIG environment secret
+3. **After main infra**: Add KUBECONFIG, CLUSTER_WIREGUARD_WG0/WG1 environment secrets
+
+**Need step-by-step help?** [Secret Generation Guide](docs/SECRET_GENERATION_GUIDE.md)
+
> **Note**: PostgreSQL secrets are no longer required! PostgreSQL setup is handled automatically by Terraform modules and Ansible scripts based on your `enable_postgresql_setup` configuration.
## Quick Start Guide
@@ -271,400 +353,681 @@ Navigate to your repository → **Settings** → **Secrets and variables** → *
Add the required secrets as follows:
- **Repository Secrets** (Settings > Secrets and variables > Actions > Repository secrets):
-
- - `GPG_PASSPHRASE`
- - `AWS_ACCESS_KEY_ID`
- - `AWS_SECRET_ACCESS_KEY`
- - `YOUR_SSH_KEY_NAME` (replace with actual ssh_key_name value from tfvars, e.g., `mosip-aws`)
+- `GPG_PASSPHRASE`
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+- `YOUR_SSH_KEY_NAME` (replace with actual ssh_key_name value from tfvars, e.g., `mosip-aws`)
- **Environment Secrets** (Settings > Secrets and variables > Actions > Environment secrets):
-
- - All other secrets mentioned in the Prerequisites section above (KUBECONFIG, WireGuard configs, etc.)
+- All other secrets mentioned in the Prerequisites section above (KUBECONFIG, WireGuard configs, etc.)
### 3. Terraform Infrastructure Deployment
+> **New to Terraform workflows?** Check our [Workflow Guide](docs/WORKFLOW_GUIDE.md) for visual step-by-step instructions on navigating GitHub Actions!
+
+#### Understanding Terraform Apply vs Dry Run
+
+Before running any Terraform workflow, understand these modes:
+
+| Mode | What It Does | When to Use | Visual |
+| ----------------------------------------- | ---------------------------------------------- | ------------------------------------------ | ------------------ |
+| **Dry Run** (checkbox unchecked ☐) | Shows what WOULD happen without making changes | Testing configurations, previewing changes | ☐ Terraform apply |
+| **Apply** (checkbox checked ✅) | Actually creates/modifies infrastructure | Real deployments, making actual changes | ✅ Terraform apply |
+
+**Tip**: Always do a dry run first to preview changes, then run with apply checked to actually deploy!
+
#### Step 3a: Base Infrastructure
+**What this creates:**
+
+- Virtual Private Cloud (VPC) - Your private network in AWS
+- Subnets - Subdivisions of your network
+- Jump Server - Secure gateway to access other servers
+- WireGuard VPN - Encrypted connection to your infrastructure
+- Security Groups - Firewall rules for network security
+
+**Time required:** 10-15 minutes
+
1. **Update terraform variables:**
- ```bash
- # Edit terraform/base-infra/aws/terraform.tfvars (or azure/gcp)
- ```
+```bash
+ # Edit terraform/base-infra/aws/terraform.tfvars (or azure/gcp)
+```
+
2. **Configure base-infra variables:**
- ```hcl
- # Example for AWS
- region = "us-west-2"
- availability_zones = ["us-west-2a", "us-west-2b"]
- vpc_cidr = "10.0.0.0/16"
- environment = "production"
- ```
+```hcl
+ # Example for AWS
+ region = "us-west-2" # Choose AWS region close to your users
+ availability_zones = ["us-west-2a", "us-west-2b"] # Multiple zones for high availability
+ vpc_cidr = "10.0.0.0/16" # Private IP address range for your network
+ environment = "production" # Name your environment
+```
+
3. **Run base-infra via GitHub Actions:**
- - Go to **Actions** → **Terraform Base Infrastructure**
- - Click **Run workflow**
- - **Configure workflow parameters:**
- - **Branch**: Select your deployment branch (e.g., `release-0.1.0`)
- - **Cloud Provider**: Select `aws` (Azure/GCP are placeholder implementations)
- - **Component**: Select `base-infra` (creates VPC, networking, jump server, WireGuard)
- - **Backend**: Choose backend configuration:
- - `local` - GPG-encrypted local state (recommended for development)
- - `s3` - Remote S3 backend (recommended for production)
- - **Action**: Select `apply` to deploy infrastructure
-
- **Component Details:**
-
- - **VPC & Networking**: Creates secure network foundation
- - **Jump Server**: Bastion host for secure access
- - **WireGuard VPN**: Encrypted private network access
- - **Security Groups**: Network access controls
- - **Route Tables**: Network traffic routing
+> **Detailed Navigation Guide**: See [Workflow Guide - Terraform Workflows](docs/WORKFLOW_GUIDE.md#workflow-1-base-infrastructure) for step-by-step screenshots
+
+- Go to **Actions** → **Terraform Base Infrastructure**
+- **Can't find it?** Look in the left sidebar under "All workflows"
+- Click **Run workflow** (green button on the right)
+- **Configure workflow parameters:**
+- **Branch**: Select your deployment branch (e.g., `release-0.1.0`)
+- **What's this?** The branch of code to use for deployment
+- **Cloud Provider**: Select `aws` (Azure/GCP are placeholder implementations)
+- **Important**: Only `aws` is fully functional
+- **Component**: Select `base-infra` (creates VPC, networking, jump server, WireGuard)
+- **What's this?** Which part of infrastructure to build
+- **Backend**: Choose backend configuration:
+- `local` - GPG-encrypted local state (recommended for development)
+- Stores state in your GitHub repository (encrypted)
+- `s3` - Remote S3 backend (recommended for production)
+- Stores state in AWS S3 bucket (centralized)
+- **SSH_PRIVATE_KEY**: GitHub secret name containing SSH private key for instance access
+- Must match the `ssh_key_name` in your terraform.tfvars
+- **Terraform apply**:
+- ☐ **Unchecked** - Dry run (preview only, no changes made)
+- ✅ **Checked** - Apply (actually creates infrastructure)
+- **First time?** Uncheck for dry run, then run again with checked
+
+ **What You Should See:**
+
+- ✅ Workflow running (yellow circle icon)
+- ✅ Steps completing one by one
+- ✅ Green checkmark when complete
+- ✅ Infrastructure created in AWS
+
+ **Component Details:**
+
+- **VPC & Networking**: Creates secure network foundation
+- **Jump Server**: Bastion host for secure access
+- **WireGuard VPN**: Encrypted private network access
+- **Security Groups**: Network access controls
+- **Route Tables**: Network traffic routing
+
+ **Need more help?** [Workflow Guide](docs/WORKFLOW_GUIDE.md)
#### GitHub Actions Workflow Parameters Reference
+> **Visual Guide:** See [Workflow Guide - Workflow Parameters Explained](docs/WORKFLOW_GUIDE.md#workflow-parameters-explained) for detailed explanations with examples!
+
**Common Parameters for All Terraform Workflows:**
- **`CLOUD_PROVIDER`**: `aws` | `azure` | `gcp` (cloud platform selection)
+- **Choose**: `aws` (only fully functional option)
+- Azure/GCP are placeholder implementations
- **`TERRAFORM_COMPONENT`**: `base-infra` | `infra` | `observ-infra` (infrastructure component)
+- **base-infra**: VPC, networking, jump server (deploy FIRST)
+- **observ-infra**: Rancher management cluster (optional)
+- **infra**: MOSIP Kubernetes cluster (main deployment)
- **`SSH_PRIVATE_KEY`**: GitHub secret name containing SSH private key for instance access
-- **`TERRAFORM_APPLY`**: `true`/`false` (apply changes or plan-only mode)
+- Must match the `ssh_key_name` in your terraform.tfvars
+- [How to create SSH keys](docs/SECRET_GENERATION_GUIDE.md#1-ssh-keys)
+- **`TERRAFORM_APPLY`**: Checkbox ☐ or ✅ (apply changes or plan-only mode)
+- ☐ **Unchecked** = Dry run (preview only, **no infrastructure changes**)
+- ✅ **Checked** = Apply (actually creates infrastructure, **real changes**)
+- **Visual Explanation:**
+
+```
+ ☐ Unchecked → Terraform Plan Only
+ → Shows: "Will create 25 resources"
+ → Does: Nothing (preview only)
+ → AWS: No changes made
+
+ ✅ Checked → Terraform Apply
+ → Shows: "Creating resources..."
+ → Does: Creates actual infrastructure
+ → AWS: Servers, networks, databases created
+ → Cost: Billing starts
+```
+
+- **Relationship with Rancher Import:**
+
+```
+ If Terraform Apply = ✅ AND Rancher Import = True
+ → Infrastructure deployed AND cluster imported to Rancher UI
+
+ If Terraform Apply = ✅ AND Rancher Import = False
+ → Infrastructure deployed but cluster runs standalone
+
+ If Terraform Apply = ☐ (unchecked - dry run)
+ → Nothing happens, just shows plan
+ → Rancher Import setting is ignored
+```
**Backend Configuration Options:**
- **`local`**: GPG-encrypted local state storage (recommended for development and small teams)
-
- - State files stored in repository with GPG encryption
- - No external dependencies required
- - Automatic encryption/decryption via GitHub Actions
+- State files stored in repository with GPG encryption
+- No external dependencies required
+- Automatic encryption/decryption via GitHub Actions
+- **Best for**: Development, testing, small teams
+- **Requires**: GPG_PASSPHRASE secret
- **`s3`**: Remote S3 backend storage (recommended for production and large teams)
-
- - Centralized state storage in AWS S3
- - DynamoDB state locking support
- - Cross-team collaboration friendly
+- Centralized state storage in AWS S3
+- DynamoDB state locking support
+- Cross-team collaboration friendly
+- **Best for**: Production, large teams, multiple environments
+- **Requires**: S3 bucket and DynamoDB table setup
#### Step 3b: WireGuard VPN Setup (Required for Private Network Access)
+> **What is WireGuard?** A modern VPN that creates a secure, encrypted "tunnel" to access your private infrastructure. Think of it like a secure phone line that only you can use to call your servers! [Learn more](docs/GLOSSARY.md#wireguard)
+
**After base infrastructure deployment**, set up WireGuard VPN for secure access to private infrastructure:
-> **📋 Detailed Setup Guide:** [WireGuard Setup Documentation](terraform/base-infra/WIREGUARD_SETUP.md)
+> **Detailed Setup Guide:** [WireGuard Setup Documentation](terraform/base-infra/WIREGUARD_SETUP.md)
+>
+> **Secret Generation:** [How to generate WireGuard configs](docs/SECRET_GENERATION_GUIDE.md#4-wireguard-vpn-configuration)
**Quick Setup Overview:**
1. **SSH to Jump Server:** Access the deployed jump server
+
+- Use the SSH key you created earlier
+- Jump server IP is in Terraform outputs
+
2. **Configure Peers:** Assign and customize WireGuard peer configurations
- - Create **peer1** configuration for Terraform access
- - Create **peer2** configuration for Helmsman access
+
+- Create **peer1** configuration for Terraform access (your computer → infrastructure)
+- Create **peer2** configuration for Helmsman access (GitHub Actions → cluster)
+- Think of peers as "authorized devices" that can connect
+
3. **Install Client:** Set up WireGuard client on your PC/Mac
+
+- **Windows**: [Download installer](https://www.wireguard.com/install/)
+- **Mac**: Install from App Store or use `brew install wireguard-tools`
+- **Linux**: `sudo apt install wireguard` (Ubuntu/Debian)
+
4. **Update Environment Secrets:** Add WireGuard configurations to your GitHub environment secrets:
- - `TF_WG_CONFIG` - For Terraform infrastructure deployments
- - `CLUSTER_WIREGUARD_WG0` - For Helmsman cluster access (peer1)
- - `CLUSTER_WIREGUARD_WG1` - For Helmsman cluster access (peer2)
+
+- `TF_WG_CONFIG` - For Terraform infrastructure deployments
+- `CLUSTER_WIREGUARD_WG0` - For Helmsman cluster access (peer1)
+- `CLUSTER_WIREGUARD_WG1` - For Helmsman cluster access (peer2, optional)
+- [How to add secrets to GitHub](docs/SECRET_GENERATION_GUIDE.md#7-how-to-add-secrets-to-github)
+
5. **Verify Connection:** Test private IP connectivity
+```bash
+ # Activate WireGuard tunnel
+ # Then test connectivity
+ ping 10.0.0.1 # Should work if VPN is connected
+```
+
**Why WireGuard is Required:**
-- **Private Network Access:** Connect to Kubernetes cluster via private IPs
-- **Enhanced Security:** Encrypted VPN tunnel for all infrastructure access
+- **Private Network Access:** Connect to Kubernetes cluster via private IPs (not exposed to internet)
+- **Enhanced Security:** Encrypted VPN tunnel for all infrastructure access (256-bit encryption)
- **Terraform Integration:** Required for subsequent infrastructure deployments
- **Helmsman Connectivity:** Enables secure cluster access for service deployments
-> **⚠️ Important:** Complete WireGuard setup and configure `TF_WG_CONFIG` environment secret before proceeding to MOSIP infrastructure deployment.
+**Visual Explanation:**
+
+```
+Without WireGuard:
+Your Computer → ❌ Can't reach private servers
+
+With WireGuard:
+Your Computer → VPN Tunnel → ✅ Access private servers securely
+```
+
+> **Important:** Complete WireGuard setup and configure `TF_WG_CONFIG` environment secret before proceeding to MOSIP infrastructure deployment.
+>
+> **Need help?** Check the [detailed WireGuard guide](terraform/base-infra/WIREGUARD_SETUP.md) with screenshots!
#### Step 3c: MOSIP Infrastructure
1. **Update infra variables in `terraform/implementations/aws/infra/aws.tfvars`:**
- Complete configuration example with detailed explanations:
-
- ```hcl
- # Environment name (infra component)
- cluster_name = "soil38"
- # MOSIP's domain (ex: sandbox.xyz.net)
- cluster_env_domain = "soil38.mosip.net"
- # Email-ID will be used by certbot to notify SSL certificate expiry via email
- mosip_email_id = "chandra.mishra@technoforte.co.in"
- # SSH login key name for AWS node instances (ex: my-ssh-key)
- ssh_key_name = "mosip-aws"
- # The AWS region for resource creation
- aws_provider_region = "ap-south-1"
-
- # Specific availability zones for VM deployment (optional)
- # If empty, uses all available AZs in the region
- # Example: ["ap-south-1a", "ap-south-1b"] for specific AZs
- # Example: [] for all available AZs in the region
- specific_availability_zones = []
-
- # The instance type for Kubernetes nodes (control plane, worker, etcd)
- k8s_instance_type = "t3a.2xlarge"
- # The instance type for Nginx server (load balancer)
- nginx_instance_type = "t3a.2xlarge"
- # The Route 53 hosted zone ID
- zone_id = "Z090954828SJIEL6P5406"
-
- ## UBUNTU 24.04
- # The Amazon Machine Image ID for the instances
- ami = "ami-0ad21ae1d0696ad58"
-
- # Repo K8S-INFRA URL
- k8s_infra_repo_url = "https://github.com/mosip/k8s-infra.git"
- # Repo K8S-INFRA branch
- k8s_infra_branch = "MOSIP-42914"
- # NGINX Node's Root volume size
- nginx_node_root_volume_size = 24
- # NGINX node's EBS volume size
- nginx_node_ebs_volume_size = 300
- # NGINX node's second EBS volume size (optional - set to 0 to disable)
- nginx_node_ebs_volume_size_2 = 200 # Enable second EBS volume for PostgreSQL testing
- # Kubernetes nodes Root volume size
- k8s_instance_root_volume_size = 64
-
- # Control-plane, ETCD, Worker
- k8s_control_plane_node_count = 3
- # ETCD, Worker
- k8s_etcd_node_count = 3
- # Worker
- k8s_worker_node_count = 2
-
- # RKE2 Version Configuration
- rke2_version = "v1.28.9+rke2r1"
-
- # Rancher Import Configuration
- enable_rancher_import = false
-
- # Security group CIDRs
- network_cidr = "10.0.0.0/8" # Use your actual VPC CIDR
- WIREGUARD_CIDR = "10.0.0.0/8" # Use your actual WireGuard VPN CIDR
-
- # Rancher Import URL
- rancher_import_url = "\"kubectl apply -f https://rancher.mosip.net/v3/import/dzshvnb6br7qtf267zsrr9xsw6tnb2vt4x68g79r2wzsnfgvkjq2jk_c-m-b5249w76.yaml\""
- # DNS Records to map
- subdomain_public = ["resident", "prereg", "esignet", "healthservices", "signup"]
- subdomain_internal = ["admin", "iam", "activemq", "kafka", "kibana", "postgres", "smtp", "pmp", "minio", "regclient", "compliance"]
-
- # PostgreSQL Configuration (used when second EBS volume is enabled)
- enable_postgresql_setup = true # Enable PostgreSQL setup for main infra
- postgresql_version = "15"
- storage_device = "/dev/nvme2n1"
- mount_point = "/srv/postgres"
- postgresql_port = "5433"
-
- # MOSIP Infrastructure Repository Configuration
- mosip_infra_repo_url = "https://github.com/mosip/mosip-infra.git"
- mosip_infra_branch = "develop"
-
- # VPC Configuration - Existing VPC to use (discovered by Name tag)
- vpc_name = "mosip-boxes"
- ```
+ Complete configuration example with detailed explanations:
- **Key Configuration Variables Explained:**
-
- | Variable | Description | Example Value |
- | -------------------------------- | ------------------------------------------ | ------------------------------------------- |
- | `cluster_name` | Unique identifier for your MOSIP cluster | `"soil38"` |
- | `cluster_env_domain` | Domain name for MOSIP services access | `"soil38.mosip.net"` |
- | `mosip_email_id` | Email for SSL certificate notifications | `"admin@example.com"` |
- | `ssh_key_name` | AWS EC2 key pair name for SSH access | `"mosip-aws"` |
- | `aws_provider_region` | AWS region for resource deployment | `"ap-south-1"` |
- | `zone_id` | Route 53 hosted zone ID for DNS management | `"Z090954828SJIEL6P5406"` |
- | `k8s_instance_type` | EC2 instance type for Kubernetes nodes | `"t3a.2xlarge"` |
- | `nginx_instance_type` | EC2 instance type for load balancer | `"t3a.2xlarge"` |
- | `ami` | Amazon Machine Image ID (Ubuntu 24.04) | `"ami-0ad21ae1d0696ad58"` |
- | `enable_postgresql_setup` | External PostgreSQL setup via Terraform | `true` (external) / `false` (container) |
- | `nginx_node_ebs_volume_size_2` | EBS volume size for PostgreSQL data (GB) | `200` |
- | `postgresql_version` | PostgreSQL version to install | `"15"` |
- | `postgresql_port` | PostgreSQL service port | `"5433"` |
- | `vpc_name` | Existing VPC name tag to use | `"mosip-boxes"` |
-
-
- > **Important Notes:**
- >
- > - Ensure `cluster_name` and `cluster_env_domain` match values used in Helmsman DSF files
- > - Set `enable_postgresql_setup = true` for production deployments with external PostgreSQL
- > - Set `enable_postgresql_setup = false` for development deployments with containerized PostgreSQL
- > - The `nginx_node_ebs_volume_size_2` is required when `enable_postgresql_setup = true`
- > - **SSH Key Configuration**: The `ssh_key_name` value must match the repository secret name containing your SSH private key (e.g., if `ssh_key_name = "mosip-aws"`, create repository secret named `mosip-aws` with your SSH private key content)
- >
-2. **Run main infra via GitHub Actions:**
+```hcl
+ # Environment name (infra component)
+ cluster_name = "soil38"
+ # MOSIP's domain (ex: sandbox.xyz.net)
+ cluster_env_domain = "soil38.mosip.net"
+ # Email-ID will be used by certbot to notify SSL certificate expiry via email
+ mosip_email_id = "chandra.mishra@technoforte.co.in"
+ # SSH login key name for AWS node instances (ex: my-ssh-key)
+ ssh_key_name = "mosip-aws"
+ # The AWS region for resource creation
+ aws_provider_region = "ap-south-1"
+
+ # Specific availability zones for VM deployment (optional)
+ # If empty, uses all available AZs in the region
+ # Example: ["ap-south-1a", "ap-south-1b"] for specific AZs
+ # Example: [] for all available AZs in the region
+ specific_availability_zones = []
+
+ # The instance type for Kubernetes nodes (control plane, worker, etcd)
+ k8s_instance_type = "t3a.2xlarge"
+ # The instance type for Nginx server (load balancer)
+ nginx_instance_type = "t3a.2xlarge"
+ # The Route 53 hosted zone ID
+ zone_id = "Z090954828SJIEL6P5406"
+
+ ## UBUNTU 24.04
+ # The Amazon Machine Image ID for the instances
+ ami = "ami-0ad21ae1d0696ad58"
+
+ # Repo K8S-INFRA URL
+ k8s_infra_repo_url = "https://github.com/mosip/k8s-infra.git"
+ # Repo K8S-INFRA branch
+ k8s_infra_branch = "MOSIP-42914"
+ # NGINX Node's Root volume size
+ nginx_node_root_volume_size = 24
+ # NGINX node's EBS volume size
+ nginx_node_ebs_volume_size = 300
+ # NGINX node's second EBS volume size (optional - set to 0 to disable)
+ nginx_node_ebs_volume_size_2 = 200 # Enable second EBS volume for PostgreSQL testing
+ # Kubernetes nodes Root volume size
+ k8s_instance_root_volume_size = 64
+
+ # Control-plane, ETCD, Worker
+ k8s_control_plane_node_count = 3
+ # ETCD, Worker
+ k8s_etcd_node_count = 3
+ # Worker
+ k8s_worker_node_count = 2
+
+ # RKE2 Version Configuration
+ rke2_version = "v1.28.9+rke2r1"
+
+ # Rancher Import Configuration
+ enable_rancher_import = false
+
+ # Security group CIDRs
+ network_cidr = "10.0.0.0/8" # Use your actual VPC CIDR
+ WIREGUARD_CIDR = "10.0.0.0/8" # Use your actual WireGuard VPN CIDR
+
+ # Rancher Import URL
+ rancher_import_url = "\"kubectl apply -f https://rancher.mosip.net/v3/import/dzshvnb6br7qtf267zsrr9xsw6tnb2vt4x68g79r2wzsnfgvkjq2jk_c-m-b5249w76.yaml\""
+ # DNS Records to map
+ subdomain_public = ["resident", "prereg", "esignet", "healthservices", "signup"]
+ subdomain_internal = ["admin", "iam", "activemq", "kafka", "kibana", "postgres", "smtp", "pmp", "minio", "regclient", "compliance"]
+
+ # PostgreSQL Configuration (used when second EBS volume is enabled)
+ enable_postgresql_setup = true # Enable PostgreSQL setup for main infra
+ postgresql_version = "15"
+ storage_device = "/dev/nvme2n1"
+ mount_point = "/srv/postgres"
+ postgresql_port = "5433"
+
+ # MOSIP Infrastructure Repository Configuration
+ mosip_infra_repo_url = "https://github.com/mosip/mosip-infra.git"
+ mosip_infra_branch = "develop"
+
+ # VPC Configuration - Existing VPC to use (discovered by Name tag)
+ vpc_name = "mosip-boxes"
+```
+
+ **Key Configuration Variables Explained:**
+
+| Variable | Description | Example Value |
+| -------------------------------- | ------------------------------------------ | ------------------------------------------- |
+| `cluster_name` | Unique identifier for your MOSIP cluster | `"soil38"` |
+| `cluster_env_domain` | Domain name for MOSIP services access | `"soil38.mosip.net"` |
+| `mosip_email_id` | Email for SSL certificate notifications | `"admin@example.com"` |
+| `ssh_key_name` | AWS EC2 key pair name for SSH access | `"mosip-aws"` |
+| `aws_provider_region` | AWS region for resource deployment | `"ap-south-1"` |
+| `zone_id` | Route 53 hosted zone ID for DNS management | `"Z090954828SJIEL6P5406"` |
+| `k8s_instance_type` | EC2 instance type for Kubernetes nodes | `"t3a.2xlarge"` |
+| `nginx_instance_type` | EC2 instance type for load balancer | `"t3a.2xlarge"` |
+| `ami` | Amazon Machine Image ID (Ubuntu 24.04) | `"ami-0ad21ae1d0696ad58"` |
+| `enable_postgresql_setup` | External PostgreSQL setup via Terraform | `true` (external) / `false` (container) |
+| `nginx_node_ebs_volume_size_2` | EBS volume size for PostgreSQL data (GB) | `200` |
+| `postgresql_version` | PostgreSQL version to install | `"15"` |
+| `postgresql_port` | PostgreSQL service port | `"5433"` |
+| `vpc_name` | Existing VPC name tag to use | `"mosip-boxes"` |
+
+> **Important Notes:**
+>
+> - Ensure `cluster_name` and `cluster_env_domain` match values used in Helmsman DSF files
+> - Set `enable_postgresql_setup = true` for production deployments with external PostgreSQL
+> - Set `enable_postgresql_setup = false` for development deployments with containerized PostgreSQL
+> - The `nginx_node_ebs_volume_size_2` is required when `enable_postgresql_setup = true`
+> - **SSH Key Configuration**: The `ssh_key_name` value must match the repository secret name containing your SSH private key (e.g., if `ssh_key_name = "mosip-aws"`, create repository secret named `mosip-aws` with your SSH private key content)
+
+**Component Details:**
+
+- **infra**: Creates MOSIP Kubernetes cluster, PostgreSQL (if enabled), networking, and application infrastructure
+
+**PostgreSQL Configuration in `aws.tfvars`:**
+
+```hcl
+# PostgreSQL Configuration (used when second EBS volume is enabled)
+enable_postgresql_setup = true # Enable PostgreSQL setup for main infra
+postgresql_version = "15"
+storage_device = "/dev/nvme2n1"
+mount_point = "/srv/postgres"
+postgresql_port = "5433"
+
+# NGINX node's second EBS volume size (required for PostgreSQL)
+nginx_node_ebs_volume_size_2 = 200 # Enable second EBS volume for PostgreSQL
+```
+
+**If `enable_postgresql_setup = true`, Terraform will automatically:**
+
+- Provision dedicated EBS volume for PostgreSQL on nginx node
+- Install and configure PostgreSQL 15 via Ansible playbooks
+- Setup security configurations and user access controls
+- Configure backup and recovery mechanisms
+- Make PostgreSQL ready for MOSIP services connectivity
+- No manual PostgreSQL secret management required!
+
+#### Rancher Import Configuration (Optional)
+
+If you have deployed **observ-infra** (Rancher management cluster), you can import your main infra cluster into Rancher for centralized monitoring and management.
+
+**Step 1: Deploy observ-infra (if not already done)**
+
+```bash
+# Deploy Rancher management cluster first
+Actions → Terraform Infrastructure → Run workflow
+Parameters:
+├─ Component: observ-infra
+├─ Cloud Provider: aws
+└─ Backend: local
+```
+
+Wait for observ-infra deployment to complete (~15-20 minutes).
+
+**Step 2: Generate Rancher Import URL**
+
+1. **Access Rancher UI:**
- - Go to **Actions** → **Terraform Infrastructure**
- - Click **Run workflow**
- - **Configure workflow parameters:**
- - **Branch**: Select your deployment branch (e.g., `release-0.1.0`)
- - **Cloud Provider**: Select `aws` (Azure/GCP are placeholder implementations)
- - **Component**: Select `infra` (MOSIP application infrastructure)
- - **Backend**: Choose backend configuration:
- - `local` - GPG-encrypted local state (recommended for development)
- - `s3` - Remote S3 backend (recommended for production)
- - **Action**: Select `apply` to deploy infrastructure
-
- **Component Details:**
-
- - **infra**: Creates MOSIP Kubernetes cluster, PostgreSQL (if enabled), networking, and application infrastructure
-
- **PostgreSQL Configuration in `aws.tfvars`:**
-
- ```hcl
- # PostgreSQL Configuration (used when second EBS volume is enabled)
- enable_postgresql_setup = true # Enable PostgreSQL setup for main infra
- postgresql_version = "15"
- storage_device = "/dev/nvme2n1"
- mount_point = "/srv/postgres"
- postgresql_port = "5433"
-
- # NGINX node's second EBS volume size (required for PostgreSQL)
- nginx_node_ebs_volume_size_2 = 200 # Enable second EBS volume for PostgreSQL
```
+ https://rancher.your-domain.net
+ ```
+
+ Login with credentials from observ-infra deployment.
+2. **Navigate to Cluster Import:**
- **If `enable_postgresql_setup = true`, Terraform will automatically:**
+ ```
+ Rancher UI → Cluster Management → Import Existing
+ ```
+3. **Select Import Method:**
- - Provision dedicated EBS volume for PostgreSQL on nginx node
- - Install and configure PostgreSQL 15 via Ansible playbooks
- - Setup security configurations and user access controls
- - Configure backup and recovery mechanisms
- - Make PostgreSQL ready for MOSIP services connectivity
- - No manual PostgreSQL secret management required!
+ ```
+ Click: "Import any Kubernetes cluster" → Generic
+ ```
+4. **Configure Cluster Import:**
-### 4. Helmsman Deployment
+ ```
+ Cluster Name: soil38 (use your cluster_name from aws.tfvars)
-#### Step 4a: Update DSF Configuration Files
+ Click: "Create"
+ ```
+5. **Copy the kubectl apply command:**
-1. **Clone the MOSIP infra repository and navigate to Helmsman directory:**
+ Rancher will generate a command like:
```bash
- git clone https://github.com/mosip/infra.git
- cd infra/Helmsman
+ kubectl apply -f https://rancher.mosip.net/v3/import/dzshvnb6br7qtf267zsrr9xsw6tnb2vt4x68g79r2wzsnfgvkjq2jk_c-m-b5249w76.yaml
```
+
+**Step 3: Update aws.tfvars**
+
+Add the generated command to your `aws.tfvars` file:
+
+```hcl
+# Enable Rancher import
+enable_rancher_import = true
+
+# Paste the kubectl apply command from Rancher UI
+# IMPORTANT: Use proper escaping - wrap the entire command in quotes with escaped inner quotes
+rancher_import_url = "\"kubectl apply -f https://rancher.mosip.net/v3/import/dzshvnb6br7qtf267zsrr9xsw6tnb2vt4x68g79r2wzsnfgvkjq2jk_c-m-b5249w76.yaml\""
+```
+
+**⚠️ Critical: Proper String Escaping**
+
+The `rancher_import_url` requires special escaping to avoid Terraform indentation errors:
+
+✅ **Correct format:**
+
+```hcl
+rancher_import_url = "\"kubectl apply -f https://rancher.example.com/v3/import/TOKEN.yaml\""
+```
+
+❌ **Wrong format (will cause errors):**
+
+```hcl
+rancher_import_url = "kubectl apply -f https://rancher.example.com/v3/import/TOKEN.yaml"
+```
+
+**Step 4: Deploy/Update Main Infra**
+
+After updating `aws.tfvars`, deploy or update your main infra cluster:
+
+2. **Run main infra via GitHub Actions:**
+
+- Go to **Actions** → **Terraform Infrastructure**
+- Click **Run workflow**
+- **Configure workflow parameters:**
+- **Branch**: Select your deployment branch (e.g., `release-0.1.0`)
+- **Cloud Provider**: Select `aws` (Azure/GCP are placeholder implementations)
+- **Component**: Select `infra` (MOSIP application infrastructure)
+- **Backend**: Choose backend configuration:
+- `local` - GPG-encrypted local state (recommended for development)
+- `s3` - Remote S3 backend (recommended for production)
+- **SSH_PRIVATE_KEY**: GitHub secret name containing SSH private key for instance access
+- Must match the `ssh_key_name` in your terraform.tfvars
+ - **Action**: Select `apply` to deploy infrastructure
+
+**Verify Rancher Import (Only if rancher_import = true):**
+
+> **Note:** Skip this entire section if you deployed without Rancher UI (`rancher_import = false`)
+
+After deployment completes:
+
+1. Go to Rancher UI: `https://rancher.your-domain.net`
+2. Navigate to: **Cluster Management**
+3. Your cluster should appear in the list with status: **Active**
+4. Click on the cluster name to view:
+ - Node status
+ - Pod metrics
+ - Resource utilization
+ - Monitoring dashboards
+
+**Troubleshooting Rancher Import:**
+
+If import fails, check:
+
+```bash
+# Verify cluster is accessible
+kubectl get nodes
+
+# Check if rancher-agent pods are running
+kubectl get pods -n cattle-system
+
+# View rancher-agent logs
+kubectl logs -n cattle-system -l app=cattle-cluster-agent
+
+# Common issues:
+# 1. Network connectivity between clusters
+# 2. Firewall rules blocking Rancher server access
+# 3. Incorrect import URL or expired token
+```
+
+To regenerate import URL if needed:
+
+1. Go to Rancher UI → Cluster Management
+2. Find your cluster (it may show as "Unavailable")
+3. Click ⋮ (three dots) → Edit Config
+4. Copy the new registration command
+
+### 4. Helmsman Deployment#### Step 4a: Update DSF Configuration Files
+
+> **What is DSF?** DSF (Desired State File) is like a recipe that tells Helmsman what applications to install and how to configure them. [Learn more](docs/GLOSSARY.md#dsf-desired-state-file)
+>
+> **Detailed DSF Guide:** [DSF Configuration Guide](docs/DSF_CONFIGURATION_GUIDE.md) - Comprehensive guide with examples and explanations!
+
+**Time required:** 20-30 minutes to update all files
+
+1. **Clone the MOSIP infra repository and navigate to Helmsman directory:**
+
+```bash
+ git clone https://github.com/mosip/infra.git
+ cd infra/Helmsman
+```
+
2. **Navigate to DSF configuration directory:**
- ```bash
- cd dsf/
- ```
+```bash
+ cd dsf/
+```
+
3. **Update prereq-dsf.yaml:**
- **Critical Updates Required:**
-
- - **Domain Validation (Double-check):**
- - `` → your cluster name (e.g., `soil`)
- - `sandbox.xyz.net` → your domain name (e.g., `soil.mosip.net`)
- - **Chart Versions:** Verify and update to latest stable versions
- - **Namespace Configuration:** Ensure proper namespace isolation
-
- > **Note:** Maintain consistency with your Terraform configuration:
- >
- > - `` should match `cluster_name` in `aws.tfvars`
- > - `sandbox.xyz.net` should match `cluster_env_domain` in `aws.tfvars`
- >
-
- ```yaml
- # Configure monitoring, Istio, logging
- helmRepos:
- rancher-latest: "https://releases.rancher.com/server-charts/latest"
-
- apps:
- rancher-monitoring:
- enabled: true
- namespace: cattle-monitoring-system
- ```
+> **IMPORTANT CONFIGURATION:** This file requires **clusterid** configuration **only if you're using Rancher UI** (when `rancher_import = true`)! See [DSF Configuration Guide - clusterid](docs/DSF_CONFIGURATION_GUIDE.md#critical-configuration-clusterid)
+
+ **Critical Updates Required:**
+
+- **clusterid Configuration (OPTIONAL - only if using Rancher):**
+- **When needed?** Only if `rancher_import = true` in your terraform configuration
+- **Skip if:** Deploying without Rancher UI (`rancher_import = false`) - ignore this entire section
+- **What is this?** Unique identifier for your Rancher-managed cluster
+- **Why needed?** Monitoring dashboards won't work without it in Rancher deployments
+- **How to find:** See [DSF Guide - Finding clusterid](docs/DSF_CONFIGURATION_GUIDE.md#how-to-find-your-clusterid)
+- **Location in file:** Around line 40-45
+- **What to change:**
+
+```yaml
+ set:
+ grafana.global.cattle.clusterId: "c-m-pbrcfglw" # ← REPLACE THIS
+ global.cattle.clusterId: "c-m-pbrcfglw" # ← REPLACE THIS
+```
+
+- **Domain Validation (Double-check):**
+- `` → your cluster name (e.g., `soil38`)
+- `sandbox.xyz.net` → your domain name (e.g., `soil38.mosip.net`)
+- **Why?** Every service needs to know its web address
+- **Chart Versions:** Verify and update to latest stable versions
+- Check [MOSIP Helm Repository](https://mosip.github.io/mosip-helm) for latest versions
+- **Namespace Configuration:** Ensure proper namespace isolation
+- **What is namespace?** Like separate folders for different applications
+
+> **Note:** Maintain consistency with your Terraform configuration:
+>
+> - `` should match `cluster_name` in `aws.tfvars`
+> - `sandbox.xyz.net` should match `cluster_env_domain` in `aws.tfvars`
+> - These MUST be identical or deployment will fail!
+
+```yaml
+ # Configure monitoring, Istio, logging
+ helmRepos:
+ rancher-latest: "https://releases.rancher.com/server-charts/latest"
+
+ apps:
+ rancher-monitoring:
+ enabled: true
+ namespace: cattle-monitoring-system
+ # DON'T FORGET: Update clusterid here! See above
+```
+
+ **Need detailed help?** [DSF Configuration Guide - Prerequisites](docs/DSF_CONFIGURATION_GUIDE.md#prerequisites-dsf-configuration)
4. **Update external-dsf.yaml:**
- **Critical Updates Required:**
-
- - **Domain Validation (Double-check):**
- - `` → your cluster name (e.g., `soil`)
- - `sandbox.xyz.net` → your domain name (e.g., `soil.mosip.net`)
- - **Chart Versions:** Update Helm chart versions to latest stable releases
- - **Database Branch:** Verify correct branch for DB scripts and schema
- - **PostgreSQL Configuration:** Match with Terraform `enable_postgresql_setup` setting
-
- > **Note:** Maintain consistency with your Terraform configuration:
- >
- > - `` should match `cluster_name` in `aws.tfvars`
- > - `sandbox.xyz.net` should match `cluster_env_domain` in `aws.tfvars`
- >
-
- - **Configure reCAPTCHA keys:**
-
- 1. **Create reCAPTCHA keys for each domain:**
-
- - Go to [Google reCAPTCHA Admin](https://www.google.com/recaptcha/admin/create)
- - Create reCAPTCHA v2 ("I'm not a robot" Checkbox) for each domain:
- - **PreReg domain**: `prereg.your-domain.net` (e.g., `prereg.soil.mosip.net`)
- - **Admin domain**: `admin.your-domain.net` (e.g., `admin.soil.mosip.net`)
- - **Resident domain**: `resident.your-domain.net` (e.g., `resident.soil.mosip.net`)
- 2. **Update captcha-setup.sh arguments in external-dsf.yaml (around line 315):**
-
- ```yaml
- hooks:
- postInstall: "$WORKDIR/hooks/captcha-setup.sh PREREG_SITE_KEY PREREG_SECRET_KEY ADMIN_SITE_KEY ADMIN_SECRET_KEY RESIDENT_SITE_KEY RESIDENT_SECRET_KEY"
- ```
-
- **Arguments order:**
-
- - **Argument 1**: PreReg site key
- - **Argument 2**: PreReg secret key
- - **Argument 3**: Admin site key
- - **Argument 4**: Admin secret key
- - **Argument 5**: Resident site key
- - **Argument 6**: Resident secret key
- 3. **Example configuration:**
-
- ```yaml
- hooks:
- postInstall: "$WORKDIR/hooks/captcha-setup.sh 6LfkAMwrAAAAAATB1WhkIhzuAVMtOs9VWabODoZ_ 6LfkAMwrAAAAAHQAT93nTGcLKa-h3XYhGoNSG-NL 6LdNAcwrAAAAAETGWvz-3I12vZ5V8vPJLu2ct9CO 6LdNAcwrAAAAAE4iWGJ-g6Dc2HreeJdIwAl5h1iL 6LdRAcwrAAAAAFUEHHKK5D_bSrwAPqdqAJqo4mCk 6LdRAcwrAAAAAOeVl6yHGBCBA8ye9GsUOy4pi9s9"
- ```
-
- ```yaml
- # Configure external dependencies
- apps:
- postgresql:
- # Set based on your Terraform configuration:
- enabled: false # false if enable_postgresql_setup = true (external PostgreSQL via Terraform)
- # true if enable_postgresql_setup = false (container PostgreSQL)
- minio:
- enabled: true
- kafka:
- enabled: true
- ```
+ **Critical Updates Required:**
+
+- **Domain Validation (Double-check):**
+- `` → your cluster name (e.g., `soil`)
+- `sandbox.xyz.net` → your domain name (e.g., `soil.mosip.net`)
+- **Chart Versions:** Update Helm chart versions to latest stable releases
+- **Database Branch:** Verify correct branch for DB scripts and schema
+- **PostgreSQL Configuration:** Match with Terraform `enable_postgresql_setup` setting
+
+> **Note:** Maintain consistency with your Terraform configuration:
+>
+> - `` should match `cluster_name` in `aws.tfvars`
+> - `sandbox.xyz.net` should match `cluster_env_domain` in `aws.tfvars`
+
+- **Configure reCAPTCHA keys:**
+
+1. **Create reCAPTCHA keys for each domain:**
+
+- Go to [Google reCAPTCHA Admin](https://www.google.com/recaptcha/admin/create)
+- Create reCAPTCHA v2 ("I'm not a robot" Checkbox) for each domain:
+- **PreReg domain**: `prereg.your-domain.net` (e.g., `prereg.soil.mosip.net`)
+- **Admin domain**: `admin.your-domain.net` (e.g., `admin.soil.mosip.net`)
+- **Resident domain**: `resident.your-domain.net` (e.g., `resident.soil.mosip.net`)
+
+2. **Update captcha-setup.sh arguments in external-dsf.yaml (around line 315):**
+
+```yaml
+ hooks:
+ postInstall: "$WORKDIR/hooks/captcha-setup.sh PREREG_SITE_KEY PREREG_SECRET_KEY ADMIN_SITE_KEY ADMIN_SECRET_KEY RESIDENT_SITE_KEY RESIDENT_SECRET_KEY"
+```
+
+ **Arguments order:**
+
+- **Argument 1**: PreReg site key
+- **Argument 2**: PreReg secret key
+- **Argument 3**: Admin site key
+- **Argument 4**: Admin secret key
+- **Argument 5**: Resident site key
+- **Argument 6**: Resident secret key
+
+3. **Example configuration:**
+
+```yaml
+ hooks:
+ postInstall: "$WORKDIR/hooks/captcha-setup.sh 6LfkAMwrAAAAAATB1WhkIhzuAVMtOs9VWabODoZ_ 6LfkAMwrAAAAAHQAT93nTGcLKa-h3XYhGoNSG-NL 6LdNAcwrAAAAAETGWvz-3I12vZ5V8vPJLu2ct9CO 6LdNAcwrAAAAAE4iWGJ-g6Dc2HreeJdIwAl5h1iL 6LdRAcwrAAAAAFUEHHKK5D_bSrwAPqdqAJqo4mCk 6LdRAcwrAAAAAOeVl6yHGBCBA8ye9GsUOy4pi9s9"
+```
+
+```yaml
+ # Configure external dependencies
+ apps:
+ postgresql:
+ # Set based on your Terraform configuration:
+ enabled: false # false if enable_postgresql_setup = true (external PostgreSQL via Terraform)
+ # true if enable_postgresql_setup = false (container PostgreSQL)
+ minio:
+ enabled: true
+ kafka:
+ enabled: true
+```
+
5. **Update mosip-dsf.yaml:**
- **Critical Updates Required:**
-
- - **Domain Validation (Double-check):**
- - `` → your cluster name (e.g., `soil`)
- - `sandbox.xyz.net` → your domain name (e.g., `soil.mosip.net`)
- - **Chart Versions:** Update MOSIP service chart versions to compatible releases
- - **Database Branch:** Ensure correct MOSIP DB scripts branch matches deployment version
- - **Service Dependencies:** Verify all required external services are properly configured
- - **Resource Limits:** Adjust CPU/memory limits based on environment requirements
-
- > **Note:** Maintain consistency with your Terraform configuration:
- >
- > - `` should match `cluster_name` in `aws.tfvars`
- > - `sandbox.xyz.net` should match `cluster_env_domain` in `aws.tfvars`
- >
-
- ```yaml
- # Configure MOSIP services
- apps:
- config-server:
- enabled: true
- artifactory:
- enabled: true
- kernel:
- enabled: true
- ```
+ **Critical Updates Required:**
+
+- **Domain Validation (Double-check):**
+- `` → your cluster name (e.g., `soil`)
+- `sandbox.xyz.net` → your domain name (e.g., `soil.mosip.net`)
+- **Chart Versions:** Update MOSIP service chart versions to compatible releases
+- **Database Branch:** Ensure correct MOSIP DB scripts branch matches deployment version
+- **Service Dependencies:** Verify all required external services are properly configured
+- **Resource Limits:** Adjust CPU/memory limits based on environment requirements
+
+> **Note:** Maintain consistency with your Terraform configuration:
+>
+> - `` should match `cluster_name` in `aws.tfvars`
+> - `sandbox.xyz.net` should match `cluster_env_domain` in `aws.tfvars`
+
+```yaml
+ # Configure MOSIP services
+ apps:
+ config-server:
+ enabled: true
+ artifactory:
+ enabled: true
+ kernel:
+ enabled: true
+```
+
6. **Update testrigs-dsf.yaml (if deploying test environment):**
- **Critical Updates Required:**
+ **Critical Updates Required:**
- - **Domain Validation (Double-check):**
- - `` → your cluster name (e.g., `soil`)
- - `sandbox.xyz.net` → your domain name (e.g., `soil.mosip.net`)
- - **Test Chart Versions:** Update test rig chart versions to match MOSIP service versions
- - **Database Branch:** Ensure test DB scripts use correct branch
- - **Test Configuration:** Update test endpoints, API versions, and test data paths
- - **Resource Allocation:** Configure appropriate test environment resource limits
+- **Domain Validation (Double-check):**
+- `` → your cluster name (e.g., `soil`)
+- `sandbox.xyz.net` → your domain name (e.g., `soil.mosip.net`)
+- **Test Chart Versions:** Update test rig chart versions to match MOSIP service versions
+- **Database Branch:** Ensure test DB scripts use correct branch
+- **Test Configuration:** Update test endpoints, API versions, and test data paths
+- **Resource Allocation:** Configure appropriate test environment resource limits
-> **⚠️ Critical Validation Checklist for All DSF Files:**
+> **Critical Validation Checklist for All DSF Files:**
>
> **Domain Configuration (Validate Twice):**
>
@@ -698,66 +1061,67 @@ Add the required secrets as follows:
1. **Update Repository Branch Configuration:**
- - Ensure your repository is configured to use the correct branch for Helmsman workflows
- - Verify GitHub Actions have access to your deployment branch
+- Ensure your repository is configured to use the correct branch for Helmsman workflows
+- Verify GitHub Actions have access to your deployment branch
+
2. **Configure KUBECONFIG Secret:**
- **Locate the Kubernetes config file:**
+ **Locate the Kubernetes config file:**
- ```bash
- # After Terraform infrastructure deployment completes, find the kubeconfig file in:
- terraform/implementations/aws/infra/
- ```
+```bash
+ # After Terraform infrastructure deployment completes, find the kubeconfig file in:
+ terraform/implementations/aws/infra/
+```
- **Add KUBECONFIG as Environment Secret:**
+**Example kubeconfig file location:**
- - Go to your GitHub repository → Settings → Environments
- - Select or create environment for your branch (e.g., `release-0.1.0`, `main`, `develop`)
- - Click "Add secret" under Environment secrets
- - Name: `KUBECONFIG`
- - Value: Copy the entire contents of the kubeconfig file from `terraform/implementations/aws/infra/`
+```
+ terraform/implementations/aws/infra/kubeconfig_
+ terraform/implementations/aws/infra/-role.yaml
+```
- **Example kubeconfig file location:**
+ **Add KUBECONFIG as Environment Secret:**
- ```
- terraform/implementations/aws/infra/kubeconfig_
- terraform/implementations/aws/infra/-role.yaml
- ```
+- Go to your GitHub repository → Settings → Environments
+- Select or create environment for your branch (e.g., `release-0.1.0`, `main`, `develop`)
+- Click "Add secret" under Environment secrets
+- Name: `KUBECONFIG`
+ - Value: Copy the entire contents of the kubeconfig file from `terraform/implementations/aws/infra/`
- **Branch Environment Configuration:**
+ **Branch Environment Configuration:- Ensure the environment name matches your deployment branch
+- Configure environment protection rules if needed
+- Verify Helmsman workflows reference the correct environment
- - Ensure the environment name matches your deployment branch
- - Configure environment protection rules if needed
- - Verify Helmsman workflows reference the correct environment
3. **Required Environment Secrets for Helmsman:**
- **Environment Secrets (branch-specific):**
+ **Environment Secrets (branch-specific):**
- ```yaml
- # Kubernetes Access (Environment Secret)
- KUBECONFIG: ""
+```yaml
+ # Kubernetes Access (Environment Secret)
+ KUBECONFIG: ""
- # WireGuard Cluster Access for Helmsman
- CLUSTER_WIREGUARD_WG0: "peer1-wireguard-config" # Helmsman cluster access (peer1)
- CLUSTER_WIREGUARD_WG1: "peer2-wireguard-config" # Helmsman cluster access (peer2)
- ```
+ # WireGuard Cluster Access for Helmsman
+ CLUSTER_WIREGUARD_WG0: "peer1-wireguard-config" # Helmsman cluster access (peer1)
+ CLUSTER_WIREGUARD_WG1: "peer2-wireguard-config" # Helmsman cluster access (peer2)
+```
- **Repository Secrets (global):**
+ **Repository Secrets (global):**
- ```yaml
- # GPG Encryption (if using encrypted backends)
- GPG_PASSPHRASE: "your-gpg-passphrase"
+```yaml
+ # GPG Encryption (if using encrypted backends)
+ GPG_PASSPHRASE: "your-gpg-passphrase"
+
+ # AWS Credentials (if not using OIDC)
+ AWS_ACCESS_KEY_ID: "AKIA..."
+ AWS_SECRET_ACCESS_KEY: "..."
+```
- # AWS Credentials (if not using OIDC)
- AWS_ACCESS_KEY_ID: "AKIA..."
- AWS_SECRET_ACCESS_KEY: "..."
- ```
4. **Verify Secret Configuration:**
- - Ensure KUBECONFIG is configured as environment secret for your branch
- - Verify repository secrets are properly configured
- - Test repository access from GitHub Actions
- - Verify KUBECONFIG provides cluster access
+- Ensure KUBECONFIG is configured as environment secret for your branch
+- Verify repository secrets are properly configured
+- Test repository access from GitHub Actions
+- Verify KUBECONFIG provides cluster access
> **Important:**
>
@@ -767,54 +1131,82 @@ Add the required secrets as follows:
#### Step 4c: Run Helmsman Deployments via GitHub Actions
+> **Need visual guidance?** See [Workflow Guide - Helmsman Workflows](docs/WORKFLOW_GUIDE.md#helmsman-workflows) for detailed step-by-step instructions!
+
The Helmsman deployment process follows a specific sequence with automated triggers and error handling mechanisms:
-> **⚠️ Important**: Always use `apply` mode for Helmsman deployments. The `dry-run` mode will fail due to dependencies on shared configmaps and secrets from other namespaces that are not available during dry-run validation.
+> **Important**: Always use `apply` mode for Helmsman deployments. The `dry-run` mode will fail due to dependencies on shared configmaps and secrets from other namespaces that are not available during dry-run validation.
+>
+> **Why does dry-run fail?** Helmsman checks if resources exist before deployment. In dry-run mode, these resources aren't created yet, so validation fails. Think of it like checking if ingredients are in the kitchen before actually cooking - but in dry-run mode, the ingredients haven't been bought yet!
+
+**Understanding Workflow Names:**
+
+| Actual Workflow Name in GitHub | Where to Find |
+| -------------------------------------------------- | ----------------------- |
+| "Deploy External services of mosip using Helmsman" | Actions → Left sidebar |
+| "Deploy MOSIP services using Helmsman" | Actions → Left sidebar |
+| "Deploy Testrigs of mosip using Helmsman" | Actions → Left sidebar |
+
+> **Can't find the workflow?** Look for keywords like "External", "MOSIP", or "Deploy" in the left sidebar. See [Workflow Guide](docs/WORKFLOW_GUIDE.md#understanding-workflow-basics) for navigation help!
1. **Deploy Prerequisites & External Dependencies:**
- - Actions → **Helmsman External Dependencies** (`helmsman_external.yml`)
- - This workflow handles both deployments in parallel:
- - **Prerequisites**: `prereq-dsf.yaml` (monitoring, Istio, logging)
- - **External Dependencies**: `external-dsf.yaml` (databases, message queues, storage)
- - Mode: `apply` (required - dry-run will fail)
- - **Automatic Trigger**: Upon successful completion, this workflow automatically triggers the MOSIP services deployment
+> **Detailed Steps:** [Workflow Guide - Prerequisites & External Dependencies](docs/WORKFLOW_GUIDE.md#workflow-1-prerequisites--external-dependencies)
+
+- Actions → **"Deploy External services of mosip using Helmsman"** (or "Helmsman External Dependencies")
+- **Can't find it?** Search for "External" in the workflows list
+- This workflow handles both deployments in parallel:
+- **Prerequisites**: `prereq-dsf.yaml` (monitoring, Istio, logging)
+- **External Dependencies**: `external-dsf.yaml` (databases, message queues, storage)
+- **Mode**: `apply` (required - dry-run will fail!)
+- **Important:** DO NOT select dry-run mode for Helmsman
+- **Time required:** 20-40 minutes
+- **Automatic Trigger**: Upon successful completion, this workflow automatically triggers the MOSIP services deployment
+
+ **What You Should See:**
- > **Note**: The `helmsman_external.yml` workflow deploys both prereq and external dependencies in parallel for optimal deployment time.
+- ✅ Monitoring stack deploying (Prometheus, Grafana)
+- ✅ Istio service mesh installing
+- ✅ PostgreSQL database starting (if container mode)
+- ✅ MinIO storage deploying
+- ✅ Kafka message queue starting
+
+> **Note**: The `helmsman_external.yml` workflow deploys both prereq and external dependencies in parallel for optimal deployment time.
2. **Deploy MOSIP Services (Automated):**
- - **Automatically triggered** after successful completion of step 1
- - Workflow: **Helmsman MOSIP Deployment** (`helmsman_mosip.yml`)
- - DSF file: `mosip-dsf.yaml`
- - Mode: `apply` (required - dry-run will fail due to namespace dependencies)
-
- **Error Handling:**
- - If the automatic trigger fails, manually trigger: Actions → **Helmsman MOSIP Deployment**
- - If onboarding processes fail during deployment, manual re-onboarding is required (see limitations section)
+- **Automatically triggered** after successful completion of step 1
+- Workflow: **Deploy MOSIP services using Helmsman** (`helmsman_mosip.yml`)
+- DSF file: `mosip-dsf.yaml`
+- Mode: `apply` (required - dry-run will fail due to namespace dependencies)
+
+ **Error Handling:**
+
+- If the automatic trigger fails, manually trigger: Actions → **Deploy MOSIP services using Helmsman**
+- If onboarding processes fail during deployment, manual re-onboarding is required (see limitations section)
3. **Verify All Pods are Running:**
- Before proceeding to test rigs, ensure all MOSIP services are properly deployed:
-
- ```bash
- # Check all MOSIP pods are running
- kubectl get pods -n mosip
- kubectl get pods -n keycloak
- kubectl get pods -n postgres
-
- # Ensure no pods are in pending/error state
- kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
- ```
+ Before proceeding to test rigs, ensure all MOSIP services are properly deployed:
+
+```bash
+ # Check all MOSIP pods are running
+ kubectl get pods -A
+ kubectl get pods -n keycloak
+ kubectl get pods -n postgres
+
+ # Ensure no pods are in pending/error state
+ kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
+```
4. **Deploy Test Rigs (Manual):**
- - **Prerequisites**: All pods from steps 1-2 must be in `Running` state
- - Actions → **Helmsman Test Rigs** (`helmsman_testrigs.yml`)
- - Select DSF file: `testrigs-dsf.yaml`
- - Mode: `apply` (required - dry-run will fail due to namespace dependencies)
-
- > **Important**: Test rigs should only be deployed after verifying all core services are running successfully. Failed onboarding processes must be manually re-executed before test rig deployment.
+- **Prerequisites**: All pods from steps 1-2 must be in `Running` state
+- Actions → **Deploy Testrigs of mosip using Helmsman** (`helmsman_testrigs.yml`)
+- Select DSF file: `testrigs-dsf.yaml`
+- Mode: `apply` (required - dry-run will fail due to namespace dependencies)
+
+> **Important**: Test rigs should only be deployed after verifying all core services are running successfully. Failed onboarding processes must be manually re-executed before test rig deployment.
### 5. Verify Deployment
@@ -824,39 +1216,82 @@ kubectl get nodes
kubectl get namespaces
# Check MOSIP services
-# Check MOSIP services
-kubectl get pods -n mosip
+kubectl get pods -A
kubectl get services -n istio-system
```
-### 6. Next Steps & Detailed Documentation
+### 6. Environment Destruction and Cleanup
+
+> **Need to destroy your environment?** See our [Environment Destruction Guide](docs/ENVIRONMENT_DESTRUCTION_GUIDE.md) for safe teardown procedures!
+
+**When you need to destroy/decommission an environment:**
+
+- Development/testing environment no longer needed
+- Failed deployment requires clean restart
+- Cost optimization (shutting down unused resources)
+- Recreating infrastructure with new configuration
+
+**What the guide covers:**
+
+- ✅ **Safe destruction order** - Destroy in reverse order to avoid orphaned resources
+- ✅ **Step-by-step commands** - Exact commands for Helmsman and Terraform cleanup
+- ✅ **Backup procedures** - How to save data before destruction
+- ✅ **Cost verification** - Ensure all resources are deleted to stop billing
+- ✅ **Troubleshooting** - Fix common deletion issues
+
+**Quick Destruction Summary:**
+
+```
+Destruction Order (Reverse of Deployment):
+
+1. Test Rigs ← Destroy FIRST
+2. MOSIP Services ← Destroy SECOND
+3. External Services ← Destroy THIRD
+4. Prerequisites ← Destroy FOURTH
+5. Infrastructure ← Destroy FIFTH (Kubernetes cluster)
+6. Base Infrastructure ← Destroy LAST (VPC, networking)
+```
+
+**Time required:** 1-2 hours for complete cleanup
+
+**Important:**
+
+- Always backup data before destroying production environments
+- Verify all AWS resources are deleted to avoid unexpected costs
+- Follow the guide's checklist to ensure complete cleanup
+
+**[View Complete Destruction Guide](docs/ENVIRONMENT_DESTRUCTION_GUIDE.md)**
+
+---
+
+### 7. Next Steps & Detailed Documentation
The Quick Start Guide provides the essential deployment flow. For comprehensive configuration options, troubleshooting, and advanced features, refer to the detailed component documentation:
-#### 📁 **Terraform Infrastructure Documentation**
+#### **Terraform Infrastructure Documentation**
- **Location**: [`terraform/README.md`](terraform/README.md)
- **Contents**: Detailed variable explanations, multi-cloud configurations, state management, security best practices
- **Use Cases**: Custom infrastructure configurations, production deployments, troubleshooting infrastructure issues
-#### 📁 **Helmsman Deployment Documentation**
+#### **Helmsman Deployment Documentation**
- **Location**: [`Helmsman/README.md`](Helmsman/README.md)
- **Contents**: Complete DSF configuration reference, hook scripts, environment management, customization options
- **Use Cases**: Custom service configurations, environment-specific deployments, service scaling and tuning
-#### 📁 **WireGuard VPN Setup Guide**
+#### **WireGuard VPN Setup Guide**
- **Location**: [`terraform/base-infra/WIREGUARD_SETUP.md`](terraform/base-infra/WIREGUARD_SETUP.md)
- **Contents**: Step-by-step VPN configuration, multi-peer setup, client installation, troubleshooting
- **Use Cases**: Private network access, secure infrastructure connectivity, peer management
-#### 📋 **Component-Specific Guides**
+#### **Component-Specific Guides**
- **GitHub Actions Workflows**: [`.github/workflows/`](.github/workflows/) - Complete CI/CD pipeline documentation
- **Security Configurations**: See respective component READMEs for security hardening options
-> **💡 Pro Tip**: Each component directory contains detailed documentation tailored to that specific technology stack. Start with this Quick Start Guide, then dive into component-specific docs as needed.
+> **Pro Tip**: Each component directory contains detailed documentation tailored to that specific technology stack. Start with this Quick Start Guide, then dive into component-specific docs as needed.
## Known Limitations
@@ -878,11 +1313,13 @@ The Quick Start Guide provides the essential deployment flow. For comprehensive
**Impact**: Additional administrator intervention needed to complete onboarding workflow.
**Details:**
+
- **Failed Onboarding Recovery**: If partner onboarding fails during the automated MOSIP deployment, manual re-onboarding is required before proceeding to test rig deployment
- **Pre-Test Rig Requirements**: All pods must be verified as running and stable before triggering test rig deployments
- **Manual Verification Steps**: Administrator must check pod status across all namespaces (mosip, keycloak, postgres) before proceeding with test rigs
**Required Actions:**
+
1. Monitor deployment logs for onboarding failures
2. Execute manual re-onboarding procedures for failed cases
3. Verify all services are operational before test rig deployment
@@ -921,13 +1358,15 @@ Failed to pull image "docker.io/mosipid/pre-registration-batchjob:1.2.0.3": fail
1. **Docker Hub Authentication**: Configure Docker Hub credentials in your cluster
2. **Retry Deployments**: Re-run failed Helmsman deployments after waiting period
3. **Manual Pod Restart**: If any pod remains in "ContainerCreating" state for more than 3 minutes:
- ```bash
- # Delete the stuck pod to trigger recreation
- kubectl delete pod -n
- # Check pod status
- kubectl get pods -n -w
- ```
+```bash
+ # Delete the stuck pod to trigger recreation
+ kubectl delete pod -n
+
+ # Check pod status
+ kubectl get pods -n -w
+```
+
4. **Mirror Registries**: Use alternative container registries or mirrors
5. **Rate Limit Increase**: Consider Docker Hub paid plans for higher limits
@@ -947,7 +1386,7 @@ status code: 500, request id: 0b0423e2-0906-4096-a03c-41df5c00f5a8
# If empty, uses all available AZs in the region
# Example: ["ap-south-1a", "ap-south-1b"] for specific AZs
# Example: [] for all available AZs in the region
-specific_availability_zones = [] # Use empty array to allow all AZs
+specific_availability_zones = [] # Use empty array to allow all AZs
```
**Best Practice**: Always set `specific_availability_zones = []` to allow AWS to select from all available zones with capacity.
@@ -995,28 +1434,28 @@ This section provides baseline test execution numbers for MOSIP test rigs to hel
### API Testing Baseline
```
-auth: T-612 | P-589 | S-0 | F-2 | I-5 | KI-16
+auth: T-612 | P-589 | S-0 | F-2 | I-5 | KI-16
-resident: T-1142| P-580 | S-535 | F-14 | I-0 | KI-12
+resident: T-1142| P-580 | S-535 | F-14 | I-0 | KI-12
-idrepo: T-414 | P-315 | S-0 | F-1 | I-78 | KI-20
+idrepo: T-414 | P-315 | S-0 | F-1 | I-78 | KI-20
-pms: T-509 | P-480 | S-0 | F-2 | I-15 | KI-12
+pms: T-509 | P-480 | S-0 | F-2 | I-15 | KI-12
-prereg: T-288 | P-277 | S-0 | F-0 | I-2 | KI-9
+prereg: T-288 | P-277 | S-0 | F-0 | I-2 | KI-9
masterdata:
- - fra: T-945 | P-907 | S-0 | F-0 | I-15 | KI-23
- - ara: T-945 | P-895 | S-0 | F-0 | I-15 | KI-35
- - eng: T-945 | P-922 | S-0 | F-1 | I-0 | KI-22
+ - fra: T-945 | P-907 | S-0 | F-0 | I-15 | KI-23
+ - ara: T-945 | P-895 | S-0 | F-0 | I-15 | KI-35
+ - eng: T-945 | P-922 | S-0 | F-1 | I-0 | KI-22
```
### DSL Testing Baseline
```
-DSL (sanity): T-3 | P-3 | S-0 | F-0 | KI-0
+DSL (sanity): T-3 | P-3 | S-0 | F-0 | KI-0
-DSL (full run): T-204 | P-163 | S-0 | F-2 | I-12 | KI-27
+DSL (full run): T-204 | P-163 | S-0 | F-2 | I-12 | KI-27
```
---
diff --git a/docs/DSF_CONFIGURATION_GUIDE.md b/docs/DSF_CONFIGURATION_GUIDE.md
new file mode 100644
index 00000000..67a258d5
--- /dev/null
+++ b/docs/DSF_CONFIGURATION_GUIDE.md
@@ -0,0 +1,814 @@
+# DSF Configuration Guide
+
+This guide explains how to configure Helmsman Desired State Files (DSF) for MOSIP deployment, including the critical `clusterid` configuration and other essential settings.
+
+## Table of Contents
+
+1. [Understanding DSF Files](#understanding-dsf-files)
+2. [Prerequisites DSF Configuration](#prerequisites-dsf-configuration)
+3. [External Dependencies DSF Configuration](#external-dependencies-dsf-configuration)
+4. [MOSIP Services DSF Configuration](#mosip-services-dsf-configuration)
+5. [Test Rigs DSF Configuration](#test-rigs-dsf-configuration)
+6. [Common Configuration Patterns](#common-configuration-patterns)
+
+---
+
+## Understanding DSF Files
+
+### What is a DSF File?
+
+DSF (Desired State File) tells Helmsman:
+- **What** applications to install
+- **Which versions** to use
+- **How** to configure them
+- **Where** to deploy them
+
+Think of it as a recipe that Helmsman follows to deploy your MOSIP platform.
+
+### DSF File Types
+
+| File | Purpose | Deploy Order |
+|------|---------|--------------|
+| `prereq-dsf.yaml` | Monitoring, Istio, logging | 1st |
+| `external-dsf.yaml` | Databases, queues, storage | 2nd |
+| `mosip-dsf.yaml` | MOSIP core services | 3rd |
+| `testrigs-dsf.yaml` | Testing infrastructure | 4th (optional) |
+
+### File Location
+
+```
+Helmsman/dsf/
+├── prereq-dsf.yaml
+├── external-dsf.yaml
+├── mosip-dsf.yaml
+└── testrigs-dsf.yaml
+```
+
+---
+
+## Prerequisites DSF Configuration
+
+**File**: `Helmsman/dsf/prereq-dsf.yaml`
+
+**Purpose**: Deploys monitoring stack, Istio service mesh, and logging
+
+### Critical Configuration: clusterid
+
+#### What is clusterid?
+
+The `clusterid` is a unique identifier for your Rancher-managed Kubernetes cluster. It's used by Rancher Monitoring to correctly identify and track your cluster's metrics.
+
+#### Why is it Important?
+
+Without the correct `clusterid`:
+- ❌ Monitoring dashboards won't display correct cluster data
+- ❌ Grafana may show metrics from wrong cluster
+- ❌ Alerts and notifications may not work properly
+
+#### How to Find Your clusterid
+
+**Option 1: From Rancher UI (Recommended)**
+
+1. Log in to Rancher UI
+ ```
+ URL: https://rancher.your-domain.net
+ ```
+
+2. Navigate to your cluster
+ ```
+ Home → Clusters → Click on your MOSIP cluster
+ ```
+
+3. Get clusterid from URL
+ ```
+ URL format: https://rancher.example.net/c/c-m-abc12xyz/explorer
+ ^^^^^^^^^^^^
+ This is your clusterid
+ ```
+
+4. Copy the clusterid
+ ```
+ Example: c-m-pbrcfglw
+ c-m-abc12xyz
+ c-m-5x9k7w3d
+ ```
+
+**Option 2: Using kubectl**
+
+```bash
+# Connect to your cluster with kubectl
+export KUBECONFIG=/path/to/kubeconfig
+
+# Get clusterid from cluster registration
+kubectl get setting cluster-id -n cattle-system -o jsonpath='{.value}'
+
+# Output example:
+# c-m-pbrcfglw
+```
+
+**Option 3: From Terraform Outputs**
+
+```bash
+# If Rancher import was enabled during Terraform deployment
+cd terraform/implementations/aws/infra/
+terraform output rancher_cluster_id
+```
+
+#### Where to Add clusterid in prereq-dsf.yaml
+
+**Location**: Around line 40-45 in `prereq-dsf.yaml`
+
+**Before (Default/Placeholder)**:
+```yaml
+apps:
+ rancher-monitoring:
+ namespace: cattle-monitoring-system
+ enabled: true
+ version: 103.1.0+up45.31.1
+ chart: mosip/rancher-monitoring
+ set:
+ grafana.global.cattle.clusterId: "c-m-pbrcfglw" # ← CHANGE THIS
+ global.cattle.clusterId: "c-m-pbrcfglw" # ← CHANGE THIS
+ wait: true
+ valuesFile: "$WORKDIR/utils/monitoring_values.yaml"
+ priority: -5
+ timeout: 600
+```
+
+**After (Your Actual clusterid)**:
+```yaml
+apps:
+ rancher-monitoring:
+ namespace: cattle-monitoring-system
+ enabled: true
+ version: 103.1.0+up45.31.1
+ chart: mosip/rancher-monitoring
+ set:
+ grafana.global.cattle.clusterId: "c-m-5x9k7w3d" # ← YOUR clusterid
+ global.cattle.clusterId: "c-m-5x9k7w3d" # ← YOUR clusterid
+ wait: true
+ valuesFile: "$WORKDIR/utils/monitoring_values.yaml"
+ priority: -5
+ timeout: 600
+```
+
+#### Configuration Snippet with Detailed Comments
+
+```yaml
+apps:
+ rancher-monitoring:
+ namespace: cattle-monitoring-system # Namespace for monitoring
+ enabled: true # Set to false to skip monitoring
+ version: 103.1.0+up45.31.1 # Chart version
+ chart: mosip/rancher-monitoring # Helm chart location
+ set:
+ # CRITICAL: Replace with your actual Rancher clusterid
+ # Find it in Rancher UI URL or using kubectl
+ grafana.global.cattle.clusterId: "c-m-5x9k7w3d" # For Grafana
+ global.cattle.clusterId: "c-m-5x9k7w3d" # For Prometheus
+ wait: true # Wait for deployment to complete
+ valuesFile: "$WORKDIR/utils/monitoring_values.yaml" # Additional config
+ priority: -5 # Deploy order (lower = earlier)
+ timeout: 600 # Max wait time in seconds
+```
+
+### Complete prereq-dsf.yaml Configuration
+
+#### Domain Replacement
+
+**Find and replace** these placeholders throughout the file:
+
+| Placeholder | Replace With | Example |
+|-------------|--------------|---------|
+| `` | Your cluster name | `soil38` |
+| `sandbox.xyz.net` | Your domain | `soil38.mosip.net` |
+
+**Example:**
+
+**Before**:
+```yaml
+hooks:
+ postInstall: "$WORKDIR/hooks/install-istio-and-httpbin.sh helmsman"
+```
+
+**After**:
+```yaml
+hooks:
+ postInstall: "$WORKDIR/hooks/install-istio-and-httpbin.sh soil38 helmsman"
+```
+
+#### Monitoring Configuration Options
+
+```yaml
+apps:
+ # CRDs (Custom Resource Definitions) - Always deploy first
+ rancher-monitoring-crd:
+ namespace: cattle-monitoring-system
+ enabled: true # Keep enabled
+ version: 103.1.1+up45.31.1
+ chart: mosip/rancher-monitoring-crd
+ priority: -6 # Highest priority (deploys first)
+
+ # Main monitoring stack
+ rancher-monitoring:
+ namespace: cattle-monitoring-system
+ enabled: true # Set to false to disable monitoring
+ # If disabled, Grafana and Prometheus won't be installed
+```
+
+#### Logging Configuration (Optional)
+
+```yaml
+apps:
+ logging:
+ namespace: cattle-logging-system
+ enabled: false # Set to true to enable logging
+ # When enabled, installs Fluentd/Elasticsearch/Kibana stack
+```
+
+**When to enable logging:**
+- Need centralized log aggregation
+- Debugging complex issues
+- Compliance requirements
+- Performance analysis
+
+**Cost considerations:**
+- Logging consumes significant storage
+- Requires additional resources (CPU/memory)
+- Recommended for production, optional for development
+
+---
+
+## External Dependencies DSF Configuration
+
+**File**: `Helmsman/dsf/external-dsf.yaml`
+
+**Purpose**: Deploys databases, message queues, storage, and identity management
+
+### Critical Configurations
+
+#### 1. Domain Replacement
+
+**Same as prereq-dsf.yaml**: Replace `` and `sandbox.xyz.net`
+
+```yaml
+# Example locations in external-dsf.yaml:
+apps:
+ postgres-init:
+ set:
+ dbUserPasswords:
+ postgres-postgresql\.sandbox\.svc\.cluster\.local: # ← Replace sandbox
+```
+
+#### 2. PostgreSQL Configuration
+
+**Location**: Around line 50-100
+
+**Decision Point**: Container PostgreSQL vs External PostgreSQL
+
+| Configuration | When to Use | Setting |
+|--------------|-------------|---------|
+| **Container** | Development, testing | `enabled: true` |
+| **External** | Production, deployed via Terraform | `enabled: false` |
+
+**For Container PostgreSQL**:
+```yaml
+apps:
+ postgres:
+ namespace: postgres
+ enabled: true # ← Set to true
+ chart: mosip/postgres
+ version: 14.0.0
+ priority: -11
+ set:
+ # PostgreSQL version
+ image.tag: "15.8.0-debian-12-r0"
+```
+
+**For External PostgreSQL (Terraform-deployed)**:
+```yaml
+apps:
+ postgres:
+ namespace: postgres
+ enabled: false # ← Set to false (skip container)
+ # External PostgreSQL connection details are in terraform outputs
+```
+
+**Important**: This setting must match your Terraform configuration:
+```hcl
+# In terraform/implementations/aws/infra/aws.tfvars
+enable_postgresql_setup = true # External → postgres.enabled = false
+enable_postgresql_setup = false # Container → postgres.enabled = true
+```
+
+#### 3. reCAPTCHA Configuration
+
+**Location**: Around line 315
+
+**Purpose**: Protects MOSIP web portals from bots
+
+**Configuration**:
+```yaml
+apps:
+ captcha:
+ enabled: true
+ hooks:
+ postInstall: "$WORKDIR/hooks/captcha-setup.sh PREREG_SITE_KEY PREREG_SECRET_KEY ADMIN_SITE_KEY ADMIN_SECRET_KEY RESIDENT_SITE_KEY RESIDENT_SECRET_KEY"
+```
+
+**How to get keys**: See [Secret Generation Guide - reCAPTCHA](SECRET_GENERATION_GUIDE.md#6-recaptcha-keys)
+
+**Example with actual keys**:
+```yaml
+hooks:
+ postInstall: "$WORKDIR/hooks/captcha-setup.sh 6LfkAMwrAAAAAATB1WhkIhzuAVMtOs9VWabODoZ_ 6LfkAMwrAAAAAHQAT93nTGcLKa-h3XYhGoNSG-NL 6LdNAcwrAAAAAETGWvz-3I12vZ5V8vPJLu2ct9CO 6LdNAcwrAAAAAE4iWGJ-g6Dc2HreeJdIwAl5h1iL 6LdRAcwrAAAAAFUEHHKK5D_bSrwAPqdqAJqo4mCk 6LdRAcwrAAAAAOeVl6yHGBCBA8ye9GsUOy4pi9s9"
+```
+
+**Key Order (CRITICAL)**:
+1. PreReg Site Key
+2. PreReg Secret Key
+3. Admin Site Key
+4. Admin Secret Key
+5. Resident Site Key
+6. Resident Secret Key
+
+#### 4. MinIO Configuration (Object Storage)
+
+**Location**: Around line 150-200
+
+```yaml
+apps:
+ minio:
+ namespace: minio
+ enabled: true # Required for MOSIP file storage
+ chart: mosip/minio
+ version: 5.0.14
+ priority: -11
+ set:
+ # Storage capacity
+ persistence.size: "100Gi" # Adjust based on needs
+
+ # Resource limits
+ resources.requests.memory: "2Gi"
+ resources.limits.memory: "4Gi"
+```
+
+**Storage recommendations:**
+- **Development**: 50-100 GB
+- **Testing**: 100-200 GB
+- **Production**: 500+ GB
+
+#### 5. Kafka Configuration (Message Queue)
+
+**Location**: Around line 250-300
+
+```yaml
+apps:
+ kafka:
+ namespace: kafka
+ enabled: true # Required for MOSIP async communication
+ chart: mosip/kafka
+ version: 22.1.5
+ priority: -11
+ set:
+ # Kafka cluster sizing
+ replicaCount: 3 # Number of Kafka brokers
+
+ # ZooKeeper configuration
+ zookeeper.replicaCount: 3 # Number of ZooKeeper nodes
+```
+
+**Sizing recommendations:**
+- **Development**: 1 broker, 1 ZooKeeper
+- **Testing**: 3 brokers, 3 ZooKeeper
+- **Production**: 3+ brokers, 3+ ZooKeeper
+
+#### 6. Keycloak Configuration (Identity Management)
+
+**Location**: Around line 350-400
+
+```yaml
+apps:
+ keycloak-init:
+ namespace: keycloak
+ enabled: true # Required for authentication
+ chart: mosip/keycloak-init
+ version: 12.0.1
+ priority: -10
+ set:
+ # Admin credentials (auto-generated)
+ # Database connection (auto-configured)
+
+ # Domain configuration
+ istio.hosts:
+ - "iam.sandbox.xyz.net" # ← Replace with your domain
+```
+
+---
+
+## MOSIP Services DSF Configuration
+
+**File**: `Helmsman/dsf/mosip-dsf.yaml`
+
+**Purpose**: Deploys all MOSIP core application services
+
+### Critical Configurations
+
+#### 1. Domain Replacement
+
+**Replace throughout file**:
+- `` → Your cluster name
+- `sandbox.xyz.net` → Your domain
+
+#### 2. Database Branch Configuration
+
+**Location**: Multiple locations, look for `dbBranch`
+
+```yaml
+apps:
+ config-server:
+ set:
+ # CRITICAL: Must match your MOSIP version
+ gitRepo.dbBranch: "v1.2.0.2" # ← Update to your MOSIP version
+```
+
+**How to determine correct branch:**
+
+| MOSIP Version | DB Branch |
+|---------------|-----------|
+| 1.2.0.2 | `v1.2.0.2` |
+| 1.2.0.3 | `v1.2.0.3` |
+| Latest | Check MOSIP releases |
+
+**Where to find your MOSIP version:**
+- Check Helm chart versions in DSF file
+- Refer to your deployment requirements
+- Match with MOSIP official releases
+
+#### 3. Chart Versions
+
+**Update all chart versions** to latest compatible releases:
+
+```yaml
+apps:
+ config-server:
+ version: 12.0.2 # Check for updates
+ chart: mosip/config-server
+
+ artifactory:
+ version: 12.0.1 # Check for updates
+ chart: mosip/artifactory
+
+ kernel:
+ version: 1.2.0.2 # Match MOSIP version
+ chart: mosip/kernel
+```
+
+**Where to find latest versions:**
+- MOSIP Helm Repository: https://mosip.github.io/mosip-helm
+- GitHub Releases: https://github.com/mosip/mosip-helm/releases
+
+#### 4. Resource Limits
+
+**Adjust based on environment:**
+
+```yaml
+apps:
+ prereg-application:
+ set:
+ # Resource requests (minimum guaranteed)
+ resources.requests.cpu: "500m"
+ resources.requests.memory: "1Gi"
+
+ # Resource limits (maximum allowed)
+ resources.limits.cpu: "2000m"
+ resources.limits.memory: "4Gi"
+```
+
+**Environment-based recommendations:**
+
+| Environment | CPU Request | Memory Request | CPU Limit | Memory Limit |
+|-------------|-------------|----------------|-----------|--------------|
+| **Development** | 250m | 512Mi | 1000m | 2Gi |
+| **Testing** | 500m | 1Gi | 2000m | 4Gi |
+| **Production** | 1000m | 2Gi | 4000m | 8Gi |
+
+#### 5. Replica Counts
+
+**Scale services based on load:**
+
+```yaml
+apps:
+ kernel-auth:
+ set:
+ replicaCount: 3 # Number of pod replicas
+```
+
+**Scaling recommendations:**
+
+| Service Type | Development | Testing | Production |
+|--------------|-------------|---------|------------|
+| **Auth Services** | 1 | 2 | 3-5 |
+| **Core Services** | 1 | 2 | 3 |
+| **Batch Jobs** | 1 | 1 | 1-2 |
+| **Web Apps** | 1 | 2 | 3-5 |
+
+---
+
+## Test Rigs DSF Configuration
+
+**File**: `Helmsman/dsf/testrigs-dsf.yaml`
+
+**Purpose**: Deploys automated testing infrastructure
+
+### When to Deploy
+
+**Deploy test rigs when:**
+- ✅ All MOSIP services are running
+- ✅ All pods show `Running` status
+- ✅ Partner onboarding completed successfully
+- ✅ You need automated testing
+
+**Skip test rigs when:**
+- ❌ This is a production environment
+- ❌ MOSIP services are not stable
+- ❌ You don't need automated testing
+
+### Critical Configurations
+
+#### 1. Test Endpoints
+
+**Location**: Throughout file, in API endpoint configurations
+
+```yaml
+apps:
+ apitestrig:
+ set:
+ # API endpoints for testing
+ apiEndpoint: "https://api.sandbox.xyz.net" # ← Replace domain
+
+ # Authentication endpoint
+ authEndpoint: "https://iam.sandbox.xyz.net" # ← Replace domain
+```
+
+#### 2. Test Data Configuration
+
+```yaml
+apps:
+ apitestrig:
+ set:
+ # Test data branch
+ testDataBranch: "v1.2.0.2" # ← Match MOSIP version
+```
+
+#### 3. Test Rig Resources
+
+```yaml
+apps:
+ apitestrig:
+ set:
+ resources.requests.memory: "2Gi"
+ resources.limits.memory: "4Gi"
+```
+
+**Recommendations:**
+- API Test Rig: 2-4 GB memory
+- DSL Test Rig: 4-8 GB memory
+- UI Test Rig: 4-8 GB memory
+
+---
+
+## Common Configuration Patterns
+
+### Pattern 1: Enabling/Disabling Services
+
+**To disable a service:**
+```yaml
+apps:
+ service-name:
+ enabled: false # Service will not be deployed
+```
+
+**To enable a service:**
+```yaml
+apps:
+ service-name:
+ enabled: true # Service will be deployed
+```
+
+**Common uses:**
+- Disable logging in development to save resources
+- Disable test rigs in production
+- Disable optional monitoring components
+
+---
+
+### Pattern 2: Domain Configuration
+
+**All DSF files** need consistent domain configuration:
+
+```yaml
+# prereq-dsf.yaml
+postInstall: "$WORKDIR/hooks/install-istio-and-httpbin.sh soil38 helmsman"
+
+# external-dsf.yaml
+istio.hosts:
+ - "postgres.soil38.mosip.net"
+
+# mosip-dsf.yaml
+istio.hosts:
+ - "prereg.soil38.mosip.net"
+
+# testrigs-dsf.yaml
+apiEndpoint: "https://api.soil38.mosip.net"
+```
+
+**Consistency checklist:**
+- [ ] Cluster name matches Terraform `cluster_name`
+- [ ] Domain matches Terraform `cluster_env_domain`
+- [ ] All internal references use same values
+- [ ] DNS records exist for all subdomains
+
+---
+
+### Pattern 3: Chart Version Management
+
+**Check and update versions regularly:**
+
+```yaml
+apps:
+ service-name:
+ version: 12.0.1 # Chart version
+ chart: mosip/service-name # Chart repository/name
+```
+
+**Version compatibility matrix:**
+
+| MOSIP Version | Config Server | Kernel | Auth | PreReg |
+|---------------|---------------|--------|------|--------|
+| 1.2.0.2 | 12.0.2 | 1.2.0.2 | 1.2.0.2 | 1.2.0.2 |
+| 1.2.0.3 | 12.0.3 | 1.2.0.3 | 1.2.0.3 | 1.2.0.3 |
+
+**Always check**: https://github.com/mosip/mosip-helm/releases
+
+---
+
+### Pattern 4: Priority Configuration
+
+**Deploy order is controlled by priority:**
+
+```yaml
+priority: -20 # Deploy first (highest priority)
+priority: -10 # Deploy second
+priority: 0 # Deploy last (lowest priority)
+```
+
+**Standard priorities:**
+- `-20` to `-15`: Infrastructure (databases, queues)
+- `-14` to `-10`: External services (Keycloak, MinIO)
+- `-9` to `-5`: Core services (config-server)
+- `-4` to `0`: Application services
+
+**Example:**
+```yaml
+apps:
+ postgres:
+ priority: -20 # Deploy first (database needed by all)
+
+ keycloak:
+ priority: -10 # Deploy second (auth needed by services)
+
+ config-server:
+ priority: -5 # Deploy third (config needed by apps)
+
+ prereg:
+ priority: 0 # Deploy last (depends on above)
+```
+
+---
+
+## Configuration Validation Checklist
+
+### Before Deploying prereq-dsf.yaml
+
+- [ ] `clusterid` updated with actual value from Rancher
+- [ ] `` replaced with cluster name
+- [ ] `sandbox.xyz.net` replaced with actual domain
+- [ ] Chart versions are latest stable
+- [ ] Monitoring enabled/disabled as needed
+
+### Before Deploying external-dsf.yaml
+
+- [ ] Domain placeholders replaced
+- [ ] PostgreSQL `enabled` matches Terraform setting
+- [ ] reCAPTCHA keys configured
+- [ ] MinIO storage size appropriate
+- [ ] Kafka cluster size appropriate
+- [ ] Chart versions updated
+
+### Before Deploying mosip-dsf.yaml
+
+- [ ] Domain placeholders replaced
+- [ ] Database branch matches MOSIP version
+- [ ] Chart versions compatible
+- [ ] Resource limits appropriate for environment
+- [ ] Replica counts set correctly
+
+### Before Deploying testrigs-dsf.yaml
+
+- [ ] All MOSIP services running
+- [ ] Domain placeholders replaced
+- [ ] Test data branch matches MOSIP version
+- [ ] Test endpoints correctly configured
+
+---
+
+## Troubleshooting DSF Configurations
+
+### Issue: clusterid Error
+
+**Symptom**: Monitoring dashboards show wrong cluster or no data
+
+**Solution**:
+1. Verify clusterid in Rancher UI
+2. Update both locations in prereq-dsf.yaml:
+ ```yaml
+ grafana.global.cattle.clusterId: "correct-id"
+ global.cattle.clusterId: "correct-id"
+ ```
+3. Redeploy prereq-dsf.yaml
+
+---
+
+### Issue: Domain Resolution Failures
+
+**Symptom**: Services can't be accessed via domain names
+
+**Solution**:
+1. Check DNS records in Route 53
+2. Verify domain in all DSF files matches Terraform
+3. Check subdomain configuration in Terraform:
+ ```hcl
+ subdomain_public = ["resident", "prereg", ...]
+ subdomain_internal = ["admin", "iam", ...]
+ ```
+
+---
+
+### Issue: PostgreSQL Connection Failures
+
+**Symptom**: Services can't connect to database
+
+**Solution**:
+1. Verify PostgreSQL enabled setting matches Terraform:
+ ```yaml
+ # If Terraform enable_postgresql_setup = true
+ postgres:
+ enabled: false # Use external
+
+ # If Terraform enable_postgresql_setup = false
+ postgres:
+ enabled: true # Use container
+ ```
+2. Check database branch matches MOSIP version
+3. Verify database init jobs completed successfully
+
+---
+
+### Issue: Version Incompatibilities
+
+**Symptom**: Services fail to start or crash repeatedly
+
+**Solution**:
+1. Check MOSIP version compatibility matrix
+2. Ensure all chart versions are compatible
+3. Update database branch to match service versions
+4. Refer to MOSIP release notes for breaking changes
+
+---
+
+## Quick Reference
+
+### Essential DSF Settings
+
+| Setting | File | Location | Purpose |
+|---------|------|----------|---------|
+| `clusterid` | prereq-dsf.yaml | Line ~40 | Rancher monitoring |
+| `` | All DSF files | Throughout | Cluster name |
+| `sandbox.xyz.net` | All DSF files | Throughout | Domain name |
+| `postgres.enabled` | external-dsf.yaml | Line ~50 | PostgreSQL mode |
+| `reCAPTCHA keys` | external-dsf.yaml | Line ~315 | Bot protection |
+| `dbBranch` | mosip-dsf.yaml | Multiple | DB scripts version |
+
+---
+
+## Need More Help?
+
+- **Workflow Execution**: [Workflow Guide](WORKFLOW_GUIDE.md)
+- **Secret Configuration**: [Secret Generation Guide](SECRET_GENERATION_GUIDE.md)
+- **Technical Terms**: [Glossary](GLOSSARY.md)
+- **Main Documentation**: [README](../README.md)
+
+---
+
+**Navigation**: [Back to Main README](../README.md) | [View All Docs](.)
diff --git a/docs/ENVIRONMENT_DESTRUCTION_GUIDE.md b/docs/ENVIRONMENT_DESTRUCTION_GUIDE.md
new file mode 100644
index 00000000..5dd1c9c1
--- /dev/null
+++ b/docs/ENVIRONMENT_DESTRUCTION_GUIDE.md
@@ -0,0 +1,981 @@
+# Environment Destruction Guide
+
+This guide provides comprehensive instructions for safely destroying and decommissioning MOSIP environments. Follow these steps carefully to ensure complete cleanup and avoid unexpected costs.
+
+## Table of Contents
+
+1. [Quick Destruction (No Data Backup Required)](#quick-destruction-no-data-backup-required)
+2. [When to Destroy an Environment](#when-to-destroy-an-environment)
+3. [Pre-Destruction Checklist](#pre-destruction-checklist)
+4. [Destruction Order (Critical!)](#destruction-order-critical)
+5. [Step-by-Step Destruction](#step-by-step-destruction)
+6. [Verification and Cleanup](#verification-and-cleanup)
+7. [Cost Monitoring](#cost-monitoring)
+
+---
+
+## Quick Destruction (No Data Backup Required)
+
+### Fast Complete Teardown: 6-10 Minutes
+
+If you **don't need to backup any data** and want to destroy everything quickly, use the Terraform Destroy workflow directly. This will delete all resources in one go.
+
+**Use this method when:**
+- ✅ Testing/development environment with no valuable data
+- ✅ Failed deployment that needs complete cleanup
+- ✅ Temporary demo environment
+- ✅ You've already backed up necessary data
+
+**⚠️ WARNING**: This is irreversible! All data, configurations, and resources will be permanently deleted.
+
+### Steps for Quick Destruction
+
+#### 1. Go to GitHub Actions
+
+Navigate to: **Repository → Actions → Terraform Infrastructure Destroy**
+
+#### 2. Run the Destroy Workflow
+
+Click **"Run workflow"** and configure:
+
+```
+Parameters:
+├─ Branch: release-0.1.0 (or your deployment branch)
+├─ Cloud Provider: aws
+├─ Component: infra
+└─ Backend: local (or s3, match your deployment configuration)
+```
+
+#### 3. Wait for Completion
+
+- **Total Time**: 6-10 minutes
+- Monitor the workflow logs in real-time
+- The workflow will automatically destroy:
+ - All Kubernetes resources (pods, services, deployments)
+ - RKE2 Kubernetes cluster
+ - EC2 instances (control plane and worker nodes)
+ - Load balancers and network interfaces
+ - Security groups
+ - EBS volumes
+ - All associated AWS resources
+
+#### 4. Verify Complete Deletion
+
+After the workflow completes successfully:
+
+```bash
+# Check AWS EC2 instances (should show terminated)
+aws ec2 describe-instances --filters "Name=tag:Name,Values=*mosip*" --query 'Reservations[].Instances[].[InstanceId,State.Name,Tags[?Key==`Name`].Value|[0]]' --output table
+
+# Check for any remaining resources
+aws ec2 describe-volumes --filters "Name=tag:Name,Values=*mosip*" --output table
+aws elb describe-load-balancers --output table | grep mosip
+```
+
+#### 5. Optional: Destroy Base Infrastructure
+
+If you also want to delete the VPC and networking (base-infra):
+
+1. Go to **Actions → Terraform Infrastructure Destroy**
+2. Run workflow with:
+ ```
+ Component: base-infra
+ ```
+3. Wait 5-8 minutes for completion
+
+### What Gets Deleted
+
+The quick destruction removes:
+- ✅ All Kubernetes namespaces and resources
+- ✅ MOSIP services (prereg, regproc, IDA, etc.)
+- ✅ External services (PostgreSQL, Keycloak, MinIO, Kafka)
+- ✅ Kubernetes cluster (RKE2)
+- ✅ All EC2 instances
+- ✅ Load balancers (nginx ingress)
+- ✅ EBS volumes (including database storage)
+- ✅ Security groups
+- ✅ Network interfaces
+
+### Cost Savings
+
+After quick destruction:
+- **Immediate**: All compute and storage costs stop
+- **Within hours**: AWS billing reflects terminated resources
+- **Monitor**: Check AWS Cost Explorer after 24 hours to confirm
+
+---
+
+## When to Destroy an Environment
+
+### Valid Reasons to Destroy
+
+✅ **Development/Testing environments**
+- No longer needed after testing complete
+- Recreating from scratch with new configuration
+- Cost optimization during non-working hours
+
+✅ **Temporary deployments**
+- Demo environments after presentation
+- Training environments after session ends
+- Proof-of-concept completed
+
+✅ **Failed deployments**
+- Major configuration errors requiring restart
+- Corrupted state that can't be recovered
+
+✅ **Cost management**
+- Shutting down unused environments
+- Scaling down during off-hours
+
+### Think Twice Before Destroying
+
+❌ **Production environments**
+- Contains live user data
+- Active identity records
+- Operational identity services
+
+❌ **Environments with valuable data**
+- Databases with test data you might need
+- Configurations you haven't backed up
+- Logs needed for troubleshooting
+
+---
+
+## Pre-Destruction Checklist
+
+### Before Destroying ANY Environment
+
+#### 1. Backup Critical Data
+
+```bash
+# Backup PostgreSQL databases
+kubectl exec -n postgres postgres-0 -- pg_dumpall -U postgres > mosip_backup_$(date +%Y%m%d).sql
+
+# Backup MinIO object storage
+mc alias set mosip https://minio.your-domain.net minioadmin minioadmin
+mc mirror mosip/mosip ./minio-backup/
+
+# Backup Keycloak realm configurations
+kubectl exec -n keycloak keycloak-0 -- /opt/jboss/keycloak/bin/standalone.sh \
+ -Djboss.socket.binding.port-offset=100 -Dkeycloak.migration.action=export \
+ -Dkeycloak.migration.provider=singleFile \
+ -Dkeycloak.migration.file=/tmp/keycloak-export.json
+```
+
+#### 2. Export Configurations
+
+```bash
+# Export Terraform state (if using local backend)
+cd terraform/implementations/aws/infra/
+tar -czf terraform-state-backup-$(date +%Y%m%d).tar.gz terraform.tfstate*
+
+# Export Kubernetes configurations
+kubectl get all --all-namespaces -o yaml > k8s-resources-backup.yaml
+
+# Export secrets (be careful with these!)
+kubectl get secrets --all-namespaces -o yaml > k8s-secrets-backup.yaml
+# Store securely, contains sensitive data!
+```
+
+#### 3. Document Current State
+
+Create a destruction log:
+
+```bash
+cat > destruction-log-$(date +%Y%m%d).txt << EOF
+Environment Destruction Log
+Date: $(date)
+Cluster Name:
+Domain:
+Reason:
+
+AWS Resources:
+- VPC ID:
+- Cluster Name:
+- Region:
+
+Backups Taken:
+- [ ] PostgreSQL databases
+- [ ] MinIO storage
+- [ ] Keycloak configurations
+- [ ] Terraform state
+- [ ] Kubernetes configs
+
+Destroyed By:
+EOF
+```
+
+#### 4. Notify Stakeholders
+
+- Inform team members
+- Update deployment status
+- Document in project management tools
+
+#### 5. Double-Check Environment
+
+```bash
+# Verify you're destroying the correct environment!
+kubectl config current-context
+terraform workspace show # If using workspaces
+
+# List EC2 instances to confirm
+aws ec2 describe-instances \
+ --filters "Name=tag:Environment,Values=" \
+ --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`].Value|[0],State.Name]' \
+ --output table
+```
+
+---
+
+## Destruction Order (Critical!)
+
+**IMPORTANT**: Destroy in reverse order of deployment to avoid orphaned resources and dependency issues.
+
+```
+Destruction Order (Reverse of Deployment):
+
+5. Test Rigs ← Destroy FIRST
+4. MOSIP Services ← Destroy SECOND
+3. External Services ← Destroy THIRD
+2. Prerequisites ← Destroy FOURTH
+1. Infrastructure ← Destroy FIFTH (Kubernetes cluster)
+0. Base Infrastructure ← Destroy LAST (VPC, networking)
+```
+
+**Why this order matters:**
+- Applications depend on infrastructure
+- Destroying infrastructure first leaves orphaned resources
+- Proper order ensures clean deletion of all dependencies
+
+---
+
+## Step-by-Step Destruction
+
+### Phase 1: Destroy Helmsman Deployments
+
+#### Step 1: Destroy Test Rigs (Optional, if deployed)
+
+**Time Required**: 5-10 minutes
+
+```bash
+# Option A: Using Helmsman (Recommended)
+cd Helmsman/
+helmsman --destroy -f dsf/testrigs-dsf.yaml
+
+# Option B: Using Helm directly
+helm list -n apitestrig
+helm uninstall -n apitestrig
+
+helm list -n dsltestrig
+helm uninstall -n dsltestrig
+
+helm list -n uitestrig
+helm uninstall -n uitestrig
+
+# Option C: Delete namespaces entirely
+kubectl delete namespace apitestrig
+kubectl delete namespace dsltestrig
+kubectl delete namespace uitestrig
+```
+
+**Verify deletion:**
+```bash
+kubectl get namespaces | grep testrig
+# Should return nothing
+```
+
+---
+
+#### Step 2: Destroy MOSIP Services
+
+**Time Required**: 10-20 minutes
+
+```bash
+# Option A: Using Helmsman (Recommended)
+cd Helmsman/
+helmsman --destroy -f dsf/mosip-dsf.yaml
+
+# Option B: Delete namespace with all resources
+kubectl delete namespace mosip
+
+# Watch deletion progress
+kubectl get namespace mosip -w
+# Wait until namespace is fully deleted
+```
+
+**Verify deletion:**
+```bash
+kubectl get all -n mosip
+# Should return "No resources found"
+
+kubectl get pvc -n mosip
+# Should return "No resources found"
+```
+
+---
+
+#### Step 3: Destroy External Dependencies
+
+**Time Required**: 10-20 minutes
+
+```bash
+# Option A: Using Helmsman (Recommended)
+cd Helmsman/
+helmsman --destroy -f dsf/external-dsf.yaml
+
+# Option B: Delete namespaces one by one
+kubectl delete namespace postgres
+kubectl delete namespace keycloak
+kubectl delete namespace minio
+kubectl delete namespace kafka
+kubectl delete namespace activemq
+
+# Watch for completion
+watch kubectl get namespaces
+```
+
+**Verify deletion:**
+```bash
+kubectl get pods --all-namespaces | grep -E "postgres|keycloak|minio|kafka|activemq"
+# Should return nothing
+```
+
+**Important: Persistent Volume Claims**
+```bash
+# PVCs may not be deleted automatically
+# List all PVCs
+kubectl get pvc --all-namespaces
+
+# Delete manually if they exist
+kubectl delete pvc --all -n postgres
+kubectl delete pvc --all -n minio
+kubectl delete pvc --all -n kafka
+```
+
+---
+
+#### Step 4: Destroy Prerequisites
+
+**Time Required**: 5-10 minutes
+
+```bash
+# Option A: Using Helmsman (Recommended)
+cd Helmsman/
+helmsman --destroy -f dsf/prereq-dsf.yaml
+
+# Option B: Delete monitoring and Istio
+kubectl delete namespace cattle-monitoring-system
+kubectl delete namespace cattle-logging-system
+kubectl delete namespace istio-system
+kubectl delete namespace istio-operator
+
+# Watch deletion
+watch kubectl get namespaces
+```
+
+**Verify deletion:**
+```bash
+kubectl get namespaces | grep -E "cattle|istio"
+# Should return nothing
+```
+
+---
+
+### Phase 2: Destroy Terraform Infrastructure
+
+#### Step 5: Destroy Main Infrastructure (Kubernetes Cluster)
+
+**Time Required**: 15-30 minutes
+
+##### Option A: Using GitHub Actions (Recommended)
+
+1. **Navigate to GitHub Actions**
+ ```
+ Repository → Actions → Terraform Infrastructure Destroy
+ ```
+
+2. **Run Workflow**
+ ```
+ Click: "Run workflow"
+
+ Parameters:
+ - Branch: release-0.1.0 (your deployment branch)
+ - Cloud Provider: aws
+ - Component: infra
+ - Backend: local (or s3, match your deployment)
+ ```
+
+3. **Monitor Progress**
+ - Watch workflow logs in real-time
+ - Destruction takes 15-30 minutes
+ - Verify successful completion
+
+##### Option B: Using Terraform CLI Locally
+
+**Prerequisites:**
+- WireGuard VPN active
+- AWS credentials configured
+- Terraform state accessible
+
+```bash
+# Navigate to infra directory
+cd terraform/implementations/aws/infra/
+
+# Verify Terraform state
+terraform state list
+
+# Preview destruction
+terraform plan -destroy
+
+# Destroy infrastructure
+terraform destroy -auto-approve
+
+# Watch for completion
+# This will delete:
+# - Kubernetes cluster (RKE2)
+# - EC2 instances (control plane, workers)
+# - Load balancer (nginx)
+# - PostgreSQL (if deployed via Terraform)
+# - Security groups
+# - Network interfaces
+# - EBS volumes
+```
+
+**Verify deletion in AWS Console:**
+```
+AWS Console → EC2 → Instances
+Look for: Cluster name tag
+Status: Should all be "terminated"
+```
+
+---
+
+#### Step 6: Destroy Observability Infrastructure (If deployed)
+
+**Time Required**: 10-20 minutes
+
+##### Using GitHub Actions
+
+```
+Actions → Terraform Observability Infrastructure Destroy
+Parameters:
+- Component: observ-infra
+- Others: same as infra destroy
+```
+
+##### Using Terraform CLI
+
+```bash
+cd terraform/observ-infra/aws/
+
+terraform plan -destroy
+terraform destroy -auto-approve
+```
+
+---
+
+#### Step 7: Destroy Base Infrastructure (VPC, Networking)
+
+**Time Required**: 10-15 minutes
+
+**DESTROY THIS LAST!** Base infrastructure includes VPC and networking used by all other resources.
+
+##### Using GitHub Actions
+
+```
+Actions → Terraform Base Infrastructure Destroy
+Parameters:
+- Component: base-infra
+- Others: same as previous destroys
+```
+
+##### Using Terraform CLI
+
+```bash
+cd terraform/base-infra/aws/
+
+# List resources to be destroyed
+terraform state list
+
+# Preview destruction
+terraform plan -destroy
+
+# Destroy base infrastructure
+terraform destroy -auto-approve
+
+# This will delete:
+# - VPC
+# - Subnets
+# - Internet Gateway
+# - NAT Gateway
+# - Route Tables
+# - Security Groups
+# - Jump Server
+# - WireGuard VPN
+# - Elastic IPs
+```
+
+---
+
+### Phase 3: Manual Cleanup (If Needed)
+
+#### Check for Orphaned Resources
+
+Sometimes resources aren't fully destroyed due to dependencies or errors.
+
+##### 1. Check EC2 Instances
+
+```bash
+# List instances by cluster tag
+aws ec2 describe-instances \
+ --filters "Name=tag:kubernetes.io/cluster/,Values=owned" \
+ --query 'Reservations[].Instances[].[InstanceId,State.Name,Tags[?Key==`Name`].Value|[0]]' \
+ --output table
+
+# Terminate any remaining instances
+aws ec2 terminate-instances --instance-ids
+```
+
+##### 2. Check Load Balancers
+
+```bash
+# List load balancers
+aws elbv2 describe-load-balancers --query 'LoadBalancers[].[LoadBalancerName,LoadBalancerArn]'
+
+# Delete if any exist
+aws elbv2 delete-load-balancer --load-balancer-arn
+```
+
+##### 3. Check Security Groups
+
+```bash
+# List security groups in VPC
+aws ec2 describe-security-groups \
+ --filters "Name=vpc-id,Values=" \
+ --query 'SecurityGroups[].[GroupId,GroupName]'
+
+# Delete security groups (may need to delete dependencies first)
+aws ec2 delete-security-group --group-id
+```
+
+##### 4. Check Network Interfaces
+
+```bash
+# List network interfaces
+aws ec2 describe-network-interfaces \
+ --filters "Name=vpc-id,Values=" \
+ --query 'NetworkInterfaces[].[NetworkInterfaceId,Status]'
+
+# Detach and delete if needed
+aws ec2 delete-network-interface --network-interface-id
+```
+
+##### 5. Check EBS Volumes
+
+```bash
+# List volumes
+aws ec2 describe-volumes \
+ --filters "Name=tag:kubernetes.io/cluster/,Values=owned" \
+ --query 'Volumes[].[VolumeId,State,Size]'
+
+# Delete available volumes
+aws ec2 delete-volume --volume-id
+```
+
+##### 6. Check Elastic IPs
+
+```bash
+# List Elastic IPs
+aws ec2 describe-addresses --query 'Addresses[].[PublicIp,AllocationId,AssociationId]'
+
+# Release unassociated Elastic IPs
+aws ec2 release-address --allocation-id
+```
+
+##### 7. Check Route 53 Records
+
+```bash
+# List DNS records
+aws route53 list-resource-record-sets \
+ --hosted-zone-id \
+ --query 'ResourceRecordSets[?Type==`A`].[Name,Type,ResourceRecords]'
+
+# Delete MOSIP-related records manually via AWS Console
+# Go to Route 53 → Hosted Zones → Your zone → Delete records
+```
+
+##### 8. Check S3 Buckets (If using S3 backend)
+
+```bash
+# List S3 buckets
+aws s3 ls | grep -i mosip
+
+# Delete bucket and contents (CAREFUL - THIS IS PERMANENT!)
+aws s3 rb s3:// --force
+```
+
+---
+
+## Verification and Cleanup
+
+### Final Verification Checklist
+
+After completing all destruction steps, verify nothing remains:
+
+#### AWS Resources
+
+```bash
+# Check for any remaining instances
+aws ec2 describe-instances \
+ --filters "Name=tag:Environment,Values=" \
+ --query 'Reservations[].Instances[?State.Name!=`terminated`].[InstanceId,State.Name]'
+
+# Should return empty
+
+# Check for VPCs
+aws ec2 describe-vpcs --query 'Vpcs[?Tags[?Key==`Name` && contains(Value, `mosip`)]].[VpcId,Tags]'
+
+# Should return empty (or only non-MOSIP VPCs)
+
+# Check for volumes
+aws ec2 describe-volumes \
+ --filters "Name=status,Values=available" \
+ --query 'Volumes[?Tags[?Key==`Environment` && Value==``]].[VolumeId,State]'
+
+# Should return empty
+```
+
+#### GitHub Cleanup
+
+```bash
+# Optional: Clean up environment secrets (if no longer needed)
+# Go to: Settings → Environments → Delete environment
+
+# Optional: Archive workflow runs
+# Go to: Actions → Select old runs → Delete workflow runs
+```
+
+#### Local Cleanup
+
+```bash
+# Remove local Terraform state (if using local backend and no longer needed)
+rm -rf terraform/implementations/aws/*/terraform.tfstate*
+rm -rf terraform/implementations/aws/*/.terraform/
+
+# Remove backed-up kubeconfig
+rm ~/.kube/config-
+
+# Remove WireGuard configs
+# On Linux
+sudo rm /etc/wireguard/wg-mosip.conf
+sudo wg-quick down wg-mosip
+
+# On Mac
+# Remove config from WireGuard app
+
+# On Windows
+# Remove tunnel from WireGuard app
+```
+
+---
+
+## Cost Monitoring
+
+### Monitor AWS Costs After Destruction
+
+#### Immediate Check (5 minutes after destruction)
+
+```bash
+# Check for running instances
+aws ec2 describe-instances \
+ --filters "Name=instance-state-name,Values=running" \
+ --query 'Reservations[].Instances[].[InstanceId,InstanceType,LaunchTime]'
+
+# If any instances are running that shouldn't be, investigate immediately
+```
+
+#### Daily Cost Monitoring (First 3 days)
+
+```
+1. Log into AWS Console
+2. Go to Cost Explorer or Billing Dashboard
+3. Check daily costs
+4. Look for unexpected charges
+
+Expected cost reduction:
+- Day 1: 50-70% reduction (some hourly services still billing)
+- Day 2: 80-90% reduction
+- Day 3+: 95%+ reduction (only long-term resources like S3, Route53)
+```
+
+#### What Costs Remain After Destruction
+
+**Normal remaining costs:**
+- ✅ Route 53 hosted zone: ~$0.50/month
+- ✅ S3 buckets (if keeping backups): Variable
+- ✅ Elastic IPs (if not released): $0.005/hour per IP
+- ✅ EBS snapshots (if created): Variable
+
+**Unexpected costs to investigate:**
+- ❌ Running EC2 instances
+- ❌ Load balancers still active
+- ❌ EBS volumes not deleted
+- ❌ NAT Gateways still running
+
+---
+
+## Troubleshooting Destruction Issues
+
+### Issue 1: Terraform Destroy Fails with Dependency Errors
+
+**Symptom:**
+```
+Error: cannot destroy X because Y depends on it
+```
+
+**Solution:**
+
+```bash
+# Option A: Destroy resources in specific order
+terraform destroy -target=
+terraform destroy -target=
+
+# Option B: Force resource removal from state (use carefully!)
+terraform state rm
+
+# Option C: Manual deletion in AWS Console, then clean state
+# Delete resource in AWS Console
+terraform refresh
+terraform state rm
+```
+
+---
+
+### Issue 2: Namespace Stuck in "Terminating" State
+
+**Symptom:**
+```bash
+kubectl get namespace
+NAME STATUS AGE
+mosip Terminating 30m # Stuck!
+```
+
+**Solution:**
+
+```bash
+# Method 1: Check for finalizers
+kubectl get namespace mosip -o json | jq '.spec.finalizers'
+
+# Remove finalizers
+kubectl patch namespace mosip -p '{"spec":{"finalizers":[]}}' --type=merge
+
+# Method 2: Force delete (if above doesn't work)
+kubectl delete namespace mosip --grace-period=0 --force
+
+# Method 3: Manual cleanup of stuck resources
+kubectl api-resources --verbs=list --namespaced -o name \
+ | xargs -n 1 kubectl get --show-kind --ignore-not-found -n mosip
+```
+
+---
+
+### Issue 3: PVCs Not Deleting
+
+**Symptom:**
+```bash
+kubectl get pvc -n postgres
+NAME STATUS AGE
+data-pvc Terminating 1h # Stuck!
+```
+
+**Solution:**
+
+```bash
+# Check what's using the PVC
+kubectl describe pvc data-pvc -n postgres
+
+# Remove protection
+kubectl patch pvc data-pvc -n postgres -p '{"metadata":{"finalizers":null}}'
+
+# If EBS volume stuck, delete in AWS
+aws ec2 describe-volumes --filters "Name=tag:kubernetes.io/created-for/pvc/name,Values=data-pvc"
+aws ec2 delete-volume --volume-id
+```
+
+---
+
+### Issue 4: Can't Delete VPC Due to Dependencies
+
+**Symptom:**
+```
+Error: DependencyViolation - The vpc 'vpc-xxx' has dependencies and cannot be deleted
+```
+
+**Solution:**
+
+```bash
+# 1. Check and delete network interfaces
+aws ec2 describe-network-interfaces --filters "Name=vpc-id,Values="
+# Delete each: aws ec2 delete-network-interface --network-interface-id
+
+# 2. Check and delete security groups
+aws ec2 describe-security-groups --filters "Name=vpc-id,Values="
+# Delete each: aws ec2 delete-security-group --group-id
+
+# 3. Check and delete subnets
+aws ec2 describe-subnets --filters "Name=vpc-id,Values="
+# Delete each: aws ec2 delete-subnet --subnet-id
+
+# 4. Check and delete internet gateways
+aws ec2 describe-internet-gateways --filters "Name=attachment.vpc-id,Values="
+# Detach: aws ec2 detach-internet-gateway --internet-gateway-id --vpc-id
+# Delete: aws ec2 delete-internet-gateway --internet-gateway-id
+
+# 5. Try VPC deletion again
+aws ec2 delete-vpc --vpc-id
+```
+
+---
+
+## Destruction Time Estimates
+
+| Phase | Component | Time Required |
+|-------|-----------|---------------|
+| **Phase 1** | Test Rigs | 5-10 minutes |
+| | MOSIP Services | 10-20 minutes |
+| | External Services | 10-20 minutes |
+| | Prerequisites | 5-10 minutes |
+| **Phase 2** | Main Infrastructure | 15-30 minutes |
+| | Observability Infra | 10-20 minutes |
+| | Base Infrastructure | 10-15 minutes |
+| **Phase 3** | Manual Cleanup | 10-30 minutes |
+| **Total** | Complete Destruction | **1-2 hours** |
+
+---
+
+## Quick Destruction Script (Use with Caution!)
+
+**WARNING**: This script will destroy EVERYTHING. Use only if you're absolutely sure!
+
+```bash
+#!/bin/bash
+# complete-destruction.sh
+# USE WITH EXTREME CAUTION!
+
+set -e
+
+CLUSTER_NAME="your-cluster-name"
+ENV_NAME="your-env-name"
+
+echo " WARNING: This will destroy the entire $ENV_NAME environment!"
+echo "Press Ctrl+C to cancel, or press Enter to continue..."
+read
+
+echo "Step 1/7: Destroying Test Rigs..."
+kubectl delete namespace apitestrig dsltestrig uitestrig --ignore-not-found
+
+echo "Step 2/7: Destroying MOSIP Services..."
+kubectl delete namespace mosip --ignore-not-found
+
+echo "Step 3/7: Destroying External Services..."
+kubectl delete namespace postgres keycloak minio kafka activemq --ignore-not-found
+
+echo "Step 4/7: Destroying Prerequisites..."
+kubectl delete namespace cattle-monitoring-system cattle-logging-system istio-system --ignore-not-found
+
+echo "Step 5/7: Waiting for namespace deletion..."
+sleep 60
+
+echo "Step 6/7: Destroying Kubernetes Infrastructure..."
+cd terraform/implementations/aws/infra/
+terraform destroy -auto-approve
+
+echo "Step 7/7: Destroying Base Infrastructure..."
+cd ../../base-infra/aws/
+terraform destroy -auto-approve
+
+echo "✅ Destruction complete!"
+echo "Please verify manually that all resources are deleted."
+```
+
+---
+
+## Recovery After Accidental Destruction
+
+### If You Accidentally Destroyed Production
+
+1. **Stop immediately** - Don't panic, assess damage
+2. **Check backups** - Locate most recent backups
+3. **Restore from backup**:
+
+```bash
+# Restore PostgreSQL
+kubectl exec -n postgres postgres-0 -- psql -U postgres < mosip_backup_YYYYMMDD.sql
+
+# Restore MinIO
+mc mirror ./minio-backup/ mosip/mosip
+
+# Restore Keycloak
+# Re-import realm configuration through Keycloak admin UI
+```
+
+4. **Document incident** - Write post-mortem
+5. **Implement safeguards**:
+ - Add production environment protection in GitHub
+ - Require manual approval for destruction workflows
+ - Regular backup schedules
+
+---
+
+## Best Practices
+
+### Protection Mechanisms
+
+1. **GitHub Environment Protection**
+ ```
+ Settings → Environments → Production → Protection rules
+ - Required reviewers: 2
+ - Wait timer: 5 minutes
+ - Restrict to specific branches
+ ```
+
+2. **Terraform State Locking**
+ ```hcl
+ # Use S3 backend with DynamoDB locking for production
+ backend "s3" {
+ bucket = "terraform-state"
+ key = "prod/terraform.tfstate"
+ region = "us-west-2"
+ dynamodb_table = "terraform-locks"
+ }
+ ```
+
+3. **AWS Resource Tags**
+ ```hcl
+ tags = {
+ Environment = "production"
+ Protected = "true"
+ Team = "ops"
+ }
+ ```
+
+4. **Deletion Protection**
+ ```hcl
+ # Enable termination protection on critical resources
+ resource "aws_instance" "k8s_master" {
+ disable_api_termination = true # For production only
+ }
+ ```
+
+---
+
+## Need Help?
+
+- **Before destroying production**: Get team approval
+- **Stuck during destruction**: Check troubleshooting section
+- **Unexpected costs after destruction**: Review Cost Monitoring section
+- **Need to recover**: See Recovery section
+
+---
+
+**Navigation**: [Back to Main README](../README.md) | [View All Docs](.)
diff --git a/docs/GLOSSARY.md b/docs/GLOSSARY.md
new file mode 100644
index 00000000..baf13827
--- /dev/null
+++ b/docs/GLOSSARY.md
@@ -0,0 +1,363 @@
+# Glossary of Technical Terms
+
+This glossary provides simple, beginner-friendly explanations of technical terms used throughout the MOSIP deployment documentation.
+
+## Table of Contents
+
+- [Cloud & Infrastructure Terms](#cloud--infrastructure-terms)
+- [Kubernetes & Container Terms](#kubernetes--container-terms)
+- [Deployment & Automation Terms](#deployment--automation-terms)
+- [Security & Networking Terms](#security--networking-terms)
+- [Database & Storage Terms](#database--storage-terms)
+
+---
+
+## Cloud & Infrastructure Terms
+
+### AWS (Amazon Web Services)
+A cloud computing platform provided by Amazon that offers servers, storage, databases, and other computing resources over the internet. Think of it as renting computer resources instead of buying physical servers.
+
+**Example**: Instead of buying a physical server for $10,000, you rent AWS computing power for $100/month.
+
+### Azure
+Microsoft's cloud computing platform, similar to AWS. It provides virtual machines, databases, and other cloud services.
+
+### GCP (Google Cloud Platform)
+Google's cloud computing platform, similar to AWS and Azure.
+
+### VPC (Virtual Private Cloud)
+A private, isolated section of a cloud provider's network that you control. It's like having your own private network in the cloud.
+
+**Why it matters**: VPC keeps your MOSIP deployment separate and secure from other users on the same cloud platform.
+
+### EC2 (Elastic Compute Cloud)
+AWS's virtual server service. An EC2 instance is like a computer running in the cloud that you can configure and use remotely.
+
+**Example**: Instead of a physical server in your office, you get a virtual server running in AWS's data center.
+
+### Availability Zones (AZs)
+Separate data centers within a cloud region. Using multiple availability zones provides backup if one data center has issues.
+
+**Example**: `us-east-1a`, `us-east-1b` are different data centers in the US East region.
+
+### Region
+A geographic location where cloud providers have data centers.
+
+**Example**: `us-west-2` (Oregon, USA), `ap-south-1` (Mumbai, India), `eu-west-1` (Ireland)
+
+### EBS (Elastic Block Store)
+Virtual hard drives that attach to AWS EC2 instances for data storage.
+
+**Example**: Like adding an external hard drive to your computer, but in the cloud.
+
+### AMI (Amazon Machine Image)
+A pre-configured template that contains the operating system and software needed to launch an EC2 instance.
+
+**Example**: A snapshot of a computer with Ubuntu Linux already installed and configured.
+
+### Load Balancer
+A system that distributes incoming traffic across multiple servers to prevent any single server from getting overloaded.
+
+**Example**: Like a receptionist directing customers to different service counters to avoid long queues.
+
+---
+
+## Kubernetes & Container Terms
+
+### Kubernetes (K8s)
+An open-source system for managing containerized applications across multiple servers. It automates deployment, scaling, and management of applications.
+
+**Why it matters**: MOSIP runs on Kubernetes to ensure high availability and easy scaling.
+
+### Cluster
+A group of servers (called nodes) that work together to run applications. In Kubernetes, a cluster consists of control plane nodes and worker nodes.
+
+**Example**: Instead of one powerful server, you have 5-10 servers working together as a team.
+
+### Node
+A single server (physical or virtual) in a Kubernetes cluster. Each node can run multiple application containers.
+
+### Pod
+The smallest deployable unit in Kubernetes. A pod contains one or more containers running your application.
+
+**Example**: A pod is like a house, and containers are like rooms in that house.
+
+### Namespace
+A way to divide a single Kubernetes cluster into multiple virtual clusters for organization and isolation.
+
+**Example**: Like having separate folders on your computer for work files, personal files, and photos.
+
+### Container
+A lightweight, standalone package that includes everything needed to run a piece of software (code, runtime, libraries, settings).
+
+**Example**: Like a shipping container that holds everything needed to run your application, ensuring it works the same everywhere.
+
+### Docker
+A platform for creating, deploying, and running applications in containers.
+
+### Helm
+A package manager for Kubernetes that helps you install and manage applications on Kubernetes clusters.
+
+**Example**: Like an app store for Kubernetes applications - you can install, update, and remove applications easily.
+
+### Helm Chart
+A package of pre-configured Kubernetes resources that defines how to install and configure an application.
+
+**Example**: Like a recipe that tells Helm exactly how to install and set up an application.
+
+### RKE2 (Rancher Kubernetes Engine 2)
+A Kubernetes distribution that is easy to install and manage, with enhanced security features.
+
+**Why it matters**: MOSIP uses RKE2 to create and manage Kubernetes clusters.
+
+### Rancher
+A management platform for Kubernetes that provides a user-friendly interface to manage multiple clusters.
+
+**Example**: Like a dashboard that lets you control and monitor all your Kubernetes clusters from one place.
+
+---
+
+## Deployment & Automation Terms
+
+### Terraform
+An infrastructure-as-code tool that lets you define and create cloud resources using configuration files instead of clicking buttons in a web interface.
+
+**Why it matters**: Terraform automates the creation of AWS resources (VPCs, servers, databases) needed for MOSIP.
+
+**Example**: Instead of manually clicking 50 buttons to create a server, you write a config file and Terraform creates everything automatically.
+
+### Infrastructure as Code (IaC)
+Managing and provisioning infrastructure through code files instead of manual configuration.
+
+**Benefits**: Version control, repeatability, documentation, automation.
+
+### Helmsman
+A tool that manages Kubernetes deployments using Helm charts. It reads configuration files (DSF) and ensures your desired applications are installed correctly.
+
+**Why it matters**: Helmsman automates MOSIP service deployment on Kubernetes.
+
+### DSF (Desired State File)
+A configuration file that tells Helmsman what applications to install, what versions to use, and how to configure them.
+
+**Example**: Like a shopping list that tells Helmsman: "Install PostgreSQL version 15, MinIO with 100GB storage, Kafka with 3 replicas."
+
+### GitHub Actions
+An automation platform built into GitHub that runs workflows (automated tasks) based on triggers like pushing code or clicking buttons.
+
+**Why it matters**: MOSIP uses GitHub Actions to automate Terraform and Helmsman deployments.
+
+### Workflow
+An automated process defined in GitHub Actions that performs tasks like deploying infrastructure or running tests.
+
+**Example**: A workflow might: 1) Connect to AWS, 2) Run Terraform to create servers, 3) Send you a notification when done.
+
+### CI/CD (Continuous Integration/Continuous Deployment)
+A method of frequently deploying code changes through automated testing and deployment pipelines.
+
+---
+
+## Security & Networking Terms
+
+### SSH (Secure Shell)
+A secure protocol for connecting to remote servers over a network. SSH keys are cryptographic keys used for authentication.
+
+**Example**: Like a secure tunnel between your computer and a server, where only you have the key.
+
+### SSH Key Pair
+Two related keys: a private key (kept secret) and a public key (shared with servers). Used for secure server access without passwords.
+
+**Example**: Private key is like your house key, public key is like your door lock. Only your key opens your lock.
+
+### GPG (GNU Privacy Guard)
+A tool for encrypting and signing data to keep it secure.
+
+**Why it matters**: MOSIP uses GPG to encrypt Terraform state files containing sensitive information.
+
+### WireGuard
+A modern, fast, and secure VPN (Virtual Private Network) protocol.
+
+**Why it matters**: MOSIP uses WireGuard to create secure connections to private infrastructure that isn't exposed to the internet.
+
+### Jump Server (Bastion Host)
+A special server that acts as a gateway to access other servers in a private network.
+
+**Example**: Like a secure lobby where you check in before accessing the main building.
+
+### VPN (Virtual Private Network)
+A secure, encrypted connection over the internet that makes your traffic private.
+
+**Why it matters**: WireGuard VPN lets you securely access MOSIP infrastructure from anywhere.
+
+### SSL/TLS Certificate
+Digital certificates that enable encrypted HTTPS connections to websites and services.
+
+**Example**: The padlock icon you see in your browser's address bar.
+
+### Route 53
+AWS's DNS (Domain Name System) service that translates domain names into IP addresses.
+
+**Example**: Translates `sandbox.mosip.net` into the actual server IP address like `52.23.45.67`.
+
+### Security Group
+A virtual firewall that controls what network traffic is allowed to reach your servers.
+
+**Example**: Like a security guard checking IDs - only allows traffic from approved sources.
+
+### IAM (Identity and Access Management)
+AWS's system for managing user permissions and access to cloud resources.
+
+**Example**: Like employee badges in an office - different badges give access to different rooms.
+
+---
+
+## Database & Storage Terms
+
+### PostgreSQL (Postgres)
+A powerful, open-source relational database management system.
+
+**Why it matters**: MOSIP stores all identity data, configurations, and transactions in PostgreSQL.
+
+### Database Schema
+The structure of a database - defines tables, columns, relationships, and constraints.
+
+**Example**: Like a blueprint that shows how data is organized in the database.
+
+### S3 (Simple Storage Service)
+AWS's object storage service for storing files, backups, and static assets.
+
+**Example**: Like an unlimited cloud hard drive for storing files.
+
+### MinIO
+An open-source object storage system compatible with AWS S3.
+
+**Why it matters**: MOSIP uses MinIO to store documents, photos, and files.
+
+### NFS (Network File System)
+A protocol that allows multiple servers to share and access the same files over a network.
+
+**Example**: Like a shared network drive that multiple computers can access.
+
+### Persistent Volume
+Storage in Kubernetes that persists even if containers or pods are deleted.
+
+**Example**: Like saving a file to your hard drive instead of keeping it in RAM - it survives restarts.
+
+---
+
+## MOSIP-Specific Terms
+
+### MOSIP (Modular Open Source Identity Platform)
+An open-source platform for building national identity systems. It provides services for enrollment, authentication, and identity management.
+
+### Config Server
+A centralized service that provides configuration settings to all MOSIP services.
+
+**Example**: Like a settings manager that tells all applications how to behave.
+
+### Keycloak
+An open-source identity and access management solution used by MOSIP for authentication and authorization.
+
+**Example**: The login system that checks usernames and passwords.
+
+### Kafka
+A distributed messaging system that handles communication between MOSIP services.
+
+**Example**: Like a post office that delivers messages between different applications.
+
+### ActiveMQ
+Another messaging system used by MOSIP for service-to-service communication.
+
+### Istio
+A service mesh that manages communication between microservices with features like traffic control, security, and monitoring.
+
+**Why it matters**: Istio provides networking, security, and observability for MOSIP services.
+
+### reCAPTCHA
+Google's anti-bot service that verifies users are human (the "I'm not a robot" checkbox).
+
+**Why it matters**: Protects MOSIP web interfaces from automated attacks and spam.
+
+---
+
+## Monitoring & Operations Terms
+
+### Prometheus
+An open-source monitoring system that collects and stores metrics from applications and infrastructure.
+
+**Example**: Like a health monitoring system that tracks server CPU, memory, and application performance.
+
+### Grafana
+A visualization tool that creates dashboards and charts from monitoring data.
+
+**Example**: Like a dashboard in your car showing speed, fuel, temperature - but for servers and applications.
+
+### Kibana
+A visualization tool for searching and analyzing log data from applications and servers.
+
+**Example**: Like a search engine for application logs and error messages.
+
+### Logging
+Recording events, errors, and activities that happen in applications and systems.
+
+**Why it matters**: Logs help troubleshoot issues and understand what happened when things go wrong.
+
+---
+
+## Additional Terms
+
+### API (Application Programming Interface)
+A way for different software applications to communicate with each other.
+
+**Example**: Like a waiter taking your order (request) to the kitchen (server) and bringing back your food (response).
+
+### Endpoint
+A specific URL where an API can be accessed.
+
+**Example**: `https://api.mosip.net/authenticate` is an endpoint for authentication.
+
+### Repository Secret
+A secure variable stored in GitHub that workflows can access but are hidden from view.
+
+**Example**: Storing your AWS password securely so GitHub Actions can use it without exposing it.
+
+### Environment Secret
+A secret that is specific to a deployment environment (like development, staging, production).
+
+**Example**: Different database passwords for development vs. production environments.
+
+### Dry Run
+A test mode where commands are simulated without making actual changes.
+
+**Example**: Like a dress rehearsal before the actual performance - you see what would happen without actually doing it.
+
+### State File (Terraform)
+A file that tracks what infrastructure Terraform has created so it knows what to update or delete.
+
+**Example**: Like a inventory list of everything Terraform has built.
+
+---
+
+## Quick Reference
+
+| Term | Simple Explanation | Why It Matters |
+|------|-------------------|----------------|
+| **AWS** | Amazon's cloud computing platform | Where MOSIP infrastructure runs |
+| **Kubernetes** | Container orchestration platform | Manages MOSIP applications |
+| **Terraform** | Infrastructure automation tool | Creates AWS resources automatically |
+| **Helmsman** | Kubernetes deployment manager | Deploys MOSIP services |
+| **VPC** | Private cloud network | Keeps MOSIP secure and isolated |
+| **WireGuard** | VPN protocol | Secure access to private infrastructure |
+| **PostgreSQL** | Database system | Stores MOSIP data |
+| **DSF** | Deployment configuration file | Tells Helmsman what to install |
+
+---
+
+## Need More Help?
+
+- **Read the Documentation**: Each section of the README provides detailed explanations
+- **Ask Questions**: Open a GitHub issue if you need clarification
+- **Community Support**: Join MOSIP community channels for assistance
+
+---
+
+**Navigation**: [Back to Main README](../README.md)
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 00000000..b80cb5e5
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,306 @@
+# MOSIP Infrastructure Documentation Index
+
+Welcome to the MOSIP Infrastructure documentation! This index helps you find exactly what you need, whether you're a complete beginner or an experienced DevOps engineer.
+
+## Quick Navigation
+
+### For Complete Beginners
+
+**Start your MOSIP deployment journey here:**
+
+1. **[Glossary](GLOSSARY.md)** - Learn all the technical terms
+ - What is AWS? What is Kubernetes? What is Terraform?
+ - Plain language explanations for every concept
+ - No prior cloud knowledge required
+
+2. **[Secret Generation Guide](SECRET_GENERATION_GUIDE.md)** - Create required credentials
+ - Step-by-step SSH key generation
+ - How to get AWS credentials
+ - Creating passwords and VPN configs
+ - Includes links to official documentation
+
+3. **[Workflow Guide](WORKFLOW_GUIDE.md)** - Run deployments through GitHub
+ - Visual walkthrough of GitHub Actions interface
+ - Where to click and what to select
+ - Understanding dry-run vs actual deployment
+ - Screenshots and examples
+
+4. **[DSF Configuration Guide](DSF_CONFIGURATION_GUIDE.md)** - Configure MOSIP services
+ - What is a DSF file and why you need it
+ - How to find and update clusterid
+ - Domain configuration explained
+ - Examples for each setting
+
+5. **[Environment Destruction Guide](ENVIRONMENT_DESTRUCTION_GUIDE.md)** - Clean up resources
+ - Safe teardown procedures
+ - How to avoid unexpected costs
+ - Backup before destruction
+ - Complete cleanup verification
+
+### For Experienced Users
+
+**Jump directly to what you need:**
+
+- **Terraform Infrastructure**: See [terraform/README.md](../terraform/README.md)
+- **Helmsman Deployment**: See [Helmsman/README.md](../Helmsman/README.md)
+- **WireGuard VPN Setup**: See [terraform/base-infra/WIREGUARD_SETUP.md](../terraform/base-infra/WIREGUARD_SETUP.md)
+- **GitHub Actions Workflows**: See [.github/workflows/]../.github/workflows/)
+
+---
+
+## Complete Documentation List
+
+### Core Deployment Guides
+
+| Document | Description | Who Should Read |
+|----------|-------------|-----------------|
+| **[Main README](../README.md)** | Complete deployment overview and quick start guide | Everyone - start here |
+| **[Glossary](GLOSSARY.md)** | Technical terms explained in plain language | Beginners |
+| **[Secret Generation Guide](SECRET_GENERATION_GUIDE.md)** | How to create all required secrets and credentials | Everyone before deployment |
+| **[Workflow Guide](WORKFLOW_GUIDE.md)** | Visual GitHub Actions workflow navigation | Everyone during deployment |
+| **[DSF Configuration Guide](DSF_CONFIGURATION_GUIDE.md)** | Helmsman configuration file explained in detail | Everyone before Helmsman deployment |
+| **[Environment Destruction Guide](ENVIRONMENT_DESTRUCTION_GUIDE.md)** | Safe resource cleanup and cost management | Everyone when decommissioning |
+
+### Component-Specific Documentation
+
+| Component | Document | What It Covers |
+|-----------|----------|----------------|
+| **Terraform** | [terraform/README.md](../terraform/README.md) | Infrastructure as code, tfvars, modules, state management |
+| **Helmsman** | [Helmsman/README.md](../Helmsman/README.md) | Application deployment, DSF files, hooks, troubleshooting |
+| **WireGuard** | [terraform/base-infra/WIREGUARD_SETUP.md](../terraform/base-infra/WIREGUARD_SETUP.md) | VPN setup, peer configuration, client installation |
+| **Workflows** | [Helmsman/helmsman-workflow-guide.md](../Helmsman/helmsman-workflow-guide.md) | Helmsman workflow details |
+
+---
+
+## Learning Paths
+
+### Path 1: "I'm New to Cloud/DevOps"
+
+**Time required:** 2-3 days for first deployment
+
+1. **Day 1: Learning & Setup**
+ - [ ] Read [Glossary](GLOSSARY.md) - Understand all terms (2 hours)
+ - [ ] Read [Main README](../README.md) - Get deployment overview (1 hour)
+ - [ ] Follow [Secret Generation Guide](SECRET_GENERATION_GUIDE.md) - Create all secrets (2 hours)
+ - [ ] Set up AWS account and configure IAM permissions
+
+2. **Day 2: Infrastructure Deployment**
+ - [ ] Follow [Workflow Guide](WORKFLOW_GUIDE.md) - Deploy base-infra (1 hour)
+ - [ ] Follow [WireGuard Setup](../terraform/base-infra/WIREGUARD_SETUP.md) - Configure VPN (1 hour)
+ - [ ] Deploy main infrastructure (1 hour)
+ - [ ] Verify all components working
+
+3. **Day 3: Application Deployment**
+ - [ ] Follow [DSF Configuration Guide](DSF_CONFIGURATION_GUIDE.md) - Update configs (2 hours)
+ - [ ] Deploy prerequisites and external services (1 hour)
+ - [ ] Deploy MOSIP core services (2 hours)
+ - [ ] Verify deployment and test access
+
+### Path 2: "I Know Cloud but New to MOSIP"
+
+**Time required:** 4-6 hours for first deployment
+
+1. **Planning (30 minutes)**
+ - [ ] Skim [Main README](../README.md) - Understand MOSIP architecture
+ - [ ] Review [Glossary](GLOSSARY.md) - MOSIP-specific terms only
+ - [ ] Check [Secret Generation Guide](SECRET_GENERATION_GUIDE.md) - What secrets are needed
+
+2. **Infrastructure (2 hours)**
+ - [ ] Configure terraform.tfvars files
+ - [ ] Run Terraform workflows
+ - [ ] Set up WireGuard VPN
+
+3. **Applications (2-3 hours)**
+ - [ ] Update DSF files using [DSF Configuration Guide](DSF_CONFIGURATION_GUIDE.md)
+ - [ ] Run Helmsman workflows
+ - [ ] Verify deployment
+
+### Path 3: "I've Deployed MOSIP Before"
+
+**Quick reference for common tasks:**
+
+- **New deployment**: Follow Quick Start in [Main README](../README.md)
+- **Updating configurations**: [DSF Configuration Guide](DSF_CONFIGURATION_GUIDE.md)
+- **Troubleshooting**: Check component-specific READMEs
+- **Cleanup**: [Environment Destruction Guide](ENVIRONMENT_DESTRUCTION_GUIDE.md)
+
+---
+
+## Find What You Need
+
+### By Task
+
+| I Want To... | Read This |
+|--------------|-----------|
+| Understand what "Kubernetes" means | [Glossary](GLOSSARY.md#kubernetes-k8s) |
+| Generate SSH keys | [Secret Generation Guide - SSH Keys](SECRET_GENERATION_GUIDE.md#1-ssh-keys) |
+| Get AWS credentials | [Secret Generation Guide - AWS Credentials](SECRET_GENERATION_GUIDE.md#3-aws-credentials) |
+| Run my first workflow | [Workflow Guide - Base Infrastructure](WORKFLOW_GUIDE.md#workflow-1-base-infrastructure) |
+| Understand dry-run vs apply | [Workflow Guide - Parameters](WORKFLOW_GUIDE.md#terraform-apply-checkbox) |
+| Fix "clusterid not found" error | [DSF Configuration Guide - clusterid](DSF_CONFIGURATION_GUIDE.md#critical-configuration-clusterid) |
+| Configure PostgreSQL | [DSF Configuration Guide - PostgreSQL](DSF_CONFIGURATION_GUIDE.md#2-postgresql-configuration) |
+| Add reCAPTCHA keys | [DSF Configuration Guide - reCAPTCHA](DSF_CONFIGURATION_GUIDE.md#3-recaptcha-configuration) |
+| Delete my environment | [Environment Destruction Guide](ENVIRONMENT_DESTRUCTION_GUIDE.md) |
+| Check costs after deletion | [Environment Destruction Guide - Cost Monitoring](ENVIRONMENT_DESTRUCTION_GUIDE.md#cost-monitoring) |
+| Understand workflow names | [Workflow Guide - Understanding Workflow Names](WORKFLOW_GUIDE.md#understanding-workflow-names) |
+
+### By Technology
+
+| Technology | Learn About It | Use It |
+|------------|----------------|--------|
+| **AWS** | [Glossary - AWS](GLOSSARY.md#aws-amazon-web-services) | [Main README - AWS Setup](../README.md#prerequisites) |
+| **Terraform** | [Glossary - Terraform](GLOSSARY.md#terraform) | [Terraform README](../terraform/README.md) |
+| **Kubernetes** | [Glossary - Kubernetes](GLOSSARY.md#kubernetes-k8s) | All deployment guides |
+| **Helmsman** | [Glossary - Helmsman](GLOSSARY.md#helmsman) | [DSF Configuration Guide](DSF_CONFIGURATION_GUIDE.md) |
+| **WireGuard** | [Glossary - WireGuard](GLOSSARY.md#wireguard) | [WireGuard Setup Guide](../terraform/base-infra/WIREGUARD_SETUP.md) |
+| **GitHub Actions** | [Glossary - GitHub Actions](GLOSSARY.md#github-actions) | [Workflow Guide](WORKFLOW_GUIDE.md) |
+
+### By Error Message
+
+| Error | Solution |
+|-------|----------|
+| "clusterid not found" | [DSF Configuration Guide - clusterid](DSF_CONFIGURATION_GUIDE.md#critical-configuration-clusterid) |
+| "Authentication failed" | [Secret Generation Guide - AWS Credentials](SECRET_GENERATION_GUIDE.md#3-aws-credentials) |
+| "InsufficientInstanceCapacity" | [Main README - AWS Capacity Issues](../README.md#aws-capacity-issues) |
+| "Namespace stuck in Terminating" | [Environment Destruction Guide - Troubleshooting](ENVIRONMENT_DESTRUCTION_GUIDE.md#issue-2-namespace-stuck-in-terminating-state) |
+| "Can't find workflow" | [Workflow Guide - Issue 1](WORKFLOW_GUIDE.md#issue-1-workflow-not-found) |
+| "Helmsman dry-run fails" | [Workflow Guide - Issue 5](WORKFLOW_GUIDE.md#issue-5-helmsman-dry-run-fails) |
+
+---
+
+## Documentation Features
+
+### What Makes Our Docs Beginner-Friendly?
+
+✅ **Plain Language**
+- Every technical term explained
+- No assumptions about prior knowledge
+- Real-world analogies and examples
+
+✅ **Step-by-Step Instructions**
+- Numbered steps you can follow exactly
+- "What you should see" at each step
+- Clear success/failure indicators
+
+✅ **Visual Guidance**
+- Where to click in GitHub interface
+- What buttons to press
+- Expected output examples
+
+✅ **Comprehensive Examples**
+- Real configuration examples
+- Before/after comparisons
+- Common patterns explained
+
+✅ **Links to Official Docs**
+- Every tool linked to official documentation
+- Additional learning resources
+- Community support channels
+
+✅ **Troubleshooting Sections**
+- Common errors and solutions
+- "What went wrong" explanations
+- Recovery procedures
+
+---
+
+## Getting Help
+
+### When You're Stuck
+
+1. **Check the relevant guide** - Use the tables above to find what you need
+2. **Search for error messages** - Use Ctrl+F in documentation
+3. **Review troubleshooting sections** - Each guide has a troubleshooting section
+4. **Check official documentation** - Follow links to tool-specific docs
+5. **Ask the community** - Open a GitHub issue with details
+
+### What to Include When Asking for Help
+
+- **What you're trying to do**: "Deploy base infrastructure"
+- **What guide you're following**: "Step 3a in Main README"
+- **What happened**: Error message or unexpected behavior
+- **What you expected**: What you thought would happen
+- **What you've tried**: Steps you took to fix it
+
+---
+
+## Documentation Conventions
+
+### Symbols Used
+
+| Symbol | Meaning |
+|--------|---------|
+| | Important for beginners |
+| | Helpful tip or explanation |
+| | Warning - pay attention! |
+| ✅ | Recommended action |
+| ❌ | Action to avoid |
+| | Link to external documentation |
+| | Link to our documentation |
+| | Step-by-step guide available |
+| | Security-related |
+| | Deletion/cleanup related |
+
+### Text Formatting
+
+- **Bold**: Important terms, action items
+- `Code`: Commands, file names, values to copy
+- > Blockquotes: Important notes, warnings
+- ```code blocks```: Multi-line commands, configuration examples
+
+---
+
+## Contributing to Documentation
+
+Found something unclear? Want to add more examples? Contributions welcome!
+
+1. **Report issues**: Open a GitHub issue describing what's confusing
+2. **Suggest improvements**: What would make docs clearer?
+3. **Share your experience**: What worked? What didn't?
+
+---
+
+## Quick Reference Cards
+
+### Essential Commands Cheat Sheet
+
+```bash
+# Check Kubernetes cluster
+kubectl get nodes
+kubectl get namespaces
+kubectl get pods --all-namespaces
+
+# Check specific services
+kubectl get pods -A
+kubectl get pods -n postgres
+kubectl get svc -n istio-system
+
+# View logs
+kubectl logs -n
+kubectl logs -n --follow
+
+# Describe resources
+kubectl describe pod -n
+kubectl describe svc -n
+```
+
+### Deployment Checklist
+
+- [ ] Read [Glossary](GLOSSARY.md) if new to cloud
+- [ ] Generate all secrets ([Guide](SECRET_GENERATION_GUIDE.md))
+- [ ] Configure terraform.tfvars files
+- [ ] Deploy base infrastructure ([Workflow Guide](WORKFLOW_GUIDE.md))
+- [ ] Set up WireGuard VPN ([Setup Guide](../terraform/base-infra/WIREGUARD_SETUP.md))
+- [ ] Deploy main infrastructure ([Workflow Guide](WORKFLOW_GUIDE.md))
+- [ ] Update DSF files ([DSF Guide](DSF_CONFIGURATION_GUIDE.md))
+- [ ] Deploy prerequisites & external services
+- [ ] Deploy MOSIP core services
+- [ ] Deploy test rigs (optional)
+- [ ] Verify deployment
+- [ ] Document your deployment
+
+---
+
+**Need to go back?** [Return to Main README](../README.md)
+
+**Have questions?** Open an issue on GitHub!
diff --git a/docs/SECRET_GENERATION_GUIDE.md b/docs/SECRET_GENERATION_GUIDE.md
new file mode 100644
index 00000000..f59a6d5f
--- /dev/null
+++ b/docs/SECRET_GENERATION_GUIDE.md
@@ -0,0 +1,666 @@
+# Secret Generation and Configuration Guide
+
+This guide provides step-by-step instructions for generating all required secrets for MOSIP deployment. Each section explains what the secret is for, how to generate it, and where to use it.
+
+## Table of Contents
+
+1. [SSH Keys](#1-ssh-keys)
+2. [GPG Passphrase](#2-gpg-passphrase)
+3. [AWS Credentials](#3-aws-credentials)
+4. [WireGuard VPN Configuration](#4-wireguard-vpn-configuration)
+5. [Kubernetes Config (KUBECONFIG)](#5-kubernetes-config-kubeconfig)
+6. [reCAPTCHA Keys](#6-recaptcha-keys)
+7. [How to Add Secrets to GitHub](#how-to-add-secrets-to-github)
+
+---
+
+## 1. SSH Keys
+
+### What is it?
+SSH (Secure Shell) keys are used to securely connect to your servers without using passwords. You need SSH keys to access EC2 instances created by Terraform.
+
+### Why do you need it?
+- To access jump servers and Kubernetes nodes
+- For automated deployments via GitHub Actions
+- For troubleshooting and maintenance tasks
+
+### How to Generate SSH Keys
+
+#### On Linux/Mac:
+
+```bash
+# Open terminal and run:
+ssh-keygen -t rsa -b 4096 -C "your-email@example.com" -f ~/.ssh/mosip-aws
+
+# This creates two files:
+# - ~/.ssh/mosip-aws (private key - keep this SECRET)
+# - ~/.ssh/mosip-aws.pub (public key - this is safe to share)
+```
+
+#### On Windows:
+
+**Option 1: Using Git Bash or WSL**
+```bash
+# Open Git Bash or WSL terminal
+ssh-keygen -t rsa -b 4096 -C "your-email@example.com" -f ~/.ssh/mosip-aws
+```
+
+**Option 2: Using PuTTYgen**
+1. Download and install [PuTTY](https://www.putty.org/)
+2. Open PuTTYgen
+3. Click "Generate" and move mouse randomly
+4. Set key passphrase (optional but recommended)
+5. Save private key (keep secret)
+6. Copy public key text for AWS
+
+### Official Documentation
+- **GitHub SSH Guide**: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
+- **AWS EC2 Key Pairs**: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html
+
+### Where to Use It
+
+#### 1. Add Public Key to AWS
+```bash
+# First, upload the PUBLIC key to AWS:
+# 1. Go to AWS Console > EC2 > Key Pairs
+# 2. Click "Actions" > "Import key pair"
+# 3. Name it (e.g., "mosip-aws")
+# 4. Paste contents of ~/.ssh/mosip-aws.pub
+# 5. Click "Import"
+```
+
+#### 2. Add Private Key to GitHub Secrets
+```bash
+# Copy private key content
+cat ~/.ssh/mosip-aws
+
+# Add as Repository Secret in GitHub:
+# Name: mosip-aws (must match ssh_key_name in terraform.tfvars)
+# Value: (paste the entire private key including BEGIN and END lines)
+```
+
+#### 3. Update Terraform Configuration
+```hcl
+# In terraform/implementations/aws/infra/aws.tfvars
+ssh_key_name = "mosip-aws" # Must match the name in AWS and GitHub secret
+```
+
+### Common Pitfalls
+- ❌ Using the public key as GitHub secret (use private key!)
+- ❌ Mismatched names between AWS, GitHub secret, and tfvars
+- ❌ Not including BEGIN/END lines when copying private key
+- ❌ Adding extra spaces or newlines when pasting key
+
+---
+
+## 2. GPG Passphrase
+
+### What is it?
+GPG (GNU Privacy Guard) is used to encrypt sensitive Terraform state files when using local backend storage.
+
+### Why do you need it?
+- To encrypt Terraform state files that contain sensitive information
+- To prevent unauthorized access to infrastructure secrets
+- Required when using `local` backend option
+
+### How to Generate
+
+**Choose a strong passphrase that:**
+- Is at least 16 characters long
+- Contains uppercase and lowercase letters
+- Contains numbers and special characters
+- Is unique (not used elsewhere)
+- Is memorable (you may need it for recovery)
+
+**Example Strong Passphrase:**
+```
+MyM0s1p$ecur3!2024Deploy
+```
+
+**Password Generation Tools:**
+- Online: https://www.random.org/passwords/
+- Command line (Linux/Mac): `openssl rand -base64 32`
+- Password managers: 1Password, LastPass, Bitwarden
+
+### Official Documentation
+- **GPG Official Guide**: https://gnupg.org/gph/en/manual/c14.html
+
+### Where to Use It
+Add as **Repository Secret** in GitHub:
+- **Name**: `GPG_PASSPHRASE`
+- **Value**: Your generated passphrase
+
+### Common Pitfalls
+- ❌ Using a weak/short passphrase
+- ❌ Forgetting the passphrase (store it in a password manager!)
+- ❌ Sharing the passphrase in insecure locations
+
+---
+
+## 3. AWS Credentials
+
+### What is it?
+AWS Access Keys are credentials that allow programmatic access to your AWS account.
+
+### Why do you need it?
+- To allow Terraform to create AWS resources
+- To allow GitHub Actions to deploy infrastructure
+- For automated AWS API calls
+
+### How to Generate AWS Access Keys
+
+#### Step-by-Step:
+
+1. **Log in to AWS Console**
+ - Go to https://console.aws.amazon.com/
+
+2. **Navigate to IAM**
+ - Search for "IAM" in the top search bar
+ - Click "IAM" (Identity and Access Management)
+
+3. **Create or Select User**
+ - Option A: Use existing IAM user
+ - Option B: Create new user:
+ - Click "Users" > "Create user"
+ - Enter username (e.g., "mosip-deployer")
+ - Click "Next"
+
+4. **Attach Permissions**
+ - Click "Attach policies directly"
+ - Search and select these policies:
+ - `AmazonEC2FullAccess`
+ - `AmazonVPCFullAccess`
+ - `AmazonRoute53FullAccess`
+ - `IAMFullAccess`
+ - `AmazonS3FullAccess`
+ - Click "Next" > "Create user"
+
+5. **Create Access Key**
+ - Click on the user name
+ - Click "Security credentials" tab
+ - Scroll to "Access keys" section
+ - Click "Create access key"
+ - Select use case: "Application running outside AWS"
+ - Click "Next"
+ - Add description tag (optional): "MOSIP Terraform Deployment"
+ - Click "Create access key"
+
+6. **Save Credentials Securely**
+ - **IMPORTANT**: This is the ONLY time you can view the secret key!
+ - Download CSV file or copy both keys
+ - Store in password manager
+
+**You will get:**
+```
+Access Key ID: AKIAIOSFODNN7EXAMPLE
+Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+```
+
+### Official Documentation
+- **AWS Access Keys Guide**: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
+- **AWS IAM Best Practices**: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html
+
+### Where to Use It
+Add as **Repository Secrets** in GitHub:
+
+1. **AWS_ACCESS_KEY_ID**
+ - Value: `AKIAIOSFODNN7EXAMPLE` (your actual access key ID)
+
+2. **AWS_SECRET_ACCESS_KEY**
+ - Value: `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` (your actual secret)
+
+### Security Best Practices
+- ✅ Use IAM user (not root account) for deployment
+- ✅ Enable MFA (Multi-Factor Authentication) on IAM user
+- ✅ Rotate access keys every 90 days
+- ✅ Use least-privilege permissions (only required policies)
+- ✅ Monitor AWS CloudTrail for unauthorized usage
+- ❌ Never commit keys to Git repositories
+- ❌ Never share keys via email or chat
+- ❌ Never use root account credentials
+
+### Common Pitfalls
+- ❌ Using root account credentials (use IAM user instead)
+- ❌ Not saving the secret access key (can't retrieve it later!)
+- ❌ Insufficient permissions (deployment will fail)
+- ❌ Committing credentials to Git (use secrets!)
+
+---
+
+## 4. WireGuard VPN Configuration
+
+### What is it?
+WireGuard is a modern VPN that creates secure connections to your private infrastructure.
+
+### Why do you need it?
+- To access Kubernetes clusters via private IP addresses
+- To deploy services securely without exposing infrastructure to the internet
+- For encrypted communication with infrastructure
+
+### How to Generate WireGuard Config
+
+WireGuard configuration is generated **AFTER** deploying base infrastructure. Follow these steps:
+
+#### Step 1: Deploy Base Infrastructure First
+```bash
+# Complete Terraform base-infra deployment
+# This creates the jump server with WireGuard installed
+```
+
+#### Step 2: Follow Detailed Setup Guide
+**Complete WireGuard Setup Guide**: [WIREGUARD_SETUP.md](../terraform/base-infra/WIREGUARD_SETUP.md)
+
+#### Quick Overview:
+
+1. **SSH to Jump Server**
+ ```bash
+ ssh -i ~/.ssh/mosip-aws ubuntu@
+ ```
+
+2. **Generate Peer Configurations**
+ ```bash
+ # On jump server
+ sudo wg-quick down wg0
+ sudo nano /etc/wireguard/wg0.conf
+
+ # Add peer configuration
+ [Peer]
+ PublicKey =
+ AllowedIPs = 10.0.1.2/32
+ ```
+
+3. **Get Client Configuration**
+ - The jump server provides pre-configured peer templates
+ - Customize with your private key and IP address
+
+4. **Install WireGuard Client**
+ - **Windows**: https://www.wireguard.com/install/ (download MSI installer)
+ - **Mac**: `brew install wireguard-tools` or Mac App Store
+ - **Linux**: `sudo apt install wireguard` (Ubuntu/Debian)
+
+5. **Import Configuration**
+ - Open WireGuard client
+ - Import tunnel from file or paste configuration
+ - Activate tunnel
+
+### Official Documentation
+- **WireGuard Official**: https://www.wireguard.com/quickstart/
+- **WireGuard Installation**: https://www.wireguard.com/install/
+
+### Where to Use It
+
+Add as **Environment Secrets** in GitHub (not repository secrets):
+
+1. **TF_WG_CONFIG** - For Terraform deployments
+ ```ini
+ [Interface]
+ PrivateKey =
+ Address = 10.0.1.2/24
+ DNS = 10.0.0.2
+
+ [Peer]
+ PublicKey =
+ Endpoint = :51820
+ AllowedIPs = 10.0.0.0/16
+ PersistentKeepalive = 25
+ ```
+
+2. **CLUSTER_WIREGUARD_WG0** - For Helmsman cluster access (peer1)
+3. **CLUSTER_WIREGUARD_WG1** - For Helmsman cluster access (peer2, optional)
+
+### Common Pitfalls
+- ❌ Trying to create WireGuard config before deploying base-infra
+- ❌ Not activating WireGuard tunnel when testing connectivity
+- ❌ Using same peer configuration for different purposes
+- ❌ Incorrect AllowedIPs range (should match VPC CIDR)
+- ❌ Firewall blocking UDP port 51820
+
+---
+
+## 5. Kubernetes Config (KUBECONFIG)
+
+### What is it?
+KUBECONFIG is a configuration file that contains credentials and connection details for your Kubernetes cluster.
+
+### Why do you need it?
+- To allow Helmsman to deploy applications to your Kubernetes cluster
+- For kubectl command-line access
+- For automated deployments via GitHub Actions
+
+### How to Get KUBECONFIG
+
+KUBECONFIG is **automatically generated** by Terraform after deploying infrastructure.
+
+#### Step 1: Deploy Infrastructure First
+```bash
+# Complete Terraform infra deployment
+# Wait for workflow to complete successfully
+```
+
+#### Step 2: Locate KUBECONFIG File
+
+The file is created in your Terraform outputs:
+
+```bash
+# Location in repository:
+terraform/implementations/aws/infra/kubeconfig_
+
+# Example:
+terraform/implementations/aws/infra/kubeconfig_soil38
+```
+
+#### Step 3: Download KUBECONFIG
+
+**Option 1: From GitHub Actions Artifacts**
+1. Go to your GitHub repository
+2. Click "Actions" tab
+3. Find the completed "Terraform Infrastructure" workflow
+4. Scroll to "Artifacts" section at the bottom
+5. Download artifact containing kubeconfig
+
+**Option 2: From Terraform Outputs**
+```bash
+# View kubeconfig content
+cd terraform/implementations/aws/infra/
+cat kubeconfig_
+```
+
+#### Step 4: Test KUBECONFIG Locally (Optional)
+
+```bash
+# Set kubeconfig path
+export KUBECONFIG=/path/to/kubeconfig_soil38
+
+# Test connectivity
+kubectl get nodes
+
+# You should see your cluster nodes listed
+```
+
+### Official Documentation
+- **Kubernetes Config**: https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/
+- **kubectl Installation**: https://kubernetes.io/docs/tasks/tools/
+
+### Where to Use It
+
+Add as **Environment Secret** in GitHub (not repository secret):
+
+**Secret Name**: `KUBECONFIG`
+
+**Secret Value**: Complete contents of the kubeconfig file
+
+```yaml
+# Example format (your actual values will be different):
+apiVersion: v1
+clusters:
+- cluster:
+ certificate-authority-data: LS0tLS1CRUdJTi...
+ server: https://10.0.1.10:6443
+ name: soil38
+contexts:
+- context:
+ cluster: soil38
+ user: soil38
+ name: soil38
+current-context: soil38
+kind: Config
+preferences: {}
+users:
+- name: soil38
+ user:
+ client-certificate-data: LS0tLS1CRUdJTi...
+ client-key-data: LS0tLS1CRUdJTi...
+```
+
+### Common Pitfalls
+- ❌ Trying to get KUBECONFIG before infrastructure is deployed
+- ❌ Adding KUBECONFIG as repository secret (should be environment secret)
+- ❌ Not including complete file content (missing BEGIN/END lines)
+- ❌ Using KUBECONFIG from wrong cluster
+- ❌ WireGuard VPN not active when testing kubectl access
+
+---
+
+## 6. reCAPTCHA Keys
+
+### What is it?
+reCAPTCHA is Google's service that protects websites from bots and spam by verifying users are human.
+
+### Why do you need it?
+- To protect MOSIP web portals (PreReg, Admin, Resident) from automated attacks
+- To prevent spam registrations and malicious bots
+- Required for production deployments
+
+### How to Generate reCAPTCHA Keys
+
+You need **separate reCAPTCHA keys for each MOSIP portal**:
+- PreReg portal: `prereg.your-domain.net`
+- Admin portal: `admin.your-domain.net`
+- Resident portal: `resident.your-domain.net`
+
+#### Step-by-Step for Each Domain:
+
+1. **Go to Google reCAPTCHA Admin Console**
+ - Visit: https://www.google.com/recaptcha/admin/create
+ - Sign in with your Google account
+
+2. **Create New Site**
+ - **Label**: `MOSIP PreReg` (or Admin/Resident)
+ - **reCAPTCHA type**: Select **"reCAPTCHA v2"**
+ - **Sub-type**: Select **"I'm not a robot" Checkbox**
+
+3. **Add Domain**
+ - Enter your domain: `prereg.your-domain.net`
+ - Example: `prereg.soil.mosip.net`
+ - Click "+" to add
+
+4. **Accept Terms**
+ - Check "Accept the reCAPTCHA Terms of Service"
+ - Click "Submit"
+
+5. **Save Keys**
+ You will receive two keys:
+ ```
+ Site Key: 6LfkAMwrAAAAAATB1WhkIhzuAVMtOs9VWabODoZ_
+ Secret Key: 6LfkAMwrAAAAAHQAT93nTGcLKa-h3XYhGoNSG-NL
+ ```
+
+6. **Repeat for Other Domains**
+ - Create separate reCAPTCHA for Admin portal
+ - Create separate reCAPTCHA for Resident portal
+
+### Official Documentation
+- **Google reCAPTCHA Documentation**: https://developers.google.com/recaptcha/intro
+- **reCAPTCHA v2 Guide**: https://developers.google.com/recaptcha/docs/display
+
+### Where to Use It
+
+Update in `Helmsman/dsf/external-dsf.yaml` file:
+
+```yaml
+# Around line 315 in external-dsf.yaml
+hooks:
+ postInstall: "$WORKDIR/hooks/captcha-setup.sh PREREG_SITE_KEY PREREG_SECRET_KEY ADMIN_SITE_KEY ADMIN_SECRET_KEY RESIDENT_SITE_KEY RESIDENT_SECRET_KEY"
+```
+
+**Replace placeholders with actual keys:**
+
+```yaml
+hooks:
+ postInstall: "$WORKDIR/hooks/captcha-setup.sh 6LfkAMwrAAAAAATB1WhkIhzuAVMtOs9VWabODoZ_ 6LfkAMwrAAAAAHQAT93nTGcLKa-h3XYhGoNSG-NL 6LdNAcwrAAAAAETGWvz-3I12vZ5V8vPJLu2ct9CO 6LdNAcwrAAAAAE4iWGJ-g6Dc2HreeJdIwAl5h1iL 6LdRAcwrAAAAAFUEHHKK5D_bSrwAPqdqAJqo4mCk 6LdRAcwrAAAAAOeVl6yHGBCBA8ye9GsUOy4pi9s9"
+```
+
+**Key Order:**
+1. PreReg Site Key
+2. PreReg Secret Key
+3. Admin Site Key
+4. Admin Secret Key
+5. Resident Site Key
+6. Resident Secret Key
+
+### Common Pitfalls
+- ❌ Using the same reCAPTCHA for all three portals (need separate ones!)
+- ❌ Wrong domain name in reCAPTCHA setup
+- ❌ Mixing up site keys and secret keys
+- ❌ Wrong order of keys in captcha-setup.sh arguments
+- ❌ Not updating keys after changing domain names
+
+---
+
+## 7. How to Add Secrets to GitHub
+
+### Understanding Secret Types
+
+#### Repository Secrets
+- **Scope**: Available to all branches and environments
+- **Use case**: Cloud credentials, SSH keys, GPG passphrase
+- **Where to add**: Settings > Secrets and variables > Actions > Repository secrets
+
+#### Environment Secrets
+- **Scope**: Specific to a deployment environment/branch
+- **Use case**: KUBECONFIG, WireGuard configs (different per environment)
+- **Where to add**: Settings > Secrets and variables > Actions > Environments
+
+### Step-by-Step: Adding Repository Secrets
+
+1. **Navigate to Repository Settings**
+ ```
+ Your Repository > Settings > Secrets and variables > Actions
+ ```
+
+2. **Click "Repository secrets" Tab**
+
+3. **Click "New repository secret"**
+
+4. **Add Secret**
+ - **Name**: Enter secret name (e.g., `AWS_ACCESS_KEY_ID`)
+ - **Value**: Paste secret value
+ - Click "Add secret"
+
+5. **Repeat for All Repository Secrets**
+ - `GPG_PASSPHRASE`
+ - `AWS_ACCESS_KEY_ID`
+ - `AWS_SECRET_ACCESS_KEY`
+ - `mosip-aws` (or your SSH key name)
+
+### Step-by-Step: Adding Environment Secrets
+
+1. **Create Environment (if not exists)**
+ ```
+ Settings > Environments > New environment
+ ```
+ - **Name**: Your branch name (e.g., `release-0.1.0`, `main`, `develop`)
+ - Click "Configure environment"
+
+2. **Add Environment Secrets**
+ - Scroll to "Environment secrets" section
+ - Click "Add secret"
+ - **Name**: Enter secret name (e.g., `KUBECONFIG`)
+ - **Value**: Paste secret value
+ - Click "Add secret"
+
+3. **Repeat for All Environment Secrets**
+ - `KUBECONFIG`
+ - `TF_WG_CONFIG`
+ - `CLUSTER_WIREGUARD_WG0`
+ - `CLUSTER_WIREGUARD_WG1` (optional)
+
+### Official Documentation
+- **GitHub Secrets Documentation**: https://docs.github.com/en/actions/security-guides/encrypted-secrets
+
+### Visual Guide
+
+```
+Repository Structure:
+├── Repository Secrets (global, all branches)
+│ ├── GPG_PASSPHRASE
+│ ├── AWS_ACCESS_KEY_ID
+│ ├── AWS_SECRET_ACCESS_KEY
+│ └── mosip-aws (SSH private key)
+│
+└── Environments
+ ├── release-0.1.0 (environment)
+ │ ├── KUBECONFIG
+ │ ├── TF_WG_CONFIG
+ │ ├── CLUSTER_WIREGUARD_WG0
+ │ └── CLUSTER_WIREGUARD_WG1
+ │
+ ├── main (environment)
+ │ └── (same secrets as above)
+ │
+ └── develop (environment)
+ └── (same secrets as above)
+```
+
+### Common Pitfalls
+- ❌ Adding KUBECONFIG as repository secret (should be environment secret)
+- ❌ Environment name doesn't match branch name
+- ❌ Copy-paste errors (extra spaces, newlines)
+- ❌ Not updating secrets after rotation
+- ❌ Typos in secret names (case-sensitive!)
+
+---
+
+## Quick Reference Checklist
+
+Use this checklist to ensure you've generated and configured all required secrets:
+
+### Repository Secrets (Do Once)
+- [ ] SSH Key Pair generated
+- [ ] SSH Public Key added to AWS EC2 Key Pairs
+- [ ] SSH Private Key added to GitHub Repository Secret
+- [ ] GPG Passphrase generated and added
+- [ ] AWS Access Key ID obtained and added
+- [ ] AWS Secret Access Key obtained and added
+
+### Infrastructure Deployment
+- [ ] Terraform base-infra deployed successfully
+- [ ] WireGuard VPN configured on jump server
+- [ ] WireGuard client installed on your computer
+- [ ] TF_WG_CONFIG environment secret added
+
+### Main Infrastructure
+- [ ] Terraform infra deployed successfully
+- [ ] KUBECONFIG file downloaded from Terraform outputs
+- [ ] KUBECONFIG added as environment secret
+- [ ] WireGuard cluster access configs added
+
+### MOSIP Services
+- [ ] reCAPTCHA keys generated for PreReg portal
+- [ ] reCAPTCHA keys generated for Admin portal
+- [ ] reCAPTCHA keys generated for Resident portal
+- [ ] reCAPTCHA keys added to external-dsf.yaml
+
+---
+
+## Troubleshooting
+
+### Secret Not Working
+
+**Check these common issues:**
+
+1. **Typo in secret name**
+ - Secret names are case-sensitive
+ - Verify exact name matches workflow configuration
+
+2. **Wrong secret type**
+ - Verify if it should be repository or environment secret
+ - Check if workflow is using correct environment
+
+3. **Invalid format**
+ - Ensure no extra spaces or newlines
+ - Include complete content (BEGIN/END lines for keys)
+
+4. **Secret not accessible**
+ - Verify workflow has permission to access secrets
+ - Check environment protection rules
+
+### Need Help?
+
+- **GitHub Issues**: Report problems in repository issues
+- **MOSIP Community**: Join community channels for support
+- **Documentation**: Refer to component-specific guides
+
+---
+
+**Navigation**: [Back to Main README](../README.md) | [View Glossary](GLOSSARY.md)
diff --git a/docs/WORKFLOW_GUIDE.md b/docs/WORKFLOW_GUIDE.md
new file mode 100644
index 00000000..1bd31489
--- /dev/null
+++ b/docs/WORKFLOW_GUIDE.md
@@ -0,0 +1,763 @@
+# GitHub Actions Workflow Guide
+
+This guide provides detailed, step-by-step instructions for running Terraform and Helmsman workflows through GitHub Actions. Perfect for beginners who need visual guidance on navigating the GitHub interface.
+
+## Table of Contents
+
+1. [Understanding Workflow Basics](#understanding-workflow-basics)
+2. [Terraform Workflows](#terraform-workflows)
+3. [Helmsman Workflows](#helmsman-workflows)
+4. [Workflow Parameters Explained](#workflow-parameters-explained)
+5. [Common Issues and Solutions](#common-issues-and-solutions)
+
+---
+
+## Understanding Workflow Basics
+
+### What You'll See
+When you run workflows, you'll interact with GitHub's web interface. Here's what to expect:
+
+```
+Your Repository
+└── Actions Tab (at the top)
+ ├── All workflows (left sidebar)
+ ├── Workflow runs (center)
+ └── Run workflow button (right side)
+```
+
+### Key Concepts
+
+#### Terraform Apply vs Dry Run
+
+| Mode | What It Does | When to Use |
+|------|--------------|-------------|
+| **Apply** (checkbox ✅) | Actually creates/modifies infrastructure | Production deployments, real changes |
+| **Dry Run** (checkbox ☐) | Shows what WOULD happen without making changes | Testing configurations, previewing changes |
+
+**Example:**
+- ✅ **Checked** → Terraform will create actual AWS servers
+- ☐ **Unchecked** → Terraform only shows you the plan (no changes made)
+
+#### Rancher Import Option
+
+| Setting | What It Does | When to Use |
+|---------|--------------|-------------|
+| **True** | Automatically imports cluster into Rancher UI | If you want centralized cluster management |
+| **False** | Cluster runs independently | For standalone deployments |
+
+**Relationship with Terraform Apply:**
+```
+If Terraform Apply = ✅ AND Rancher Import = True
+ → Cluster is deployed AND imported into Rancher
+
+If Terraform Apply = ✅ AND Rancher Import = False
+ → Cluster is deployed but NOT imported
+
+If Terraform Apply = ☐ (unchecked)
+ → Dry run only, nothing happens (Rancher import setting is ignored)
+```
+
+---
+
+## Terraform Workflows
+
+### Workflow 1: Base Infrastructure
+
+**What it does**: Creates the foundation - VPC, networking, jump server, WireGuard VPN
+
+#### Step-by-Step Navigation
+
+1. **Open GitHub Actions**
+ ```
+ Click: Your Repository → Actions (top menu bar)
+ ```
+
+2. **Find the Workflow**
+ ```
+ Left Sidebar: Look for "Terraform Base Infrastructure"
+ Click on it
+ ```
+
+3. **Start the Workflow**
+ ```
+ Right Side: Click "Run workflow" button (dropdown)
+ ```
+
+4. **Configure Parameters**
+ You'll see a form with these fields:
+
+ | Parameter | What to Select | Example | Notes |
+ |-----------|---------------|---------|-------|
+ | **Use workflow from** | `Branch: release-0.1.0` | Your deployment branch | Dropdown at top |
+ | **Cloud Provider** | `aws` | `aws` | Azure/GCP not fully implemented |
+ | **Component** | `base-infra` | `base-infra` | Creates VPC & networking |
+ | **Backend** | `local` or `s3` | `local` for dev, `s3` for prod | Where Terraform stores state |
+ | **Terraform apply** | ✅ Check this box | ✅ | Leave unchecked for dry run |
+
+5. **Run the Workflow**
+ ```
+ Bottom of form: Click green "Run workflow" button
+ ```
+
+6. **Monitor Progress**
+ ```
+ Watch the workflow run in real-time
+ Yellow circle = Running
+ Green checkmark = Success
+ Red X = Failed
+ ```
+
+#### What You Should See
+
+**During Execution:**
+```
+✓ Setup environment
+✓ Configure WireGuard
+✓ Run Terraform init
+✓ Run Terraform plan
+→ Run Terraform apply (if checked)
+✓ Complete
+```
+
+**After Success:**
+- VPC created in AWS
+- Jump server running
+- WireGuard VPN configured
+- Security groups configured
+- Output shows server IP addresses
+
+---
+
+### Workflow 2: Main Infrastructure
+
+**What it does**: Creates MOSIP Kubernetes cluster, PostgreSQL (optional), application infrastructure
+
+#### Step-by-Step Navigation
+
+1. **Open GitHub Actions**
+ ```
+ Click: Your Repository → Actions
+ ```
+
+2. **Find the Workflow**
+ ```
+ Left Sidebar: Look for "Terraform Infrastructure"
+ Click on it
+ ```
+
+ **Note**: The workflow name might also appear as:
+ - "Terraform"
+ - "Deploy Infrastructure"
+ - Check for keywords: "Infrastructure" or "Main Infra"
+
+3. **Start the Workflow**
+ ```
+ Click: "Run workflow" button (right side)
+ ```
+
+4. **Configure Parameters**
+
+ | Parameter | What to Select | Example | Why? |
+ |-----------|---------------|---------|------|
+ | **Use workflow from** | `Branch: release-0.1.0` | Your branch | Dropdown at top |
+ | **Cloud Provider** | `aws` | `aws` | Where to deploy |
+ | **Component** | `infra` | `infra` | Main MOSIP infrastructure |
+ | **Backend** | `local` or `s3` | `local` | State storage location |
+ | **Terraform apply** | ✅ | ✅ | Check to deploy, uncheck for dry run |
+
+5. **Run the Workflow**
+ ```
+ Click: Green "Run workflow" button
+ ```
+
+6. **Monitor Progress** (This takes 15-30 minutes)
+ ```
+ → Creating Kubernetes cluster
+ → Installing RKE2
+ → Configuring networking
+ → Setting up PostgreSQL (if enabled)
+ → Importing to Rancher (if enabled)
+ ```
+
+#### What You Should See
+
+**Success Indicators:**
+- ✅ Kubernetes cluster created
+- ✅ Multiple nodes visible in AWS EC2
+- ✅ KUBECONFIG file generated
+- ✅ PostgreSQL running (if enabled)
+
+**Outputs to Save:**
+- KUBECONFIG file location
+- Cluster endpoint URL
+- Node IP addresses
+
+---
+
+### Workflow 3: Observability Infrastructure (Optional)
+
+**What it does**: Creates Rancher management cluster for monitoring multiple MOSIP clusters
+
+#### When to Use
+- Managing multiple MOSIP environments
+- Need centralized cluster management UI
+- Want advanced monitoring dashboards
+
+#### Step-by-Step Navigation
+
+1. **Open GitHub Actions**
+ ```
+ Click: Repository → Actions
+ ```
+
+2. **Find the Workflow**
+ ```
+ Left Sidebar: "Terraform Observability Infrastructure"
+ OR look for: "Observ Infra" / "Monitoring Infrastructure"
+ ```
+
+3. **Configure Parameters**
+
+ | Parameter | What to Select | Example |
+ |-----------|---------------|---------|
+ | **Branch** | `release-0.1.0` | Your deployment branch |
+ | **Cloud Provider** | `aws` | `aws` |
+ | **Component** | `observ-infra` | `observ-infra` |
+ | **Backend** | `local` or `s3` | `local` for dev |
+ | **Terraform apply** | ✅ | Check to deploy |
+
+4. **Run and Monitor**
+ - Deployment takes 10-20 minutes
+ - Creates separate Rancher management cluster
+
+---
+
+## Helmsman Workflows
+
+### Understanding Helmsman Modes
+
+**IMPORTANT**: For Helmsman, ALWAYS use `apply` mode. Dry-run mode will fail!
+
+**Why dry-run fails:**
+- Helmsman checks dependencies between namespaces
+- Dry-run doesn't create shared configmaps/secrets
+- Validation fails when resources don't exist yet
+
+**Always use:**
+```
+Mode: apply ✅
+```
+
+---
+
+### Workflow 1: Prerequisites & External Dependencies
+
+**What it does**: Deploys monitoring, Istio, databases, message queues, storage
+
+**Workflow Name in GitHub**: `Deploy External services of mosip using Helmsman`
+
+**Note**: The documentation might refer to this as "Helmsman External Dependencies" but the actual workflow file is named differently. Look for either name!
+
+#### Step-by-Step Navigation
+
+1. **Open GitHub Actions**
+ ```
+ Click: Repository → Actions
+ ```
+
+2. **Find the Workflow**
+ ```
+ Left Sidebar: Look for EITHER:
+ - "Deploy External services of mosip using Helmsman" (actual name)
+ - "Helmsman External Dependencies" (documentation name)
+ - Keywords: "External" or "Dependencies"
+ ```
+
+3. **Start the Workflow**
+ ```
+ Click: "Run workflow" button
+ ```
+
+4. **Configure Parameters**
+
+ | Parameter | What to Select | Why? |
+ |-----------|---------------|------|
+ | **Branch** | `release-0.1.0` | Your deployment branch |
+ | **Mode** | `apply` | MUST be apply, not dry-run! |
+
+5. **What Happens Automatically**
+
+ This workflow deploys TWO DSF files in parallel:
+
+ **Part 1: Prerequisites** (`prereq-dsf.yaml`)
+ ```
+ → Monitoring stack (Prometheus, Grafana)
+ → Istio service mesh
+ → Logging stack (optional)
+ ```
+
+ **Part 2: External Dependencies** (`external-dsf.yaml`)
+ ```
+ → PostgreSQL (if container mode)
+ → MinIO (object storage)
+ → Kafka (messaging)
+ → ActiveMQ (messaging)
+ → Keycloak (identity management)
+ ```
+
+6. **Monitor Progress** (Takes 20-40 minutes)
+ ```
+ → Installing monitoring
+ → Deploying Istio
+ → Creating databases
+ → Setting up message queues
+ ```
+
+7. **Automatic Trigger**
+ ```
+ ✅ On success → Automatically triggers MOSIP Services deployment
+ ❌ On failure → You must manually trigger next workflow
+ ```
+
+#### What You Should See
+
+**Check Pod Status:**
+```bash
+kubectl get pods -n cattle-monitoring-system
+kubectl get pods -n istio-system
+kubectl get pods -n postgres
+kubectl get pods -n kafka
+```
+
+**All pods should show:**
+```
+STATUS: Running
+READY: X/X (e.g., 1/1, 2/2)
+```
+
+---
+
+### Workflow 2: MOSIP Core Services
+
+**What it does**: Deploys all MOSIP application services
+
+**Workflow Name**: `Deploy MOSIP services using Helmsman`
+
+#### Step-by-Step Navigation
+
+1. **Verify Prerequisites**
+ ```bash
+ # FIRST: Ensure all external services are running
+ kubectl get pods --all-namespaces | grep -v Running
+ # Should return nothing (or only Completed pods)
+ ```
+
+2. **Find the Workflow**
+ ```
+ GitHub Actions → Left Sidebar → Look for:
+ - "Deploy MOSIP services using Helmsman"
+ - "Deploy MOSIP services using Helmsman" (alternative)
+ - Keywords: "MOSIP" or "Services"
+ ```
+
+3. **Trigger Method**
+
+ **Option A: Automatic (Recommended)**
+ - Workflow triggers automatically after external dependencies succeed
+ - No action needed if previous workflow passed
+
+ **Option B: Manual**
+ - Click "Run workflow"
+ - Select branch: `release-0.1.0`
+ - Select mode: `apply`
+ - Click "Run workflow"
+
+4. **Monitor Progress** (Takes 30-60 minutes)
+ ```
+ → Config Server
+ → Artifactory Server
+ → Kernel services
+ → Pre-registration services
+ → Registration Processor
+ → ID Repository
+ → Authentication services
+ → Partner Management
+ → Resident Services
+ ```
+
+5. **Handle Onboarding Failures**
+
+ **Known Issue**: Partner onboarding may fail on first attempt
+
+ **What to do:**
+ ```
+ 1. Check logs for onboarding errors
+ 2. Follow manual onboarding procedure
+ 3. Verify all services are running before test rigs
+ ```
+
+#### What You Should See
+
+**Check MOSIP Pods:**
+```bash
+kubectl get pods -A
+```
+
+**Expected Output:**
+```
+NAME READY STATUS RESTARTS
+prereg-application-0 1/1 Running 0
+regproc-stage-group1-0 1/1 Running 0
+kernel-auth-0 1/1 Running 0
+idrepo-identity-0 1/1 Running 0
+...
+```
+
+**All pods should be Running before proceeding!**
+
+---
+
+### Workflow 3: Test Rigs (Optional)
+
+**What it does**: Deploys automated testing infrastructure
+
+**IMPORTANT**: Only run after ALL MOSIP services are running successfully!
+
+#### Step-by-Step Navigation
+
+1. **Verify All Services Running**
+ ```bash
+ # Check all namespaces
+ kubectl get pods -A | grep -v Running | grep -v Completed
+ # Should return nothing!
+
+ # Check external services
+ kubectl get pods -n postgres | grep -v Running
+ kubectl get pods -n keycloak | grep -v Running
+ # All should return nothing!
+ ```
+
+2. **Find the Workflow**
+ ```
+ GitHub Actions → "Helmsman Test Rigs" or "Deploy Test Rigs"
+ ```
+
+3. **Run the Workflow**
+ ```
+ Click: "Run workflow"
+ Branch: release-0.1.0
+ Mode: apply
+ DSF File: testrigs-dsf.yaml
+ ```
+
+4. **Monitor Progress** (Takes 15-30 minutes)
+ ```
+ → API Test Rig
+ → DSL Test Rig
+ → UI Test Rig
+ ```
+
+#### What You Should See
+
+**Check Test Rigs:**
+```bash
+kubectl get pods -n apitestrig
+kubectl get pods -n dsltestrig
+kubectl get pods -n uitestrig
+```
+
+---
+
+## Workflow Parameters Explained
+
+### Common to All Terraform Workflows
+
+#### Branch Selection
+```
+Use workflow from: Branch [dropdown]
+```
+**What it does**: Selects which branch's code to use
+
+**Choose**:
+- `release-0.1.0` - Stable release branch
+- `main` - Main development branch
+- `develop` - Latest development
+
+**Recommendation**: Use release branches for production
+
+---
+
+#### Cloud Provider
+```
+Cloud Provider: [aws | azure | gcp]
+```
+**What it does**: Selects cloud platform
+
+**Status**:
+- ✅ `aws` - Fully functional
+- `azure` - Placeholder only
+- `gcp` - Placeholder only
+
+**Choose**: `aws` (only fully implemented option)
+
+---
+
+#### Component
+```
+Component: [base-infra | infra | observ-infra]
+```
+**What it does**: Selects which infrastructure layer to deploy
+
+**Options**:
+| Component | Creates | Run Order |
+|-----------|---------|-----------|
+| `base-infra` | VPC, networking, jump server | **1st** (foundation) |
+| `observ-infra` | Rancher management cluster | **2nd** (optional) |
+| `infra` | MOSIP Kubernetes cluster | **3rd** (main deployment) |
+
+---
+
+#### Backend
+```
+Backend: [local | s3]
+```
+**What it does**: Determines where Terraform stores state files
+
+| Backend | Storage Location | Best For | Encryption |
+|---------|-----------------|----------|------------|
+| `local` | GitHub repository | Development, small teams | GPG encrypted |
+| `s3` | AWS S3 bucket | Production, large teams | S3 server-side encryption |
+
+**Recommendations**:
+- Development → `local`
+- Production → `s3`
+
+---
+
+#### Terraform Apply Checkbox
+```
+☐ Terraform apply
+```
+**What it does**: Controls whether changes are actually made
+
+| State | Effect | Use Case |
+|-------|--------|----------|
+| ☐ **Unchecked** | Dry run - shows plan only | Testing configurations |
+| ✅ **Checked** | Applies changes - creates resources | Actual deployment |
+
+**Visual Guide**:
+```
+☐ Unchecked → terraform plan → Shows what WOULD happen → No changes
+✅ Checked → terraform apply → Actually creates resources → Real changes
+```
+
+---
+
+### Helmsman-Specific Parameters
+
+#### Mode Selection
+```
+Mode: [apply | dry-run]
+```
+
+**IMPORTANT**: Always use `apply` for Helmsman!
+
+**Why?**
+```
+apply ✅ Works correctly
+dry-run ❌ Fails due to missing shared resources
+```
+
+**Choose**: `apply` (always)
+
+---
+
+#### DSF File Selection
+```
+DSF File: [prereq-dsf.yaml | external-dsf.yaml | mosip-dsf.yaml | testrigs-dsf.yaml]
+```
+
+**Deployment Order**:
+1. `prereq-dsf.yaml` - Monitoring, Istio
+2. `external-dsf.yaml` - Databases, queues
+3. `mosip-dsf.yaml` - MOSIP services
+4. `testrigs-dsf.yaml` - Testing infrastructure
+
+**Note**: Some workflows handle multiple DSFs automatically!
+
+---
+
+## Common Issues and Solutions
+
+### Issue 1: Workflow Not Found
+
+**Problem**: Can't find the workflow in GitHub Actions
+
+**Solution**:
+1. Check left sidebar - workflows are listed by name
+2. Try searching for keywords: "Terraform", "Helmsman", "Deploy"
+3. Verify you're on the "Actions" tab
+4. Check if workflows are in `.github/workflows/` directory
+
+**Workflow Name Variations**:
+| Documentation Says | Actual Workflow Name Might Be |
+|--------------------|------------------------------|
+| "Helmsman External Dependencies" | "Deploy External services of mosip using Helmsman" |
+| "Terraform Infrastructure" | "Terraform" or "Deploy Infrastructure" |
+
+---
+
+### Issue 2: Workflow Run Fails
+
+**Problem**: Red X appears, workflow failed
+
+**Solution**:
+1. **Click on the failed run**
+ ```
+ Actions → Click the failed run → Click failed job
+ ```
+
+2. **Read error messages**
+ ```
+ Scroll through logs
+ Look for red ERROR or FAILED messages
+ ```
+
+3. **Common Errors**:
+
+ | Error Message | Solution |
+ |--------------|----------|
+ | "Authentication failed" | Check AWS credentials in secrets |
+ | "InsufficientInstanceCapacity" | Set `specific_availability_zones = []` |
+ | "Permission denied" | Check IAM permissions |
+ | "Secret not found" | Add missing secret to GitHub |
+
+---
+
+### Issue 3: Terraform Apply Checkbox Confusion
+
+**Problem**: Not sure when to check or uncheck
+
+**Simple Rule**:
+```
+Want to see what would happen? → ☐ Uncheck (dry run)
+Want to actually deploy? → ✅ Check (apply)
+```
+
+**Example Scenarios**:
+```
+Testing new configuration → ☐ Uncheck first to verify
+First time deploying → ☐ Uncheck first, then ✅ check
+Updating existing deployment → ☐ Uncheck first to see changes
+```
+
+---
+
+### Issue 4: Environment Not Found
+
+**Problem**: Workflow can't find environment secrets
+
+**Solution**:
+1. **Verify environment name matches branch**
+ ```
+ Branch: release-0.1.0
+ Environment name: release-0.1.0 (must match exactly!)
+ ```
+
+2. **Create environment if missing**
+ ```
+ Settings → Environments → New environment
+ Name: release-0.1.0 (match your branch)
+ ```
+
+3. **Add secrets to environment**
+ ```
+ Configure environment → Add secrets
+ ```
+
+---
+
+### Issue 5: Helmsman Dry-Run Fails
+
+**Problem**: Helmsman workflow fails with validation errors
+
+**Solution**:
+```
+❌ Don't use: Mode = dry-run
+✅ Always use: Mode = apply
+```
+
+**Why?**
+- Dry-run validates resources that don't exist yet
+- Shared configmaps/secrets aren't created in dry-run
+- Dependencies between namespaces can't be validated
+
+---
+
+## Workflow Execution Checklist
+
+### Before Running ANY Workflow
+
+- [ ] All required secrets configured
+- [ ] Branch selected correctly
+- [ ] Environment matches branch name
+- [ ] AWS/cloud credentials valid
+- [ ] Previous steps completed successfully
+
+### For Terraform Workflows
+
+- [ ] tfvars file updated with correct values
+- [ ] Cloud provider = `aws`
+- [ ] Backend choice made (`local` or `s3`)
+- [ ] Understand dry-run vs apply
+- [ ] WireGuard configured (for infra deployment)
+
+### For Helmsman Workflows
+
+- [ ] KUBECONFIG secret added
+- [ ] WireGuard cluster access configured
+- [ ] Previous Helmsman steps completed
+- [ ] All pods from previous steps are Running
+- [ ] Mode set to `apply` (not dry-run!)
+- [ ] DSF files updated with correct domains
+
+---
+
+## Visual Workflow Summary
+
+```
+DEPLOYMENT FLOW:
+
+1. Terraform: Base Infrastructure
+ └── Creates VPC, networking, jump server
+ └── PAUSE: Configure WireGuard VPN
+
+2. Terraform: Main Infrastructure
+ └── Creates Kubernetes cluster
+ └── PAUSE: Get KUBECONFIG, add to secrets
+
+3. Helmsman: External Dependencies
+ └── Deploys monitoring, Istio, databases
+ └── ✅ Auto-triggers next step on success
+
+4. Helmsman: MOSIP Services (auto or manual)
+ └── Deploys MOSIP applications
+ └── PAUSE: Verify all pods Running
+
+5. Helmsman: Test Rigs (manual)
+ └── Deploys testing infrastructure
+ └── ✅ Deployment Complete!
+```
+
+---
+
+## Need More Help?
+
+- **Detailed Configurations**: See [DSF Configuration Guide](DSF_CONFIGURATION_GUIDE.md)
+- **Secret Setup**: See [Secret Generation Guide](SECRET_GENERATION_GUIDE.md)
+- **Troubleshooting**: See [Main README Troubleshooting Section](../README.md#troubleshooting-guides)
+- **Report Issues**: Open GitHub issue with workflow logs
+
+---
+
+**Navigation**: [Back to Main README](../README.md) | [View Glossary](GLOSSARY.md)
diff --git a/docs/_images/ARCHITECTURE_DIAGRAMS.md b/docs/_images/ARCHITECTURE_DIAGRAMS.md
index 36858e99..a7918f22 100644
--- a/docs/_images/ARCHITECTURE_DIAGRAMS.md
+++ b/docs/_images/ARCHITECTURE_DIAGRAMS.md
@@ -11,26 +11,26 @@ MOSIP Infrastructure Components (Updated)
========================================
GitHub Actions (Central Orchestration)
- |
- v
+ |
+ v
┌─────────────┬─────────────┬─────────────┐
-│ AWS Cloud │Azure Cloud │ GCP Cloud │
-│ Full Impl │ Placeholder │ Placeholder │
+│ AWS Cloud │Azure Cloud │ GCP Cloud │
+│ Full Impl │ Placeholder │ Placeholder │
└─────────────┴─────────────┴─────────────┘
- | | |
-┌─────┼─────┐ ┌─┼─┐ ┌─┼─┐
-│ │ │ │ │ │ │ │ │
-v v v v v v v v v
-base obs infra base infra base infra
-│ │ │ │ │ │ │
-│ │ │ │ ├──PostgreSQL Module (optional)
-│ │ ├──────┼───┘
-└─────┼─────┘ └───┘ └───┘
- | | |
- v v v
-[State Files] [State Files] [State Files]
-(Branch/Cloud (Branch/Cloud (Branch/Cloud
- Isolated) Isolated) Isolated)
+ | | |
+┌─────┼─────┐ ┌─┼─┐ ┌─┼─┐
+│ │ │ │ │ │ │ │ │
+v v v v v v v v v
+base obs infra base infra base infra
+│ │ │ │ │ │ │
+│ │ │ │ ├──PostgreSQL Module (optional)
+│ │ ├──────┼───┘
+└─────┼─────┘ └───┘ └───┘
+ | | |
+ v v v
+[State Files] [State Files] [State Files]
+(Branch/Cloud (Branch/Cloud (Branch/Cloud
+ Isolated) Isolated) Isolated)
```
### Component Relationships (Updated)
@@ -48,289 +48,289 @@ Terraform Infra Module
├── RKE2 Kubernetes Cluster
├── Node Groups & Networking
└── PostgreSQL Module (Conditional)
- ├── enable_postgresql_setup = true → External PostgreSQL
- │ ├── Dedicated EBS Volume
- │ ├── Ansible Installation Script
- │ └── PostgreSQL 15 Configuration
- └── enable_postgresql_setup = false → Container PostgreSQL via Helmsman
+ ├── enable_postgresql_setup = true → External PostgreSQL
+ │ ├── Dedicated EBS Volume
+ │ ├── Ansible Installation Script
+ │ └── PostgreSQL 15 Configuration
+ └── enable_postgresql_setup = false → Container PostgreSQL via Helmsman
```
## Multi-Cloud Deployment Architecture
```mermaid
graph TB
- subgraph "AWS Deployment"
- subgraph "AWS Base-Infra (One-time)"
- AWS_VPC[VPC 10.0.0.0/16
Public/Private Subnets
Security Groups
WireGuard Jumpserver]
- end
- subgraph "AWS Observ-Infra (Optional)"
- AWS_MON[Management Cluster
Rancher UI + Keycloak
RBAC Integration]
- end
- subgraph "AWS Infra (Multiple)"
- AWS_RKE1[MOSIP Cluster 1
Production Environment
+ External PostgreSQL]
- AWS_RKE2[MOSIP Cluster 2
Staging Environment
+ Optional PostgreSQL]
- end
- AWS_VPC --> AWS_MON
- AWS_VPC --> AWS_RKE1
- AWS_VPC --> AWS_RKE2
- AWS_MON -.->|Import| AWS_RKE1
- AWS_MON -.->|Import| AWS_RKE2
- end
-
- subgraph "Azure Deployment"
- subgraph "Azure Base-Infra (One-time)"
- AZ_VNET[VNet 10.1.0.0/16
Public/Private Subnets
Network Security Groups
WireGuard Jumpserver]
- end
- subgraph "Azure Observ-Infra (Optional)"
- AZ_MON[Management Cluster
Rancher UI + Keycloak
RBAC Integration]
- end
- subgraph "Azure Infra (Multiple)"
- AZ_RKE1[MOSIP Cluster 1
Production Environment
+ External PostgreSQL]
- AZ_RKE2[MOSIP Cluster 2
Staging Environment
+ Optional PostgreSQL]
- end
- AZ_VNET --> AZ_MON
- AZ_VNET --> AZ_RKE1
- AZ_VNET --> AZ_RKE2
- AZ_MON -.->|Import| AZ_RKE1
- AZ_MON -.->|Import| AZ_RKE2
- end
-
- subgraph "GCP Deployment"
- subgraph "GCP Base-Infra (One-time)"
- GCP_VPC[VPC 10.2.0.0/16
Public/Private Subnets
Firewall Rules
WireGuard Jumpserver]
- end
- subgraph "GCP Observ-Infra (Optional)"
- GCP_MON[Management Cluster
Rancher UI + Keycloak
RBAC Integration]
- end
- subgraph "GCP Infra (Multiple)"
- GCP_RKE1[MOSIP Cluster 1
Production Environment
+ External PostgreSQL]
- GCP_RKE2[MOSIP Cluster 2
Staging Environment
+ Optional PostgreSQL]
- end
- GCP_VPC --> GCP_MON
- GCP_VPC --> GCP_RKE1
- GCP_VPC --> GCP_RKE2
- GCP_MON -.->|Import| GCP_RKE1
- GCP_MON -.->|Import| GCP_RKE2
- end
-
- style AWS_VPC fill:#e1f5fe,stroke:#01579b,color:#000000
- style AWS_MON fill:#fff3e0,stroke:#f57c00,color:#000000
- style AWS_RKE1 fill:#f3e5f5,stroke:#4a148c,color:#000000
- style AWS_RKE2 fill:#e8f5e8,stroke:#1b5e20,color:#000000
- style AZ_VNET fill:#e1f5fe,stroke:#01579b,color:#000000
- style AZ_MON fill:#fff3e0,stroke:#f57c00,color:#000000
- style AZ_RKE1 fill:#f3e5f5,stroke:#4a148c,color:#000000
- style AZ_RKE2 fill:#e8f5e8,stroke:#1b5e20,color:#000000
- style GCP_VPC fill:#e1f5fe,stroke:#01579b,color:#000000
- style GCP_MON fill:#fff3e0,stroke:#f57c00,color:#000000
- style GCP_RKE1 fill:#f3e5f5,stroke:#4a148c,color:#000000
- style GCP_RKE2 fill:#e8f5e8,stroke:#1b5e20,color:#000000
+ subgraph "AWS Deployment"
+ subgraph "AWS Base-Infra (One-time)"
+ AWS_VPC[VPC 10.0.0.0/16
Public/Private Subnets
Security Groups
WireGuard Jumpserver]
+ end
+ subgraph "AWS Observ-Infra (Optional)"
+ AWS_MON[Management Cluster
Rancher UI + Keycloak
RBAC Integration]
+ end
+ subgraph "AWS Infra (Multiple)"
+ AWS_RKE1[MOSIP Cluster 1
Production Environment
+ External PostgreSQL]
+ AWS_RKE2[MOSIP Cluster 2
Staging Environment
+ Optional PostgreSQL]
+ end
+ AWS_VPC --> AWS_MON
+ AWS_VPC --> AWS_RKE1
+ AWS_VPC --> AWS_RKE2
+ AWS_MON -.->|Import| AWS_RKE1
+ AWS_MON -.->|Import| AWS_RKE2
+ end
+
+ subgraph "Azure Deployment"
+ subgraph "Azure Base-Infra (One-time)"
+ AZ_VNET[VNet 10.1.0.0/16
Public/Private Subnets
Network Security Groups
WireGuard Jumpserver]
+ end
+ subgraph "Azure Observ-Infra (Optional)"
+ AZ_MON[Management Cluster
Rancher UI + Keycloak
RBAC Integration]
+ end
+ subgraph "Azure Infra (Multiple)"
+ AZ_RKE1[MOSIP Cluster 1
Production Environment
+ External PostgreSQL]
+ AZ_RKE2[MOSIP Cluster 2
Staging Environment
+ Optional PostgreSQL]
+ end
+ AZ_VNET --> AZ_MON
+ AZ_VNET --> AZ_RKE1
+ AZ_VNET --> AZ_RKE2
+ AZ_MON -.->|Import| AZ_RKE1
+ AZ_MON -.->|Import| AZ_RKE2
+ end
+
+ subgraph "GCP Deployment"
+ subgraph "GCP Base-Infra (One-time)"
+ GCP_VPC[VPC 10.2.0.0/16
Public/Private Subnets
Firewall Rules
WireGuard Jumpserver]
+ end
+ subgraph "GCP Observ-Infra (Optional)"
+ GCP_MON[Management Cluster
Rancher UI + Keycloak
RBAC Integration]
+ end
+ subgraph "GCP Infra (Multiple)"
+ GCP_RKE1[MOSIP Cluster 1
Production Environment
+ External PostgreSQL]
+ GCP_RKE2[MOSIP Cluster 2
Staging Environment
+ Optional PostgreSQL]
+ end
+ GCP_VPC --> GCP_MON
+ GCP_VPC --> GCP_RKE1
+ GCP_VPC --> GCP_RKE2
+ GCP_MON -.->|Import| GCP_RKE1
+ GCP_MON -.->|Import| GCP_RKE2
+ end
+
+ style AWS_VPC fill:#e1f5fe,stroke:#01579b,color:#000000
+ style AWS_MON fill:#fff3e0,stroke:#f57c00,color:#000000
+ style AWS_RKE1 fill:#f3e5f5,stroke:#4a148c,color:#000000
+ style AWS_RKE2 fill:#e8f5e8,stroke:#1b5e20,color:#000000
+ style AZ_VNET fill:#e1f5fe,stroke:#01579b,color:#000000
+ style AZ_MON fill:#fff3e0,stroke:#f57c00,color:#000000
+ style AZ_RKE1 fill:#f3e5f5,stroke:#4a148c,color:#000000
+ style AZ_RKE2 fill:#e8f5e8,stroke:#1b5e20,color:#000000
+ style GCP_VPC fill:#e1f5fe,stroke:#01579b,color:#000000
+ style GCP_MON fill:#fff3e0,stroke:#f57c00,color:#000000
+ style GCP_RKE1 fill:#f3e5f5,stroke:#4a148c,color:#000000
+ style GCP_RKE2 fill:#e8f5e8,stroke:#1b5e20,color:#000000
```
## PostgreSQL Integration Architecture (NEW)
```mermaid
graph TD
- A[Terraform Infra Deployment] --> B{enable_postgresql_setup}
-
- B -->|true| C[External PostgreSQL Path]
- B -->|false| D[Container PostgreSQL Path]
-
- C --> E[Provision EBS/Disk Volume]
- E --> F[Create Dedicated PostgreSQL Node]
- F --> G[Execute Ansible PostgreSQL Setup]
- G --> H[Install PostgreSQL 15]
- H --> I[Configure Security & Networking]
- I --> J[Setup Data Directory & Permissions]
- J --> K[Configure PostgreSQL Port 5433]
- K --> L[External PostgreSQL Ready]
-
- D --> M[Skip PostgreSQL Infrastructure]
- M --> N[Configure Helmsman external-dsf.yaml]
- N --> O[Enable PostgreSQL Container Deployment]
- O --> P[Container PostgreSQL via Kubernetes]
-
- L --> Q[Update Helmsman Configuration]
- P --> Q[Update Helmsman Configuration]
-
- Q --> R[postgresql.enabled = false (External)]
- Q --> S[postgresql.enabled = true (Container)]
-
- R --> T[Deploy MOSIP Services]
- S --> T[Deploy MOSIP Services]
-
- T --> U[MOSIP Uses Configured PostgreSQL]
-
- style C fill:#e8f5e8,stroke:#2e7d32,color:#000000
- style D fill:#fff3e0,stroke:#f57c00,color:#000000
- style L fill:#c8e6c9,stroke:#1b5e20,color:#000000
- style P fill:#ffe0b2,stroke:#e65100,color:#000000
+ A[Terraform Infra Deployment] --> B{enable_postgresql_setup}
+
+ B -->|true| C[External PostgreSQL Path]
+ B -->|false| D[Container PostgreSQL Path]
+
+ C --> E[Provision EBS/Disk Volume]
+ E --> F[Create Dedicated PostgreSQL Node]
+ F --> G[Execute Ansible PostgreSQL Setup]
+ G --> H[Install PostgreSQL 15]
+ H --> I[Configure Security & Networking]
+ I --> J[Setup Data Directory & Permissions]
+ J --> K[Configure PostgreSQL Port 5433]
+ K --> L[External PostgreSQL Ready]
+
+ D --> M[Skip PostgreSQL Infrastructure]
+ M --> N[Configure Helmsman external-dsf.yaml]
+ N --> O[Enable PostgreSQL Container Deployment]
+ O --> P[Container PostgreSQL via Kubernetes]
+
+ L --> Q[Update Helmsman Configuration]
+ P --> Q[Update Helmsman Configuration]
+
+ Q --> R[postgresql.enabled = false (External)]
+ Q --> S[postgresql.enabled = true (Container)]
+
+ R --> T[Deploy MOSIP Services]
+ S --> T[Deploy MOSIP Services]
+
+ T --> U[MOSIP Uses Configured PostgreSQL]
+
+ style C fill:#e8f5e8,stroke:#2e7d32,color:#000000
+ style D fill:#fff3e0,stroke:#f57c00,color:#000000
+ style L fill:#c8e6c9,stroke:#1b5e20,color:#000000
+ style P fill:#ffe0b2,stroke:#e65100,color:#000000
```
## Updated Deployment Flow with PostgreSQL
```mermaid
graph TD
- A[1. Deploy base-infra
VPC + WireGuard
One-time setup] --> B{Deploy observ-infra?}
-
- B -->|Yes| C[2. Deploy observ-infra
Rancher + Keycloak]
- B -->|No| D[3. Configure PostgreSQL in Terraform]
-
- C --> D[3. Configure PostgreSQL in Terraform]
-
- D --> E{enable_postgresql_setup?}
-
- E -->|true| F[3a. Set EBS Volume Size
nginx_node_ebs_volume_size_2 = 200]
- E -->|false| G[3b. Plan for Container PostgreSQL]
-
- F --> H[4. Deploy infra via Terraform
RKE2 + PostgreSQL + Networking]
- G --> I[4. Deploy infra via Terraform
RKE2 + Networking only]
-
- H --> J[Terraform Auto-executes:
- Provision PostgreSQL node
- Install via Ansible
- Configure PostgreSQL 15]
- I --> K[PostgreSQL will be containerized]
-
- J --> L[5. Update Helmsman DSF
postgresql.enabled = false]
- K --> M[5. Update Helmsman DSF
postgresql.enabled = true]
-
- L --> N[6. Deploy Prerequisites
Monitoring + Istio + Logging]
- K --> M[6. Deploy Prerequisites
Monitoring + Istio + Logging]
-
- L --> O[6. Deploy External Dependencies (Parallel)
PostgreSQL + MinIO + Keycloak + Kafka]
- K --> P[6. Deploy External Dependencies (Parallel)
PostgreSQL + MinIO + Keycloak + Kafka]
-
- N --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
- O --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
- M --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
- P --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
- P --> Q[9. Optional: Deploy Test Rigs]
- Q --> R[MOSIP Platform Ready]
-
- style F fill:#c8e6c9,stroke:#1b5e20,color:#000000
- style G fill:#ffe0b2,stroke:#e65100,color:#000000
- style H fill:#c8e6c9,stroke:#1b5e20,color:#000000
- style I fill:#ffe0b2,stroke:#e65100,color:#000000
- style J fill:#a5d6a7,stroke:#2e7d32,color:#000000
- style K fill:#ffcc02,stroke:#f57c00,color:#000000
+ A[1. Deploy base-infra
VPC + WireGuard
One-time setup] --> B{Deploy observ-infra?}
+
+ B -->|Yes| C[2. Deploy observ-infra
Rancher + Keycloak]
+ B -->|No| D[3. Configure PostgreSQL in Terraform]
+
+ C --> D[3. Configure PostgreSQL in Terraform]
+
+ D --> E{enable_postgresql_setup?}
+
+ E -->|true| F[3a. Set EBS Volume Size
nginx_node_ebs_volume_size_2 = 200]
+ E -->|false| G[3b. Plan for Container PostgreSQL]
+
+ F --> H[4. Deploy infra via Terraform
RKE2 + PostgreSQL + Networking]
+ G --> I[4. Deploy infra via Terraform
RKE2 + Networking only]
+
+ H --> J[Terraform Auto-executes:
- Provision PostgreSQL node
- Install via Ansible
- Configure PostgreSQL 15]
+ I --> K[PostgreSQL will be containerized]
+
+ J --> L[5. Update Helmsman DSF
postgresql.enabled = false]
+ K --> M[5. Update Helmsman DSF
postgresql.enabled = true]
+
+ L --> N[6. Deploy Prerequisites
Monitoring + Istio + Logging]
+ K --> M[6. Deploy Prerequisites
Monitoring + Istio + Logging]
+
+ L --> O[6. Deploy External Dependencies (Parallel)
PostgreSQL + MinIO + Keycloak + Kafka]
+ K --> P[6. Deploy External Dependencies (Parallel)
PostgreSQL + MinIO + Keycloak + Kafka]
+
+ N --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
+ O --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
+ M --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
+ P --> Q[7. Deploy MOSIP Services
Core services using PostgreSQL]
+ P --> Q[9. Optional: Deploy Test Rigs]
+ Q --> R[MOSIP Platform Ready]
+
+ style F fill:#c8e6c9,stroke:#1b5e20,color:#000000
+ style G fill:#ffe0b2,stroke:#e65100,color:#000000
+ style H fill:#c8e6c9,stroke:#1b5e20,color:#000000
+ style I fill:#ffe0b2,stroke:#e65100,color:#000000
+ style J fill:#a5d6a7,stroke:#2e7d32,color:#000000
+ style K fill:#ffcc02,stroke:#f57c00,color:#000000
## Deployment Flow & Dependencies
```mermaid
graph TD
- A[Start Deployment] --> B{Choose Cloud Provider}
- B -->|AWS| C[AWS Deployment]
- B -->|Azure| D[Azure Deployment]
- B -->|GCP| E[GCP Deployment]
-
- C --> F[1. Deploy base-infra
VPC + WireGuard
One-time setup]
- D --> G[1. Deploy base-infra
VNet + WireGuard
One-time setup]
- E --> H[1. Deploy base-infra
VPC + WireGuard
One-time setup]
-
- F --> I{Deploy observ-infra?}
- G --> J{Deploy observ-infra?}
- H --> K{Deploy observ-infra?}
-
- I -->|Yes| L[2. Deploy observ-infra
Rancher + Keycloak]
- I -->|No| M[3. Deploy infra
RKE2 + NGINX + NFS]
- J -->|Yes| N[2. Deploy observ-infra
Rancher + Keycloak]
- J -->|No| O[3. Deploy infra
RKE2 + NGINX + NFS]
- K -->|Yes| P[2. Deploy observ-infra
Rancher + Keycloak]
- K -->|No| Q[3. Deploy infra
RKE2 + NGINX + NFS]
-
- L --> M
- N --> O
- P --> Q
-
- M --> R[MOSIP Cluster Ready]
- O --> S[MOSIP Cluster Ready]
- Q --> T[MOSIP Cluster Ready]
-
- L -.->|Optional Import| R
- N -.->|Optional Import| S
- P -.->|Optional Import| T
-
- style F fill:#e1f5fe,stroke:#01579b,color:#000000
- style G fill:#e1f5fe,stroke:#01579b,color:#000000
- style H fill:#e1f5fe,stroke:#01579b,color:#000000
- style L fill:#fff3e0,stroke:#f57c00,color:#000000
- style N fill:#fff3e0,stroke:#f57c00,color:#000000
- style P fill:#fff3e0,stroke:#f57c00,color:#000000
- style M fill:#f3e5f5,stroke:#4a148c,color:#000000
- style O fill:#f3e5f5,stroke:#4a148c,color:#000000
- style Q fill:#f3e5f5,stroke:#4a148c,color:#000000
+ A[Start Deployment] --> B{Choose Cloud Provider}
+ B -->|AWS| C[AWS Deployment]
+ B -->|Azure| D[Azure Deployment]
+ B -->|GCP| E[GCP Deployment]
+
+ C --> F[1. Deploy base-infra
VPC + WireGuard
One-time setup]
+ D --> G[1. Deploy base-infra
VNet + WireGuard
One-time setup]
+ E --> H[1. Deploy base-infra
VPC + WireGuard
One-time setup]
+
+ F --> I{Deploy observ-infra?}
+ G --> J{Deploy observ-infra?}
+ H --> K{Deploy observ-infra?}
+
+ I -->|Yes| L[2. Deploy observ-infra
Rancher + Keycloak]
+ I -->|No| M[3. Deploy infra
RKE2 + NGINX + NFS]
+ J -->|Yes| N[2. Deploy observ-infra
Rancher + Keycloak]
+ J -->|No| O[3. Deploy infra
RKE2 + NGINX + NFS]
+ K -->|Yes| P[2. Deploy observ-infra
Rancher + Keycloak]
+ K -->|No| Q[3. Deploy infra
RKE2 + NGINX + NFS]
+
+ L --> M
+ N --> O
+ P --> Q
+
+ M --> R[MOSIP Cluster Ready]
+ O --> S[MOSIP Cluster Ready]
+ Q --> T[MOSIP Cluster Ready]
+
+ L -.->|Optional Import| R
+ N -.->|Optional Import| S
+ P -.->|Optional Import| T
+
+ style F fill:#e1f5fe,stroke:#01579b,color:#000000
+ style G fill:#e1f5fe,stroke:#01579b,color:#000000
+ style H fill:#e1f5fe,stroke:#01579b,color:#000000
+ style L fill:#fff3e0,stroke:#f57c00,color:#000000
+ style N fill:#fff3e0,stroke:#f57c00,color:#000000
+ style P fill:#fff3e0,stroke:#f57c00,color:#000000
+ style M fill:#f3e5f5,stroke:#4a148c,color:#000000
+ style O fill:#f3e5f5,stroke:#4a148c,color:#000000
+ style Q fill:#f3e5f5,stroke:#4a148c,color:#000000
```
## Terraform Module Structure
```mermaid
graph TB
- subgraph "Terraform Directory Structure"
- subgraph "implementations/"
- subgraph "aws/"
- AWS_BASE[base-infra/
Foundation setup]
- AWS_OBS[observ-infra/
Management cluster]
- AWS_INF[infra/
MOSIP clusters]
- end
- subgraph "azure/"
- AZ_BASE[base-infra/
Foundation setup]
- AZ_OBS[observ-infra/
Management cluster]
- AZ_INF[infra/
MOSIP clusters]
- end
- subgraph "gcp/"
- GCP_BASE[base-infra/
Foundation setup]
- GCP_OBS[observ-infra/
Management cluster]
- GCP_INF[infra/
MOSIP clusters]
- end
- end
-
- subgraph "modules/"
- subgraph "AWS Modules"
- AWS_VPC[aws-resource-creation/
VPC, subnets, security]
- AWS_RKE[rke2-cluster/
Kubernetes setup]
- AWS_NGINX[nginx-setup/
Load balancer]
- AWS_NFS[nfs-setup/
Storage]
- end
- subgraph "Azure Modules"
- AZ_VNET[azure-resource-creation/
VNet, NSG, security]
- AZ_RKE[rke2-cluster/
Kubernetes setup]
- AZ_LB[lb-setup/
Load balancer]
- AZ_STOR[storage-setup/
Storage]
- end
- subgraph "GCP Modules"
- GCP_VPC_MOD[gcp-resource-creation/
VPC, firewall]
- GCP_RKE[rke2-cluster/
Kubernetes setup]
- GCP_LB[lb-setup/
Load balancer]
- GCP_STOR[storage-setup/
Storage]
- end
- end
- end
-
- AWS_BASE --> AWS_VPC
- AWS_OBS --> AWS_RKE
- AWS_INF --> AWS_RKE
- AWS_INF --> AWS_NGINX
- AWS_INF --> AWS_NFS
-
- AZ_BASE --> AZ_VNET
- AZ_OBS --> AZ_RKE
- AZ_INF --> AZ_RKE
- AZ_INF --> AZ_LB
- AZ_INF --> AZ_STOR
-
- GCP_BASE --> GCP_VPC_MOD
- GCP_OBS --> GCP_RKE
- GCP_INF --> GCP_RKE
- GCP_INF --> GCP_LB
- GCP_INF --> GCP_STOR
-
- style AWS_BASE fill:#e1f5fe,stroke:#01579b,color:#000000
- style AWS_OBS fill:#fff3e0,stroke:#f57c00,color:#000000
- style AWS_INF fill:#f3e5f5,stroke:#4a148c,color:#000000
- style AZ_BASE fill:#e1f5fe,stroke:#01579b,color:#000000
- style AZ_OBS fill:#fff3e0,stroke:#f57c00,color:#000000
- style AZ_INF fill:#f3e5f5,stroke:#4a148c,color:#000000
- style GCP_BASE fill:#e1f5fe,stroke:#01579b,color:#000000
- style GCP_OBS fill:#fff3e0,stroke:#f57c00,color:#000000
- style GCP_INF fill:#f3e5f5,stroke:#4a148c,color:#000000
+ subgraph "Terraform Directory Structure"
+ subgraph "implementations/"
+ subgraph "aws/"
+ AWS_BASE[base-infra/
Foundation setup]
+ AWS_OBS[observ-infra/
Management cluster]
+ AWS_INF[infra/
MOSIP clusters]
+ end
+ subgraph "azure/"
+ AZ_BASE[base-infra/
Foundation setup]
+ AZ_OBS[observ-infra/
Management cluster]
+ AZ_INF[infra/
MOSIP clusters]
+ end
+ subgraph "gcp/"
+ GCP_BASE[base-infra/
Foundation setup]
+ GCP_OBS[observ-infra/
Management cluster]
+ GCP_INF[infra/
MOSIP clusters]
+ end
+ end
+
+ subgraph "modules/"
+ subgraph "AWS Modules"
+ AWS_VPC[aws-resource-creation/
VPC, subnets, security]
+ AWS_RKE[rke2-cluster/
Kubernetes setup]
+ AWS_NGINX[nginx-setup/
Load balancer]
+ AWS_NFS[nfs-setup/
Storage]
+ end
+ subgraph "Azure Modules"
+ AZ_VNET[azure-resource-creation/
VNet, NSG, security]
+ AZ_RKE[rke2-cluster/
Kubernetes setup]
+ AZ_LB[lb-setup/
Load balancer]
+ AZ_STOR[storage-setup/
Storage]
+ end
+ subgraph "GCP Modules"
+ GCP_VPC_MOD[gcp-resource-creation/
VPC, firewall]
+ GCP_RKE[rke2-cluster/
Kubernetes setup]
+ GCP_LB[lb-setup/
Load balancer]
+ GCP_STOR[storage-setup/
Storage]
+ end
+ end
+ end
+
+ AWS_BASE --> AWS_VPC
+ AWS_OBS --> AWS_RKE
+ AWS_INF --> AWS_RKE
+ AWS_INF --> AWS_NGINX
+ AWS_INF --> AWS_NFS
+
+ AZ_BASE --> AZ_VNET
+ AZ_OBS --> AZ_RKE
+ AZ_INF --> AZ_RKE
+ AZ_INF --> AZ_LB
+ AZ_INF --> AZ_STOR
+
+ GCP_BASE --> GCP_VPC_MOD
+ GCP_OBS --> GCP_RKE
+ GCP_INF --> GCP_RKE
+ GCP_INF --> GCP_LB
+ GCP_INF --> GCP_STOR
+
+ style AWS_BASE fill:#e1f5fe,stroke:#01579b,color:#000000
+ style AWS_OBS fill:#fff3e0,stroke:#f57c00,color:#000000
+ style AWS_INF fill:#f3e5f5,stroke:#4a148c,color:#000000
+ style AZ_BASE fill:#e1f5fe,stroke:#01579b,color:#000000
+ style AZ_OBS fill:#fff3e0,stroke:#f57c00,color:#000000
+ style AZ_INF fill:#f3e5f5,stroke:#4a148c,color:#000000
+ style GCP_BASE fill:#e1f5fe,stroke:#01579b,color:#000000
+ style GCP_OBS fill:#fff3e0,stroke:#f57c00,color:#000000
+ style GCP_INF fill:#f3e5f5,stroke:#4a148c,color:#000000
```
## State File Isolation
@@ -342,15 +342,15 @@ State Management Structure
Production (main branch):
├── mosip-terraform-bucket-main/
-│ ├── aws-base-infra-main-terraform.tfstate
-│ ├── aws-observ-infra-main-terraform.tfstate
-│ ├── aws-infra-main-terraform.tfstate
-│ ├── azure-base-infra-main-terraform.tfstate
-│ ├── azure-observ-infra-main-terraform.tfstate
-│ ├── azure-infra-main-terraform.tfstate
-│ ├── gcp-base-infra-main-terraform.tfstate
-│ ├── gcp-observ-infra-main-terraform.tfstate
-│ └── gcp-infra-main-terraform.tfstate
+│ ├── aws-base-infra-main-terraform.tfstate
+│ ├── aws-observ-infra-main-terraform.tfstate
+│ ├── aws-infra-main-terraform.tfstate
+│ ├── azure-base-infra-main-terraform.tfstate
+│ ├── azure-observ-infra-main-terraform.tfstate
+│ ├── azure-infra-main-terraform.tfstate
+│ ├── gcp-base-infra-main-terraform.tfstate
+│ ├── gcp-observ-infra-main-terraform.tfstate
+│ └── gcp-infra-main-terraform.tfstate
Staging (staging branch):
├── mosip-terraform-bucket-staging/
diff --git a/terraform/README.md b/terraform/README.md
index 3aa24d26..618e68d3 100644
--- a/terraform/README.md
+++ b/terraform/README.md
@@ -28,68 +28,68 @@ The MOSIP infrastructure follows a clean three-component architecture with isola
```mermaid
graph TB
- subgraph "GitHub Actions Orchestration"
- GHA[GitHub Actions
Central Deployment Control]
- end
-
- subgraph "Multi-Cloud Infrastructure"
- subgraph "AWS Cloud"
- AWS_BASE[base-infra
VPC + WireGuard]
- AWS_OBS[observ-infra
Rancher + Keycloak]
- AWS_INFRA[infra
MOSIP K8s Cluster]
- AWS_STATE[(AWS S3
State Files)]
- end
-
- subgraph "Azure Cloud"
- AZ_BASE[base-infra
VNet + WireGuard]
- AZ_OBS[observ-infra
Rancher + Keycloak]
- AZ_INFRA[infra
MOSIP RKE2 Cluster]
- AZ_STATE[(Azure Storage
State Files)]
- end
-
- subgraph "GCP Cloud"
- GCP_BASE[base-infra
VPC + WireGuard]
- GCP_OBS[observ-infra
Rancher + Keycloak]
- GCP_INFRA[infra
MOSIP RKE2 Cluster]
- GCP_STATE[(GCS
State Files)]
- end
- end
-
- GHA --> AWS_BASE
- GHA --> AZ_BASE
- GHA --> GCP_BASE
-
- AWS_BASE --> AWS_OBS
- AWS_BASE --> AWS_INFRA
- AWS_OBS -.->|Import| AWS_INFRA
- AWS_BASE -.-> AWS_STATE
- AWS_OBS -.-> AWS_STATE
- AWS_INFRA -.-> AWS_STATE
-
- AZ_BASE --> AZ_OBS
- AZ_BASE --> AZ_INFRA
- AZ_OBS -.->|Import| AZ_INFRA
- AZ_BASE -.-> AZ_STATE
- AZ_OBS -.-> AZ_STATE
- AZ_INFRA -.-> AZ_STATE
-
- GCP_BASE --> GCP_OBS
- GCP_BASE --> GCP_INFRA
- GCP_OBS -.->|Import| GCP_INFRA
- GCP_BASE -.-> GCP_STATE
- GCP_OBS -.-> GCP_STATE
- GCP_INFRA -.-> GCP_STATE
-
- style GHA fill:#2196F3,stroke:#1976D2,stroke-width:2px,color:#fff
- style AWS_BASE fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
- style AWS_OBS fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
- style AWS_INFRA fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
- style AZ_BASE fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
- style AZ_OBS fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
- style AZ_INFRA fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
- style GCP_BASE fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
- style GCP_OBS fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
- style GCP_INFRA fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
+ subgraph "GitHub Actions Orchestration"
+ GHA[GitHub Actions
Central Deployment Control]
+ end
+
+ subgraph "Multi-Cloud Infrastructure"
+ subgraph "AWS Cloud"
+ AWS_BASE[base-infra
VPC + WireGuard]
+ AWS_OBS[observ-infra
Rancher + Keycloak]
+ AWS_INFRA[infra
MOSIP K8s Cluster]
+ AWS_STATE[(AWS S3
State Files)]
+ end
+
+ subgraph "Azure Cloud"
+ AZ_BASE[base-infra
VNet + WireGuard]
+ AZ_OBS[observ-infra
Rancher + Keycloak]
+ AZ_INFRA[infra
MOSIP RKE2 Cluster]
+ AZ_STATE[(Azure Storage
State Files)]
+ end
+
+ subgraph "GCP Cloud"
+ GCP_BASE[base-infra
VPC + WireGuard]
+ GCP_OBS[observ-infra
Rancher + Keycloak]
+ GCP_INFRA[infra
MOSIP RKE2 Cluster]
+ GCP_STATE[(GCS
State Files)]
+ end
+ end
+
+ GHA --> AWS_BASE
+ GHA --> AZ_BASE
+ GHA --> GCP_BASE
+
+ AWS_BASE --> AWS_OBS
+ AWS_BASE --> AWS_INFRA
+ AWS_OBS -.->|Import| AWS_INFRA
+ AWS_BASE -.-> AWS_STATE
+ AWS_OBS -.-> AWS_STATE
+ AWS_INFRA -.-> AWS_STATE
+
+ AZ_BASE --> AZ_OBS
+ AZ_BASE --> AZ_INFRA
+ AZ_OBS -.->|Import| AZ_INFRA
+ AZ_BASE -.-> AZ_STATE
+ AZ_OBS -.-> AZ_STATE
+ AZ_INFRA -.-> AZ_STATE
+
+ GCP_BASE --> GCP_OBS
+ GCP_BASE --> GCP_INFRA
+ GCP_OBS -.->|Import| GCP_INFRA
+ GCP_BASE -.-> GCP_STATE
+ GCP_OBS -.-> GCP_STATE
+ GCP_INFRA -.-> GCP_STATE
+
+ style GHA fill:#2196F3,stroke:#1976D2,stroke-width:2px,color:#fff
+ style AWS_BASE fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
+ style AWS_OBS fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
+ style AWS_INFRA fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
+ style AZ_BASE fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
+ style AZ_OBS fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
+ style AZ_INFRA fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
+ style GCP_BASE fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
+ style GCP_OBS fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#000000
+ style GCP_INFRA fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000000
```
**Component Relationships:**
@@ -102,40 +102,40 @@ graph TB
```mermaid
graph TD
- A["terraform/"] --> B["base-infra/"]
- A --> C["infra/"]
- A --> D["observ-infra/"]
- A --> E["modules/"]
- A --> F["implementations/"]
-
- E --> E1["aws/"]
- E --> E2["azure/"]
- E --> E3["gcp/"]
-
- E1 --> E1A["aws-resource-creation/"]
- E1 --> E1B["nginx-setup/"]
- E1 --> E1C["rke2-cluster/"]
- E1 --> E1D["nfs-setup/"]
- E1 --> E1E["postgresql-setup/"]
- E1 --> E1F["rancher-keycloak-setup/"]
-
- F --> F1["aws/"]
- F --> F2["azure/"]
- F --> F3["gcp/"]
-
- F1 --> F1A["base-infra/"]
- F1 --> F1B["infra/"]
- F1 --> F1C["observ-infra/"]
-
- classDef root fill:#2196F3,stroke:#1976D2,stroke-width:2px,color:#fff
- classDef component fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
- classDef modules fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
- classDef impl fill:#fce4ec,stroke:#880e4f,stroke-width:2px,color:#000000
-
- class A root
- class B,C,D component
- class E,E1,E2,E3,E1A,E1B,E1C,E1D,E1E,E1F modules
- class F,F1,F2,F3,F1A,F1B,F1C impl
+ A["terraform/"] --> B["base-infra/"]
+ A --> C["infra/"]
+ A --> D["observ-infra/"]
+ A --> E["modules/"]
+ A --> F["implementations/"]
+
+ E --> E1["aws/"]
+ E --> E2["azure/"]
+ E --> E3["gcp/"]
+
+ E1 --> E1A["aws-resource-creation/"]
+ E1 --> E1B["nginx-setup/"]
+ E1 --> E1C["rke2-cluster/"]
+ E1 --> E1D["nfs-setup/"]
+ E1 --> E1E["postgresql-setup/"]
+ E1 --> E1F["rancher-keycloak-setup/"]
+
+ F --> F1["aws/"]
+ F --> F2["azure/"]
+ F --> F3["gcp/"]
+
+ F1 --> F1A["base-infra/"]
+ F1 --> F1B["infra/"]
+ F1 --> F1C["observ-infra/"]
+
+ classDef root fill:#2196F3,stroke:#1976D2,stroke-width:2px,color:#fff
+ classDef component fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000000
+ classDef modules fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px,color:#000000
+ classDef impl fill:#fce4ec,stroke:#880e4f,stroke-width:2px,color:#000000
+
+ class A root
+ class B,C,D component
+ class E,E1,E2,E3,E1A,E1B,E1C,E1D,E1E,E1F modules
+ class F,F1,F2,F3,F1A,F1B,F1C impl
```
**Color Legend:**
@@ -178,34 +178,34 @@ graph TD
```mermaid
sequenceDiagram
- participant User
- participant GitHub Actions
- participant Cloud Provider
- participant Terraform State
-
- Note over User,Terraform State: Phase 1: Base Infrastructure (One-time)
- User->>GitHub Actions: Trigger terraform.yml
(base-infra)
- GitHub Actions->>Cloud Provider: Deploy VPC, Jumpserver, WireGuard
- Cloud Provider-->>Terraform State: Store base-infra state
- GitHub Actions-->>User: Base infrastructure ready
-
- Note over User,Terraform State: Phase 2: MOSIP Infrastructure (Repeatable)
- User->>GitHub Actions: Trigger terraform.yml
(infra)
- GitHub Actions->>Cloud Provider: Deploy K8s for MOSIP core/external services
- Cloud Provider-->>Terraform State: Store infra state
- GitHub Actions-->>User: MOSIP infrastructure ready
-
- Note over User,Terraform State: Phase 3: Observation Infrastructure (Independent)
- User->>GitHub Actions: Trigger terraform.yml
(observ-infra)
- GitHub Actions->>Cloud Provider: Deploy K8s for Rancher UI, Keycloak
- Cloud Provider-->>Terraform State: Store observ-infra state
- GitHub Actions-->>User: Observation infrastructure ready
-
- Note over User,Terraform State: Phase 4: Safe Destruction (When needed)
- User->>GitHub Actions: Trigger terraform-destroy.yml
(infra or observ-infra)
- GitHub Actions->>Cloud Provider: Destroy selected resources
- Cloud Provider-->>Terraform State: Clear selected state
- GitHub Actions-->>User: Resources destroyed, base intact
+ participant User
+ participant GitHub Actions
+ participant Cloud Provider
+ participant Terraform State
+
+ Note over User,Terraform State: Phase 1: Base Infrastructure (One-time)
+ User->>GitHub Actions: Trigger terraform.yml
(base-infra)
+ GitHub Actions->>Cloud Provider: Deploy VPC, Jumpserver, WireGuard
+ Cloud Provider-->>Terraform State: Store base-infra state
+ GitHub Actions-->>User: Base infrastructure ready
+
+ Note over User,Terraform State: Phase 2: MOSIP Infrastructure (Repeatable)
+ User->>GitHub Actions: Trigger terraform.yml
(infra)
+ GitHub Actions->>Cloud Provider: Deploy K8s for MOSIP core/external services
+ Cloud Provider-->>Terraform State: Store infra state
+ GitHub Actions-->>User: MOSIP infrastructure ready
+
+ Note over User,Terraform State: Phase 3: Observation Infrastructure (Independent)
+ User->>GitHub Actions: Trigger terraform.yml
(observ-infra)
+ GitHub Actions->>Cloud Provider: Deploy K8s for Rancher UI, Keycloak
+ Cloud Provider-->>Terraform State: Store observ-infra state
+ GitHub Actions-->>User: Observation infrastructure ready
+
+ Note over User,Terraform State: Phase 4: Safe Destruction (When needed)
+ User->>GitHub Actions: Trigger terraform-destroy.yml
(infra or observ-infra)
+ GitHub Actions->>Cloud Provider: Destroy selected resources
+ Cloud Provider-->>Terraform State: Clear selected state
+ GitHub Actions-->>User: Resources destroyed, base intact
```
## Quick Start Guide
@@ -213,50 +213,50 @@ sequenceDiagram
### Prerequisites
1. **GitHub Secrets Configuration** (per cloud):
- ```yaml
- # AWS (Complete Implementation)
- AWS_ACCESS_KEY_ID: your-aws-access-key
- AWS_SECRET_ACCESS_KEY: your-aws-secret-key
-
- # Azure (Placeholder Implementation)
- AZURE_CLIENT_ID: your-azure-client-id
- AZURE_CLIENT_SECRET: your-azure-client-secret
- AZURE_SUBSCRIPTION_ID: your-azure-subscription-id
- AZURE_TENANT_ID: your-azure-tenant-id
-
- # GCP (Placeholder Implementation)
- GOOGLE_CREDENTIALS: your-gcp-service-account-json
-
- # Common Requirements
- YOUR_SSH_KEY_NAME: your-ssh-private-key-content
- WG_CONFIG: wireguard-configuration
-
- # GPG Encryption (Required for State Management)
- GPG_PRIVATE_KEY: |
- -----BEGIN PGP PRIVATE KEY BLOCK-----
- your-gpg-private-key-content
- -----END PGP PRIVATE KEY BLOCK-----
-
- # Optional
- SLACK_WEBHOOK_URL: slack-webhook-for-notifications
- ```
+ ```yaml
+ # AWS (Complete Implementation)
+ AWS_ACCESS_KEY_ID: your-aws-access-key
+ AWS_SECRET_ACCESS_KEY: your-aws-secret-key
+
+ # Azure (Placeholder Implementation)
+ AZURE_CLIENT_ID: your-azure-client-id
+ AZURE_CLIENT_SECRET: your-azure-client-secret
+ AZURE_SUBSCRIPTION_ID: your-azure-subscription-id
+ AZURE_TENANT_ID: your-azure-tenant-id
+
+ # GCP (Placeholder Implementation)
+ GOOGLE_CREDENTIALS: your-gcp-service-account-json
+
+ # Common Requirements
+ YOUR_SSH_KEY_NAME: your-ssh-private-key-content
+ WG_CONFIG: wireguard-configuration
+
+ # GPG Encryption (Required for State Management)
+ GPG_PRIVATE_KEY: |
+ -----BEGIN PGP PRIVATE KEY BLOCK-----
+ your-gpg-private-key-content
+ -----END PGP PRIVATE KEY BLOCK-----
+
+ # Optional
+ SLACK_WEBHOOK_URL: slack-webhook-for-notifications
+ ```
2. **GPG Key Setup** (for state file encryption):
- ```bash
- # Generate GPG key pair (if you don't have one)
- gpg --full-generate-key
-
- # Export private key for GitHub secret
- gpg --armor --export-secret-keys your-key-id
-
- # Export public key (for team sharing)
- gpg --armor --export your-key-id
- ```
+ ```bash
+ # Generate GPG key pair (if you don't have one)
+ gpg --full-generate-key
+
+ # Export private key for GitHub secret
+ gpg --armor --export-secret-keys your-key-id
+
+ # Export public key (for team sharing)
+ gpg --armor --export your-key-id
+ ```
3. **No Remote State Storage Required**:
- - **Local Backend**: State files encrypted with GPG and stored in repository
- - **No S3/Azure Storage/GCS needed**: Eliminates remote storage dependencies
- - **Enhanced Security**: GPG encryption provides better access control than cloud storage
+ - **Local Backend**: State files encrypted with GPG and stored in repository
+ - **No S3/Azure Storage/GCS needed**: Eliminates remote storage dependencies
+ - **Enhanced Security**: GPG encryption provides better access control than cloud storage
### Step 1: Deploy Base Infrastructure
@@ -264,34 +264,34 @@ sequenceDiagram
# Via GitHub Actions UI:
# 1. Go to Actions → terraform plan / apply
# 2. Select:
-# - CLOUD_PROVIDER: aws
-# - TERRAFORM_COMPONENT: base-infra
-# - SSH_PRIVATE_KEY: YOUR_SSH_KEY_NAME
-# - TERRAFORM_APPLY: true
+# - CLOUD_PROVIDER: aws
+# - TERRAFORM_COMPONENT: base-infra
+# - SSH_PRIVATE_KEY: YOUR_SSH_KEY_NAME
+# - TERRAFORM_APPLY: true
```
### Step 2: Deploy MOSIP Infrastructure
```bash
# Via GitHub Actions UI:
-# 1. Go to Actions → terraform plan / apply
+# 1. Go to Actions → terraform plan / apply
# 2. Select:
-# - CLOUD_PROVIDER: aws
-# - TERRAFORM_COMPONENT: infra
-# - SSH_PRIVATE_KEY: YOUR_SSH_KEY_NAME
-# - TERRAFORM_APPLY: true
+# - CLOUD_PROVIDER: aws
+# - TERRAFORM_COMPONENT: infra
+# - SSH_PRIVATE_KEY: YOUR_SSH_KEY_NAME
+# - TERRAFORM_APPLY: true
```
### Step 3: Deploy Observation Infrastructure (Optional)
```bash
# Via GitHub Actions UI:
-# 1. Go to Actions → terraform plan / apply
+# 1. Go to Actions → terraform plan / apply
# 2. Select:
-# - CLOUD_PROVIDER: aws
-# - TERRAFORM_COMPONENT: observ-infra
-# - SSH_PRIVATE_KEY: YOUR_SSH_KEY_NAME
-# - TERRAFORM_APPLY: true
+# - CLOUD_PROVIDER: aws
+# - TERRAFORM_COMPONENT: observ-infra
+# - SSH_PRIVATE_KEY: YOUR_SSH_KEY_NAME
+# - TERRAFORM_APPLY: true
```
### Step 4: Access Your MOSIP Deployment
@@ -300,88 +300,88 @@ After successful deployment, access your MOSIP services:
```bash
# MOSIP Services (from infra)
-https://your-domain.mosip.net # MOSIP Landing Page
-https://api.your-domain.mosip.net # API Gateway
-https://admin.your-domain.mosip.net # Admin Console
+https://your-domain.mosip.net # MOSIP Landing Page
+https://api.your-domain.mosip.net # API Gateway
+https://admin.your-domain.mosip.net # Admin Console
# Observation Services (from observ-infra)
-https://rancher.your-domain.mosip.net # Rancher UI
-https://keycloak.your-domain.mosip.net # Keycloak Management
+https://rancher.your-domain.mosip.net # Rancher UI
+https://keycloak.your-domain.mosip.net # Keycloak Management
```
## Directory Structure
```
terraform/
-├── README.md # This file
-├── base-infra/ # Foundational infrastructure
-│ ├── main.tf # Cloud selector for base infrastructure
-│ ├── variables.tf # Common base infrastructure variables
-│ ├── outputs.tf # Common base infrastructure outputs
-│ ├── aws/ # AWS base infrastructure
-│ │ ├── main.tf # AWS VPC, subnets, jumpserver, WireGuard
-│ │ ├── variables.tf # AWS-specific base variables
-│ │ └── outputs.tf # AWS base infrastructure outputs
-│ ├── azure/ # Azure base infrastructure (placeholder)
-│ └── gcp/ # GCP base infrastructure (placeholder)
-├── infra/ # MOSIP infrastructure interface
-│ ├── main.tf # Cloud selector for MOSIP infrastructure
-│ ├── variables.tf # Common MOSIP variables
-│ ├── outputs.tf # Common MOSIP outputs
-│ ├── aws/ # AWS MOSIP infrastructure interface
-│ │ ├── main.tf # Calls AWS modules for MOSIP services
-│ │ ├── variables.tf # AWS-specific MOSIP variables
-│ │ └── outputs.tf # AWS MOSIP outputs
-│ ├── azure/ # Azure MOSIP infrastructure (placeholder)
-│ └── gcp/ # GCP MOSIP infrastructure (placeholder)
-├── observ-infra/ # Observation infrastructure interface
-│ ├── main.tf # Cloud selector for observation infrastructure
-│ ├── variables.tf # Common observation variables
-│ ├── outputs.tf # Common observation outputs
-│ ├── aws/ # AWS observation infrastructure interface
-│ │ ├── main.tf # Calls AWS modules for Rancher UI, Keycloak
-│ │ ├── variables.tf # AWS-specific observation variables
-│ │ └── outputs.tf # AWS observation outputs
-│ ├── azure/ # Azure observation infrastructure (placeholder)
-│ └── gcp/ # GCP observation infrastructure (placeholder)
-├── modules/ # Reusable infrastructure modules
-│ ├── aws/ # AWS-specific modules
-│ │ ├── main.tf # AWS module interface
-│ │ ├── variables.tf # AWS module variables
-│ │ ├── outputs.tf # AWS module outputs
-│ │ ├── aws-resource-creation/ # Core AWS resources
-│ │ ├── nginx-setup/ # NGINX load balancer setup
-│ │ ├── rke2-cluster/ # RKE2 Kubernetes cluster
-│ │ ├── nfs-setup/ # NFS storage setup
-│ │ ├── postgresql-setup/ # External PostgreSQL 15 database setup
-│ │ └── rancher-keycloak-setup/ # Rancher UI + Keycloak integration
-│ ├── azure/ # Azure modules (placeholder)
-│ └── gcp/ # GCP modules (placeholder)
-├── implementations/ # Deployable configurations
-│ ├── aws/ # AWS implementations
-│ │ ├── base-infra/ # AWS base infrastructure deployment
-│ │ │ ├── main.tf # Points to base-infra/aws
-│ │ │ ├── variables.tf # AWS base configuration
-│ │ │ ├── outputs.tf # AWS base outputs
-│ │ │ └── aws.tfvars # AWS base configuration values
-│ │ ├── infra/ # AWS MOSIP infrastructure deployment
-│ │ │ ├── main.tf # Points to infra/aws
-│ │ │ ├── variables.tf # AWS MOSIP configuration
-│ │ │ ├── outputs.tf # AWS MOSIP outputs
-│ │ │ └── aws.tfvars # AWS MOSIP configuration values
-│ │ └── observ-infra/ # AWS observation infrastructure deployment
-│ │ ├── main.tf # Points to observ-infra/aws
-│ │ ├── variables.tf # AWS observation configuration
-│ │ ├── outputs.tf # AWS observation outputs
-│ │ └── aws.tfvars # AWS observation configuration values
-│ ├── azure/ # Azure implementations
-│ │ ├── base-infra/ # Azure base infrastructure
-│ │ ├── infra/ # Azure MOSIP infrastructure
-│ │ └── observ-infra/ # Azure observation infrastructure
-│ └── gcp/ # GCP implementations
-│ ├── base-infra/ # GCP base infrastructure
-│ ├── infra/ # GCP MOSIP infrastructure
-│ └── observ-infra/ # GCP observation infrastructure
+├── README.md # This file
+├── base-infra/ # Foundational infrastructure
+│ ├── main.tf # Cloud selector for base infrastructure
+│ ├── variables.tf # Common base infrastructure variables
+│ ├── outputs.tf # Common base infrastructure outputs
+│ ├── aws/ # AWS base infrastructure
+│ │ ├── main.tf # AWS VPC, subnets, jumpserver, WireGuard
+│ │ ├── variables.tf # AWS-specific base variables
+│ │ └── outputs.tf # AWS base infrastructure outputs
+│ ├── azure/ # Azure base infrastructure (placeholder)
+│ └── gcp/ # GCP base infrastructure (placeholder)
+├── infra/ # MOSIP infrastructure interface
+│ ├── main.tf # Cloud selector for MOSIP infrastructure
+│ ├── variables.tf # Common MOSIP variables
+│ ├── outputs.tf # Common MOSIP outputs
+│ ├── aws/ # AWS MOSIP infrastructure interface
+│ │ ├── main.tf # Calls AWS modules for MOSIP services
+│ │ ├── variables.tf # AWS-specific MOSIP variables
+│ │ └── outputs.tf # AWS MOSIP outputs
+│ ├── azure/ # Azure MOSIP infrastructure (placeholder)
+│ └── gcp/ # GCP MOSIP infrastructure (placeholder)
+├── observ-infra/ # Observation infrastructure interface
+│ ├── main.tf # Cloud selector for observation infrastructure
+│ ├── variables.tf # Common observation variables
+│ ├── outputs.tf # Common observation outputs
+│ ├── aws/ # AWS observation infrastructure interface
+│ │ ├── main.tf # Calls AWS modules for Rancher UI, Keycloak
+│ │ ├── variables.tf # AWS-specific observation variables
+│ │ └── outputs.tf # AWS observation outputs
+│ ├── azure/ # Azure observation infrastructure (placeholder)
+│ └── gcp/ # GCP observation infrastructure (placeholder)
+├── modules/ # Reusable infrastructure modules
+│ ├── aws/ # AWS-specific modules
+│ │ ├── main.tf # AWS module interface
+│ │ ├── variables.tf # AWS module variables
+│ │ ├── outputs.tf # AWS module outputs
+│ │ ├── aws-resource-creation/ # Core AWS resources
+│ │ ├── nginx-setup/ # NGINX load balancer setup
+│ │ ├── rke2-cluster/ # RKE2 Kubernetes cluster
+│ │ ├── nfs-setup/ # NFS storage setup
+│ │ ├── postgresql-setup/ # External PostgreSQL 15 database setup
+│ │ └── rancher-keycloak-setup/ # Rancher UI + Keycloak integration
+│ ├── azure/ # Azure modules (placeholder)
+│ └── gcp/ # GCP modules (placeholder)
+├── implementations/ # Deployable configurations
+│ ├── aws/ # AWS implementations
+│ │ ├── base-infra/ # AWS base infrastructure deployment
+│ │ │ ├── main.tf # Points to base-infra/aws
+│ │ │ ├── variables.tf # AWS base configuration
+│ │ │ ├── outputs.tf # AWS base outputs
+│ │ │ └── aws.tfvars # AWS base configuration values
+│ │ ├── infra/ # AWS MOSIP infrastructure deployment
+│ │ │ ├── main.tf # Points to infra/aws
+│ │ │ ├── variables.tf # AWS MOSIP configuration
+│ │ │ ├── outputs.tf # AWS MOSIP outputs
+│ │ │ └── aws.tfvars # AWS MOSIP configuration values
+│ │ └── observ-infra/ # AWS observation infrastructure deployment
+│ │ ├── main.tf # Points to observ-infra/aws
+│ │ ├── variables.tf # AWS observation configuration
+│ │ ├── outputs.tf # AWS observation outputs
+│ │ └── aws.tfvars # AWS observation configuration values
+│ ├── azure/ # Azure implementations
+│ │ ├── base-infra/ # Azure base infrastructure
+│ │ ├── infra/ # Azure MOSIP infrastructure
+│ │ └── observ-infra/ # Azure observation infrastructure
+│ └── gcp/ # GCP implementations
+│ ├── base-infra/ # GCP base infrastructure
+│ ├── infra/ # GCP MOSIP infrastructure
+│ └── observ-infra/ # GCP observation infrastructure
```
## State Management
@@ -396,17 +396,17 @@ Encrypted State Management
Local State Files (Encrypted with GPG):
├── .terraform-state/
-│ ├── aws-base-infra-testgrid-terraform.tfstate.gpg ← Encrypted
-│ ├── aws-infra-testgrid-terraform.tfstate.gpg ← Encrypted
-│ └── aws-observ-infra-testgrid-terraform.tfstate.gpg ← Encrypted
+│ ├── aws-base-infra-testgrid-terraform.tfstate.gpg ← Encrypted
+│ ├── aws-infra-testgrid-terraform.tfstate.gpg ← Encrypted
+│ └── aws-observ-infra-testgrid-terraform.tfstate.gpg ← Encrypted
Temporary Decrypted Files (During Operations):
├── terraform/base-infra/
-│ └── aws-base-infra-testgrid-terraform.tfstate ← Temporary
+│ └── aws-base-infra-testgrid-terraform.tfstate ← Temporary
├── terraform/infra/
-│ └── aws-infra-testgrid-terraform.tfstate ← Temporary
+│ └── aws-infra-testgrid-terraform.tfstate ← Temporary
└── terraform/observ-infra/
- └── aws-observ-infra-testgrid-terraform.tfstate ← Temporary
+ └── aws-observ-infra-testgrid-terraform.tfstate ← Temporary
```
### GPG Encryption Benefits
@@ -424,10 +424,10 @@ State files use descriptive naming pattern: `{provider}-{component}-{branch}-ter
```
Examples:
-├── aws-base-infra-main-terraform.tfstate.gpg # Production base infrastructure
-├── aws-infra-staging-terraform.tfstate.gpg # Staging MOSIP cluster
+├── aws-base-infra-main-terraform.tfstate.gpg # Production base infrastructure
+├── aws-infra-staging-terraform.tfstate.gpg # Staging MOSIP cluster
├── aws-observ-infra-development-terraform.tfstate.gpg # Dev monitoring cluster
-└── azure-infra-testgrid-terraform.tfstate.gpg # Test environment
+└── azure-infra-testgrid-terraform.tfstate.gpg # Test environment
```
### Isolated State Files
@@ -439,25 +439,25 @@ Encrypted State File Isolation Structure
=========================================
AWS Encrypted States (.gpg files):
-├── aws-base-infra-{branch}-terraform.tfstate.gpg (VPC, Jumpserver, WireGuard)
-├── aws-infra-{branch}-terraform.tfstate.gpg (K8s for MOSIP Core/External)
-└── aws-observ-infra-{branch}-terraform.tfstate.gpg (K8s for Rancher UI, Keycloak)
+├── aws-base-infra-{branch}-terraform.tfstate.gpg (VPC, Jumpserver, WireGuard)
+├── aws-infra-{branch}-terraform.tfstate.gpg (K8s for MOSIP Core/External)
+└── aws-observ-infra-{branch}-terraform.tfstate.gpg (K8s for Rancher UI, Keycloak)
Azure Encrypted States (.gpg files):
-├── azure-base-infra-{branch}-terraform.tfstate.gpg (VNet, Jumpserver, WireGuard)
-├── azure-infra-{branch}-terraform.tfstate.gpg (RKE2 for MOSIP Core/External)
+├── azure-base-infra-{branch}-terraform.tfstate.gpg (VNet, Jumpserver, WireGuard)
+├── azure-infra-{branch}-terraform.tfstate.gpg (RKE2 for MOSIP Core/External)
└── azure-observ-infra-{branch}-terraform.tfstate.gpg (RKE2 for Rancher UI, Keycloak)
GCP Encrypted States (.gpg files):
-├── gcp-base-infra-{branch}-terraform.tfstate.gpg (VPC, Jumpserver, WireGuard)
-├── gcp-infra-{branch}-terraform.tfstate.gpg (RKE2 for MOSIP Core/External)
-└── gcp-observ-infra-{branch}-terraform.tfstate.gpg (RKE2 for Rancher UI, Keycloak)
+├── gcp-base-infra-{branch}-terraform.tfstate.gpg (VPC, Jumpserver, WireGuard)
+├── gcp-infra-{branch}-terraform.tfstate.gpg (RKE2 for MOSIP Core/External)
+└── gcp-observ-infra-{branch}-terraform.tfstate.gpg (RKE2 for Rancher UI, Keycloak)
```
### Benefits of GPG Encrypted Isolated States
- **Enhanced Security**: All state files encrypted before storage
-- **Branch Isolation**: Each branch has separate encrypted state files
+- **Branch Isolation**: Each branch has separate encrypted state files
- **No Cloud Dependencies**: No need for S3/Azure Storage/GCS setup
- **Access Control**: Only users with GPG private key can decrypt states
- **Git-Safe Storage**: Encrypted states safely committed to repository
@@ -547,7 +547,7 @@ Deployment Sequence:
terraform.yml → CLOUD_PROVIDER: aws, TERRAFORM_COMPONENT: base-infra
terraform.yml → CLOUD_PROVIDER: aws, TERRAFORM_COMPONENT: infra
-# Deploy to Azure
+# Deploy to Azure
terraform.yml → CLOUD_PROVIDER: azure, TERRAFORM_COMPONENT: base-infra
terraform.yml → CLOUD_PROVIDER: azure, TERRAFORM_COMPONENT: infra
@@ -563,10 +563,10 @@ Create multiple environments by duplicating implementation directories:
```
implementations/
├── aws/
-│ ├── base-infra/ # Shared base infrastructure
-│ ├── infra-prod/ # Production environment
-│ ├── infra-staging/ # Staging environment
-│ └── infra-dev/ # Development environment
+│ ├── base-infra/ # Shared base infrastructure
+│ ├── infra-prod/ # Production environment
+│ ├── infra-staging/ # Staging environment
+│ └── infra-dev/ # Development environment
```
### Custom Module Development
@@ -576,10 +576,10 @@ Add new modules following the established pattern:
```
modules/
├── aws/
-│ ├── your-custom-module/
-│ │ ├── main.tf
-│ │ ├── variables.tf
-│ │ └── outputs.tf
+│ ├── your-custom-module/
+│ │ ├── main.tf
+│ │ ├── variables.tf
+│ │ └── outputs.tf
```
## Troubleshooting
@@ -588,25 +588,25 @@ modules/
1. **State File Conflicts**
- ```
- Error: Resource already exists in state
- ```
+ ```
+ Error: Resource already exists in state
+ ```
- **Solution**: Ensure you're not mixing old and new directory structures
+ **Solution**: Ensure you're not mixing old and new directory structures
2. **Backend Configuration Issues**
- ```
- Error: Backend configuration changed
- ```
+ ```
+ Error: Backend configuration changed
+ ```
- **Solution**: Run `terraform init -reconfigure` in the implementation directory
+ **Solution**: Run `terraform init -reconfigure` in the implementation directory
3. **Missing Dependencies**
- ```
- Error: base-infra resources not found
- ```
+ ```
+ Error: base-infra resources not found
+ ```
- **Solution**: Deploy base-infra before application infrastructure
+ **Solution**: Deploy base-infra before application infrastructure
### Debugging Tips
diff --git a/terraform/base-infra/WIREGUARD_SETUP.md b/terraform/base-infra/WIREGUARD_SETUP.md
index dcd2c36c..a395de38 100644
--- a/terraform/base-infra/WIREGUARD_SETUP.md
+++ b/terraform/base-infra/WIREGUARD_SETUP.md
@@ -37,7 +37,7 @@ ls
**Expected output:**
```
-peer1/ peer2/ peer3/ peer4/ peer5/ ...
+peer1/ peer2/ peer3/ peer4/ peer5/ ...
```
### Step 3: Assign and Configure a Peer
@@ -52,7 +52,7 @@ nano assigned.txt
**Example assigned.txt content:**
```
peer1 : john.doe
-peer2 : jane.smith
+peer2 : jane.smith
peer3 : admin.user
peer4 : available
peer5 : available
@@ -88,12 +88,12 @@ AllowedIPs = 0.0.0.0/0
[Interface]
PrivateKey =
Address = 10.0.1.4/24
-# DNS = 1.1.1.1 <-- DELETE THIS LINE
+# DNS = 1.1.1.1 <-- DELETE THIS LINE
[Peer]
PublicKey =
Endpoint = :51820
-AllowedIPs = 10.10.20.0/23 # <-- UPDATE TO YOUR SUBNET CIDR
+AllowedIPs = 10.10.20.0/23 # <-- UPDATE TO YOUR SUBNET CIDR
```
**Required Changes:**
@@ -178,19 +178,19 @@ After setting up WireGuard, you need to create **multiple peer configurations**
#### Required Peer Configurations
1. **Create Multiple Peers:**
- ```bash
- # Configure peer1 for Terraform access
- cd /home/ubuntu/wireguard/config/peer1
- nano peer1.conf
-
- # Configure peer2 for Helmsman access
- cd /home/ubuntu/wireguard/config/peer2
- nano peer2.conf
- ```
+ ```bash
+ # Configure peer1 for Terraform access
+ cd /home/ubuntu/wireguard/config/peer1
+ nano peer1.conf
+
+ # Configure peer2 for Helmsman access
+ cd /home/ubuntu/wireguard/config/peer2
+ nano peer2.conf
+ ```
2. **Apply Same Configuration Changes to Both Peers:**
- - Delete the DNS IP line
- - Update AllowedIPs to your subnet CIDR (e.g., `10.10.20.0/23`)
+ - Delete the DNS IP line
+ - Update AllowedIPs to your subnet CIDR (e.g., `10.10.20.0/23`)
#### GitHub Environment Secrets Setup
@@ -200,27 +200,27 @@ After setting up WireGuard, you need to create **multiple peer configurations**
**Add the following Environment Secrets:**
1. **TF_WG_CONFIG Secret:**
- - Name: `TF_WG_CONFIG`
- - Value: Contents of `peer1.conf`
- - Purpose: Terraform infrastructure deployments via private IPs
+ - Name: `TF_WG_CONFIG`
+ - Value: Contents of `peer1.conf`
+ - Purpose: Terraform infrastructure deployments via private IPs
2. **CLUSTER_WIREGUARD_WG0 Secret:**
- - Name: `CLUSTER_WIREGUARD_WG0`
- - Value: Contents of `peer1.conf`
- - Purpose: Helmsman cluster access (primary connection)
+ - Name: `CLUSTER_WIREGUARD_WG0`
+ - Value: Contents of `peer1.conf`
+ - Purpose: Helmsman cluster access (primary connection)
3. **CLUSTER_WIREGUARD_WG1 Secret:**
- - Name: `CLUSTER_WIREGUARD_WG1`
- - Value: Contents of `peer2.conf`
- - Purpose: Helmsman cluster access (secondary connection)
+ - Name: `CLUSTER_WIREGUARD_WG1`
+ - Value: Contents of `peer2.conf`
+ - Purpose: Helmsman cluster access (secondary connection)
#### Secret Configuration Summary
```yaml
# Required Environment Secrets for WireGuard Access
-TF_WG_CONFIG: "" # Terraform workflows
-CLUSTER_WIREGUARD_WG0: "" # Helmsman primary
-CLUSTER_WIREGUARD_WG1: "" # Helmsman secondary
+TF_WG_CONFIG: "" # Terraform workflows
+CLUSTER_WIREGUARD_WG0: "" # Helmsman primary
+CLUSTER_WIREGUARD_WG1: "" # Helmsman secondary
```
#### Purpose of Multiple Configurations
diff --git a/terraform/modules/aws/README.md b/terraform/modules/aws/README.md
index 8dbc8517..59d2ba33 100644
--- a/terraform/modules/aws/README.md
+++ b/terraform/modules/aws/README.md
@@ -9,72 +9,72 @@ The infrastructure consists of four key components:
1. **AWS Resource Creation:**
- * This component is responsible for creating all the necessary AWS infrastructure that other components depend on. It sets up:
- * AWS Security Groups to control network traffic.
- * IAM roles and policies are used to grant permission to the Nginx EC2 instance to update DNS records for certbot validation in order to generate an SSL certificate.
- * EC2 Instances for Nginx load balancer, Kubernetes Control Plane, ETCD, and Worker nodes.
- * DNS Records (Route53) for domain name resolution.
+ * This component is responsible for creating all the necessary AWS infrastructure that other components depend on. It sets up:
+ * AWS Security Groups to control network traffic.
+ * IAM roles and policies are used to grant permission to the Nginx EC2 instance to update DNS records for certbot validation in order to generate an SSL certificate.
+ * EC2 Instances for Nginx load balancer, Kubernetes Control Plane, ETCD, and Worker nodes.
+ * DNS Records (Route53) for domain name resolution.
2. **NGINX setup:**
- * The NGINX setup component is responsible for deploying an NGINX server and updating its configuration file to act as a load balancer.
- * This ensures that external traffic can be routed to Istio node ports, which then route the request to the services running on the Kubernetes cluster.
+ * The NGINX setup component is responsible for deploying an NGINX server and updating its configuration file to act as a load balancer.
+ * This ensures that external traffic can be routed to Istio node ports, which then route the request to the services running on the Kubernetes cluster.
3. **RKE2 Setup**
- * This component focuses on deploying the `Rancher Kubernetes Engine 2 (RKE2) cluster` and importing it into the `Rancher UI` dashboard.
- * It also handles downloading necessary files such as the `kubectl` binary and `kubeconfig` file from the control plane nodes, using the infrastructure provisioned by the AWS component. It manages:
+ * This component focuses on deploying the `Rancher Kubernetes Engine 2 (RKE2) cluster` and importing it into the `Rancher UI` dashboard.
+ * It also handles downloading necessary files such as the `kubectl` binary and `kubeconfig` file from the control plane nodes, using the infrastructure provisioned by the AWS component. It manages:
4. **NFS Setup**
- * The NFS setup component provides shared file storage for the Kubernetes cluster. This involves:
- * **NFS Server Setup** for hosting the storage.
- * **NFS Client Configuration** for enabling Kubernetes workloads to use NFS volumes as storage class.
- * By default, the Nginx node is used as the NFS server.
- If you wish to designate a separate node for this purpose, please update the following variables in the `main.tf` file for the `nfs-setup` module:
- * `NFS_SERVER_LOCATION`
- * `NFS_SERVER`
- * `SSH_PRIVATE_KEY`
+ * The NFS setup component provides shared file storage for the Kubernetes cluster. This involves:
+ * **NFS Server Setup** for hosting the storage.
+ * **NFS Client Configuration** for enabling Kubernetes workloads to use NFS volumes as storage class.
+ * By default, the Nginx node is used as the NFS server.
+ If you wish to designate a separate node for this purpose, please update the following variables in the `main.tf` file for the `nfs-setup` module:
+ * `NFS_SERVER_LOCATION`
+ * `NFS_SERVER`
+ * `SSH_PRIVATE_KEY`
## Create MOSIP Infrastructure
### Prerequisites
* Import `infra` repository to your GitHub account and set its visibility to private to ensure confidentiality.
- - Go to the repository URL and click "Use this template" or "Fork"
- - Set repository visibility to "Private"
- - Import may take some time to prepare the repository
+ - Go to the repository URL and click "Use this template" or "Fork"
+ - Set repository visibility to "Private"
+ - Import may take some time to prepare the repository
* Create a new branch `env-` from master branch.
* Goto `terraform/` location and update environment related details in `env.tfvars` file.
- * `CLUSTER_NAME`: The name of the Kubernetes cluster.Example: `sandbox`
- * `CLUSTER_ENV_DOMAIN`: The domain name for MOSIP.Example: `sandbox.xyz.net`
- * `MOSIP_EMAIL_ID`: The email address used by Certbot to send SSL certificate expiry notifications.
- * `SSH_KEY_NAME`: The SSH key name used for accessing AWS node instances via SSH. Ensure an SSH key pair is created/exists on AWS, and provide the key pair name in this field.Example: `my-ssh-key`
- * `AWS_PROVIDER_REGION`: The AWS region where resources will be created.Example: `ap-south-1`
- * `K8S_INSTANCE_TYPE`: The instance type for Kubernetes nodes.Default: `t3a.2xlarge`
- * `NGINX_INSTANCE_TYPE`: The instance type for the Nginx server.Default: `t3a.medium`
- * `ZONE_ID`: The Route 53 hosted zone ID associated with the domain.
- * `AMI`: The Amazon Machine Image (AMI) ID for the instances.Default: `ami-0ad21ae1d0696ad58`Note: `This is specific to Ubuntu 24.04.`
- * `K8S_INFRA_REPO_URL`: The URL of the Kubernetes infrastructure repository.Default: `https://github.com/mosip/k8s-infra.git`
- * `K8S_INFRA_BRANCH`: The branch of the Kubernetes infrastructure repository to be used.Default: `MOSIP-34911`
- * `NGINX_NODE_ROOT_VOLUME_SIZE`: The root volume size (in GB) for the Nginx node.Default: `24`
- * `NGINX_NODE_EBS_VOLUME_SIZE`: The EBS volume size (in GB) for the Nginx node.
- This volume will be used as a NFS server location for kubernetes storage class.Default: `300`
- * `K8S_INSTANCE_ROOT_VOLUME_SIZE`: The root volume size (in GB) for the Kubernetes nodes.Default: `64`
- * `K8S_CONTROL_PLANE_NODE_COUNT`: The number of control-plane nodes for the Kubernetes cluster. These nodes will serve as a control-plane, ETCD, and worker node within the Kubernetes cluster.Default: `4`
- * `K8S_ETCD_NODE_COUNT`: The number of ETCD nodes in the Kubernetes cluster. These nodes will serve as a ETCD and worker node within the Kubernetes cluster.Default: `2`
- * `K8S_WORKER_NODE_COUNT`: The number of worker nodes in the Kubernetes cluster. These nodes will serve as a worker node within the Kubernetes cluster.Default: `2`
- * `RANCHER_IMPORT_URL`: The Rancher import URL used to import the Kubernetes cluster into Rancher.
- Default: `"kubectl apply -f "`
+ * `CLUSTER_NAME`: The name of the Kubernetes cluster.Example: `sandbox`
+ * `CLUSTER_ENV_DOMAIN`: The domain name for MOSIP.Example: `sandbox.xyz.net`
+ * `MOSIP_EMAIL_ID`: The email address used by Certbot to send SSL certificate expiry notifications.
+ * `SSH_KEY_NAME`: The SSH key name used for accessing AWS node instances via SSH. Ensure an SSH key pair is created/exists on AWS, and provide the key pair name in this field.Example: `my-ssh-key`
+ * `AWS_PROVIDER_REGION`: The AWS region where resources will be created.Example: `ap-south-1`
+ * `K8S_INSTANCE_TYPE`: The instance type for Kubernetes nodes.Default: `t3a.2xlarge`
+ * `NGINX_INSTANCE_TYPE`: The instance type for the Nginx server.Default: `t3a.medium`
+ * `ZONE_ID`: The Route 53 hosted zone ID associated with the domain.
+ * `AMI`: The Amazon Machine Image (AMI) ID for the instances.Default: `ami-0ad21ae1d0696ad58`Note: `This is specific to Ubuntu 24.04.`
+ * `K8S_INFRA_REPO_URL`: The URL of the Kubernetes infrastructure repository.Default: `https://github.com/mosip/k8s-infra.git`
+ * `K8S_INFRA_BRANCH`: The branch of the Kubernetes infrastructure repository to be used.Default: `MOSIP-34911`
+ * `NGINX_NODE_ROOT_VOLUME_SIZE`: The root volume size (in GB) for the Nginx node.Default: `24`
+ * `NGINX_NODE_EBS_VOLUME_SIZE`: The EBS volume size (in GB) for the Nginx node.
+ This volume will be used as a NFS server location for kubernetes storage class.Default: `300`
+ * `K8S_INSTANCE_ROOT_VOLUME_SIZE`: The root volume size (in GB) for the Kubernetes nodes.Default: `64`
+ * `K8S_CONTROL_PLANE_NODE_COUNT`: The number of control-plane nodes for the Kubernetes cluster. These nodes will serve as a control-plane, ETCD, and worker node within the Kubernetes cluster.Default: `4`
+ * `K8S_ETCD_NODE_COUNT`: The number of ETCD nodes in the Kubernetes cluster. These nodes will serve as a ETCD and worker node within the Kubernetes cluster.Default: `2`
+ * `K8S_WORKER_NODE_COUNT`: The number of worker nodes in the Kubernetes cluster. These nodes will serve as a worker node within the Kubernetes cluster.Default: `2`
+ * `RANCHER_IMPORT_URL`: The Rancher import URL used to import the Kubernetes cluster into Rancher.
+ Default: `"kubectl apply -f "`
### Run `terraform plan / apply` workflow to set up MOSIP infrastructure
* This GitHub Action automates the Terraform workflow,
- allowing users to run `terraform plan` and optionally `terraform apply` commands within a CI/CD pipeline.
- The workflow is triggered manually via workflow_dispatch.
+ allowing users to run `terraform plan` and optionally `terraform apply` commands within a CI/CD pipeline.
+ The workflow is triggered manually via workflow_dispatch.
* To trigger this workflow:
- 1. Go to the `Actions` tab in your GitHub repository
- 2. Select `terraform plan / apply` workflow
- 3. Click `Run workflow`
- 4. Provide the required inputs
- 5. Click `Run workflow` to start the workflow
+ 1. Go to the `Actions` tab in your GitHub repository
+ 2. Select `terraform plan / apply` workflow
+ 3. Click `Run workflow`
+ 4. Provide the required inputs
+ 5. Click `Run workflow` to start the workflow
### Inputs
@@ -92,13 +92,13 @@ The infrastructure consists of four key components:
### Run `terraform destroy` workflow
* This GitHub Action automates the `Terraform destroy` command within a CI/CD pipeline.
- The workflow can be manually triggered to destroy infrastructure managed by Terraform.
+ The workflow can be manually triggered to destroy infrastructure managed by Terraform.
* To trigger this workflow:
- 1. Go to the `Actions` tab in your GitHub repository
- 2. Select `terraform destroy` workflow
- 3. Click `Run workflow`
- 4. Provide the required inputs
- 5. Click `Run workflow` to start the workflow
+ 1. Go to the `Actions` tab in your GitHub repository
+ 2. Select `terraform destroy` workflow
+ 3. Click `Run workflow`
+ 4. Provide the required inputs
+ 5. Click `Run workflow` to start the workflow
### Inputs
@@ -129,11 +129,11 @@ The setup includes:
* Terraform version: `v1.8.4`
* AWS Account
* AWS CLI configured with appropriate credentials
- ```
- $ export AWS_ACCESS_KEY_ID=
- $ export AWS_SECRET_ACCESS_KEY=
- $ export TF_VAR_SSH_PRIVATE_KEY=
- ```
+ ```
+ $ export AWS_ACCESS_KEY_ID=
+ $ export AWS_SECRET_ACCESS_KEY=
+ $ export TF_VAR_SSH_PRIVATE_KEY=
+ ```
## Files
@@ -147,36 +147,36 @@ The setup includes:
* Initialize Terraform.
- ```
- terraform init
- ```
+ ```
+ terraform init
+ ```
* Review and modify variable values:
- * Ensure `locals.tf` contains correct values for your setup.
- * Update values in `env.tfvars` as per your organization requirement.
+ * Ensure `locals.tf` contains correct values for your setup.
+ * Update values in `env.tfvars` as per your organization requirement.
* Terraform validate & plan the terraform scripts:
- ```
- terraform validate
- ```
+ ```
+ terraform validate
+ ```
- ```
- terraform plan -var-file="./aws.tfvars
- ```
+ ```
+ terraform plan -var-file="./aws.tfvars
+ ```
* Apply the Terraform configuration:
- ```
- terraform apply -var-file="./aws.tfvars
- ```
+ ```
+ terraform apply -var-file="./aws.tfvars
+ ```
## Destroy
To destroy AWS resources, follow the steps below:
* Ensure to have `terraform.tfstate` file.
- ```
- terraform destroy -var-file=./aws.tfvars
- ```
+ ```
+ terraform destroy -var-file=./aws.tfvars
+ ```
## COMPONENTS
@@ -185,41 +185,41 @@ To destroy AWS resources, follow the steps below:
This module is responsible for creating the AWS resources needed for the MOSIP platform, including security groups, an NGINX server, and a Kubernetes cluster nodes.
* Inputs:
- * `CLUSTER_NAME`: The name of the Kubernetes cluster.
- * `AWS_PROVIDER_REGION`: The AWS region for resource creation.
- * `SSH_KEY_NAME`: The name of the SSH key for accessing instances.
- * `K8S_INSTANCE_TYPE`: The instance type for Kubernetes nodes.
- * `NGINX_INSTANCE_TYPE`: The instance type for the NGINX server.
- * `CLUSTER_ENV_DOMAIN`: The domain name for the MOSIP platform.
- * `ZONE_ID`: The Route 53 hosted zone ID.
- * `AMI`: The Amazon Machine Image ID for the instances.
- * `SECURITY_GROUP`: Security group configurations.
+ * `CLUSTER_NAME`: The name of the Kubernetes cluster.
+ * `AWS_PROVIDER_REGION`: The AWS region for resource creation.
+ * `SSH_KEY_NAME`: The name of the SSH key for accessing instances.
+ * `K8S_INSTANCE_TYPE`: The instance type for Kubernetes nodes.
+ * `NGINX_INSTANCE_TYPE`: The instance type for the NGINX server.
+ * `CLUSTER_ENV_DOMAIN`: The domain name for the MOSIP platform.
+ * `ZONE_ID`: The Route 53 hosted zone ID.
+ * `AMI`: The Amazon Machine Image ID for the instances.
+ * `SECURITY_GROUP`: Security group configurations.
#### nginx-setup
This module sets up NGINX and configures it with the provided domain and SSL certificates.
* Inputs:
- * `NGINX_PUBLIC_IP`: The public IP address of the NGINX server.
- * `CLUSTER_ENV_DOMAIN`: The domain name for the MOSIP platform.
- * `MOSIP_K8S_CLUSTER_NODES_PRIVATE_IP_LIST`: List of private IP addresses of the Kubernetes nodes.
- * `MOSIP_PUBLIC_DOMAIN_LIST`: List of public domain names.
- * `CERTBOT_EMAIL`: The email ID for SSL certificate generation.
- * `SSH_KEY_NAME`: SSH private key used for login (i.e., file content of SSH pem key).
+ * `NGINX_PUBLIC_IP`: The public IP address of the NGINX server.
+ * `CLUSTER_ENV_DOMAIN`: The domain name for the MOSIP platform.
+ * `MOSIP_K8S_CLUSTER_NODES_PRIVATE_IP_LIST`: List of private IP addresses of the Kubernetes nodes.
+ * `MOSIP_PUBLIC_DOMAIN_LIST`: List of public domain names.
+ * `CERTBOT_EMAIL`: The email ID for SSL certificate generation.
+ * `SSH_KEY_NAME`: SSH private key used for login (i.e., file content of SSH pem key).
#### rke2-setup
This module sets up RKE2 kubernetes cluster.
* **Primary Control Plane Node Setup :**
- This resource sets up the primary control plane node.
- It connects to the node via SSH, uploads the rke2-setup.sh script, and executes it to configure the node.
+ This resource sets up the primary control plane node.
+ It connects to the node via SSH, uploads the rke2-setup.sh script, and executes it to configure the node.
* **Additional Nodes Setup :**
- This resource sets up additional nodes (control plane, ETCD, and worker nodes) in the cluster.
- It follows a similar process to the primary node setup.
+ This resource sets up additional nodes (control plane, ETCD, and worker nodes) in the cluster.
+ It follows a similar process to the primary node setup.
* **Rancher Import :**
- This resource imports the RKE2 cluster into Rancher.
- It connects to the primary control plane node, configures kubectl, and applies the Rancher import URL.
+ This resource imports the RKE2 cluster into Rancher.
+ It connects to the primary control plane node, configures kubectl, and applies the Rancher import URL.
#### nfs-setup
@@ -227,12 +227,12 @@ This Terraform module, nfs-setup, is designed to configure and deploy an NFS (Ne
The module depends on other infrastructure modules for AWS resource creation and RKE2 setup.
* Inputs:
- * `NFS_SERVER_LOCATION`: The location on the NFS server where the files will be stored. This is dynamically set based on the `CLUSTER_ENV_DOMAIN` variable.
- * `NFS_SERVER`: The private IP address of the NGINX server, which acts as the NFS server. This value is retrieved from the aws-resource-creation module.
- * `SSH_PRIVATE_KEY`: The SSH private key used for accessing the NFS server.
- * `K8S_INFRA_REPO_URL`: The URL of the Kubernetes infrastructure repository where the configuration files are stored.
- * `K8S_INFRA_BRANCH`: The branch of the Kubernetes infrastructure repository to be used.
- * `CLUSTER_NAME`: The name of the Kubernetes cluster that will use the NFS setup.
+ * `NFS_SERVER_LOCATION`: The location on the NFS server where the files will be stored. This is dynamically set based on the `CLUSTER_ENV_DOMAIN` variable.
+ * `NFS_SERVER`: The private IP address of the NGINX server, which acts as the NFS server. This value is retrieved from the aws-resource-creation module.
+ * `SSH_PRIVATE_KEY`: The SSH private key used for accessing the NFS server.
+ * `K8S_INFRA_REPO_URL`: The URL of the Kubernetes infrastructure repository where the configuration files are stored.
+ * `K8S_INFRA_BRANCH`: The branch of the Kubernetes infrastructure repository to be used.
+ * `CLUSTER_NAME`: The name of the Kubernetes cluster that will use the NFS setup.
## Outputs
diff --git a/terraform/modules/aws/aws-resource-creation/README.md b/terraform/modules/aws/aws-resource-creation/README.md
index cf9bfe95..c2a5e1ab 100644
--- a/terraform/modules/aws/aws-resource-creation/README.md
+++ b/terraform/modules/aws/aws-resource-creation/README.md
@@ -15,10 +15,10 @@ This Terraform script sets up an AWS infrastructure that includes:
* Terraform version: `v1.8.4`
* AWS Account
* AWS CLI configured with appropriate credentials
- ```
- $ export AWS_ACCESS_KEY_ID=
- $ export AWS_SECRET_ACCESS_KEY=
- ```
+ ```
+ $ export AWS_ACCESS_KEY_ID=
+ $ export AWS_SECRET_ACCESS_KEY=
+ ```
* Ensure SSH key created for accessing EC2 instances on AWS.
## Files
@@ -32,46 +32,46 @@ This Terraform script sets up an AWS infrastructure that includes:
* Initialize Terraform
- ```
- terraform init
- ```
+ ```
+ terraform init
+ ```
* Review and modify variable values:
- * Ensure `aws.tfvars` contains correct values for your setup.
- * Verify `variables.tf` for any additional configuration needs.
+ * Ensure `aws.tfvars` contains correct values for your setup.
+ * Verify `variables.tf` for any additional configuration needs.
* Terraform validate & plan the terraform scripts:
- ```
- terraform validate
- ```
+ ```
+ terraform validate
+ ```
- ```
- terraform plan -var-file="aws.tfvars"
- ```
+ ```
+ terraform plan -var-file="aws.tfvars"
+ ```
* Apply the Terraform configuration:
- ```
- terraform apply -var-file="aws.tfvars"
- ```
+ ```
+ terraform apply -var-file="aws.tfvars"
+ ```
## Destroy
To destroy AWS resources, follow the steps below:
* Ensure to have `terraform.tfstate` file.
- ```
- terraform destroy
- ```
+ ```
+ terraform destroy
+ ```
## Terraform Scripts
#### certbot-ssl-certgen.tf
* Defines resources for setting up IAM roles and policies for Certbot:
- * `aws_iam_role.certbot_role`: IAM role for Certbot with EC2 assume role policy.
- * `aws_iam_policy.certbot_policy`: IAM policy allowing Certbot to modify Route 53 records.
- * `aws_iam_role_policy_attachment.certbot_policy_attachment`: Attaches the policy to the role.
- * `aws_iam_instance_profile.certbot_profile`: Creates an instance profile for the IAM role.
+ * `aws_iam_role.certbot_role`: IAM role for Certbot with EC2 assume role policy.
+ * `aws_iam_policy.certbot_policy`: IAM policy allowing Certbot to modify Route 53 records.
+ * `aws_iam_role_policy_attachment.certbot_policy_attachment`: Attaches the policy to the role.
+ * `aws_iam_instance_profile.certbot_profile`: Creates an instance profile for the IAM role.
#### aws.tfvars
@@ -83,14 +83,14 @@ To destroy AWS resources, follow the steps below:
#### aws-main.tf
* Defines the main resources and provider configuration:
- * `Providers`: AWS provider configuration.
- * `Security Groups`: aws_security_group.security-group for NGINX and Kubernetes.
- * `EC2 Instances`:
- * **aws_instance.NGINX_EC2_INSTANCE** for NGINX.
- * **aws_instance.K8S_CLUSTER_EC2_INSTANCE** for Kubernetes.
- * `Route 53 Records`:
- * **aws_route53_record.MAP_DNS_TO_IP** for A records.
- * **aws_route53_record.MAP_DNS_TO_CNAME** for CNAME records.
+ * `Providers`: AWS provider configuration.
+ * `Security Groups`: aws_security_group.security-group for NGINX and Kubernetes.
+ * `EC2 Instances`:
+ * **aws_instance.NGINX_EC2_INSTANCE** for NGINX.
+ * **aws_instance.K8S_CLUSTER_EC2_INSTANCE** for Kubernetes.
+ * `Route 53 Records`:
+ * **aws_route53_record.MAP_DNS_TO_IP** for A records.
+ * **aws_route53_record.MAP_DNS_TO_CNAME** for CNAME records.
#### outputs.tf
diff --git a/terraform/modules/aws/nfs-setup/README.md b/terraform/modules/aws/nfs-setup/README.md
index 1aec9043..688fd3f0 100644
--- a/terraform/modules/aws/nfs-setup/README.md
+++ b/terraform/modules/aws/nfs-setup/README.md
@@ -10,10 +10,10 @@ The module configures the NFS server and deploys the necessary components to ena
* Terraform version: `v1.8.4`
* AWS Account
* AWS CLI configured with appropriate credentials
- ```
- $ export AWS_ACCESS_KEY_ID=
- $ export AWS_SECRET_ACCESS_KEY=
- ```
+ ```
+ $ export AWS_ACCESS_KEY_ID=
+ $ export AWS_SECRET_ACCESS_KEY=
+ ```
* Ensure SSH key created for accessing EC2 instances on AWS.
* Ensure you have access to the private SSH key that corresponds to the public key used when launching the EC2 instance.
* Domain and DNS: Ensure that you have a domain and that its DNS is managed by Route 53.
@@ -28,32 +28,32 @@ The module configures the NFS server and deploys the necessary components to ena
* Initialize Terraform
- ```
- terraform init
- ```
+ ```
+ terraform init
+ ```
* Terraform validate & plan the terraform scripts:
- ```
- terraform validate
- ```
+ ```
+ terraform validate
+ ```
- ```
- terraform plan
- ```
+ ```
+ terraform plan
+ ```
* Apply the Terraform configuration:
- ```
- terraform apply
- ```
+ ```
+ terraform apply
+ ```
## Destroy
To destroy AWS resources, follow the steps below:
* Ensure to have `terraform.tfstate` file.
- ```
- terraform destroy
- ```
+ ```
+ terraform destroy
+ ```
## Input Variables
@@ -69,11 +69,11 @@ To destroy AWS resources, follow the steps below:
#### nfs-setup-main.tf
* **null_resource.nfs-server-setup**:
- This resource uses SSH to connect to the NFS server and runs the install-nfs-server.sh script to configure the NFS server.
- The script logs its output to a file `/tmp/nfs-server-log` on the server.
+ This resource uses SSH to connect to the NFS server and runs the install-nfs-server.sh script to configure the NFS server.
+ The script logs its output to a file `/tmp/nfs-server-log` on the server.
* **null_resource.nfs-csi-setup**:
- This resource runs the nfs-csi.sh script locally after the NFS server has been set up.
- The script configures the NFS CSI driver and logs its output locally `./tmp/nfs-csi-log`.
+ This resource runs the nfs-csi.sh script locally after the NFS server has been set up.
+ The script configures the NFS CSI driver and logs its output locally `./tmp/nfs-csi-log`.
#### nfs-csi.sh
diff --git a/terraform/modules/aws/nginx-setup/README.md b/terraform/modules/aws/nginx-setup/README.md
index 07f7f3d6..29c47570 100644
--- a/terraform/modules/aws/nginx-setup/README.md
+++ b/terraform/modules/aws/nginx-setup/README.md
@@ -10,10 +10,10 @@ It fetches SSL certificates using Certbot and integrates with Kubernetes infrast
* Terraform version: `v1.8.4`
* AWS Account
* AWS CLI configured with appropriate credentials
- ```
- $ export AWS_ACCESS_KEY_ID=
- $ export AWS_SECRET_ACCESS_KEY=
- ```
+ ```
+ $ export AWS_ACCESS_KEY_ID=
+ $ export AWS_SECRET_ACCESS_KEY=
+ ```
* Ensure SSH key created for accessing EC2 instances on AWS.
* Ensure you have access to the private SSH key that corresponds to the public key used when launching the EC2 instance.
* Domain and DNS: Ensure that you have a domain and that its DNS is managed by Route 53.
@@ -28,32 +28,32 @@ It fetches SSL certificates using Certbot and integrates with Kubernetes infrast
* Initialize Terraform
- ```
- terraform init
- ```
+ ```
+ terraform init
+ ```
* Terraform validate & plan the terraform scripts:
- ```
- terraform validate
- ```
+ ```
+ terraform validate
+ ```
- ```
- terraform plan -var-file="aws.tfvars"
- ```
+ ```
+ terraform plan -var-file="aws.tfvars"
+ ```
* Apply the Terraform configuration:
- ```
- terraform apply -var-file="aws.tfvars"
- ```
+ ```
+ terraform apply -var-file="aws.tfvars"
+ ```
## Destroy
To destroy AWS resources, follow the steps below:
* Ensure to have `terraform.tfstate` file.
- ```
- terraform destroy
- ```
+ ```
+ terraform destroy
+ ```
## Input Variables
@@ -72,12 +72,12 @@ The script `main.tf` defines a local variable NGINX_CONFIG containing various co
#### main.tf
* **null_resource "Nginx-setup"**: This resource performs the following actions:
- * `Triggers`: Sets up triggers based on the hash of the Kubernetes cluster nodes' private IP list and the public domain list.
- * `Connection`: Defines the SSH connection parameters for the EC2 instance.
- * `File Provisioner`: Uploads the nginx-setup.sh script to the EC2 instance.
- * `Remote Exec Provisioner`: Executes the necessary commands to:
- * Set environment variables.
- * Run the nginx-setup.sh script.
+ * `Triggers`: Sets up triggers based on the hash of the Kubernetes cluster nodes' private IP list and the public domain list.
+ * `Connection`: Defines the SSH connection parameters for the EC2 instance.
+ * `File Provisioner`: Uploads the nginx-setup.sh script to the EC2 instance.
+ * `Remote Exec Provisioner`: Executes the necessary commands to:
+ * Set environment variables.
+ * Run the nginx-setup.sh script.
#### nginx-setup.sh:
diff --git a/terraform/modules/aws/rancher-keycloak-setup/FINAL_CLEAN_RANCHER_KEYCLOAK_IMPLEMENTATION.md b/terraform/modules/aws/rancher-keycloak-setup/FINAL_CLEAN_RANCHER_KEYCLOAK_IMPLEMENTATION.md
index 759c6ab6..5d5c4cf7 100644
--- a/terraform/modules/aws/rancher-keycloak-setup/FINAL_CLEAN_RANCHER_KEYCLOAK_IMPLEMENTATION.md
+++ b/terraform/modules/aws/rancher-keycloak-setup/FINAL_CLEAN_RANCHER_KEYCLOAK_IMPLEMENTATION.md
@@ -25,48 +25,48 @@ Successfully implemented automated Rancher UI and Keycloak installation **ONLY f
```
terraform/
-├── modules/aws/ # CLEAN - No Rancher/Keycloak
-│ ├── aws-main.tf # ✅ Clean, ends at nfs-setup
-│ ├── variables.tf # ✅ No Rancher/Keycloak vars
-│ ├── outputs.tf # ✅ No Rancher/Keycloak outputs
-│ └── rancher-keycloak-setup/ # ✅ Isolated module
-│ ├── main.tf # Rancher/Keycloak logic
-│ ├── variables.tf # Setup variables
-│ └── outputs.tf # Setup outputs
-├── observ-infra/ # ONLY PLACE WITH INTEGRATION
-│ ├── variables.tf # ✅ Has Rancher/Keycloak vars
-│ ├── outputs.tf # ✅ Has Rancher/Keycloak outputs
-│ └── aws/
-│ ├── main.tf # ✅ Calls rancher-keycloak-setup
-│ ├── variables.tf # ✅ Has Rancher/Keycloak vars
-│ └── outputs.tf # ✅ Has Rancher/Keycloak outputs
-├── infra/ # ✅ CLEAN - No Rancher/Keycloak
-│ ├── main.tf # Only calls main AWS module
-│ ├── variables.tf # No Rancher/Keycloak vars
-│ └── outputs.tf # No Rancher/Keycloak outputs
+├── modules/aws/ # CLEAN - No Rancher/Keycloak
+│ ├── aws-main.tf # ✅ Clean, ends at nfs-setup
+│ ├── variables.tf # ✅ No Rancher/Keycloak vars
+│ ├── outputs.tf # ✅ No Rancher/Keycloak outputs
+│ └── rancher-keycloak-setup/ # ✅ Isolated module
+│ ├── main.tf # Rancher/Keycloak logic
+│ ├── variables.tf # Setup variables
+│ └── outputs.tf # Setup outputs
+├── observ-infra/ # ONLY PLACE WITH INTEGRATION
+│ ├── variables.tf # ✅ Has Rancher/Keycloak vars
+│ ├── outputs.tf # ✅ Has Rancher/Keycloak outputs
+│ └── aws/
+│ ├── main.tf # ✅ Calls rancher-keycloak-setup
+│ ├── variables.tf # ✅ Has Rancher/Keycloak vars
+│ └── outputs.tf # ✅ Has Rancher/Keycloak outputs
+├── infra/ # ✅ CLEAN - No Rancher/Keycloak
+│ ├── main.tf # Only calls main AWS module
+│ ├── variables.tf # No Rancher/Keycloak vars
+│ └── outputs.tf # No Rancher/Keycloak outputs
└── implementations/
- ├── aws/infra/ # ✅ CLEAN - No Rancher/Keycloak
- └── aws/observ-infra/ # ✅ ONLY PLACE WITH INTEGRATION
- ├── aws.tfvars # Has Rancher/Keycloak config
- ├── variables.tf # Has Rancher/Keycloak vars
- └── outputs.tf # Has Rancher/Keycloak outputs
+ ├── aws/infra/ # ✅ CLEAN - No Rancher/Keycloak
+ └── aws/observ-infra/ # ✅ ONLY PLACE WITH INTEGRATION
+ ├── aws.tfvars # Has Rancher/Keycloak config
+ ├── variables.tf # Has Rancher/Keycloak vars
+ └── outputs.tf # Has Rancher/Keycloak outputs
```
## Execution Flow (observ-infra only)
```
1. AWS Infrastructure (via main AWS module)
- ├── VPC/Subnet discovery
- ├── EC2 instances creation
- ├── NGINX setup
- ├── RKE2 cluster setup
- └── NFS setup
-
+ ├── VPC/Subnet discovery
+ ├── EC2 instances creation
+ ├── NGINX setup
+ ├── RKE2 cluster setup
+ └── NFS setup
+
2. Rancher-Keycloak Setup (observ-infra specific)
- ├── Install cert-manager
- ├── Install Rancher UI via Helm
- ├── Clone k8s-infra repository
- └── Install Keycloak via script
+ ├── Install cert-manager
+ ├── Install Rancher UI via Helm
+ ├── Clone k8s-infra repository
+ └── Install Keycloak via script
```
## Configuration (observ-infra only)
@@ -74,20 +74,20 @@ terraform/
### Enable/Disable Integration
```hcl
# In aws.tfvars (observ-infra)
-enable_rancher_keycloak_integration = true # Enable for observ-infra
-rancher_hostname = "" # Defaults to rancher.testvpc.mosip.net
-keycloak_hostname = "" # Defaults to iam.testvpc.mosip.net
+enable_rancher_keycloak_integration = true # Enable for observ-infra
+rancher_hostname = "" # Defaults to rancher.testvpc.mosip.net
+keycloak_hostname = "" # Defaults to iam.testvpc.mosip.net
rancher_bootstrap_password = "admin"
```
### Disable Integration
```hcl
-enable_rancher_keycloak_integration = false # No Rancher/Keycloak installation
+enable_rancher_keycloak_integration = false # No Rancher/Keycloak installation
```
## What Each Deployment Does
-### 🏗️ **Main Infra Deployment** (`terraform/implementations/aws/infra/`)
+### **Main Infra Deployment** (`terraform/implementations/aws/infra/`)
```bash
terraform apply -var-file="aws.tfvars"
```
@@ -108,7 +108,7 @@ terraform apply -var-file="aws.tfvars"
- **+ Keycloak** at https://iam.testvpc.mosip.net
- Ready for observability tools integration
-### 🔧 **Base-Infra Deployment** (`terraform/implementations/aws/base-infra/`)
+### **Base-Infra Deployment** (`terraform/implementations/aws/base-infra/`)
```bash
terraform apply -var-file="aws.tfvars"
```
@@ -154,7 +154,7 @@ terraform apply -var-file="aws.tfvars"
### 2. **Verify Main Infra Remains Clean**
```bash
cd terraform/implementations/aws/infra
-terraform plan -var-file="aws.tfvars" # Should show NO Rancher/Keycloak
+terraform plan -var-file="aws.tfvars" # Should show NO Rancher/Keycloak
```
### 3. **Access Applications (observ-infra only)**
@@ -169,4 +169,4 @@ terraform plan -var-file="aws.tfvars" # Should show NO Rancher/Keycloak
- Professional, isolated, maintainable implementation
- Ready for production deployment
-The implementation now perfectly meets the requirement: **Rancher-Keycloak integration only for observ-infra, not for infra or base-infra.** 🎯
+The implementation now perfectly meets the requirement: **Rancher-Keycloak integration only for observ-infra, not for infra or base-infra.**
diff --git a/terraform/modules/aws/rke2-cluster/README.md b/terraform/modules/aws/rke2-cluster/README.md
index e0601d91..cd526377 100644
--- a/terraform/modules/aws/rke2-cluster/README.md
+++ b/terraform/modules/aws/rke2-cluster/README.md
@@ -9,10 +9,10 @@ The RKE2 configuration is managed through a GitHub repository.
* Terraform version: `v1.8.4`
* AWS Account
* AWS CLI configured with appropriate credentials
- ```
- $ export AWS_ACCESS_KEY_ID=
- $ export AWS_SECRET_ACCESS_KEY=
- ```
+ ```
+ $ export AWS_ACCESS_KEY_ID=
+ $ export AWS_SECRET_ACCESS_KEY=
+ ```
* Ensure SSH key created for accessing EC2 instances on AWS.
* Ensure you have access to the private SSH key that corresponds to the public key used when launching the EC2 instance.
* Git is installed on the EC2 instance.
@@ -23,27 +23,27 @@ The RKE2 configuration is managed through a GitHub repository.
## Setup
* Initialize Terraform
- ```
- terraform init
- ```
+ ```
+ terraform init
+ ```
* Terraform validate & plan the terraform scripts:
- ```
- terraform validate
- ```
- ```
- terraform plan -var-file="aws.tfvars"
- ```
+ ```
+ terraform validate
+ ```
+ ```
+ terraform plan -var-file="aws.tfvars"
+ ```
* Apply the Terraform configuration:
- ```
- terraform apply -var-file="aws.tfvars"
- ```
+ ```
+ terraform apply -var-file="aws.tfvars"
+ ```
## Destroy
To destroy AWS resources, follow the steps below:
* Ensure to have `terraform.tfstate` file.
- ```
- terraform destroy
- ```
+ ```
+ terraform destroy
+ ```
## Input Variables
* `K8S_CLUSTER_PUBLIC_IPS`: Map of public IP addresses for the Kubernetes cluster nodes.
diff --git a/terraform/modules/aws/rke2-cluster/ansible/README.md b/terraform/modules/aws/rke2-cluster/ansible/README.md
index f39d5118..94ac2fe2 100644
--- a/terraform/modules/aws/rke2-cluster/ansible/README.md
+++ b/terraform/modules/aws/rke2-cluster/ansible/README.md
@@ -13,8 +13,8 @@ This module has been converted from using Terraform's unreliable `remote-exec` p
- ✅ **Internet connectivity**
**Auto-Installed:**
-- 📦 **Ansible** - automatically installed during `terraform apply`
-- 📦 **pip3** - installed with Ansible if needed
+- **Ansible** - automatically installed during `terraform apply`
+- **pip3** - installed with Ansible if needed
### **Supported Operating Systems for Terraform Runner:**
- ✅ **Ubuntu/Debian** (apt-get)
@@ -24,7 +24,7 @@ This module has been converted from using Terraform's unreliable `remote-exec` p
- ✅ **macOS** (brew)
- ✅ **Any Linux with pip3**
-## 🚀 Quick Start
+## Quick Start
### **1. Pre-flight Check (Optional but Recommended)**
```bash
@@ -36,23 +36,23 @@ cd terraform/implementations/aws/infra
```bash
terraform init
terraform plan
-terraform apply # Ansible will be auto-installed if needed
+terraform apply # Ansible will be auto-installed if needed
```
That's it! No manual Ansible installation required.
-## 🔄 How Ansible Installation Works
+## How Ansible Installation Works
### **Automatic Installation Flow:**
```mermaid
graph TD
- A[terraform apply] --> B[local-exec: run-ansible.sh]
- B --> C{Ansible installed?}
- C -->|No| D[Auto-install Ansible]
- C -->|Yes| E[Use existing Ansible]
- D --> F[Run Ansible Playbook]
- E --> F
- F --> G[RKE2 Cluster Ready]
+ A[terraform apply] --> B[local-exec: run-ansible.sh]
+ B --> C{Ansible installed?}
+ C -->|No| D[Auto-install Ansible]
+ C -->|Yes| E[Use existing Ansible]
+ D --> F[Run Ansible Playbook]
+ E --> F
+ F --> G[RKE2 Cluster Ready]
```
### **Installation Methods by OS:**
@@ -78,7 +78,7 @@ graph TD
- ✅ Kubernetes cluster formation
- ✅ kubectl configuration
-## 🛠️ Manual Ansible Installation (if preferred)
+## Manual Ansible Installation (if preferred)
If you prefer to install Ansible manually before running terraform:
@@ -131,14 +131,14 @@ export PATH="$HOME/.local/bin:$PATH"
```
ansible/
-├── inventory.yml.tpl # Terraform template for Ansible inventory
-├── rke2-playbook.yml # Main Ansible playbook for RKE2 installation
-├── run-ansible.sh # Script to execute Ansible with proper settings
-├── test-cluster.sh # Test script to verify cluster health
-├── inventory.yml # Generated inventory file (created by Terraform)
-├── ssh_key # SSH private key file (created by Terraform)
+├── inventory.yml.tpl # Terraform template for Ansible inventory
+├── rke2-playbook.yml # Main Ansible playbook for RKE2 installation
+├── run-ansible.sh # Script to execute Ansible with proper settings
+├── test-cluster.sh # Test script to verify cluster health
+├── inventory.yml # Generated inventory file (created by Terraform)
+├── ssh_key # SSH private key file (created by Terraform)
├── primary-kubeconfig.yaml # Downloaded kubeconfig (created by Ansible)
-└── kubeconfigs/ # Directory with all node kubeconfigs
+└── kubeconfigs/ # Directory with all node kubeconfigs
```
## Usage
@@ -179,7 +179,7 @@ ansible all -i inventory.yml -u ubuntu --private-key=ssh_key -m ping
# Check service status
ansible rke2_cluster -i inventory.yml -u ubuntu --private-key=ssh_key \
- -m shell -a "sudo systemctl status rke2-server || sudo systemctl status rke2-agent"
+ -m shell -a "sudo systemctl status rke2-server || sudo systemctl status rke2-agent"
# Re-run the playbook
ansible-playbook -i inventory.yml -u ubuntu --private-key=ssh_key rke2-playbook.yml
@@ -227,7 +227,7 @@ The following variables are automatically passed from Terraform to Ansible:
Node roles are automatically detected based on hostname patterns:
- `*CONTROL-PLANE-NODE*`: Control plane nodes
-- `*ETCD-NODE*`: ETCD nodes
+- `*ETCD-NODE*`: ETCD nodes
- `*WORKER-NODE*`: Worker nodes
## Next Steps
diff --git a/terraform/modules/azure/README.md b/terraform/modules/azure/README.md
index 42130e72..31070f12 100644
--- a/terraform/modules/azure/README.md
+++ b/terraform/modules/azure/README.md
@@ -6,8 +6,8 @@ This module facilitates the deployment of MOSIP (Modular Open Source Identity Pl
- Resource Creation: Establishes foundational Azure resources necessary for MOSIP operations.
- NGINX Setup: Configures NGINX as a reverse proxy or load balancer to manage and distribute incoming traffic efficiently.
-- RKE2 (Rancher Kubernetes Engine 2) Cluster Setup: Deploys a Kubernetes cluster using RKE2, ensuring scalable and reliable orchestration of containerized applications.
-- NFS (Network File System) Server Setup: Implements an NFS server to provide shared storage solutions across the infrastructure.
+- RKE2 (Rancher Kubernetes Engine 2) Cluster Setup: Deploys a Kubernetes cluster using RKE2, ensuring scalable and reliable orchestration of containerized applications.
+- NFS (Network File System) Server Setup: Implements an NFS server to provide shared storage solutions across the infrastructure.
## Community Contributions
diff --git a/terraform/modules/gcp/README.md b/terraform/modules/gcp/README.md
index 6b889f3b..203da843 100644
--- a/terraform/modules/gcp/README.md
+++ b/terraform/modules/gcp/README.md
@@ -6,8 +6,8 @@ This module facilitates the deployment of MOSIP (Modular Open Source Identity Pl
- Resource Creation: Establishes foundational Azure resources necessary for MOSIP operations.
- NGINX Setup: Configures NGINX as a reverse proxy or load balancer to manage and distribute incoming traffic efficiently.
-- RKE2 (Rancher Kubernetes Engine 2) Cluster Setup: Deploys a Kubernetes cluster using RKE2, ensuring scalable and reliable orchestration of containerized applications.
-- NFS (Network File System) Server Setup: Implements an NFS server to provide shared storage solutions across the infrastructure.
+- RKE2 (Rancher Kubernetes Engine 2) Cluster Setup: Deploys a Kubernetes cluster using RKE2, ensuring scalable and reliable orchestration of containerized applications.
+- NFS (Network File System) Server Setup: Implements an NFS server to provide shared storage solutions across the infrastructure.
## Community Contributions
From 201d3305857674f3f5d96f33f7b5a6e608f0c4ab Mon Sep 17 00:00:00 2001
From: bhumi46 <111699703+bhumi46@users.noreply.github.com>
Date: Fri, 10 Oct 2025 13:21:10 +0530
Subject: [PATCH 2/3] [MOSIP-43149]Docs improvements bug fixes (#86)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
* [MOSIP-43211]Add image registry to multiple chart configurations
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* [MOSIP-43211]Set image registry for multiple MOSIP charts
Added image registry configuration for various charts to use 'harbor.mosip.net'.
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* [MOSIP-43211]Update image registry for Elasticsearch and Kibana
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* [MOSIP-43211]Add image registry to multiple Helm chart configurations
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update image registry in mosip-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update image repository paths in mosip-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update image repository paths in external-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update testrigs-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* [MOSIP-43149] Add comprehensive beginner-friendly documentation
- Added 6 new documentation guides (Glossary, Secret Generation, Workflow, DSF Configuration, Environment Destruction)
- Enhanced all README files with plain language explanations
- Added links to official documentation throughout
- Removed decorative emojis for professional appearance
- Preserved functional symbols (✅/❌) for clarity
- Created documentation index for easy navigation
Fixes: MOSIP-43149
Signed-off-by: bhumi46
* Fix kubectl command: Replace 'kubectl get pods -n mosip' with 'kubectl get pods -A'
- Updated all documentation to use correct command for listing all pods
- Changed from namespace-specific to all-namespaces listing
- Affected files: README.md, docs/WORKFLOW_GUIDE.md, docs/README.md, Helmsman/helmsman-workflow-guide.md
Signed-off-by: bhumi46
* Remove marketing-style 'What Makes These Guides Beginner-Friendly' section
- Removed redundant promotional content
- Keep documentation professional and concise
- The guides speak for themselves
Signed-off-by: bhumi46
* Add Quick Destruction section for fast teardown (6-10 minutes)
- Added fast destruction method using Terraform Destroy workflow
- No backup required for test/dev environments
- Complete teardown in 6-10 minutes
- Clear warnings about data loss
- Step-by-step instructions for GitHub Actions workflow
- Verification steps after destruction
Signed-off-by: bhumi46
* Add comprehensive Rancher import configuration documentation
- Document how to generate Rancher import URL from observ-infra
- Step-by-step guide for cluster import in Rancher UI
- Critical string escaping requirements for rancher_import_url
- Troubleshooting section for import failures
- Benefits of Rancher cluster management
- Fixes missing documentation bug for enable_rancher_import
Fixes: MOSIP-43149 (additional bug found during QA)
Signed-off-by: bhumi46
* Remove AWS KMS reference - not configured in current setup
- Changed 'AWS KMS' to 'S3 server-side encryption' for S3 backend
- Reflects actual encryption method used
- AWS KMS is not configured in this infrastructure
Signed-off-by: bhumi46
* Update mosip-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update mosip-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Remove image registry from multiple Helm charts
Removed image registry references for various charts.
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update mosip-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Remove image registry from Elasticsearch and Kibana
Removed image registry entries for Elasticsearch and Kibana.
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update testrigs-dsf.yaml
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* Update default value for ENABLE_STATE_LOCKING
Changed default value of ENABLE_STATE_LOCKING from true to false.
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* [MOSIP-43149] updated docs
Signed-off-by: bhumi46
* [MOSIP-43149] updated docs
Signed-off-by: bhumi46
* [MOSIP-43149] updated docs
Signed-off-by: bhumi46
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* Add workflow screenshots and update documentation
- Add annotated screenshots for base-infra terraform apply/destroy
- Add annotated screenshots for infra terraform apply/destroy
- Add annotated screenshots for Helmsman external services and testrigs deployments
- Update README.md with screenshot links and DSL orchestrator instructions
- Update ENVIRONMENT_DESTRUCTION_GUIDE.md with screenshot links
- Fix cronjob name: cronjob-dslorchestrator-full
- Add post-deployment steps for test rigs (cron updates, manual job trigger)
- Remove observ-infra deployment step from Rancher import section
Signed-off-by: bhumi46
* [MOSIP-43149]updated docs
Signed-off-by: bhumi46
* Add workflow screenshots and update documentation
- Add annotated screenshots for base-infra terraform apply/destroy
- Add annotated screenshots for infra terraform apply/destroy
- Add annotated screenshots for Helmsman external services and testrigs deployments
- Update README.md with screenshot links and DSL orchestrator instructions
- Update ENVIRONMENT_DESTRUCTION_GUIDE.md with screenshot links
- Fix cronjob name: cronjob-dslorchestrator-full
- Add post-deployment steps for test rigs (cron updates, manual job trigger)
- Remove observ-infra deployment step from Rancher import section
Signed-off-by: bhumi46
* Add workflow screenshots and update documentation
- Add annotated screenshots for base-infra terraform apply/destroy
- Add annotated screenshots for infra terraform apply/destroy
- Add annotated screenshots for Helmsman external services and testrigs deployments
- Update README.md with screenshot links and DSL orchestrator instructions
- Update ENVIRONMENT_DESTRUCTION_GUIDE.md with screenshot links
- Fix cronjob name: cronjob-dslorchestrator-full
- Add post-deployment steps for test rigs (cron updates, manual job trigger)
- Remove observ-infra deployment step from Rancher import section
Signed-off-by: bhumi46
---------
Signed-off-by: bhumi46 <111699703+bhumi46@users.noreply.github.com>
Signed-off-by: bhumi46
---
Helmsman/helmsman-workflow-guide.md | 2 +-
README.md | 134 +++++----
docs/ENVIRONMENT_DESTRUCTION_GUIDE.md | 4 +
docs/ONBOARDING_GUIDE.md | 266 ++++++++++++++++++
docs/_images/base-infra-terraform-apply.png | Bin 0 -> 644614 bytes
docs/_images/base-infra-terraform-destroy.png | Bin 0 -> 631199 bytes
docs/_images/helmsman-external-services.png | Bin 0 -> 665024 bytes
docs/_images/helmsman-testrigs.png | Bin 0 -> 661022 bytes
docs/_images/infra-terraform-apply.png | Bin 0 -> 644008 bytes
docs/_images/infra-terraform-destroy.png | Bin 0 -> 623088 bytes
10 files changed, 350 insertions(+), 56 deletions(-)
create mode 100644 docs/ONBOARDING_GUIDE.md
create mode 100644 docs/_images/base-infra-terraform-apply.png
create mode 100644 docs/_images/base-infra-terraform-destroy.png
create mode 100644 docs/_images/helmsman-external-services.png
create mode 100644 docs/_images/helmsman-testrigs.png
create mode 100644 docs/_images/infra-terraform-apply.png
create mode 100644 docs/_images/infra-terraform-destroy.png
diff --git a/Helmsman/helmsman-workflow-guide.md b/Helmsman/helmsman-workflow-guide.md
index d85b537e..91f2bdce 100644
--- a/Helmsman/helmsman-workflow-guide.md
+++ b/Helmsman/helmsman-workflow-guide.md
@@ -90,7 +90,7 @@ The following secrets are required to run this workflow:
**Error Recovery:**
- If automatic trigger fails: Manually run "Deploy MOSIP services using Helmsman" from Actions tab
-- If onboarding processes fail: Manual re-onboarding is required (see limitations in main README)
+- If onboarding processes fail: Manual re-onboarding is required (see [MOSIP Onboarding Guide](../docs/ONBOARDING_GUIDE.md))
- Monitor deployment logs for any failures requiring intervention
### Triggering the Workflow Manually
diff --git a/README.md b/README.md
index 130289ed..fad3f8cf 100644
--- a/README.md
+++ b/README.md
@@ -28,29 +28,29 @@ We've created comprehensive beginner-friendly guides to help you succeed:
graph TB
%% Prerequisites
A[Fork Repository] --> B[Configure Secrets]
- B --> C[Select Cloud Provider
AWS/Azure/GCP/etc.]
+ B --> C[Select Cloud Provider]
%% Infrastructure Phase
- C --> D[Run Terraform: base-infra
VPC, Networking, Jumpserver, WireGuard
One-time deployment]
- D --> E{Deploy Observability?}
- E -->|Yes| F[Run Terraform: observ-infra
Rancher UI, Keycloak, Monitoring
Can be destroyed/recreated]
- E -->|No| G[Run Terraform: infra
Complete MOSIP Infrastructure Setup
Can be destroyed/recreated]
+ C --> D[Terraform: base-infra
VPC, Networking, WireGuard]
+ D --> E{Deploy
Observability?}
+ E -->|Yes| F[Terraform: observ-infra
Rancher UI, Monitoring]
+ E -->|No| G[Terraform: infra
MOSIP Infrastructure]
F --> G
%% Helmsman Deployment Phase
- G --> H[Deploy Helmsman: Prerequisites
Monitoring, Istio, Logging]
- H --> I[Deploy Helmsman: External Dependencies
PostgreSQL containers, Keycloak, MinIO, Kafka]
+ G --> H[Helmsman: Prerequisites
Monitoring, Istio, Logging]
+ H --> I[Helmsman: External Deps
PostgreSQL, Keycloak, MinIO]
%% MOSIP Services
- I --> J[Deploy Helmsman: MOSIP Core Services]
- J --> K{Deploy Test Rigs?}
- K -->|Yes| L[Deploy Helmsman: Test Rigs
API Testing, UI Testing, DSL Testing]
+ I --> J[Helmsman: MOSIP Services]
+ J --> K{Deploy
Test Rigs?}
+ K -->|Yes| L[Helmsman: Test Rigs
API, UI, DSL Testing]
K -->|No| M[Verify Deployment]
L --> M
%% Final Verification
- M --> N[Access MOSIP Platform
Web UI, APIs, Admin Console]
- N --> O[Complete MOSIP Platform]
+ M --> N[Access MOSIP Platform]
+ N --> O[Deployment Complete]
%% Styling
classDef prereq fill:#fff3e0,stroke:#ff8f00,stroke-width:2px
@@ -364,16 +364,16 @@ Add the required secrets as follows:
> **New to Terraform workflows?** Check our [Workflow Guide](docs/WORKFLOW_GUIDE.md) for visual step-by-step instructions on navigating GitHub Actions!
-#### Understanding Terraform Apply vs Dry Run
+#### Understanding Terraform Apply vs Terraform Plan
Before running any Terraform workflow, understand these modes:
-| Mode | What It Does | When to Use | Visual |
-| ----------------------------------------- | ---------------------------------------------- | ------------------------------------------ | ------------------ |
-| **Dry Run** (checkbox unchecked ☐) | Shows what WOULD happen without making changes | Testing configurations, previewing changes | ☐ Terraform apply |
-| **Apply** (checkbox checked ✅) | Actually creates/modifies infrastructure | Real deployments, making actual changes | ✅ Terraform apply |
+| Mode | What It Does | When to Use | Visual |
+| ------------------------------------------------ | ---------------------------------------------- | ------------------------------------------ | ------------------ |
+| **Terraform Plan** (checkbox unchecked ☐) | Shows what WOULD happen without making changes | Testing configurations, previewing changes | ☐ Terraform apply |
+| **Apply** (checkbox checked ✅) | Actually creates/modifies infrastructure | Real deployments, making actual changes | ✅ Terraform apply |
-**Tip**: Always do a dry run first to preview changes, then run with apply checked to actually deploy!
+**Tip**: Always run terraform plan first to preview changes, then run with apply checked to actually deploy!
#### Step 3a: Base Infrastructure
@@ -407,6 +407,8 @@ Before running any Terraform workflow, understand these modes:
> **Detailed Navigation Guide**: See [Workflow Guide - Terraform Workflows](docs/WORKFLOW_GUIDE.md#workflow-1-base-infrastructure) for step-by-step screenshots
+
+
- Go to **Actions** → **Terraform Base Infrastructure**
- **Can't find it?** Look in the left sidebar under "All workflows"
- Click **Run workflow** (green button on the right)
@@ -425,9 +427,9 @@ Before running any Terraform workflow, understand these modes:
- **SSH_PRIVATE_KEY**: GitHub secret name containing SSH private key for instance access
- Must match the `ssh_key_name` in your terraform.tfvars
- **Terraform apply**:
-- ☐ **Unchecked** - Dry run (preview only, no changes made)
+- ☐ **Unchecked** - Terraform plan (preview only, no changes made)
- ✅ **Checked** - Apply (actually creates infrastructure)
-- **First time?** Uncheck for dry run, then run again with checked
+- **First time?** Uncheck for terraform plan, then run again with checked
**What You Should See:**
@@ -718,20 +720,7 @@ nginx_node_ebs_volume_size_2 = 200 # Enable second EBS volume for PostgreSQL
If you have deployed **observ-infra** (Rancher management cluster), you can import your main infra cluster into Rancher for centralized monitoring and management.
-**Step 1: Deploy observ-infra (if not already done)**
-
-```bash
-# Deploy Rancher management cluster first
-Actions → Terraform Infrastructure → Run workflow
-Parameters:
-├─ Component: observ-infra
-├─ Cloud Provider: aws
-└─ Backend: local
-```
-
-Wait for observ-infra deployment to complete (~15-20 minutes).
-
-**Step 2: Generate Rancher Import URL**
+**Step 1: Generate Rancher Import URL**
1. **Access Rancher UI:**
@@ -800,6 +789,8 @@ After updating `aws.tfvars`, deploy or update your main infra cluster:
2. **Run main infra via GitHub Actions:**
+
+
- Go to **Actions** → **Terraform Infrastructure**
- Click **Run workflow**
- **Configure workflow parameters:**
@@ -811,7 +802,7 @@ After updating `aws.tfvars`, deploy or update your main infra cluster:
- `s3` - Remote S3 backend (recommended for production)
- **SSH_PRIVATE_KEY**: GitHub secret name containing SSH private key for instance access
- Must match the `ssh_key_name` in your terraform.tfvars
- - **Action**: Select `apply` to deploy infrastructure
+- **Action**: Select `apply` to deploy infrastructure
**Verify Rancher Import (Only if rancher_import = true):**
@@ -855,15 +846,18 @@ To regenerate import URL if needed:
3. Click ⋮ (three dots) → Edit Config
4. Copy the new registration command
-### 4. Helmsman Deployment#### Step 4a: Update DSF Configuration Files
+ ```
+ Cluster Name: soil38 (use your cluster_name from aws.tfvars)
+
+### 4. Helmsman Deployment
> **What is DSF?** DSF (Desired State File) is like a recipe that tells Helmsman what applications to install and how to configure them. [Learn more](docs/GLOSSARY.md#dsf-desired-state-file)
>
> **Detailed DSF Guide:** [DSF Configuration Guide](docs/DSF_CONFIGURATION_GUIDE.md) - Comprehensive guide with examples and explanations!
-**Time required:** 20-30 minutes to update all files
+#### Step 4a: Update DSF Configuration Files
-1. **Clone the MOSIP infra repository and navigate to Helmsman directory:**
+1. **Clone the repository (if not already done):**
```bash
git clone https://github.com/mosip/infra.git
@@ -925,6 +919,7 @@ To regenerate import URL if needed:
```
**Need detailed help?** [DSF Configuration Guide - Prerequisites](docs/DSF_CONFIGURATION_GUIDE.md#prerequisites-dsf-configuration)
+
4. **Update external-dsf.yaml:**
**Critical Updates Required:**
@@ -1086,9 +1081,10 @@ To regenerate import URL if needed:
- Select or create environment for your branch (e.g., `release-0.1.0`, `main`, `develop`)
- Click "Add secret" under Environment secrets
- Name: `KUBECONFIG`
- - Value: Copy the entire contents of the kubeconfig file from `terraform/implementations/aws/infra/`
+- Value: Copy the entire contents of the kubeconfig file from `terraform/implementations/aws/infra/`
**Branch Environment Configuration:- Ensure the environment name matches your deployment branch
+
- Configure environment protection rules if needed
- Verify Helmsman workflows reference the correct environment
@@ -1105,17 +1101,6 @@ To regenerate import URL if needed:
CLUSTER_WIREGUARD_WG1: "peer2-wireguard-config" # Helmsman cluster access (peer2)
```
- **Repository Secrets (global):**
-
-```yaml
- # GPG Encryption (if using encrypted backends)
- GPG_PASSPHRASE: "your-gpg-passphrase"
-
- # AWS Credentials (if not using OIDC)
- AWS_ACCESS_KEY_ID: "AKIA..."
- AWS_SECRET_ACCESS_KEY: "..."
-```
-
4. **Verify Secret Configuration:**
- Ensure KUBECONFIG is configured as environment secret for your branch
@@ -1153,6 +1138,8 @@ The Helmsman deployment process follows a specific sequence with automated trigg
> **Detailed Steps:** [Workflow Guide - Prerequisites & External Dependencies](docs/WORKFLOW_GUIDE.md#workflow-1-prerequisites--external-dependencies)
+
+
- Actions → **"Deploy External services of mosip using Helmsman"** (or "Helmsman External Dependencies")
- **Can't find it?** Search for "External" in the workflows list
- This workflow handles both deployments in parallel:
@@ -1183,7 +1170,6 @@ The Helmsman deployment process follows a specific sequence with automated trigg
**Error Handling:**
- If the automatic trigger fails, manually trigger: Actions → **Deploy MOSIP services using Helmsman**
-- If onboarding processes fail during deployment, manual re-onboarding is required (see limitations section)
3. **Verify All Pods are Running:**
@@ -1199,16 +1185,54 @@ The Helmsman deployment process follows a specific sequence with automated trigg
kubectl get pods --all-namespaces | grep -v Running | grep -v Completed
```
-4. **Deploy Test Rigs (Manual):**
+4. **Handle Onboarding Failures (If Required):**
+
+> **⚠️ Important**: The partner-onboarder pod will run successfully, but you must check the onboarding reports in MinIO to verify if all partners were onboarded correctly. Failed onboardings must be manually re-executed before deploying test rigs.
+
+**When to check and rerun onboarding:**
+
+- After the partner-onboarder pod completes (check MinIO reports for failures)
+- When onboarding reports show failed partner registrations
+- Before deploying test rigs to ensure all prerequisites are met
+
+**How to check onboarding status and rerun if needed:**
+
+Refer to the comprehensive [MOSIP Onboarding Guide](docs/ONBOARDING_GUIDE.md) for detailed instructions on:
+
+- [Verifying partner-onboarder pod completion](docs/ONBOARDING_GUIDE.md#step-1-verify-partner-onboarder-pod-completion)
+- [How to access MinIO and check onboarding reports](docs/ONBOARDING_GUIDE.md#step-2-access-minio-to-check-onboarding-reports)
+- [Checking onboarding reports in MinIO](docs/ONBOARDING_GUIDE.md#step-3-check-onboarding-reports-in-minio)
+- [How to delete failed jobs and rerun partner-onboarder](docs/ONBOARDING_GUIDE.md#manual-partner-re-onboarding-procedures)
+- [Updating mosip-dsf.yaml to retry failed modules](docs/ONBOARDING_GUIDE.md#step-2-update-mosip-dsfyaml-configuration)
+- [Verification steps after rerunning](docs/ONBOARDING_GUIDE.md#step-4-verify-success-in-minio-reports)
-- **Prerequisites**: All pods from steps 1-2 must be in `Running` state
+5. **Deploy Test Rigs (Manual):**
+
+
+
+- **Prerequisites**: All pods from steps 1-2 must be in `Running` state and onboarding completed successfully
- Actions → **Deploy Testrigs of mosip using Helmsman** (`helmsman_testrigs.yml`)
- Select DSF file: `testrigs-dsf.yaml`
- Mode: `apply` (required - dry-run will fail due to namespace dependencies)
-> **Important**: Test rigs should only be deployed after verifying all core services are running successfully. Failed onboarding processes must be manually re-executed before test rig deployment.
+**Post-Deployment Steps:**
+
+After test rigs deployment completes:
+
+1. **Update cron schedules**: Update the cron time for all CronJobs in the `dslrig`, `apitestrig`, and `uitestrig` namespaces as needed
+2. **Trigger DSL orchestrator**:
+ ```bash
+ kubectl create job --from=cronjob/cronjob-dslorchestrator-full dslrig-manual-run -n dslrig
+ ```
+
+ > **Note**: This job will run for more than 3 hours. Monitor progress with:
+ >
+ > ```bash
+ > kubectl logs -f job/dslrig-manual-run -n dslrig
+ > ```
+ >
-### 5. Verify Deployment
+### 6. Verify Deployment
```bash
# Check cluster status
diff --git a/docs/ENVIRONMENT_DESTRUCTION_GUIDE.md b/docs/ENVIRONMENT_DESTRUCTION_GUIDE.md
index 5dd1c9c1..59419a0d 100644
--- a/docs/ENVIRONMENT_DESTRUCTION_GUIDE.md
+++ b/docs/ENVIRONMENT_DESTRUCTION_GUIDE.md
@@ -36,6 +36,8 @@ Navigate to: **Repository → Actions → Terraform Infrastructure Destroy**
#### 2. Run the Destroy Workflow
+
+
Click **"Run workflow"** and configure:
```
@@ -76,6 +78,8 @@ aws elb describe-load-balancers --output table | grep mosip
If you also want to delete the VPC and networking (base-infra):
+
+
1. Go to **Actions → Terraform Infrastructure Destroy**
2. Run workflow with:
```
diff --git a/docs/ONBOARDING_GUIDE.md b/docs/ONBOARDING_GUIDE.md
new file mode 100644
index 00000000..89b225e5
--- /dev/null
+++ b/docs/ONBOARDING_GUIDE.md
@@ -0,0 +1,266 @@
+# MOSIP Onboarding Guide
+
+This guide provides detailed instructions for handling partner onboarding processes in MOSIP deployments, including troubleshooting failed automated onboarding and performing manual re-onboarding.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Understanding MOSIP Onboarding](#understanding-mosip-onboarding)
+- [Identifying Failed Onboarding Processes](#identifying-failed-onboarding-processes)
+- [Manual Partner Re-onboarding Procedures](#manual-partner-re-onboarding-procedures)
+- [Verification Steps](#verification-steps)
+
+## Overview
+
+Partner onboarding is a critical process in MOSIP deployments that configures authentication and integration partners required for the system to function properly. During automated Helmsman deployments, onboarding processes may occasionally fail due to timing issues, resource constraints, or configuration dependencies.
+
+**When manual onboarding is required:**
+
+- Partner-onboarder pod completes successfully, but MinIO reports show onboarding failures
+- Onboarding reports indicate some partners failed to register
+- Test rig deployment prerequisites fail due to incomplete onboarding
+- Partner-related services are not responding correctly
+
+> **Important**: The partner-onboarder pod will typically complete with a "Completed" status, but this does NOT guarantee all partners were onboarded successfully. You MUST check the onboarding reports in MinIO to verify actual onboarding status.
+
+## Understanding MOSIP Onboarding
+
+### What is Partner Onboarding?
+
+Partner onboarding in MOSIP involves:
+
+- Registering authentication partners (IDA, eSignet, etc.)
+- Configuring partner policies and certificates
+- Setting up partner API access and credentials
+- Establishing trust relationships between MOSIP components
+
+## Identifying Failed Onboarding Processes
+
+### Step 1: Verify Partner-Onboarder Pod Completion
+
+```bash
+# Check partner-onboarder pod status
+kubectl get pod -n onboarder
+
+# Expected output: Status should be "Completed"
+# NAME READY STATUS RESTARTS AGE
+# partner-onboarder-xxxxx 0/1 Completed 0 5m
+```
+
+> **Note**: A "Completed" status only means the pod finished execution, NOT that all onboarding succeeded!
+
+### Step 2: Access MinIO to Check Onboarding Reports
+
+Onboarding reports are stored in MinIO and contain detailed success/failure information for each partner.
+
+#### Access MinIO UI
+
+**Option 1: Via MinIO Direct URL**
+
+- Access MinIO at: `https://minio.sandbox.xyz.net` (replace `sandbox.xyz.net` with your domain)
+
+**Option 2: Via Landing Page**
+
+- Access landing page at: `https://sandbox.xyz.net` (replace with your domain)
+- Click on MinIO link
+
+#### Get MinIO Credentials
+
+**If using Rancher UI:**
+
+```bash
+# Navigate in Rancher UI:
+# 1. Go to your cluster
+# 2. Navigate to minio namespace
+# 3. Go to Storage → Secrets
+# 4. Select s3 secret to find credentials
+```
+
+**If deployed without observ-infra:**
+
+```bash
+# Get MinIO password using kubectl
+kubectl get secret minio -n minio -o jsonpath='{.data.root-password}' | base64 -d
+```
+
+**Login credentials:**
+
+- Username: `admin`
+- Password: Use the secret retrieved above
+
+### Step 3: Check Onboarding Reports in MinIO
+
+1. **Login to MinIO Console** using credentials from above
+2. **Navigate to the `onboarder` bucket**
+3. **Go to the `reports` folder**
+4. **Download all report files:**
+ - `abis` report
+ - `print` report
+ - `ida` report
+ - Other partner reports
+5. **Check whether all reports show PASSED status**
+ - Open each downloaded report
+ - Verify that all partners have successfully onboarded
+ - Look for any FAILED or ERROR entries
+
+### Step 4: Review Partner-Onboarder Pod Logs (Optional)
+
+While MinIO reports are the primary source of truth, pod logs can provide additional debugging context:
+
+```bash
+# Get partner-onboarder pod name
+POD_NAME=$(kubectl get pod -n onboarder -o jsonpath='{.items[0].metadata.name}')
+
+# View pod logs
+kubectl logs -n onboarder $POD_NAME
+
+# Search for specific errors
+kubectl logs -n onboarder $POD_NAME | grep -i "error\|failed\|exception"
+
+# Check if reports were uploaded to MinIO
+kubectl logs -n onboarder $POD_NAME | grep -i "minio\|report uploaded"
+```
+
+## Manual Partner Re-onboarding Procedures
+
+### Method 1: Rerun Partner-Onboarder Job (Recommended)
+
+This is the primary method to retry failed onboarding after reviewing MinIO reports.
+
+#### Step 1: Delete the Completed Partner-Onboarder Job
+
+**Option 1: If deployed with observ-infra (Rancher UI available)**
+
+1. **Access Rancher UI**
+2. **Navigate to your cluster**
+3. **Go to the `onboarder` namespace**
+4. **Navigate to Workloads → Jobs**
+5. **Delete all jobs that have failed reports**
+ - Identify jobs related to failed onboarding reports
+ - Select the jobs and delete them
+
+**Option 2: If deployed without Rancher UI (kubectl method)**
+
+```bash
+# Find the partner-onboarder job
+kubectl get jobs -n onboarder
+
+# Example output:
+# NAME COMPLETIONS DURATION AGE
+# partner-onboarder-xxxxx 1/1 2m5s 10m
+
+# Delete the job (replace with actual job name)
+kubectl delete job -n onboarder
+
+# Example:
+kubectl delete job partner-onboarder -n onboarder
+
+# Verify deletion
+kubectl get jobs -n onboarder
+# Should show no results or remaining jobs only
+```
+
+#### Step 2: Update mosip-dsf.yaml Configuration
+
+After deleting the failed report jobs, you need to update the `mosip-dsf.yaml` file to configure which modules should be re-onboarded:
+
+1. **Navigate to the mosip-dsf.yaml file:**
+ - Location: `Helmsman/dsf/mosip-dsf.yaml`
+
+2. **Find the partner-onboarder app section:**
+ - Search for `partner-onboarder:` in the file
+
+3. **Update module enabled status based on report results:**
+ - **For successful reports (PASSED)**: Set `enabled: false`
+ - **For failed reports (FAILED/ERROR)**: Set `enabled: true`
+
+**Example Configuration:**
+
+If your MinIO reports show:
+- ✅ `ida` report: SUCCESS → Set `enabled: false`
+- ❌ `print` report: FAILED → Set `enabled: true`
+- ❌ `abis` report: FAILED → Set `enabled: true`
+- ❌ `resident` report: FAILED → Set `enabled: true`
+
+```yaml
+partner-onboarder:
+ namespace: onboarder
+ enabled: true
+ version: 12.0.1
+ chart: mosip/partner-onboarder
+ set:
+ onboarding.configmaps.s3.s3-host: "http://minio.minio:9000"
+ onboarding.configmaps.s3.s3-user-key: "admin"
+ onboarding.configmaps.s3.s3-region: ""
+ onboarding.configmaps.s3.s3-bucket-name: "onboarder"
+ extraEnvVarsCM[0]: "global"
+ extraEnvVarsCM[1]: "keycloak-env-vars"
+ extraEnvVarsCM[2]: "keycloak-host"
+ onboarding.modules[0].name: "ida"
+ onboarding.modules[0].enabled: false # SUCCESS - skip
+ onboarding.modules[1].name: "print"
+ onboarding.modules[1].enabled: true # FAILED - retry
+ onboarding.modules[2].name: "abis"
+ onboarding.modules[2].enabled: true # FAILED - retry
+ onboarding.modules[3].name: "resident"
+ onboarding.modules[3].enabled: true # FAILED - retry
+ onboarding.modules[4].name: "mobileid"
+ onboarding.modules[4].enabled: true
+ onboarding.modules[5].name: "digitalcard"
+ onboarding.modules[5].enabled: true
+ onboarding.modules[6].name: "esignet"
+ onboarding.modules[6].enabled: false
+```
+
+4. **Commit the changes:**
+ - Commit the updated `mosip-dsf.yaml` file to your repository
+ - This will automatically trigger the Helmsman workflow
+
+5. **The workflow will automatically:**
+ - Detect the changes in `mosip-dsf.yaml`
+ - Recreate the partner-onboarder job
+ - Re-run onboarding only for the modules with `enabled: true`
+
+**Alternative Option: Run Partner-Onboarder Without DSF**
+
+If you prefer to run partner-onboarder directly without using Helmsman DSF, you can use the standalone deployment method:
+
+📖 **Reference Guide:** [Partner-Onboarder Standalone Deployment](https://github.com/mosip/mosip-infra/tree/v1.2.0.2/deployment/v3/mosip/partner-onboarder)
+
+This method allows you to deploy and configure partner-onboarder independently using Helm charts.
+
+#### Step 3: Monitor the Rerun
+
+```bash
+# Watch pod status
+kubectl get pod -n onboarder -w
+
+# Follow logs in real-time
+POD_NAME=$(kubectl get pod -n onboarder -o jsonpath='{.items[0].metadata.name}')
+kubectl logs -n onboarder -f $POD_NAME
+
+# Wait for completion
+kubectl wait --for=condition=complete job/partner-onboarder -n onboarder --timeout=15m
+```
+
+#### Step 4: Verify Success in MinIO Reports
+
+After the rerun completes:
+
+1. **Login to MinIO Console** using credentials from above
+2. **Navigate to the `onboarder` bucket**
+3. **Go to the `reports` folder**
+4. **Download the latest report files and verify all show PASSED status**
+
+## Getting Help
+
+If you continue to experience issues:
+
+1. **Check Logs**: Review all relevant logs thoroughly
+2. **Consult Documentation**: Refer to MOSIP official documentation
+3. **Community Support**: Reach out to MOSIP community forums
+4. **Issue Tracker**: Report persistent issues on the GitHub repository
+
+---
+
+**Note**: This guide is based on MOSIP deployment patterns using Helmsman and Kubernetes. Specific steps may vary based on your environment configuration and MOSIP version.
diff --git a/docs/_images/base-infra-terraform-apply.png b/docs/_images/base-infra-terraform-apply.png
new file mode 100644
index 0000000000000000000000000000000000000000..659368da0f269be3ce4c386a7f370c38a4a988b1
GIT binary patch
literal 644614
zcmYg&N3QJ3vfTw_Abj9!0P0^rYUqL9H0fFN-gN|eZ;HM+gFh<`K{Jw)=lzCo_NmQH
zCX>m`$cPmyBI@5PL;BZ$`EUR7*I$4A>qq9DzyA79|M#!I{=@!X{xdl9FaO*5uYdi=
z|HJM=w2yby4&`5equf7_{zjl?-uAyy{%-_Hm*pQto7QVOZvU9d9h?H!)4H7gy@5VI
z?J%T&BO?9}2weK_5e)y0vfxN-r{e~{Z;JR|1ol6`5wk2q8J3yaf__9CCI5jE?=7nA
z>Du222abofFTpRV%=`Jhg=^M%1ZPATrT+oF|G-H%teJ?YQe~|TWggwsh`-`$3hI=Mq>BY@l&>S(R&^Y9i
zW{;Ba@v+`$S2mE7s(txw52lQbz&q7H#^*a49ND=A2bXn!2`T=9znw?`pEr6DC9#19
ztO1$eM7a9j2X`oW-7I_EBkZA)p0Ktptlpc%4SjO!c{cf;!Qbc<>F)PeJpt8!JGvnD
zXbDd+RqPM4se?w$^iZiUMP8Rjrx5iHDT*%gDUx4X7hMNhroQ=(MCEhq0>78_`yBYn
z1T+o5O9g)T#d<%%{|)?4D0w6ea3eU5=pVDKLiTgh;CAVJ8*l3qC}Txw%Y+xnJI0i(
z@(lb&@B}`N)0*0YLOqni1o}6whBcag&7-modStDT91yb7m8wg-q`F{s8sKE3cO<*N
zA%f2h=xCbkZ#~aoEbkB&@E=9J%)pvtfa9m{_xu?iR(|0ZHyB
zwA6>)gP3_>M5Z#xx^u`HS2&y3JGs7L3npEh_22nmEwwne%67
z;a?|3PkUTfeaIn@1NIe9vG5%uu^S_~_ngF{hUatU9XnfmHx~wD%cikw?V$<)AbMWaSiSq}uC)CVROc);R-C9{?
z=c|Uc>%!_=d#qk*`pD?u)H8j|?+2CE^~A{sN6*zlkDTa$USr{^t;YDhd#i#YoR(Dc
zT_T{4!ule=G2QenVO|9Yeh!vMCH(+L;06}yJro><^u7#-^|0o-Dou3Abb0XV-FIP8
zVO@57mrojeY4?nt#WWx+`tfTM?N4&x;MrITkXhHlnsEUul++cLYtetOE2srwktK3)<}vogp61!wklMlr7{9^onz#
z`__t{;rvcG=yqHqd_TrtH^J}YBz<*#qz6x8Kl@oxhq;xnqB&k`Q4ig`f51USIN)4<
z!v3=u3eX=cae~v6MBhURLldpTNb*g}^ZSg15A+Tjg3G&%g)w$Wl);yTFy~q5R$uDkk}2{%AUFj&j7Z)h?XTA!zLfeYZoSRhZ3kb
z_IbLLA8h5Nr_#|jtxN%dMe39Uv3#gJb=&~95#2y-^;HO3`b8+M$?uC)R$o1AjWFYX
zZbNpR1HQDmZl>S4i>lMXABbZ1eAvl=(z6B_GO;PZhZbZ7v=(>M?Ke!6s5fX!9A2Nu
zj^*2nzpi-K&9!o`ZUT}ansTtTu=t>GhRO4YLd(UAj83A`ocrQ_}VusW&
zTlv-jP3mKP$~gz;ZfY2#?w;-G#u9jRZ?4oKt$j_Ko>@wrN_;-yjElEkI^t>
z7~;GqsZyX>#rxfBUSY%h!4LbOz+i|#4bqeNYseo*6qFU2j2p&+?lzU-
zxoIL_D5)WS9ihKp&6k_tDz#2chML$}U#>dc#_2cnM8biNhnk*SS_*rgXgg*jPR~X8
z;Lf09ae=Q!s{+=8A>RT2?)LF$V3hQtw+UxalB(?Qy<>mT1%0zKLGP~cs;!Ku`XRdD$5e#WUyba
z9!x`&(oUqOZ2=Uec!O=q!V`UDF!S#deUqt(yXZB9y1t1fjOCD2xsU!(B+>AquH@|o
zHRf=WJ3_mi+IRfVo-PZawlYZZwx7FLl0mmi+N6gj)+iUY75-=Y8-|!;G$)xRXF9Z&
z$T%J*3rDp%vYq;Ti)F`?AJl`0B}on^)}ukO21TSq;fKWNIQUxP;XLy!ljhws;x93+D-sQ-0o^0I&E`
z9m{^1KzDo?%B6_DD)vta2^?QRNjwSgFZn{CPxFV2v-~^zzRY0f-RJZYnNLAL7)d9<
z#?27EJ_cGG^Atj)W!`y{vV1hwyn(7+n8ra+=vv2y5(!*W;&o>Q5N
z8g*rXZ9=uuhB}H;erZ|7YfGm#PP3_B1jFKizEuFbT-KK0+1<8=*e~+zsFz&uU3Nz`
zbZ@)5JnM@U`9}nzDtbN`8L|u{1sS+20R9Wc6r)mr!BHc~;U-S?He*!0lWrs|OrZ3P?{Ay*$Z~KJYk~d=z
z)I4PVF&yi-NFfPRIA02Ot582>`WS0pg$!)l{`_$sYeePr(%5@2;tevWiTmOoK{;XW
z3Xvexgx4-al@@i)?$fi)6(WW>v98cXW>?%3wz^D&%9~SLOvl
zbEC6Fmn5*Uw7Zbwuc=|PR2|#f>tDl0&*W)3#Y!IvT8C!c2(JJG6Nws0wT_|@X`%V~
z=t=Gw0#beRD>N|F+pnJo+9}v{B=F~U=XbOJH
z_4%IpYRZb4eLl{wuG=W|7l~)U?uz4)v4oVtHyeBSv?1krXfPB7iNNG{DkF0s$K*=X|;Ji?lzOP9rNB=b%1>mxDlsck#s8K>
z;xzKTntc4?9XtZ?f&C69*-V_OGntoU@8GHO+~9t+2Ya%O84jD#Ka;e$CK4xl>}bIX
zg8M+j{^QbHxiQQ`V!mZ?JJo$+Xfwm8!hS(ZsDh2DD_P7Q0TzfAT}C3_4Uni6nag-BtQV206>;_kA_3?Y$>Z>d84E8g9QCKu~g3R!36q>MVUuMOU`U0R^74G-0(O|$f9L%07&b%UjYFlCc$vm&x8F_LV~0PTNZ^YUe=**k^wQP9eWPLZxEx4u
zBDcn$f*A;w{MhIBUVopSo1!Qui*_BRr0FT0|5yU>)|u8Sso@}_{d~3%dlbSGtzWa-
z7@%Pgwm0iN>HF7%#wWP^u-*k3UT)*B2l8mrG_HPO7)x2|ECR6UAa^Q~x!imy39L8$
z8EBc)C}{OedYW1@xnD%`DXpCSvz(vpo(i=bk@Mh~GoJZ@?wqSPm+t`Y{3bVyJ&q@$
zal0Qbfau_!Rh@z_kQ;Wsx;}}GQbn-~GY$_JG+S|P**HfMx6?qgctf_G=FY%K^)CAX
z$>tH5h=S{MND*%KmL0R}6)Hb1fJdJCN@Z7zmdeL4zW&yU@>L%PA86aa2IIOt`=W6~
zpI>xuHkWv(2mD;Yy^1wyL&rfkdY44SO7;}|{4%WuGuuW#Z5VpJNt5Ii`By^`#$8x4
z!;}}q;;P4t@xAavU6B_g-YpcNKrzH$h6;8;RG3n4yppFLwyXA}p^l$M@3?FFd=|Wq
z-_f7vZ`y0h10CCoV%gD4b{tvisbof44|68?Bkte+yFKKQm!U9Gn{j0e;^;%@CKFdM
z=KgWrkDXIebeB~%w(E`|w91suT(Wih;7jmmyzUM)ud|B~;W)G-uSt}`U
zRqOdd38q$qqyqrd4t_Q@S{uftP=tM@z^sbZOHk#t46(`}=-LVEAiS$7vL|_wseDC9
z^ZUjUTAMC&6$_SHU#0C_r$m^(wW7=paNrio3_cPQ%h@ACw=;QMUGN+qnd*_56^n9_
z49Scb{}Bt~$F&03^gr&-_1)hii@7@BQ$^XUxA>vM;2Nr~k2v+u0bE0H5)p`}n&_aP
z?1n@KLnNxg&f42Q=W%pR2T`lls`wr^l4v)5$Wbck)eR#Z>(rTG>LO3c;X-#Q@z>{d
zjqtBH-2Kfo)hs$l!BjuR5xJ%uDiS7OO2t`C68%kZhTz5fmqyFV=XB*9roZE8X)!o?
zGiRbk!^79MLmz!*fAhfBE7OXMwOa1omJbKxl#<7aqQXUpjlr&VMlJ
zmI3OZF0eYNB63##2s-oOP>uHh4c$F2S^xJ#G%jLRrZz*
zrRStib(M!)V-IZrM-liA;_o=aq=r?T#)MS=tk_9@+U*087mVu|f%)>7s$9anh}wk?
z>4)!gb!NsA?OH>-6c8aXUy2l29cdKaR>JBJsN!@~emN9r>{Lqn&*eq>6KyQquGmB&
z0As97mUZ+8LA-YjKOB!S@$dEXZ3?Ro&-D@j`{>=uU%XQa`L_K4K;-5%65rR{d%%jK
z`iUyaD4!YK+a{n|*u;>z@Ape;2|~alRS{=+}|vd>$ywX^`HcjqP7CX7$eguZ$RfvB*rS^yh+`v6K(U_rDWK%SO}-X*t_uXFKciN)(MTYSiN^0FI)Cvh`E!Q9#b2
z0{0qJ@J-cPsvBJyKW1Wd5fB`-$VXCw
z@)hY5113zAIS398Ez#x7P#cVZT>qeIvXLt*x(w@B>Pr`y{Jcl+0_>#>9BLPa)MD7+
zU(qv2JPt%>htf!I0^UaAK32jF_8{Np0|C6^kh+`s#IRS@y(of?oUolHAFUf;szv7x
zMA^#bI|0*os6KHSp~u?$*iRuxdb@<R)`(fhdcM1;
zCXsqVGFH`TZOXj{>*b@QSBqLK`Bh;Ss=3&R5$)Gsl9o}v*LCZQB|l0C_kKxZ10cC4
zhC55G&JnCW6~Ozd#z@dn+Z4r4ZR~8})+!gUz8q<;7_Bnm3%C4UTM!;W%v(~x4g&H3
znYx5WDe(w!c?hcWN&55hxaotqdQ|;K=hHz>hPkbig~3?*(Dn-(WVE0q$dFX)mPe1ja>cjK+10j3PfnqjGTe
zoM={v8sAJXAt_{st2%OH3YOR2%|dT7Dp0E;>ly@4=?dKWCe~}H0mwu$A+@@D*s1p;
z?~+%!g%94PTMiIwO&S#j-D&Y;X0gH9z(|0MROG}I#As-wnaJaPg*OitP8_I
z%d7%jQ2}anJ7z4m`!jK907GHNL8Z5Wj|RG|Zhxgj$g=_yDM-l^6qrRVaUgFSYA~5>
z$!0id`uecdw6{_8I<Q@B?^EjGg^;wlY!VG{F6M!Lg3d@2S%XXSnP)E8J(p<5VzYr%pjpo%86^#Nx-
zKIuHshlC;F6Ub|&F;7u?2(ue8#x}v-DFfpA#aaZN)|<{j*cHySoQlUy=ZM2h-sJ0@
zBLxEH-`q!TDOx8M4Hn5;;;YWNm@Km^-{ST=gDjgBAAQ)R(iT;quNmhSIDNffo%fv2
z{u$rnhj20j#Agum!+4?uFsTTk>T8=yZ7k=xV}2}`b>mvvF`c>-tbP^t6RnDb=)bFV
z6o-Up1>hd>Ra%O=wNVkaRK%6knk*X89-jNo_2^WG>_rHWnaS3nZnsp+j4
zj}C_#cYOwFfM$OubVj)zwnVMhMX?D$^L&Y$1fH-9?VWku5T%yNe8Uf+^0(yxMJ)E6
zs_&(+jdNcb4aotqEc49^Tu(FX-Ez^bqyUe~C$FVtF1YNz}iv?5Fr0e5Eh
z8z|E`nbLt{e0fiVJxARxCoJp)
zFnAH;j{l51YqNwdkm&B1<54>_i-1!NU!^HNKJdt$f$R|9v>MgZgG||g!iiQQ^U-bq
zYz!FT`4Q6b*t>Z$zg{?L0t9vGCIo>NW*YLQQ>fNfwmu^_$km1iipl(@z+=yUG*ux!
z>SXtNaPw=rfZo_;B^H^f+h6AjCJd53fFzhORdGAq&l`CTxhm~Hf0tvlmvJi&Om|x<
zDWC%N(9+mZto^63tB>4^w!BvCK$-xAM0!C7@X8fKV1GmtcxQ+wK!X{@-z0Xq961xP
zs+cpnoXZgY9VyIQFM-&WUmnd~qeI~Q%qfSe13R{nAW+qr;YJHD^HZ&_e3ZqTw!n6x
zY62FpJl(dPKK0J{^S49u=lH5xmW)8Ni+P$hgOGjoCe8^!@@v@O`eEa@hAN0xwZqOY
z)J2M)siBdQen|`t4HnLwF;M{)?pjn^pOioimh&nZLcQH0BX+!{R0*D@GutY
ziMblrZ)2!2+vfBl&a9uV!3u5RJ6UKsrsT+Dhl=8T#EBpi0M|b7`k7GM@Uuwqm@|U*
z_UITw6JGxM_~za2f`h()iG6S6U7V>p;kT$bvB7kcByKc#g0p?Ao{9ZusPbO)aHHqc
zUAy48uY?kb^ei;a4;~)GZE{P}P}UbpKM=f5(>g+|NIH7po;swy;s58^`kPnonz2(z>?l7sS|5
zLQZz4k1?7a@k;~&WAG-nZ(%oEPk>O%zG+u{l3sV~0w`P7scY@0^5;x@mUZt1?U@l(
zezmMMn|Pno1o)?H*)0l)P>$00ow0AZE0i%yIB?j9_EOd0=&1`h6luOBPy5i0Uj5P3q-$jlSF30qwyS#-*|^-zWRwd5*?)1==V<*Bp$H3
zstf17turnUu6{(1{F=YoCV>L!h%MNIWIhSXGTH0mn)%i94{~8%|EW~vlJW;+gp9Lw
zDMC%D8e0`&hKU8aqaUW!tQp^V`JID8$lVIe)lxrk!+qg``skNvk7u)}&C|pDt4b90
zJiwZGlgaaW<%O;tVF2XYz)u~CRsE%q&|K4RG~B$vYUZRw
z$_dEE0vnHRtqlHjUmB@Ed^`H^Q}FUSf!aC_P)no};j@^JKPfdQ5cI}QpojruBi7;%
zrM7hdBwrLI7w@9)OBx+HeKWF@-3&}AW}8?p?5`ZwsDjgn>hnhcKYIMI(_KxV15OQ)XY66gXId|r3S^1uwG@@ddKJ2A`rA0bjb?jK
zC0AH7y`!jT?@YJZ*AU+3o(Ecixn;+a1Y#xvDfumfGdNRrOc-DNBfiGoH2NCFFvVgY>*+t+*Ff^nT1yvbX`N*;
zH`po*qdro-d8Oi%oN=mj4b&TpeW`wY5cxVwo>&X!OR!P4OGUMc2IkKFCR$
z@lT^SVT^Y!U;^qB|CH!b=rKrabFqC}&5lhbB!M9*EA>lNq9O>W4&?aA6Tjz%Zqnro
zg|4sp3KvxmW8&Vg0!_G_Ms6FN90?Uzpp#J48ju!zmwb@PB`Tcu_0kFRWhHe%GlqqG
z-x;*e%TX))67C#-Qx;sAc}Opz2a0^amH5av6}oEyghXv{d=>{1FpFY7!OZz`%$xuH
zkX)b3HLy`EOo$*Z7p)UWnvuI5INe_7;0clkcm0b`#YCPfOr(Lj0r{RcTMkkHXhmy=
z_>3bTSBB9Hc}^R&Ko>npG#+ir@?Sb!KVN=Fn^@I3?q9xkqT|GjF;chPQrK)sUOf
z&~HM`2?lHdv5&z-0S~<+ofJC5G!-z2>jmTo7m|$S-bW)yaXKczLb$k-0*%N%xu+&I
z7qyojv%qIyUo{{l$~?g}!85p7fwWA=k6;W37f;5ErVs2$jm%B59UTv9*(6MwRl1X3
zIKBO(OB&WLv(R~vV8M!3R`3T+o;}IbOeuhj#u-b
zmI{2n1ssZn%l|CtqjiZmo~R4XKpiz8bB_KDZp^oe1O8|aneV>I;y9qiTb~@zta+Gj
zFR+#2gkcinEN_Uc34Z++n@F|X@8okrpuaSc+t+Ry<~EBZ$hggW^%w_?Fl&l?UjE7j`^4OgQ(A1{0t
z1{*-sT-cb)IO91|yK`X4!Mz83wW@!<=0D>~$asjDgv+GyQV>o`lC3T-ig9qMKl_HV
zf(F~1wPK2QU)vLK)ETOsyiCN=H?>W^kIUn%y?pn}r_N&b*
zFsvt!L*Mp#?5U1`Q`J~1f*dPTtOwf+op><8a
zvWOxtYA`wcSq98*$9S_WDV<0*Dg=69hCG*46(+fWRr}{g5+;<6Z=5-DV>vj69WSx{
zomu+^Q3PLI7hOQSc}GB!4TJj3m%2zg5S58Dz|x6L_(m|r13*>~g9^qL
z#0c^V=y#$a+snQ^H)-Vgi_umNH5BPO^~JfD5_Sgi?ivevogLJ5#9Ga*MH^1#1A}<$
z0CHf=qT-8X4u)?{oR$s!iRYglXpi;T#7wcdh~vC-R*dTIHh}I#N4XP1`Y4Uz=c5|>
z33?i?9KBZ@ioJiE3(HeSN3YQwoND?A7I%iY02nPIxam)jzS?#mZ7ey?)=I8XG>JkE
zSc`x^*3xK-v*uuzU39*WF{XS=GZ^9U2h!BQec&eqyIF+mwNYxY0+NY!dQB(f2X4e0
zHc@T!PHJl(5f_lW2}+X>c#w?!=34_6wEvjDrbs`W`6s_id`$S=)&-86GaX$135x(X
z!^)zx3D$@eNG39NLBPLLM@7^ABZHmd8`+ZgaY0Gg2G9oSQNoznE>sI0!P~Wj}!-0iVD$B-P>Y#Pn0U
zSSWi+-jlOsGdbkC>6%z?_R1odmU%Q>koSV&pSE_ShdMV31J2wAKi-@7rg{3veqla*
zJv102M@hUjqKot*vtYyh=z0Li4WKnyE1rqN)jSNOjF_-b{=mQ$pHw+R>7QW4!iuCO
zJ+_h;CQ*0(Ss61m?#J&aBGl!dDXy)qyDuZh%rk$gWByj)po7oZfFN;KxCl-nKnI
z3FImq{jN)nM3!uG-x}Kk6vVuS_ax=@U8T6pznQp{wn(Ibq1b4+Qca?KR`jX%S!nOh
z1G$VY1!AeM&frhWy_=Zf`D)U-Mk?VVX{Yb9FSqZLfB*xBCj`sI0N
zd8I{>zV@-wo-j4G>uDVi>FwD61dc?ISU1!+NK5Mg*>gp+v{}(>N@Lb8Y-{*-k8i&(YG&XT@cSxiO&outj0+7%*ts?j%Of}@JPyebbEMu<+aNS1}BAhYTyXC&
z$d8*69KV(X<3f-SQbvIu)8L7|bo>N^BaLN|mIdMGJ+?H-mebxD%>>tm9WyuBE}e&4
zZ9kZSPth)I?Ft{WurC9V4&7#{^ZMLzdU95qt0aXNH@p8LXWWj?OnzGiVmg#WvQCoY
zdkprCSXh|>*Ss4JNW8%pwDKb2OH%i939YB$wYzl?(xhhV$yOBPk=5q{lJAG&=~Ii0
z6HNuFi4=kK%9%=dGXx(LU5Kw9_nx(=dMfBC{(*$VdR4=>7{Y!#w!L_MdZ!J+`~_il
z=sT@Z{DZY$lg!6!$ia^cL_%ahT96YIur?Es5K-L3W4)cYCU9X{+U8n}9*b`NNu{6v
zMg_k?B~IC*9Oow_O6c8eZGOXH5vGUcUL2f9Y-Fv8h_WOg7i4i5_NSR@o{3P@|8__7
z1))dK%vb3iYt=C9v}h-WMKKwhrRxxf)DA{_WAdsfAW)3-K@EUP!IWGYH$ZOek!@3=
zS3O_7FQIKDi-G{UQIWtoq*&aP;)ol{O~UTT%;T_ow*g?orl0a#W0@(g@TYm)3C?C9
z$n8AOU4N?xR&6o)6Wc;?kv%XA?$q2t7|dVUF;TGUw+%#e!7Y
z#DSdh@Awlv5maVm+c{B93}X6{11rD?v!NT?u$!C>EwNY2SQg|tl-7kz?}gWT>#B``
zTqx`Q
z-xs%CgBgYJ(s82}Gq;|0_2X?}muc9a0GF8{)q$UkRjNh6fOykeKEUSvIaqaMjxj^TK~k=_R$EbojldI+R&tet?>UDFQ6*H!Eb&9QuAl3J91M%B=MC?>X*8^IB{rsW
zVvXmyz>$J}AU|Wa>_*ClL}bz=d~cEvz>eaJiKHFn(J^r9RfnZGAm0-M!(Z?W2}FE2
zt1ofm*VlDW{^$uAh6{yC#rvo~{OKk=0>SgpQ!BoX!wC1;gKyE!BBc+=)2(MZmnUQK
zeSVz%_r;S^rEc(W*0c=3utZ=th*2mOg*D*7im+pD=_G~Iy&WRvQ_LZd_T1D(i}j0KSxU>&r`8KoMnS&
z<+u){cMoa^Ip$dLX1f
z3bVAgUt}#iUg>sLB!`UsaRK+Ld#%Xeryr-7Qm?(6hIm1uCv8jIRk*SvnioD5`f#!+us79em?N!1F(xF#pg@d___jh+3dDR+V>&Y4$XiShnq96JKs
z%>vHbFK$L2d=yK$k6^&YM#ENfR>B2Wu~$2e>7axPq~EYJBie{f*3ctcuV~vc%4TdA
z_!R$vA7C!n3dUFT!)vpT=!@Nf2#e)P`VYQx{#^RQ8NsrNU4cL%e>ZuODc|oV&MxHq
zSZ6qRiy18wI1W7jmL!Qi1-$Azr!|%RwqVV?N5>??6ya1^b7U;=v?L%g_QG;=JHkQLHk0fseggas&e2)#g%5}
zxnJDZw0yd63)|k-Oj~YUPwA^VZjMGr8-$HjP=IUTN~tXuaA8HIHbGe^OWE`a58kXg
zLwtoU@>w4Ky9iGRYV&rmfyFgy>&1esw7{-5F8Vxxfxc{l(*~JkE*TugH+ZW)-^xRf
zMdMOGwz|E~#GkD2fb4?;kVf9HJg$vcqV77DqWjo2-X}sB-=9IBa!i48pYINx#Lj_0
zldOY%#CbFj#a<)IS=cH`;}v>Yk#RlgNtQ#Qv^+pblXFo##8d0
z({2;1^RPYIVRst%;n%o!@UnuSs@o5iQ=|qP-yUAP27xY>qef_S-D_{36OV;)d-*{(
zeCWXMXW5tK!vpkjSJ;|gFOVU^8Ms1WdD1+fqecJBH;X(Be^&yAM|xQtzN;cA7tDKq
zB6K>S7`}rzF$WgOINbp-YM&Dbq5w{RyCV|*N#=N->v&A04x&w!BwrWuYDxe)#dhI=
z>GU@0WJu`Pj^cQW1cHB6}HxULn9gvz|JdYSta}M$xV>(7G%_jVrP*-;W6ji%vG~
z`>X;)fI8|uZw3;wNd}S0)jqAxs6J1d*WXz(0H$F3qX1~;cD2~fy}T0N7rQ<2oV$Lx
z74vkm-%vO~smeQG*VL%^>D<|b1O7@Jmrumw5X
z9m@@`4%y>Uh>A2)hBZ?`wGD_}@AvY1^~_h|%7R>q?3daToj`-K~
z`qOmLH%MBfeqZaS9iMp{uPi+)9N|fdU&7zk=z1$@@G31}D
zdB-!aI$HyBCZR2qwg)rzR8YLbsFyICUkBVPa<=2QHLcSX@ch7lG4arLz@mRZB}87%
z<_ERS$gf{S1(pgY3zY+-LZ+0(&|k
z2LLSM0e#$8V$vXMs%#EGf2UkD{*+th$Ye7V6ZP-_f=0#{Ec|O|
zz&+xQ@*VQyZJ=b{FA2zb3ati#9nnQ5Nl->R?)@vc-?~7dh+98uf2q7ZIwEs+pR~@f
zn|p=F45}GDKeV+gH&5a@C}*QyJR#1^;73{&SteA}N`MN>E;|~(@LDXt7~?pCw%%rJ
zpTfKIIatE8P0EL6mUD2G_|d;S@)#gcf*q$wl=U~2Z3i;2*bH|M7$lzkG}-z9HU@kj#xJpi7zF7+7EU*U
z0ED*AEr2zog-!A#$*HD1ngZe-&Vu&Kj#Ws5!O18LNA7P$2Eaq^ghK+o5)!jRYw`~F
zl>wR@{6sMp@wS9js@PH>7!{Apwbtq_l?!>X34Vm$pa<&NNb1wJ^
z9Dd0J1*%eb<#uT{s$aE4I)D3563!cqQXq|@Q)*<{y}OB1{eZ_g)}%m!Xza}bMX*-qb%{9_n2QV~7*=us8RQmdfwj5h631%*HU8-~qG4kUFp)s%yXQOhkL
z>CskwjKbMm^VbNCDl$#j2k=n`gR+GhfdZZZCqSw!3hTx-m$w%Xu&(F)3n_0&uM%CF
z1(5g{T;AXj_kDnbEAXH|K4s*=)Xp<{Nknip2pa#x}ZxC@Dp;rcJ_XldRC6Latfejx%ymT{bvs6dCoFzr+
zk0$CLP}PoD8v|eV<^V&Ok{Qbi&7;<}{pk=Ke511VtTp4?X>t
zKUFrbs_FpB=f+-Oa5_9=dKUD)v{wY5ayo45ONH^kiKN~b(W|bSE^pmP&-b1vNE@^)
z7zKP~0%gX*ZKM~1s^17?GQAQDya*u9~2Z8fvE=_gq@Hb1MmX50Y>jCxI|3O;GzGb#yPU_^ubLLlSj
zh!F2GON(HDHDn_M10^T2H<%uN>TBe3DB>T06?{xPMe24I`@BJ0d1NK`1Pj~uY$V&S
z{zFw4W2b49{!|KRsd#*4C*_Nfa1cQIvyKiz
z>Jc)Y(@|NYv}ZW4ZZ}ZuXPmO~kmau@S9vI7fMR+p|Fw7pv_lf-u0x9ZxysD@Wgy~S
zb?<75Um%&RT11brW*-FWM;UOIR4^DIX6Hx4uVtm@F1%+3U#f#4al0sVcY}B$RB=-ziaMbUL?^aexo{|29eG9*7?mwe+WCQ?x)Xx
z5QqP+e75@T8KzmzKBMH=3LCPle}Kgg->-P$sVAI*h5L!C&1H8|qcS=AEQNWUxcxvq
zsn79TbaG4R6IM(vao;YzP5N{B=sOn{qmDdC^tl0a1N)vI+2?;V`(W=a?>Ldyma={u
zB5@yNg({Q`HE*ayvcm{<6eKzuq6iYX-Hfe!-StUsb*lTuhyceJKhSs};th)bDBv4|
z!{S{44uq}to79A#Op!YP`OFYA1+oii^!qlKs-cGC
zDnIE=47>{eS6K-3G%WsJZ9k$VTGj#01Y+7(M6lg4kx8l1J(9sE?}_Z@$pL1v4?G+ERB~fbZ?Lr)t;siR-R4$fSCfv
zd*34h{$vK&xpDoV+Y~VCba5Gl3oLW-={5~Y=omz-+$0Y@#>ahu!WYS>&~JT4KoPEi
z7(;k{_7la~OoGq)73M!zcCsxF&TGt$;=Z>R!^#Gzn20`X5?XKQXu=D~jFI9|Q;qxl
zkbQc&$p~W=_QF+IAo(Y`qCcyK?EX9Jj2SR!wUbQV4h9LcFl7W|fL=xK8uqKqh7H;4
zl>H-X*kLByS;Au{#~FJR4xC^N1eVwBJx*4kOv84sWcCgY>C4Apo!}n+Yr1b5NcE&^
z6FJWg@NUUZ?DYm}t#U|A%snP^4|qwG3$L6r{7~pl!%M420B-UE%o-tmuwKs}_o746
zTBt<&a};t9&lF;s(YXq
zMx^p}$HFRem~bmOl4%}We*uTK{8$-;{eWs9zbS_hTucJ_KC`fn6KTkb(R$d;khEaj
z->XJgatczygdHuP0UkmrRI)eq;zr7~W>*;oO55Y5?2-pg9Tu8rNcm+RSW8-t<~T2>
z00d-I=(KAubx%>PMoypk=yurkp)Drw7N2YAI~)Y+N-g~ga0JxK-O76@pcUcOxxH2(
zMwT`8l=6z7yu2?#Y>*>La)}lMGzjB|2n;n+1X|5K8sFRkkQt|F2DL##I~lF6Qq$
zmNE!GN%S(GJy7)>y9{I(6a?VY)x%sDpHrQR5h&gQj|1L3?{aCklF#^;K}oja+J`II
z)g6!T&k=G?34K4s&`yO2rayYStAi{Ic*%k6VP`-_C8*;dQ83Eg1QOC}nNV=0MHsqA
z>4{(_jJk(KG>CHFXxQ&MYTj!p0WEgCu$@^ge1wTav3fGtB^~LL3NIJ5K3WFufh5>;
zv_|)wja2eQ?xJQj#*=;_7SepQ;6`C_*pu0*f$Z8y&BF)?Q*SDrL8#{)yizLcJb7C0
zUEyvR*|kjE8;-
zhAryv$c|)_$Q+nFXcS)idv>OpgB%B#F)i+2Wae)j@~~$RVgsMNus8Kf7S?JS)FOtj%6F9+h%%Ped1Q&Qd3E
zUJt4`5E4nl-EEO@k)%we{~^4wn$B5S9gEkLMlr#x%f^e)cq$-q6Ee4u(8W3rV&$X^
zeg>645Yo#PBo1~7{W^F3Ts_2<-&U^93?-2)*R%G@@uGX>-pM-9^&G4H)z5`YnNhKH
zU%g#Se$LD-6Yq=LDRF=5)fHsn-(V^?lm`A}-C027e7R{xifd^g
zM3=M}lN0ts(I~+LZNLxTVl_)ZU?&taxZ$q(Hu5ejPm@$6wcYKb_`o{@R+;}|OV$d6
zN0x*Hy5>Sy*~ZC;dqhFxMNOAcwVSdcJUy@uuN&jE25>T7?xyZQ({iv?ITyn_!*wrC
zTvJ{otAR$^S#f$qsW}}~f21Az0VN@h6h<)_!!K~>AoyA80b3}l4luxt0^dCF`#V&1
zsN_u7=ZkX#?4_6u6cl*u{oclfe;XaaE%MSkzub8UZn|xv%J<~dx0;lF2{?r8Ar8PW
z8DbN@A9rHw!xge9q+rM8Y!ATlxTb#X3IJG&CSBDM=I7P`8d5+Q(D$#Ry%srDw>Eag
zgrPSb-58G9Gzp|=%2%cfH!0y-PocB`7g+)77d4h&Y?o_aeeN<0mp9*fITd`!@d8K`
zSK!${5fBjgs#>QX6ax8}UgfB2NNZf^5g|4$6deFWTk3fRIoeD;dk$2^hfCu<71hE1xZOZ0#sP
zBDCf5W!?eTPaYN&iAZH|6u2=`c5A6JFjysavQw6Hi0bE`1(aSU12?Ve6zhB7Vv=^>
z{=RNX8qD@v7ht`*3;47)*Qz!V0cdT=@HS)0>I5c
zKKq59$ncLa#tkNa|V?{Hk?hMZ3QNBbt)XK@pL0sze_
zlF*MiE6-nF1d+B@`t90rPwf`9lF|K8-lq(I9#vNGpQBQ#C4`7q(g;2IB7T6$Js`-w
z;}6H7tNJ+7+MGf-#*<5rq7ZsOf81Y8`dmY3c#ycj?kf@ieekA9aCKv8--p?|ug04x
zt7oBu^OIuHdyL{bnUiTmhioF>umD}N=jVjl%*aZrDZTt`75FYTA_Y#DQY#X-yBhuN
z`D&g{(-K_s3NgJl&c+0_l6zT0(5*~Qe;E+7(9>2T#YfpYQ@pn7Yd~1KJ`qq^1xd2H
zLGf0atvxZ<9$HP`P}t5If+v_p%)R-tDrNz7ldl%2)1CEGS`(0H)$gGAIseiY;yn(K
zWHu=mtS94tYOyL4H8mLhjA|DxjNeilW?scyvhpHH#tO$``7Dfj9ioN{dc>B
zj>cen*!qro2X14tcRUUOtnb2?^10`1l?XpN1z{Y
z>bCn~Ku|;g*;eVkzkZ1t+e63hGNgvTD&b0<(Ss*tpA@(Q#P?n&Pby-DeCj%>HbzJh
zDIiea0=v?kSYU-{_Dqt}{F#QSbuLd0mlly#a#Ot*?jKYRxJF!FtPufy&5k`4Igz{&
z(`n16;l3efAueAXH-7f>1wiSZJ)*S`uOpzB4`v0N|%?Un;
zbMIcIG3lP!J}A_;5%~ZCUsZou6w-i<)zAH{zq%jr+|0DKk~$J#&Iahm^+dd~NEybf
zS*WgokaQ=i^pacq&H&U5$|p0_Kw0UIS2=Ys^Uf+8_rhM*C32tzc;;ziqM$%^$2$-(Sr06
zPBN?s>Pz(p7Hi=-DeYspCrGtQM-bx!OmKC8+T*-)N>AlFza_Fx009HPgi5b0XM9av3s;2v
zV!rb%L_lU1S&|^^?~%ZoJM2E!^E0UW4DXTWgS9_#@lzfhJ>oUjp@|LGDkpb{ViLemn(uV&eM?jDqii1b7(g;Tiss8=6>}|M!;0z~0
zIZo~e*32K_TNSa-+B(}~uCvIzL+EN5u0C>ttroA1;~YTQZF~0FInae#kheEt`aN(Q
zigN3Yy>pq1DiUY-XfY^+iBXf)zy$JJcAhGr61VquMrHxb@7yE3XC8A{k9DVvlu^#*?RhXGa9;KT
zZt*!kMiT+pdc^+7DP(M3vk-QB)T9THQpaC0P+XKV1*;N&oU6*34ttA49l$NgQgN&O
ziJWd$JbQUek~V;ZeptFu%p5gWJ7P5=<-y7@>!IsC4NH$>-rns!