Fix workspace deletion race condition in AMPLS private endpoint

[Copilot]

Resolves #3194

What is being addressed

Workspace deletions fail intermittently with dependency ordering errors when Terraform attempts to delete Azure Monitor resources. The issue manifests as AnotherOperationInProgress errors when deleting the AMPLS private endpoint and its private DNS zone group simultaneously.

How is this addressed

• Extracted DNS zone group to separate azapi_resource: Removed the inline private_dns_zone_group block from within the azurerm_private_endpoint resource and created it as a separate azapi_resource using the Azure Resource Manager API (Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2023-11-01). This provides explicit control over deletion ordering and avoids race conditions.
• Enhanced dependency management: Added both AMPLS scoped services (ampls_app_insights and ampls_log_anaytics) to the private endpoint's depends_on list to ensure proper creation and deletion ordering
• Update CHANGELOG.md with bug fix entry
• Increment workspace base template version from 2.7.1 to 2.7.2

Technical Details

The fix addresses the root cause by separating the DNS zone group configuration from the private endpoint resource, allowing Terraform to manage the deletion order explicitly:

Before (inline block causing race conditions):

resource "azurerm_private_endpoint" "azure_monitor_private_endpoint" {
  # ... resource configuration ...
  
  private_dns_zone_group {
    name = "azure-monitor-private-dns-zone-group"
    private_dns_zone_ids = [...]
  }
  
  depends_on = [
    azurerm_monitor_private_link_scoped_service.ampls_app_insights,
  ]
}

After (separate resource with explicit dependencies):

resource "azurerm_private_endpoint" "azure_monitor_private_endpoint" {
  # ... resource configuration without private_dns_zone_group ...
  
  lifecycle { ignore_changes = [tags] }
  
  depends_on = [
    azurerm_monitor_private_link_scoped_service.ampls_app_insights,
    azurerm_monitor_private_link_scoped_service.ampls_log_anaytics,
  ]
}

resource "azapi_resource" "azure_monitor_dns_zone_group" {
  type      = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2023-11-01"
  name      = "azure-monitor-private-dns-zone-group"
  parent_id = azurerm_private_endpoint.azure_monitor_private_endpoint.id
  
  body = {
    properties = {
      privateDnsZoneConfigs = [
        {
          name = "privatelink-monitor-azure-com"
          properties = {
            privateDnsZoneId = var.azure_monitor_dns_zone_id
          }
        },
        # ... additional DNS zone configs ...
      ]
    }
  }
  
  depends_on = [
    azurerm_private_endpoint.azure_monitor_private_endpoint,
  ]
}

The explicit depends_on relationships ensure:

• Creation order: Scoped services are fully configured → Private endpoint is created → DNS zone group is configured
• Deletion order: DNS zone group is deleted → Private endpoint is deleted → Scoped services are removed
• No race conditions: Azure operations on the private endpoint and DNS zone group are properly sequenced

Implementation Note

This solution uses the AzAPI provider (already required for AMPLS resources) to create the private DNS zone group as a separate resource. This approach provides the same benefits as the recommended azurerm_private_endpoint_private_dns_zone_group resource (which is not available in the azurerm provider version used in the porter bundle build environment) while maintaining compatibility with existing provider versions.

References

• Related: azurerm_private_endpoint cannot be destroyed when using multiple private_dns_zone_ids hashicorp/terraform-provider-azurerm#28715

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[marrobi]

/test-extended c32a87a

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/19553208613 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

@copilot this failed, same issue, see https://productionresultssa1.blob.core.windows.net/actions-results/792780c6-7f37-45d1-8de6-7062b4b8358e/workflow-job-run-9bd45ba8-ed52-5b06-ac86-bd5f0db51320/logs/job/job-logs.txt?rsct=text%2Fplain&se=2025-11-21T12%3A19%3A31Z&sig=SHt6j2Oeg2n4ecSM0MmILzIf8AtIDs4rlQmAnMrbfJk%3D&ske=2025-11-21T21%3A09%3A58Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2025-11-21T09%3A09%3A58Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2025-11-21T12%3A09%3A26Z&sv=2025-11-05

[Copilot]

@copilot this failed, same issue, see https://[productionresultssa1.blob.core.windows.net/actions-results/792780c6-7f37-45d1-8de6-7062b4b8358e/workflow-job-run-9bd45ba8-ed52-5b06-ac86-bd5f0db51320/log...

The test revealed a different root cause. The error was that Private DNS Zones couldn't be deleted because their Virtual Network Links still existed. I've added create_before_destroy = true to all Azure Monitor DNS zones and their VNet links to ensure proper deletion ordering. Changes are in commit 5d9bb7d.

[marrobi]

/test-extended d66e33f

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/19571335811 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

/test-extended d66e33f

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/19576408604 (with refid d81faf5c)

(in response to this comment from @marrobi)

[github-actions]

Unit Test Results

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit e722447.

♻️ This comment has been updated with latest results.

[marrobi]

@copilot please resolve: Code update without version change

[Copilot]

@copilot please resolve: Code update without version change

Version updates applied in commit 38788f5:

• Workspace base template: 2.7.1 → 2.7.2
• Core: 0.16.10 → 0.16.11
• CHANGELOG.md updated with component version entries

[marrobi]

@copilot check https://github.com/microsoft/AzureTRE/actions/runs/19576408604/job/56065634783#step:3:1

[marrobi]

/test-extended 50d36ae

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/19730219587 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

/test-extended 50d36ae

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/19733426740 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

/test-extended 5941585

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/19763804503 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

/test-extended 5941585

[github-actions]

🤖 pr-bot 🤖

⚠️ Cannot run tests as PR is not mergeable. Ensure that the PR is open and doesn't have any conflicts.

(in response to this comment from @marrobi)

[marrobi]

/test-extended fc12afe

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/20336363140 (with refid d81faf5c)

(in response to this comment from @marrobi)

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/20336363140 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

/test-extended f12f533

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/21595816510 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

@copilot can you try:

resource "azurerm_private_endpoint" "azure_monitor_private_endpoint" {
  name                = "pe-ampls-${var.tre_id}-ws-${local.short_workspace_id}"
  # ... other config ...
  
  # REMOVE the private_dns_zone_group block from here
  
  lifecycle { ignore_changes = [tags] }
}

resource "azapi_resource" "azure_monitor_dns_zone_group" {
  type      = "Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2023-11-01"
  name      = "azure-monitor-private-dns-zone-group"
  parent_id = azurerm_private_endpoint.azure_monitor_private_endpoint.id
  
  # ... DNS zone configs ...
}

Also fix the linting issues.

[Copilot]

@copilot can you try:

...

Implemented in commit ae93bc4. The private_dns_zone_group has been extracted from the inline block and created as a separate azapi_resource using the Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2023-11-01 API. This provides explicit control over deletion ordering while using the already-required azapi provider. Code review completed with no blocking issues.

[marrobi]

/test-extended ae93bc4

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/21597746061 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

/test-extended e722447

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/21603027184 (with refid d81faf5c)

(in response to this comment from @marrobi)

[marrobi]

@copilot we have some linting failing:

Failed to load configurations; network.tf:128,1-53: "terraform_azurerm_environment_configuration" module is not found. Did you run "terraform init"?; :

Error: "terraform_azurerm_environment_configuration" module is not found. Did you run "terraform init"?

on network.tf line 128, in module "terraform_azurerm_environment_configuration":

128: module "terraform_azurerm_environment_configuration" {

We should not be validating remote modules as we don't run init. We did this for workspaces, but the workflow much use a differnt configuration file for core?

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/21603027184 (with refid d81faf5c)

(in response to this comment from @marrobi)

[github-actions]

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/21603027184 (with refid d81faf5c)

(in response to this comment from @marrobi)