Plan data not cleaned up when PR is closed before Atlantis finishes processing

[albertofp]

Atlantis version:

v0.35.0

Deployment method:

EKS / Helm (using custom helm chart which wraps the official runatlantis helm chart)

If not running the latest Atlantis version have you tried to reproduce this issue on the latest version:

No - we are running version v0.35.0. We have not yet tested with the latest version.

We run a cleanup post_workflow_hook to clean up the temporary plan data left behind in /atlantis-data/repos/ after the plan is completed:

      post_workflow_hooks:
        - run: rm -rf /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM

Any other information you can provide about the environment/deployment:

• Cloud provider: AWS (EKS)
• Storage: StatefulSet with PersistentVolumeClaim (750Gi)
• Data directory: /atlantis-data (mounted via PVC)
• Node type: m5.8xlarge
• Workflows used: terragrunt and helmfile

---

The Issue:

When a pull request is closed before Atlantis finishes processing it (either before plan completes, or after plan but before apply), the temporary plan data left behind is not cleaned up automatically. Over time, these leftover plans accumulate and fill up the disk, requiring manual intervention to clean up.

Expected Behavior:

When a PR is closed, Atlantis should clean up all associated plan data (including locks, working directories, and any cached data) regardless of whether the plan/apply cycle completed.

Current Workaround:

We manually clean up the data directory, but this is not sustainable as the volume continues to fill up.

Additional Context:

• Looking at issue UI Jobs streams are not cleaned up after PR is closed #5348, this seems related to the PullClosedExecutor not properly cleaning up all resources?

Questions:

1. Is there a suggested workaround for this scenario while waiting for a fix?
2. Is this something that can be tackled by the Atlantis team to ensure proper cleanup on PR closure regardless of processing state?

[dosubot]

This is a known limitation with Atlantis's cleanup mechanism. The PullClosedExecutor performs cleanup immediately without waiting for in-progress operations, creating a race condition where:

1. The working directory is deleted via os.RemoveAll while Terraform may still be running
2. Running Terraform processes aren't signaled to terminate - they continue and fail with I/O errors
3. If cleanup fails for any reason, there's no retry mechanism or automatic recovery

Regarding issue #5348 you mentioned - that's specifically about job streams not being cleaned up from the UI, and a community fix exists but was still pending merge as of February 2025.

Workarounds:

1. Cron-based cleanup - Your post_workflow_hook approach is correct, but you could also add a scheduled cron job to remove old PR directories:

   # Remove PR directories older than X days
   find /atlantis-data/repos -maxdepth 3 -mindepth 3 -type d -mtime +7 -exec rm -rf {} \;

2. Monitor for stale locks - You can inspect the lock database directly:

   go run go.etcd.io/bbolt/cmd/bbolt@latest keys /path/to/atlantis.db runLocks

3. Set up disk space alerts - Given there's no built-in retention policy, monitoring /atlantis-data size is essential for production deployments

Unfortunately, Atlantis has no built-in cleanup command or automated maintenance utility, so custom scripts are currently the only sustainable solution.

Written by @dosu, approved by jamengual

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.