Skip to content

Update Airflow provider: cloud auth now supported in Secrets Backend #319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
EldadH89 merged 4 commits into from
Jul 15, 2026
Merged

Update Airflow provider: cloud auth now supported in Secrets Backend #319

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
feab01e
Update Airflow provider docs: cloud auth now supported in Secrets Bac…
baraka-akeyless Jul 12, 2026
33074b8
Address CodeRabbit review feedback
baraka-akeyless Jul 12, 2026
f527186
Merge branch 'v1.0' into update-airflow-cloud-auth-backend
baraka-akeyless Jul 14, 2026
622028c
Merge branch 'v1.0' into update-airflow-cloud-auth-backend
EldadH89 Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 89 additions & 15 deletions docs/Integrations & Plugins/plugins-overview/apache-airflow-provider.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The provider is maintained in the [apache/airflow](https://github.com/apache/air
| --- | --- | --- |
| **Hook** | `airflow.providers.akeyless.hooks.akeyless.AkeylessHook` | Interact with Akeyless directly from Directed Acyclic Graph (DAG) code — fetch static, dynamic, and rotated secrets; create, update, or delete items; list paths. |
| **Connection type** | `akeyless` | Airflow connection type identifier. Create a connection with this type in the Airflow UI or environment to supply credentials to the hook. |
| **Secrets Backend** | `airflow.providers.akeyless.secrets.akeyless.AkeylessBackend` | Transparently resolve Airflow Connections, Variables, and Config from Akeyless — no DAG code changes required. Supports `api_key` and `uid` authentication only. |
| **Secrets Backend** | `airflow.providers.akeyless.secrets.akeyless.AkeylessBackend` | Transparently resolve Airflow Connections, Variables, and Config from Akeyless — no DAG code changes required. Supports `api_key`, `uid`, `aws_iam`, `gcp`, and `azure_ad` authentication. |

## Requirements

Expand Down Expand Up @@ -54,16 +54,16 @@ pip install apache-airflow-providers-akeyless[cloud_id]

The provider supports the following Akeyless [Authentication Methods](https://docs.akeyless.io/docs/access-and-authentication-methods):

| `access_type` | Required fields | Supported by |
| --- | --- | --- |
| `api_key` _default_ | `access_id`, `access_key` | Hook, Secrets Backend |
| `aws_iam` | `access_id` + `cloud_id` extras package | Hook only |
| `gcp` | `access_id` + `cloud_id` extras package; optional: `gcp_audience` | Hook only |
| `azure_ad` | `access_id` + `cloud_id` extras package; optional: `azure_object_id` | Hook only |
| `uid` | `uid_token` | Hook, Secrets Backend |
| `jwt` | `access_id`, `jwt` | Hook only |
| `k8s` | `access_id`, `k8s_auth_config_name` | Hook only |
| `certificate` | `access_id`, `certificate_data`, `private_key_data` | Hook only |
| `access_type` | Required fields | Extras package | Supported by |
| --- | --- | --- | --- |
| `api_key` _default_ | `access_id`, `access_key` | — | Hook, Secrets Backend |
| `aws_iam` | `access_id` | `cloud_id` | Hook, Secrets Backend |
| `gcp` | `access_id`; optional: `gcp_audience` | `cloud_id` | Hook, Secrets Backend |
| `azure_ad` | `access_id`; optional: `azure_object_id` | `cloud_id` | Hook, Secrets Backend |
| `uid` | `uid_token` | — | Hook, Secrets Backend |
| `jwt` | `access_id`, `jwt` | — | Hook only |
| `k8s` | `access_id`, `k8s_auth_config_name` | — | Hook only |
| `certificate` | `access_id`, `certificate_data`, `private_key_data` | — | Hook only |

> ⚠️ **Unsupported authentication methods:** The following Akeyless authentication methods are **not** supported by this provider: OCI IAM, Kerberos, LDAP, SAML, OIDC, and Email.

Expand Down Expand Up @@ -187,6 +187,80 @@ export AIRFLOW__SECRETS__BACKEND="airflow.providers.akeyless.secrets.akeyless.Ak
export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_path": "/airflow/connections", "variables_path": "/airflow/variables", "config_path": "/airflow/config", "api_url": "https://api.akeyless.io", "access_id": "<Access ID>", "access_key": "<Access Key>", "access_type": "api_key"}'
```

#### Cloud-based authentication in the Secrets Backend

The Secrets Backend supports `aws_iam`, `gcp`, and `azure_ad` authentication, allowing managed Airflow services to authenticate using their workload identity — no static API keys required.

**AWS IAM** (for [Amazon MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) and EC2/ECS/EKS workloads):

```text
[secrets]
backend = airflow.providers.akeyless.secrets.akeyless.AkeylessBackend
backend_kwargs = {
"connections_path": "/airflow/connections",
"variables_path": "/airflow/variables",
"config_path": "/airflow/config",
"api_url": "https://api.akeyless.io",
"access_id": "<Access ID>",
"access_type": "aws_iam"
}
```

**GCP** (for [Managed Service for Apache Airflow](https://cloud.google.com/composer/docs) and GCE/GKE workloads):

```text
[secrets]
backend = airflow.providers.akeyless.secrets.akeyless.AkeylessBackend
backend_kwargs = {
"connections_path": "/airflow/connections",
"variables_path": "/airflow/variables",
"config_path": "/airflow/config",
"api_url": "https://api.akeyless.io",
"access_id": "<Access ID>",
"access_type": "gcp",
"gcp_audience": "akeyless.io"
}
```

**Azure AD** (for Azure-hosted workloads):

```text
[secrets]
backend = airflow.providers.akeyless.secrets.akeyless.AkeylessBackend
backend_kwargs = {
"connections_path": "/airflow/connections",
"variables_path": "/airflow/variables",
"config_path": "/airflow/config",
"api_url": "https://api.akeyless.io",
"access_id": "<Access ID>",
"access_type": "azure_ad",
"azure_object_id": "<Azure Object ID>"
}
```

> ℹ️ Cloud-based authentication requires the `cloud_id` extras package. See [Installation](#installation).

##### Using with Amazon MWAA

1. Upload a `requirements.txt` to your MWAA S3 bucket containing:

```text
apache-airflow-providers-akeyless[cloud_id]
```

2. In the MWAA console under **Airflow configuration options**, add:

| Key | Value |
| --- | --- |
| `secrets.backend` | `airflow.providers.akeyless.secrets.akeyless.AkeylessBackend` |
| `secrets.backend_kwargs` | `{"api_url": "https://api.akeyless.io", "access_id": "<Access ID>", "access_type": "aws_iam"}` |

3. Ensure the MWAA VPC has outbound HTTPS access to your Akeyless API endpoint (`api.akeyless.io` or your Akeyless Gateway).
Comment on lines +256 to +258

@coderabbitai coderabbitai Bot Jul 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant section with line numbers.
sed -n '230,280p' 'docs/Integrations & Plugins/plugins-overview/apache-airflow-provider.md'

Repository: akeylesslabs/technical-documentation

Length of output: 1779

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Search for the Akeyless Airflow provider connection docs referenced by the comment.
rg -n "api_url|Gateway|secrets.backend_kwargs|access_type" 'docs/Integrations & Plugins/plugins-overview/apache-airflow-provider.md' 'docs' -g '!**/*.png'

Repository: akeylesslabs/technical-documentation

Length of output: 50393

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# If the provider docs exist in-repo, inspect the relevant guidance for backend_kwargs / api_url.
fd -a 'connections.md' .

Repository: akeylesslabs/technical-documentation

Length of output: 174

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read the exact snippet context from the review target file with numbers.
nl -ba 'docs/Integrations & Plugins/plugins-overview/apache-airflow-provider.md' | sed -n '248,265p'

Repository: akeylesslabs/technical-documentation

Length of output: 214

🌐 Web query:

airflow.apache.org docs akeyless provider connections api_url gateway backend_kwargs

💡 Result:

The apache-airflow-providers-akeyless package allows you to connect to the Akeyless Vault Platform using a dedicated akeyless connection type or as a secrets backend [1][2][3]. Akeyless Connection To configure an Akeyless connection, use the akeyless connection type [1][3]. The api_url field is used to specify the Akeyless API endpoint (e.g., https://api.akeyless.io) or your specific Akeyless Gateway URL [1][3]. Authentication details are provided via the login (Access ID), password (Access Key), and an Extra JSON field [1][3]. The Extra field supports various authentication methods defined by the access_type parameter (e.g., api_key, aws_iam, gcp, azure_ad, uid, jwt, k8s, certificate) [1][3]. Akeyless Secrets Backend When configuring Akeyless as a secrets backend in airflow.cfg or via environment variables, you use backend_kwargs to pass necessary configuration [4][5]. Example backend_kwargs configuration in airflow.cfg: [secrets] backend = airflow.providers.akeyless.secrets.akeyless.AkeylessBackend backend_kwargs = { "connections_path": "/airflow/connections", "variables_path": "/airflow/variables", "api_url": "https://api.akeyless.io", "access_id": "p-xxxxxxxxx", "access_key": "your-access-key", "access_type": "api_key" } Key parameters for backend_kwargs include [4][5]: - api_url: The Akeyless API endpoint (default: https://api.akeyless.io) [4][5]. - access_id: Your Akeyless Access ID [4][5]. - access_key: Your Akeyless Access Key (required for api_key authentication) [4][5]. - access_type: The authentication method (default: api_key) [4][5]. - connections_path, variables_path, config_path: Path prefixes in Akeyless for retrieving these Airflow objects [4][5]. Note: The Secrets Backend implementation supports api_key and uid authentication methods [3]. Additionally, in the connection settings, the field previously known as jwt was renamed to jwt_token to ensure proper masking in logs [6].

Citations:

Add a Gateway-specific api_url example. The MWAA snippet still points to https://api.akeyless.io; if users follow the Gateway path, they need to replace api_url with their Gateway URL or the backend will keep using the SaaS endpoint.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/Integrations` & Plugins/plugins-overview/apache-airflow-provider.md
around lines 256 - 258, Update the MWAA Akeyless configuration example around
secrets.backend_kwargs to include the Gateway-specific api_url value or an
explicit placeholder for the user’s Akeyless Gateway URL, while retaining the
SaaS endpoint example for the standard path. Ensure the surrounding instructions
clearly indicate that Gateway users must replace api_url with their Gateway URL.


4. In Akeyless, create an `aws_iam` [Authentication Method](https://docs.akeyless.io/docs/aws-iam-auth-method) bounded to the MWAA execution role ARN.

5. Assign the Authentication Method to an Akeyless [Access Role](https://docs.akeyless.io/docs/rbac#access-roles) that has **read** permissions on the configured secret paths (for example, `/airflow/connections/*`, `/airflow/variables/*`).

#### Naming Convention

Secrets are looked up by joining `<base_path>/<key>`:
Expand Down Expand Up @@ -230,9 +304,7 @@ JSON with `conn_uri`:

### Cloud-Based Authentication

For AWS IAM, GCP, or Azure AD, omit `access_key` and set the appropriate `access_type`. The provider uses the workload's cloud identity automatically.

> ⚠️ **Secrets Backend limitation:** `AkeylessBackend` only supports `api_key` and `uid` authentication. For cloud-based authentication (AWS IAM, GCP, Azure AD) use `AkeylessHook` directly in your DAGs.
For AWS IAM, GCP, or Azure AD, omit `access_key` and set the appropriate `access_type`. The provider uses the workload's cloud identity automatically. Both the **Hook** and the **Secrets Backend** support cloud-based authentication.

Example using AWS IAM with the hook:

Expand All @@ -245,6 +317,8 @@ value = hook.get_secret_value("/my/secret")

Set the connection `access_type` extra field to `aws_iam` and install the `cloud_id` extras. The hook authenticates using the workload's AWS IAM identity (EC2 instance profile, ECS task role, and so on) — no static credentials required.

For Secrets Backend cloud authentication configuration examples, see [Cloud-based authentication in the Secrets Backend](https://docs.akeyless.io/docs/apache-airflow-provider#cloud-based-authentication-in-the-secrets-backend).

## Troubleshooting

### `ImportError: akeyless_cloud_id is required`
Expand All @@ -257,7 +331,7 @@ pip install apache-airflow-providers-akeyless[cloud_id]

### `ValueError: Unsupported access_type for AkeylessBackend`

`AkeylessBackend` only supports `api_key` and `uid`. For cloud-based authentication in the Secrets Backend, use `AkeylessHook` directly in your DAGs instead.
`AkeylessBackend` supports `api_key`, `uid`, `aws_iam`, `gcp`, and `azure_ad`. Other authentication types (`jwt`, `k8s`, `certificate`) are only available through `AkeylessHook`.

### Secret not found when using Secrets Backend

Expand Down