diff --git a/.gitignore b/.gitignore index 20d5fdb4..a524c9c6 100644 --- a/.gitignore +++ b/.gitignore @@ -228,4 +228,4 @@ qdrant_storage .qtype-cache # SSL certificates (combined bundle with corporate certs) -certs/*.pem \ No newline at end of file +certs \ No newline at end of file diff --git a/DOCUMENTATION_ROADMAP.md b/DOCUMENTATION_ROADMAP.md index 1cc4019b..abd87493 100644 --- a/DOCUMENTATION_ROADMAP.md +++ b/DOCUMENTATION_ROADMAP.md @@ -110,8 +110,8 @@ Structure: - [ ] Use OAuth2 Authentication - [x] Configure AWS Authentication (Access Keys, Profile, Role) - [ ] Configure Google Vertex Authentication -- [ ] Manage Secrets with Secret Manager -- [ ] AWS Secret Manager integration with SecretReference +- [x] Manage Secrets with Secret Manager +- [x] AWS Secret Manager integration with SecretReference **Observability & Debugging** - [x] Trace Calls with Open Telemetry diff --git a/docs/Gallery/recipe_chatbot.md b/docs/Gallery/recipe_chatbot.md index ef225b71..4763866f 100644 --- a/docs/Gallery/recipe_chatbot.md +++ b/docs/Gallery/recipe_chatbot.md @@ -98,6 +98,6 @@ Then open http://localhost:8000 and ask questions like: ## Learn More -- Tutorial: [Building a Stateful Chatbot](../../Tutorials/building-a-stateful-chatbot.md) -- How-To: [Use Environment Variables](../../How-To%20Guides/Language%20Features/use-environment-variables.md) -- How-To: [Configure AWS Authentication](../../How-To%20Guides/Authentication/configure-aws-authentication.md) +- Tutorial: [Building a Stateful Chatbot](../Tutorials/02-conversational-chatbot.md) +- How-To: [Use Environment Variables](../How%20To/Language%20Features/use_environment_variables.md) +- How-To: [Configure AWS Authentication](../How%20To/Authentication/configure_aws_authentication.md) diff --git a/docs/How To/Authentication/use_aws_secrets_manager.md b/docs/How To/Authentication/use_aws_secrets_manager.md new file mode 100644 index 00000000..04eed375 --- /dev/null +++ b/docs/How To/Authentication/use_aws_secrets_manager.md @@ -0,0 +1,67 @@ +# Store Secrets in AWS Secrets Manager + +Avoid hardcoding API keys and passwords in your YAML by storing them in AWS Secrets Manager. QType resolves `SecretReference` values at runtime using the configured `secret_manager`. + +### QType YAML + +```yaml +auths: + # AWS credentials used to access Secrets Manager itself + - type: aws + id: aws_auth + region: us-east-1 + + # API key resolved from Secrets Manager at runtime + - type: api_key + id: my-api-auth + api_key: + secret_name: my-project/api-key + +# Declare the secret manager backed by AWS +secret_manager: + id: aws-secret-manager + type: aws_secret_manager + auth: aws_auth +``` + +### Explanation + +- **secret_manager**: Top-level application block that configures the secret backend +- **type: aws_secret_manager**: Uses AWS Secrets Manager as the secret store +- **auth**: References an `AWSAuthProvider` used to authenticate with Secrets Manager (must not itself use secret references) +- **secret_name**: The name, ID, or ARN of the secret in AWS Secrets Manager +- **key**: Optional — if the secret is a JSON object, extracts a specific key (e.g., `key: api_key`) + +### Creating the Secret + +```bash +aws secretsmanager create-secret \ + --name my-project/api-key \ + --secret-string "your-api-key-value" \ + --region us-east-1 +``` + +### Using a JSON Secret with Key Extraction + +If a single secret stores multiple values as a JSON object: + +```bash +aws secretsmanager create-secret \ + --name my-project/credentials \ + --secret-string '{"api_key": "sk-abc123", "space_id": "xyz"}' \ + --region us-east-1 +``` + +```yaml + - type: api_key + id: my-api-auth + api_key: + secret_name: my-project/credentials + key: api_key # Extract only the api_key field +``` + +## See Also + +- [Fields That Accept Secret References](../../Reference/secret-reference-fields.md) +- [Configure AWS Authentication](configure_aws_authentication.md) +- [Use API Key Authentication](use_api_key_authentication.md) diff --git a/docs/How To/Data Processing/load_documents.md b/docs/How To/Data Processing/load_documents.md index 13edde4b..7b5fdb39 100644 --- a/docs/How To/Data Processing/load_documents.md +++ b/docs/How To/Data Processing/load_documents.md @@ -70,5 +70,5 @@ steps: ## See Also - [DocumentSource Reference](../../components/DocumentSource.md) -- [DocumentSplitter How-To](chunk_documents.md) -- [RAG Tutorial](../../Tutorials/rag_tutorial.md) +- DocumentSplitter How-To (coming soon) +- RAG Tutorial (coming soon) diff --git a/docs/How To/Qtype Server/add_feedback_buttons.md b/docs/How To/Qtype Server/add_feedback_buttons.md index e47d6415..591735f4 100644 --- a/docs/How To/Qtype Server/add_feedback_buttons.md +++ b/docs/How To/Qtype Server/add_feedback_buttons.md @@ -38,6 +38,6 @@ telemetry: - [Serve Flows as UI](serve_flows_as_ui.md) - [Use Conversational Interfaces](use_conversational_interfaces.md) - [TelemetrySink Reference](../../components/TelemetrySink.md) -- [Example: Thumbs Feedback](../../../examples/feedback/thumbs_feedback_example.qtype.yaml) -- [Example: Rating Feedback](../../../examples/feedback/rating_feedback_example.qtype.yaml) -- [Example: Category Feedback](../../../examples/feedback/category_feedback_example.qtype.yaml) +- Example: `examples/feedback/thumbs_feedback_example.qtype.yaml` +- Example: `examples/feedback/rating_feedback_example.qtype.yaml` +- Example: `examples/feedback/category_feedback_example.qtype.yaml` diff --git a/docs/Reference/secret-reference-fields.md b/docs/Reference/secret-reference-fields.md new file mode 100644 index 00000000..777fc8e8 --- /dev/null +++ b/docs/Reference/secret-reference-fields.md @@ -0,0 +1,49 @@ +# Fields That Accept Secret References + +Any field typed as `str | SecretReference` can be supplied either as a plain string or as a reference to a secret in the configured [`secret_manager`](../components/AWSSecretManager.md). The secret is resolved at runtime. + +## Syntax + +```yaml +field_name: + secret_name: my-project/my-secret # Name, ID, or ARN + key: optional_json_key # Only needed for JSON-object secrets +``` + +## Secret-Capable Fields + +| Component | Field | Description | +|-----------|-------|-------------| +| `APIKeyAuthProvider` | `api_key` | API key passed as a header or query parameter | +| `BearerTokenAuthProvider` | `token` | Bearer token for Authorization header | +| `OAuth2AuthProvider` | `client_secret` | OAuth2 client secret | +| `AWSAuthProvider` | `access_key_id` | AWS access key ID | +| `AWSAuthProvider` | `secret_access_key` | AWS secret access key | +| `AWSAuthProvider` | `session_token` | AWS STS session token (temporary credentials) | +| `SQLSource` | `connection` | SQLAlchemy connection string (contains credentials) | +| `TelemetrySink` | `endpoint` | Telemetry collector URL (if it contains auth tokens) | + +## Example + +```yaml +auths: + - type: api_key + id: openai-auth + api_key: + secret_name: my-project/openai-key + + - type: aws + id: aws-auth + access_key_id: + secret_name: my-project/aws-creds + key: access_key_id + secret_access_key: + secret_name: my-project/aws-creds + key: secret_access_key + region: us-east-1 +``` + +## See Also + +- [Store Secrets in AWS Secrets Manager](../How%20To/Authentication/use_aws_secrets_manager.md) +- [Configure AWS Authentication](../How%20To/Authentication/configure_aws_authentication.md) diff --git a/qtype/interpreter/base/secrets.py b/qtype/interpreter/base/secrets.py index 57ae9ed7..ff494f9b 100644 --- a/qtype/interpreter/base/secrets.py +++ b/qtype/interpreter/base/secrets.py @@ -259,7 +259,10 @@ def get_secret(self, secret_ref: SecretReference) -> str: from qtype.interpreter.auth.aws import aws - with aws(self.config.auth) as creds: # type: ignore + # NoOpSecretManager is used here because the AWS credentials that + # bootstrap the secret manager cannot themselves contain secret + # references (that would be circular). + with aws(self.config.auth, NoOpSecretManager()) as creds: client = boto3.client("secretsmanager", **creds.as_kwargs()) response = client.get_secret_value(SecretId=secret_ref.secret_name)