From 3e22bb8d7b56aabb28baf5fea5b3d404782013ef Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Wed, 29 Apr 2026 18:52:05 -0700 Subject: [PATCH 1/7] docs: Add Bedrock evaluation metrics integration documentation Document the new integration metrics feature for monitoring documents: - Add "From Integration" tab flow for adding Bedrock Agent Evals - Add global time selector section for monitoring documents - Document connection detail page with metric sources - Add metric sources management section for Bedrock connections --- .../integrations/configure-connections.qmd | 41 ++++++++++++++- .../work-with-metrics-over-time.qmd | 51 ++++++++++++++++++- 2 files changed, 90 insertions(+), 2 deletions(-) diff --git a/site/guide/integrations/configure-connections.qmd b/site/guide/integrations/configure-connections.qmd index dfa716269e..c931af6f1c 100644 --- a/site/guide/integrations/configure-connections.qmd +++ b/site/guide/integrations/configure-connections.qmd @@ -163,7 +163,9 @@ Required configuration details: You can now [test the connection](#test-connections) to ensure it is working as expected. -## View details +## View connection details + +The connection detail page provides a dashboard-style view for managing a connection, its linked models, and metric sources. 1. In the left sidebar, click **{{< fa gear >}} Settings**. @@ -171,6 +173,12 @@ You can now [test the connection](#test-connections) to ensure it is working as 3. Click the connection you want to view. +The connection detail page displays: + +- **Connection status** — Shows whether the connection is operational, disabled, or experiencing issues. +- **Linked models** — Lists all models in the inventory linked to resources in this integration. +- **Metric sources** — For integrations that provide metrics (such as Amazon Bedrock), shows configured metric sources and their sync status. + ## Edit connections 1. In the left sidebar, click **{{< fa gear >}} Settings**. @@ -215,6 +223,34 @@ If the test is successful, the message **{{< fa check-circle >}} Connection succ 5. In the confirmation dialog, select **OK** to confirm deletion. +## Manage metric sources + +Some integrations, such as Amazon Bedrock, can provide metrics that sync into {{< var vm.product >}} for use in monitoring documents. Metric sources define which external evaluations or metrics are available for a linked model. + +### View metric sources + +1. In the left sidebar, click **{{< fa gear >}} Settings**. + +2. Under {{< fa puzzle-piece >}} Integrations, select **Connections**. + +3. Click on a connection that supports metric sources — for example, a **Bedrock** connection. + +4. On the connection detail page, locate the **Metric Sources** section. + + This section lists all configured metric sources, their sync status, and when they last synced. + +### How metric sources work + +When a model in the {{< var vm.product >}} inventory is linked to an external resource — such as a Bedrock agent — available evaluations from that resource become metric sources. These metrics: + +- Sync automatically on a scheduled interval. +- Appear in the **From Integration** tab when adding metrics over time to monitoring documents.[^5] +- Support threshold configuration for alerting when values exceed acceptable limits. + +::: {.callout title="Bedrock Agent Evaluations"} +For Amazon Bedrock connections, metric sources include agent evaluation results such as accuracy, helpfulness, and other assessment metrics. Link a model to a Bedrock agent to enable these metrics.[^6] +::: + [^1]: [Manage permissions](/guide/configuration/manage-permissions.qmd) @@ -225,3 +261,6 @@ If the test is successful, the message **{{< fa check-circle >}} Connection succ [^4]: [Implement custom integrations](implement-custom-integrations.qmd) +[^5]: [Work with metrics over time](/guide/monitoring/work-with-metrics-over-time.qmd#add-integration-metrics) + +[^6]: [Link external models](link-external-models.qmd) diff --git a/site/guide/monitoring/work-with-metrics-over-time.qmd b/site/guide/monitoring/work-with-metrics-over-time.qmd index b76c8f4468..2c27614836 100644 --- a/site/guide/monitoring/work-with-metrics-over-time.qmd +++ b/site/guide/monitoring/work-with-metrics-over-time.qmd @@ -67,6 +67,51 @@ Learn how to log metrics over time, set thresholds, and analyze model performanc ![Example F1 Score — Metric Over Time visualization](example-f1-score.png){fig-alt="A screenshot showing an example F1 Score — Metric Over Time visualization" .screenshot group="time-metric"} +## Add integration metrics + +You can add metrics from external integrations such as Amazon Bedrock directly to your monitoring documents. This enables you to track evaluation results from Bedrock agents alongside your other model metrics. + +::: {.callout title="Integration metrics require a configured connection."} +Before you can add metrics from an integration, ensure that the connection is configured and the model is linked to the external system.[^11] +::: + +1. In the left sidebar, click **{{< fa cubes >}} Inventory**. + +2. Select a model or find your model by applying a filter or searching for it.[^12] + +3. In the left sidebar that appears for your model, click **{{< fa file >}} Documents** and select the **Latest** tab. + +4. Select a Monitoring type file. + +5. Hover your mouse over the space where you want your new block to go until a horizontal line with a {{< fa square-plus >}} sign appears. + +6. Click {{< fa square-plus >}} and then select **Metric Over Time**. + +7. In the modal that opens, select the **From Integration** tab. + +8. Under [integration source]{.smallcaps}, available integration metrics appear grouped by source — for example, **Bedrock Agent Evals**. + +9. Select one or more evaluation metrics to add. For each metric, you can optionally configure: + + - **[threshold]{.smallcaps}** — Set an upper threshold value to flag when metrics exceed acceptable limits. + - **[time window]{.smallcaps}** — Define the time range for querying metric data. + +10. Click **Insert # Metric(s) Over Time to Document** when you are ready. + +After the metrics sync, data from the integration appears in your document alongside any metrics logged via the {{< var validmind.developer >}}. + +## Use the global time selector + +When working with monitoring documents that contain multiple time-based metrics, use the global time selector to coordinate the time range across all relevant content. + +1. Navigate to a monitoring document with metrics over time. + +2. Locate the time selector at the top of the document. + +3. Select the desired time range from the available options. + +The selected time range applies to all metrics in the document that support time-based filtering, providing a consistent view of your model's performance over the chosen period. + ## View metric over time metadata After you have added metrics over time to your document, you can view the following information attached to the result: @@ -112,4 +157,8 @@ After you have added metrics over time to your document, you can view the follow [^9]: [Work with document versions](/guide/model-documentation/work-with-document-versions.qmd) -[^10]: [Working with model documents](/guide/templates/working-with-model-documents.qmd) \ No newline at end of file +[^10]: [Working with model documents](/guide/templates/working-with-model-documents.qmd) + +[^11]: [Configure connections](/guide/integrations/configure-connections.qmd) and [Link external models](/guide/integrations/link-external-models.qmd) + +[^12]: [Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records) \ No newline at end of file From eb0812ed411e401b6da40d4f946f44c5f73d68d8 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 30 Apr 2026 01:38:08 -0700 Subject: [PATCH 2/7] Add health monitoring section from #1297 to simplify merging --- .../integrations/configure-connections.qmd | 23 ++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/site/guide/integrations/configure-connections.qmd b/site/guide/integrations/configure-connections.qmd index c931af6f1c..fb81b27dac 100644 --- a/site/guide/integrations/configure-connections.qmd +++ b/site/guide/integrations/configure-connections.qmd @@ -176,9 +176,30 @@ The connection detail page provides a dashboard-style view for managing a connec The connection detail page displays: - **Connection status** — Shows whether the connection is operational, disabled, or experiencing issues. -- **Linked models** — Lists all models in the inventory linked to resources in this integration. +- **Linked models** — Lists all {{< var vm.product >}} inventory records linked to resources in this integration. - **Metric sources** — For integrations that provide metrics (such as Amazon Bedrock), shows configured metric sources and their sync status. +## Monitor connection health + +The Connections page provides visibility into the health of your integrations at a glance. + +### Health indicators + +Each connection displays a status indicator: + +- [Operational]{.bubble .green-bg} — The connection is active and working correctly. +- [Needs Attention]{.bubble .orange-bg} — The connection has issues that need to be resolved. +- [Disabled]{.bubble} — The connection has been manually disabled. + +### View connections requiring attention + +Connections with errors or warnings display a status indicator, such as [{{< fa triangle-exclamation >}} 1 link with sync errors]{.bubble .yellow-bg}. Review these connections to: + +1. Identify the cause of the issue from the error details. +2. Check that required secrets are still valid. +3. Verify the external service is accessible. +4. Test the connection after making changes. + ## Edit connections 1. In the left sidebar, click **{{< fa gear >}} Settings**. From 782a791dd4fbf0d66dd57db2ccc64cf18bc8e7ba Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 30 Apr 2026 01:42:38 -0700 Subject: [PATCH 3/7] Add 'feature' and simplify monitoring navigation --- .../monitoring/work-with-metrics-over-time.qmd | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/site/guide/monitoring/work-with-metrics-over-time.qmd b/site/guide/monitoring/work-with-metrics-over-time.qmd index 2c27614836..45c4e948f4 100644 --- a/site/guide/monitoring/work-with-metrics-over-time.qmd +++ b/site/guide/monitoring/work-with-metrics-over-time.qmd @@ -69,7 +69,7 @@ Learn how to log metrics over time, set thresholds, and analyze model performanc ## Add integration metrics -You can add metrics from external integrations such as Amazon Bedrock directly to your monitoring documents. This enables you to track evaluation results from Bedrock agents alongside your other model metrics. +You can add metrics from external integrations such as Amazon Bedrock directly to your monitoring documents. This feature enables you to track evaluation results from Bedrock agents alongside your other model metrics. ::: {.callout title="Integration metrics require a configured connection."} Before you can add metrics from an integration, ensure that the connection is configured and the model is linked to the external system.[^11] @@ -79,24 +79,22 @@ Before you can add metrics from an integration, ensure that the connection is co 2. Select a model or find your model by applying a filter or searching for it.[^12] -3. In the left sidebar that appears for your model, click **{{< fa file >}} Documents** and select the **Latest** tab. +3. In the left sidebar, click **Monitoring** to open the monitoring report. -4. Select a Monitoring type file. +4. Hover your mouse over the space where you want your new block to go until a horizontal line with a {{< fa square-plus >}} sign appears. -5. Hover your mouse over the space where you want your new block to go until a horizontal line with a {{< fa square-plus >}} sign appears. +5. Click {{< fa square-plus >}} and then select **Metric Over Time**. -6. Click {{< fa square-plus >}} and then select **Metric Over Time**. +6. In the modal that opens, select the **From Integration** tab. -7. In the modal that opens, select the **From Integration** tab. +7. Under [integration source]{.smallcaps}, available integration metrics appear grouped by source — for example, **Bedrock Agent Evals**. -8. Under [integration source]{.smallcaps}, available integration metrics appear grouped by source — for example, **Bedrock Agent Evals**. - -9. Select one or more evaluation metrics to add. For each metric, you can optionally configure: +8. Select one or more evaluation metrics to add. For each metric, you can optionally configure: - **[threshold]{.smallcaps}** — Set an upper threshold value to flag when metrics exceed acceptable limits. - **[time window]{.smallcaps}** — Define the time range for querying metric data. -10. Click **Insert # Metric(s) Over Time to Document** when you are ready. +9. Click **Insert # Metric(s) Over Time to Document** when you are ready. After the metrics sync, data from the integration appears in your document alongside any metrics logged via the {{< var validmind.developer >}}. From a63280e97c5473f7c95e0331039dc3ec670a3cd9 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 30 Apr 2026 01:57:38 -0700 Subject: [PATCH 4/7] Update global time range section with correct terminology - Rename "time selector" to "time range" to match UI - Add clock icon reference - List available options (All Time, relative presets, Custom range) - Note that setting is only visible to the current user --- .../monitoring/work-with-metrics-over-time.qmd | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/site/guide/monitoring/work-with-metrics-over-time.qmd b/site/guide/monitoring/work-with-metrics-over-time.qmd index 45c4e948f4..00670b2062 100644 --- a/site/guide/monitoring/work-with-metrics-over-time.qmd +++ b/site/guide/monitoring/work-with-metrics-over-time.qmd @@ -98,17 +98,21 @@ Before you can add metrics from an integration, ensure that the connection is co After the metrics sync, data from the integration appears in your document alongside any metrics logged via the {{< var validmind.developer >}}. -## Use the global time selector +## Use the global time range -When working with monitoring documents that contain multiple time-based metrics, use the global time selector to coordinate the time range across all relevant content. +When working with monitoring documents that contain multiple time-based metrics, use the global time range to coordinate the time range across all relevant content. 1. Navigate to a monitoring document with metrics over time. -2. Locate the time selector at the top of the document. +2. Locate the {{< fa clock >}} time range button at the top of the document. -3. Select the desired time range from the available options. +3. Select the desired time range from the available options: -The selected time range applies to all metrics in the document that support time-based filtering, providing a consistent view of your model's performance over the chosen period. + - **All Time** — Show all available data + - Relative presets (Last 7 days, Last 30 days, etc.) + - **Custom range** — Specify exact start and end dates + +The selected time range applies to all metrics in the document that support time-based filtering, providing a consistent view of your model's performance over the chosen period. This setting is only visible to you and does not affect other users. ## View metric over time metadata From 70d0453466f855ffc1875317754ee0ff1d7dce8a Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 30 Apr 2026 02:03:23 -0700 Subject: [PATCH 5/7] Fix integration metric configuration options - Add both threshold (lower) and threshold (upper) fields - Rename "time window" to "time range" to match UI - Expand time range description with available options --- site/guide/monitoring/work-with-metrics-over-time.qmd | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/site/guide/monitoring/work-with-metrics-over-time.qmd b/site/guide/monitoring/work-with-metrics-over-time.qmd index 00670b2062..8b5e38f4ec 100644 --- a/site/guide/monitoring/work-with-metrics-over-time.qmd +++ b/site/guide/monitoring/work-with-metrics-over-time.qmd @@ -91,8 +91,9 @@ Before you can add metrics from an integration, ensure that the connection is co 8. Select one or more evaluation metrics to add. For each metric, you can optionally configure: - - **[threshold]{.smallcaps}** — Set an upper threshold value to flag when metrics exceed acceptable limits. - - **[time window]{.smallcaps}** — Define the time range for querying metric data. + - **[threshold (lower)]{.smallcaps}** — Set a lower threshold value to flag when metrics fall below acceptable limits. + - **[threshold (upper)]{.smallcaps}** — Set an upper threshold value to flag when metrics exceed acceptable limits. + - **[time range]{.smallcaps}** — Define the time range for querying metric data. Choose to inherit from the document, use a relative preset, or specify a custom date range. 9. Click **Insert # Metric(s) Over Time to Document** when you are ready. From 9a69e8f72e8e3bc6e63b1bac70f9c25e7f9eb0e5 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 30 Apr 2026 02:09:46 -0700 Subject: [PATCH 6/7] Use commas instead of em-dashes for parenthetical phrases --- site/guide/integrations/configure-connections.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/guide/integrations/configure-connections.qmd b/site/guide/integrations/configure-connections.qmd index fb81b27dac..18dc403ad3 100644 --- a/site/guide/integrations/configure-connections.qmd +++ b/site/guide/integrations/configure-connections.qmd @@ -254,7 +254,7 @@ Some integrations, such as Amazon Bedrock, can provide metrics that sync into {{ 2. Under {{< fa puzzle-piece >}} Integrations, select **Connections**. -3. Click on a connection that supports metric sources — for example, a **Bedrock** connection. +3. Click on a connection that supports metric sources, for example a **Bedrock** connection. 4. On the connection detail page, locate the **Metric Sources** section. @@ -262,7 +262,7 @@ Some integrations, such as Amazon Bedrock, can provide metrics that sync into {{ ### How metric sources work -When a model in the {{< var vm.product >}} inventory is linked to an external resource — such as a Bedrock agent — available evaluations from that resource become metric sources. These metrics: +When a model in the {{< var vm.product >}} inventory is linked to an external resource, such as a Bedrock agent, available evaluations from that resource become metric sources. These metrics: - Sync automatically on a scheduled interval. - Appear in the **From Integration** tab when adding metrics over time to monitoring documents.[^5] From 2e5d8bf95558e5fd7fc810237fe7eed8b5d3a382 Mon Sep 17 00:00:00 2001 From: Nik Richers Date: Thu, 30 Apr 2026 14:59:16 -0700 Subject: [PATCH 7/7] docs: Address review feedback for Bedrock integration docs - Update step 7 to describe Evaluators list instead of Bedrock Agent Evals - Rename Metric Sources references to Monitoring to match UI tab name --- site/guide/integrations/configure-connections.qmd | 8 ++++---- site/guide/monitoring/work-with-metrics-over-time.qmd | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/site/guide/integrations/configure-connections.qmd b/site/guide/integrations/configure-connections.qmd index 18dc403ad3..3a91ce7985 100644 --- a/site/guide/integrations/configure-connections.qmd +++ b/site/guide/integrations/configure-connections.qmd @@ -177,7 +177,7 @@ The connection detail page displays: - **Connection status** — Shows whether the connection is operational, disabled, or experiencing issues. - **Linked models** — Lists all {{< var vm.product >}} inventory records linked to resources in this integration. -- **Metric sources** — For integrations that provide metrics (such as Amazon Bedrock), shows configured metric sources and their sync status. +- **Monitoring** — For integrations that provide metrics (such as Amazon Bedrock), shows configured metric sources and their sync status. ## Monitor connection health @@ -244,11 +244,11 @@ If the test is successful, the message **{{< fa check-circle >}} Connection succ 5. In the confirmation dialog, select **OK** to confirm deletion. -## Manage metric sources +## View monitoring data Some integrations, such as Amazon Bedrock, can provide metrics that sync into {{< var vm.product >}} for use in monitoring documents. Metric sources define which external evaluations or metrics are available for a linked model. -### View metric sources +### Access the Monitoring tab 1. In the left sidebar, click **{{< fa gear >}} Settings**. @@ -256,7 +256,7 @@ Some integrations, such as Amazon Bedrock, can provide metrics that sync into {{ 3. Click on a connection that supports metric sources, for example a **Bedrock** connection. -4. On the connection detail page, locate the **Metric Sources** section. +4. On the connection detail page, select the **Monitoring** tab. This section lists all configured metric sources, their sync status, and when they last synced. diff --git a/site/guide/monitoring/work-with-metrics-over-time.qmd b/site/guide/monitoring/work-with-metrics-over-time.qmd index 8b5e38f4ec..08fd9483a7 100644 --- a/site/guide/monitoring/work-with-metrics-over-time.qmd +++ b/site/guide/monitoring/work-with-metrics-over-time.qmd @@ -87,7 +87,7 @@ Before you can add metrics from an integration, ensure that the connection is co 6. In the modal that opens, select the **From Integration** tab. -7. Under [integration source]{.smallcaps}, available integration metrics appear grouped by source — for example, **Bedrock Agent Evals**. +7. Select your integration under [integration source]{.smallcaps}, then choose from the **Evaluators** list — such as Goal Success Rate, Correctness, or Helpfulness. 8. Select one or more evaluation metrics to add. For each metric, you can optionally configure: