Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 61 additions & 1 deletion site/guide/integrations/configure-connections.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -163,14 +163,43 @@ Required configuration details:

You can now [test the connection](#test-connections) to ensure it is working as expected.

## View details
## View connection details

The connection detail page provides a dashboard-style view for managing a connection, its linked models, and metric sources.

1. In the left sidebar, click **{{< fa gear >}} Settings**.

2. Under {{< fa puzzle-piece >}} Integrations, select **Connections**.

3. Click the connection you want to view.

The connection detail page displays:

- **Connection status** — Shows whether the connection is operational, disabled, or experiencing issues.
- **Linked models** — Lists all {{< var vm.product >}} inventory records linked to resources in this integration.
- **Monitoring** — For integrations that provide metrics (such as Amazon Bedrock), shows configured metric sources and their sync status.

## Monitor connection health

The Connections page provides visibility into the health of your integrations at a glance.

### Health indicators

Each connection displays a status indicator:

- [Operational]{.bubble .green-bg} — The connection is active and working correctly.
- [Needs Attention]{.bubble .orange-bg} — The connection has issues that need to be resolved.
- [Disabled]{.bubble} — The connection has been manually disabled.

### View connections requiring attention

Connections with errors or warnings display a status indicator, such as [{{< fa triangle-exclamation >}} 1 link with sync errors]{.bubble .yellow-bg}. Review these connections to:

1. Identify the cause of the issue from the error details.
2. Check that required secrets are still valid.
3. Verify the external service is accessible.
4. Test the connection after making changes.

## Edit connections

1. In the left sidebar, click **{{< fa gear >}} Settings**.
Expand Down Expand Up @@ -215,6 +244,34 @@ If the test is successful, the message **{{< fa check-circle >}} Connection succ

5. In the confirmation dialog, select **OK** to confirm deletion.

## View monitoring data

Some integrations, such as Amazon Bedrock, can provide metrics that sync into {{< var vm.product >}} for use in monitoring documents. Metric sources define which external evaluations or metrics are available for a linked model.

### Access the Monitoring tab

1. In the left sidebar, click **{{< fa gear >}} Settings**.

2. Under {{< fa puzzle-piece >}} Integrations, select **Connections**.

3. Click on a connection that supports metric sources, for example a **Bedrock** connection.

4. On the connection detail page, select the **Monitoring** tab.

This section lists all configured metric sources, their sync status, and when they last synced.

### How metric sources work

When a model in the {{< var vm.product >}} inventory is linked to an external resource, such as a Bedrock agent, available evaluations from that resource become metric sources. These metrics:

- Sync automatically on a scheduled interval.
- Appear in the **From Integration** tab when adding metrics over time to monitoring documents.[^5]
- Support threshold configuration for alerting when values exceed acceptable limits.

::: {.callout title="Bedrock Agent Evaluations"}
For Amazon Bedrock connections, metric sources include agent evaluation results such as accuracy, helpfulness, and other assessment metrics. Link a model to a Bedrock agent to enable these metrics.[^6]
:::

<!-- FOOTNOTES -->

[^1]: [Manage permissions](/guide/configuration/manage-permissions.qmd)
Expand All @@ -225,3 +282,6 @@ If the test is successful, the message **{{< fa check-circle >}} Connection succ

[^4]: [Implement custom integrations](implement-custom-integrations.qmd)

[^5]: [Work with metrics over time](/guide/monitoring/work-with-metrics-over-time.qmd#add-integration-metrics)

[^6]: [Link external models](link-external-models.qmd)
54 changes: 53 additions & 1 deletion site/guide/monitoring/work-with-metrics-over-time.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,54 @@ Learn how to log metrics over time, set thresholds, and analyze model performanc

![Example F1 Score — Metric Over Time visualization](example-f1-score.png){fig-alt="A screenshot showing an example F1 Score — Metric Over Time visualization" .screenshot group="time-metric"}

## Add integration metrics

You can add metrics from external integrations such as Amazon Bedrock directly to your monitoring documents. This feature enables you to track evaluation results from Bedrock agents alongside your other model metrics.

::: {.callout title="Integration metrics require a configured connection."}
Before you can add metrics from an integration, ensure that the connection is configured and the model is linked to the external system.[^11]
:::

1. In the left sidebar, click **{{< fa cubes >}} Inventory**.

2. Select a model or find your model by applying a filter or searching for it.[^12]

3. In the left sidebar, click **Monitoring** to open the monitoring report.

4. Hover your mouse over the space where you want your new block to go until a horizontal line with a {{< fa square-plus >}} sign appears.

5. Click {{< fa square-plus >}} and then select **Metric Over Time**.

6. In the modal that opens, select the **From Integration** tab.

7. Select your integration under [integration source]{.smallcaps}, then choose from the **Evaluators** list — such as Goal Success Rate, Correctness, or Helpfulness.

8. Select one or more evaluation metrics to add. For each metric, you can optionally configure:

- **[threshold (lower)]{.smallcaps}** — Set a lower threshold value to flag when metrics fall below acceptable limits.
- **[threshold (upper)]{.smallcaps}** — Set an upper threshold value to flag when metrics exceed acceptable limits.
- **[time range]{.smallcaps}** — Define the time range for querying metric data. Choose to inherit from the document, use a relative preset, or specify a custom date range.

9. Click **Insert # Metric(s) Over Time to Document** when you are ready.

After the metrics sync, data from the integration appears in your document alongside any metrics logged via the {{< var validmind.developer >}}.

## Use the global time range

When working with monitoring documents that contain multiple time-based metrics, use the global time range to coordinate the time range across all relevant content.

1. Navigate to a monitoring document with metrics over time.

2. Locate the {{< fa clock >}} time range button at the top of the document.

3. Select the desired time range from the available options:

- **All Time** — Show all available data
- Relative presets (Last 7 days, Last 30 days, etc.)
- **Custom range** — Specify exact start and end dates

The selected time range applies to all metrics in the document that support time-based filtering, providing a consistent view of your model's performance over the chosen period. This setting is only visible to you and does not affect other users.

## View metric over time metadata

After you have added metrics over time to your document, you can view the following information attached to the result:
Expand Down Expand Up @@ -112,4 +160,8 @@ After you have added metrics over time to your document, you can view the follow

[^9]: [Work with document versions](/guide/model-documentation/work-with-document-versions.qmd)

[^10]: [Working with model documents](/guide/templates/working-with-model-documents.qmd)
[^10]: [Working with model documents](/guide/templates/working-with-model-documents.qmd)

[^11]: [Configure connections](/guide/integrations/configure-connections.qmd) and [Link external models](/guide/integrations/link-external-models.qmd)

[^12]: [Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records)
Loading