From fd25825b3c0d61b77cfdeaa4d667875a019d7434 Mon Sep 17 00:00:00 2001 From: Charles Yu Date: Mon, 13 Jul 2026 17:33:51 +0000 Subject: [PATCH] Update Databricks integration setup docs (DOIO-94) Reduce onboarding friction in the Databricks setup docs: - Fold the required workspace entitlements (Workspace access, Databricks SQL access, Admin access) into service principal creation, replacing the separate "add to admins group" steps; note the granular-permissions fallback when Admin access cannot be granted. - Restructure SQL Warehouse setup into a single step with ordered substeps: select a Pro/Serverless warehouse (recommend a dedicated 2XS), copy the ID, grant CAN USE, and grant system-table read access. - Document where to find the SQL Warehouse ID, and note it powers Quality Monitoring, with a cross-link to that page. - Tell users to run the system-table GRANT commands in the SQL Editor, and fix the grant syntax to use the backticked client ID instead of the service principal display name. - Surface the MANAGE-on-CATALOG-system prerequisite as a warning callout. Also updates the Cloud Cost Management (saas_costs) and Quality Monitoring Databricks docs to use the SQL Editor and backticked client ID for the system-table grants. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../cloud_cost_management/setup/saas_costs.md | 8 +-- .../jobs_monitoring/databricks/_index.md | 68 +++++++++++-------- .../data_warehouses/databricks.md | 2 + 3 files changed, 45 insertions(+), 33 deletions(-) diff --git a/content/en/cloud_cost_management/setup/saas_costs.md b/content/en/cloud_cost_management/setup/saas_costs.md index 5cf1d5bf8d9..2db6eeaf315 100644 --- a/content/en/cloud_cost_management/setup/saas_costs.md +++ b/content/en/cloud_cost_management/setup/saas_costs.md @@ -120,11 +120,11 @@ Notes: 4. Enter a `System Tables SQL Warehouse ID` corresponding to your Databricks instance's warehouse to query for system table billing data. 5. Click {{< ui >}}Save Databricks Workspace{{< /ui >}}. -Your service principal requires read access to the [system tables](https://docs.databricks.com/aws/en/admin/system-tables/) within Unity Catalog. +Your service principal requires read access to the [system tables](https://docs.databricks.com/aws/en/admin/system-tables/) within Unity Catalog. In Databricks, open the {{< ui >}}SQL Editor{{< /ui >}} and run the following commands, using the service principal's client ID (not its display name): ```sql -GRANT USE CATALOG ON CATALOG system TO ; -GRANT USE SCHEMA ON CATALOG system TO ; -GRANT SELECT ON CATALOG system TO ; +GRANT USE CATALOG ON CATALOG system TO ``; +GRANT USE SCHEMA ON CATALOG system TO ``; +GRANT SELECT ON CATALOG system TO ``; ``` Your Databricks cost data for the past 15 months can be accessed in Cloud Cost Management after 24 hours. To access the available data collected by each SaaS Cost Integration, see the [Data Collected section](#data-collected). diff --git a/content/en/data_observability/jobs_monitoring/databricks/_index.md b/content/en/data_observability/jobs_monitoring/databricks/_index.md index 2f84c2b6918..42051ff805d 100644 --- a/content/en/data_observability/jobs_monitoring/databricks/_index.md +++ b/content/en/data_observability/jobs_monitoring/databricks/_index.md @@ -37,9 +37,15 @@ Follow these steps to enable Data Observability: Jobs Monitoring for Databricks. 1. As a **Databricks workspace admin**, go to {{< ui >}}Settings{{< /ui >}} by clicking your profile in the upper-right corner of the workspace. 1. On the {{< ui >}}Identity and access{{< /ui >}} tab, click {{< ui >}}Manage{{< /ui >}} next to {{< ui >}}Service principals{{< /ui >}}. 1. Click {{< ui >}}Add service principal{{< /ui >}}, then click {{< ui >}}Add new{{< /ui >}}. -1. Enter a name, then click **Add**. +1. Enter a name and enable the following workspace entitlements for the service principal: + - {{< ui >}}Workspace access{{< /ui >}} + - {{< ui >}}Databricks SQL access{{< /ui >}} + - {{< ui >}}Admin access{{< /ui >}}: grants the workspace administrator access that Datadog requires. This is equivalent to adding the service principal to the `admins` group. + +
If you cannot grant the Admin access entitlement, provision granular access instead, as described in the Permissions section under Advanced Configuration below.
For Azure Databricks, select the "Databricks managed" management type. Datadog does NOT support "Microsoft Entra ID managed" service principals.
+1. Click **Add**. 1. Click on the name of your new service principal. Under the {{< ui >}}Secrets{{< /ui >}} tab, click {{< ui >}}Generate secret{{< /ui >}}. 1. Set {{< ui >}}Lifetime (days){{< /ui >}} to the maximum value allowed (730). @@ -51,8 +57,6 @@ Follow these steps to enable Data Observability: Jobs Monitoring for Databricks. {{< img src="data_jobs/databricks/client-id-secret.png" alt="In Databricks, a modal showing the client ID and secret associated with a new OAuth secret is displayed." style="width:70%;" >}} 1. On the {{< ui >}}Permissions{{< /ui >}} tab, click {{< ui >}}Grant access{{< /ui >}}. Search for the new service principal, grant it the {{< ui >}}Manage{{< /ui >}} permission, and click {{< ui >}}Save{{< /ui >}}. -1. Return to the {{< ui >}}Identity and access{{< /ui >}} tab and click {{< ui >}}Manage{{< /ui >}} next to {{< ui >}}Groups{{< /ui >}}. -1. Click the {{< ui >}}admins{{< /ui >}} group, then click {{< ui >}}Add members{{< /ui >}} to add the new service principal. #### Add the Databricks workspace to Datadog @@ -60,17 +64,19 @@ Follow these steps to enable Data Observability: Jobs Monitoring for Databricks. 1. On the {{< ui >}}Configure{{< /ui >}} tab, click {{< ui >}}Add Databricks Workspace{{< /ui >}}. 1. Enter a workspace name, your Databricks workspace URL, and the client ID and secret you generated. {{< img src="data_jobs/databricks/connect-workspace-form-m2m.png" alt="In the Datadog-Databricks integration tile, a Databricks workspace is displayed. This workspace has a name, URL, client ID, and client secret." style="width:100%;" >}} -1. To gain visibility into your Databricks costs in Data Observability: Jobs Monitoring or [Cloud Cost Management][18], provide the ID of a [Databricks SQL Warehouse][19] that Datadog can use to query your [system tables][20]. - - The service principal must have access to the SQL Warehouse. In the Warehouse configuration page, go to {{< ui >}}Permissions{{< /ui >}} (top right) and grant it `CAN USE` permission. - - Grant the service principal read access to the Unity Catalog [system tables][20] by running the following commands: - ```sql - GRANT USE CATALOG ON CATALOG system TO ; - GRANT SELECT ON CATALOG system TO ; - GRANT USE SCHEMA ON CATALOG system TO ; - ``` - The user granting these must have `MANAGE` privilege on `CATALOG system`. - - - The SQL Warehouse must be Pro or Serverless. Classic Warehouses are **NOT** supported. A 2XS warehouse is recommended, with Auto Stop set to 5-10 minutes to reduce cost. +1. Provide the ID of a [Databricks SQL Warehouse][19] for Datadog to query. This gives you visibility into your Databricks costs in Data Observability: Jobs Monitoring or [Cloud Cost Management][18], and powers [Data Observability: Quality Monitoring][21]. + 1. In Databricks, go to {{< ui >}}SQL Warehouses{{< /ui >}} and select the warehouse for Datadog to use. It must be Pro or Serverless; Classic Warehouses are **NOT** supported. We recommend creating a dedicated 2XS warehouse, with Auto Stop configured for 5-10 minutes to reduce cost. + 1. Copy the ID from the warehouse's overview page (it is also the last segment of the warehouse's URL) and enter it in the integration tile. + 1. On the warehouse's {{< ui >}}Permissions{{< /ui >}} tab (top right), grant the service principal `CAN USE`. + 1. Grant the service principal read access to the Unity Catalog [system tables][20]. In the {{< ui >}}SQL Editor{{< /ui >}}, run the following commands using the service principal's client ID (not its display name): + + ```sql + GRANT USE CATALOG ON CATALOG system TO ``; + GRANT SELECT ON CATALOG system TO ``; + GRANT USE SCHEMA ON CATALOG system TO ``; + ``` + +
The user running these commands must have the MANAGE privilege on CATALOG system.
1. In the **Select products to set up integration** section, ensure that Data Observability: Jobs Monitoring is {{< ui >}}Enabled{{< /ui >}}. 1. In the {{< ui >}}Datadog Agent Setup{{< /ui >}} section, choose either - [Managed by Datadog (recommended)](?tab=datadogmanagedglobalinitscriptrecommended#install-the-datadog-agent): Datadog installs and manages the Agent with a global init script in the workspace. @@ -79,6 +85,7 @@ Follow these steps to enable Data Observability: Jobs Monitoring for Databricks. [18]: https://docs.datadoghq.com/cloud_cost_management/ [19]: https://docs.databricks.com/aws/en/compute/sql-warehouse/ [20]: https://docs.databricks.com/aws/en/admin/system-tables/ +[21]: /data_observability/quality_monitoring/data_warehouses/databricks/ {{% /tab %}} @@ -110,17 +117,19 @@ See [Private Link Connectivity (Preview)][15] for full setup instructions. 1. On the {{< ui >}}Configure{{< /ui >}} tab, click {{< ui >}}Add Databricks Workspace{{< /ui >}}. 1. Enter a workspace name, your Databricks workspace URL, and the Databricks token you generated. {{< img src="data_jobs/databricks/configure-workspace-form.png" alt="In the Datadog-Databricks integration tile, a Databricks workspace is displayed. This workspace has a name, URL, and API token." style="width:100%;" >}} -1. To gain visibility into your Databricks costs in Data Observability: Jobs Monitoring or [Cloud Cost Management][18], provide the ID of a [Databricks SQL Warehouse][19] that Datadog can use to query your [system tables][20]. - - - The token's principal must have access to the SQL Warehouse. Give it `CAN USE` permission from **Permissions** at the top right of the Warehouse configuration page. - - Grant the service principal read access to the Unity Catalog [system tables][20] by running the following commands:: - ```sql - GRANT USE CATALOG ON CATALOG system TO ; - GRANT SELECT ON CATALOG system TO ; - GRANT USE SCHEMA ON CATALOG system TO ; - ``` - The user granting these must have `MANAGE` privilege on `CATALOG system`. - - The SQL Warehouse must be Pro or Serverless. Classic Warehouses are **NOT** supported. A 2XS size warehouse is recommended, with Auto Stop configured for 5-10 minutes to minimize cost. +1. Provide the ID of a [Databricks SQL Warehouse][19] for Datadog to query. This gives you visibility into your Databricks costs in Data Observability: Jobs Monitoring or [Cloud Cost Management][18], and powers [Data Observability: Quality Monitoring][21]. + 1. In Databricks, go to {{< ui >}}SQL Warehouses{{< /ui >}} and select the warehouse for Datadog to use. It must be Pro or Serverless; Classic Warehouses are **NOT** supported. We recommend creating a dedicated 2XS warehouse, with Auto Stop configured for 5-10 minutes to reduce cost. + 1. Copy the ID from the warehouse's overview page (it is also the last segment of the warehouse's URL) and enter it in the integration tile. + 1. On the warehouse's {{< ui >}}Permissions{{< /ui >}} tab (top right), grant the token's principal `CAN USE`. + 1. Grant the token's principal read access to the Unity Catalog [system tables][20]. In the {{< ui >}}SQL Editor{{< /ui >}}, run the following commands using the principal's client ID (not its display name): + + ```sql + GRANT USE CATALOG ON CATALOG system TO ``; + GRANT SELECT ON CATALOG system TO ``; + GRANT USE SCHEMA ON CATALOG system TO ``; + ``` + +
The user running these commands must have the MANAGE privilege on CATALOG system.
1. In the **Select products to set up integration** section, make sure the Data Observability: Jobs Monitoring product is **Enabled**. 1. In the {{< ui >}}Datadog Agent Setup{{< /ui >}} section, choose either - [Managed by Datadog (recommended)](?tab=datadogmanagedglobalinitscriptrecommended#install-the-datadog-agent): Datadog installs and manages the Agent with a global init script in the workspace. @@ -133,6 +142,7 @@ See [Private Link Connectivity (Preview)][15] for full setup instructions. [18]: https://docs.datadoghq.com/cloud_cost_management [19]: https://docs.databricks.com/aws/en/compute/sql-warehouse/ [20]: https://docs.databricks.com/aws/en/admin/system-tables/ +[21]: /data_observability/quality_monitoring/data_warehouses/databricks/ {{% /tab %}} @@ -464,11 +474,11 @@ If you need more granular control, grant these minimal permissions to the follow Additionally, for Datadog to access your Databricks cost data in Data Observability: Jobs Monitoring or [Cloud Cost Management][26], the user or service principal used to query [system tables][27] must have the following permissions: - `CAN USE` permission on the SQL Warehouse. - - Read access to the [system tables][27] within Unity Catalog. This can be granted with: + - Read access to the [system tables][27] within Unity Catalog. In Databricks, open the {{< ui >}}SQL Editor{{< /ui >}} and run the following commands, using the service principal's client ID (not its display name): ```sql - GRANT USE CATALOG ON CATALOG system TO ; - GRANT SELECT ON CATALOG system TO ; - GRANT USE SCHEMA ON CATALOG system TO ; + GRANT USE CATALOG ON CATALOG system TO ``; + GRANT SELECT ON CATALOG system TO ``; + GRANT USE SCHEMA ON CATALOG system TO ``; ``` The user granting these must have `MANAGE` privilege on `CATALOG system`. diff --git a/content/en/data_observability/quality_monitoring/data_warehouses/databricks.md b/content/en/data_observability/quality_monitoring/data_warehouses/databricks.md index e92086f17d9..18e220c09ca 100644 --- a/content/en/data_observability/quality_monitoring/data_warehouses/databricks.md +++ b/content/en/data_observability/quality_monitoring/data_warehouses/databricks.md @@ -37,6 +37,8 @@ If your Databricks workspace restricts network access by IP, add the Datadog web ### Step 2 - Grant access +In Databricks, open the {{< ui >}}SQL Editor{{< /ui >}} to run the following commands. Use the service principal's application (client) ID, not its display name, wherever `` appears. + First, grant access to system schemas for lineage: ```sql GRANT USE CATALOG ON CATALOG system TO ``;