Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/activity.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/advanced.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/all-operation.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/anomalyy.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/click-next.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/computed.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/daily.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/details-page.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/drop-down.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/hourly.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/incremental.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/monthly.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/next-button.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/next-buttonn.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/next.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/nextt.png
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/record-limit.png
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/result.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/run-now.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/run.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/running.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/sacn.png
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/schedule.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/select-check.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/side-menu.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/specific.png
Diff not rendered.
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/success-2.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/sucess.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/tag.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/warning.png
Diff not rendered.
Binary file removed docs/assets/source-datastores/scan/weekly.png
Diff not rendered.
185 changes: 45 additions & 140 deletions docs/enrichment-support/supported-enrichment-datastores.md
Original file line number Diff line number Diff line change
@@ -1,150 +1,55 @@
# Supported Enrichment Datastores

Qualytics supports enrichment datastore connectors that help enhance data discovery, profiling, and quality checks. Some connectors include enrichment capabilities, while others provide only standard connectivity.
An enrichment datastore is a dedicated datastore linked to a source datastore that persists scan results, anomalies, remediation data, and source record examples. Not all connectors support enrichment — the tables below show which connectors can be used as enrichment datastores.

In this guide, we will cover:
Enrichment support requires the connector to have **write capabilities** so that Qualytics can create and manage enrichment tables (source records, anomaly records, remediation tables, and metadata tables) in the target datastore.

- **[JDBC Connectors](#jdbc-connectors)**
- **[DFS Connectors](#dfs-connectors)**
!!! info "Available Datastore Connectors"
For the full list of all supported source datastore connectors (including those without enrichment support), see the [Available Datastore Connectors](../source-datastore/add-datastores/available-datastore-connectors.md){:target="_blank"} page.

## JDBC Connectors
!!! info "Enrichment Table Types"
To understand the different table types created in an enrichment datastore (source records, anomaly records, remediation, metadata), see the [Enrichment Table Types](../enrichment/table-types.md){:target="_blank"} documentation.

The table below shows the list of JDBC connectors and whether they support enrichment or not:
## JDBC Connectors

<table>
<thead style="background-color:#f2f6fc;">
<tr>
<th style="color:#222;">No.</th>
<th style="color:#222;">Connector</th>
<th style="color:#222;">Enrichment Support</th>
</tr>
</thead>
<tbody>
<tr style="background-color:#fff9db;">
<td style="color:#222;">01.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/athena/">Athena</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">02.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/bigquery/#add-enrichment-datastore">Big Query</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">03.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/databricks/#add-enrichment-datastore">Databricks</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">04.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/db2/#add-enrichment-datastore">DB2</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#fff9db;">
<td style="color:#222;">05.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/dremio/">Dremio</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#fff9db;">
<td style="color:#222;">06.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/fabric-analytics/">Fabric Analytics</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#fff9db;">
<td style="color:#222;">07.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/hive/">Hive</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">08.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/maria-db/#add-enrichment-datastore">MariaDB</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">09.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/microsoft-sql-server/#add-enrichment-datastore">Microsoft SQL Server</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">10.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/mysql/#add-enrichment-datastore">MySQL</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#fff9db;">
<td style="color:#222;">11.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/oracle/">Oracle</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">12.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/postgresql/#add-enrichment-datastore">PostgreSQL</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#fff9db;">
<td style="color:#222;">13.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/presto/">Presto</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">14.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/redshift/#add-enrichment-datastore">Redshift</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">15.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/snowflake/#add-enrichment-datastore-connection">Snowflake</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">16.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/synapse/#add-enrichment-datastore">Synapse</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#fff9db;">
<td style="color:#222;">17.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/teradata/">Teradata</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#fff9db;">
<td style="color:#222;">18.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/timescale-db/">TimescaleDB</a></td>
<td>❌</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">19.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/trino/">Trino</a></td>
<td>✅</td>
</tr>
</tbody>
</table>
JDBC connectors that support enrichment can store scan results and anomaly data directly in the relational database. Out of 19 JDBC connectors, **11 support enrichment**.

## DFS Connectors
<div class="connector-table" markdown>

The table below shows the list of DFS connectors and whether they support enrichment or not:
| No. | Connector | Logo | Enrichment Support |
| :--- | :--- | :---: | :---: |
| 1. | [Athena](../source-datastore/add-datastores/athena.md) | ![Athena](../assets/source-datastores/add-datastores/connector-logos/logo-athena.svg){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 2. | [BigQuery](../source-datastore/add-datastores/bigquery.md#add-enrichment-datastore) | ![BigQuery](../assets/source-datastores/add-datastores/connector-logos/logo-big-query.png){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 3. | [Databricks](../source-datastore/add-datastores/databricks.md#add-enrichment-datastore) | ![Databricks](../assets/source-datastores/add-datastores/connector-logos/logo-databricks.png){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 4. | [DB2](../source-datastore/add-datastores/db2.md#add-enrichment-datastore) | ![DB2](../assets/source-datastores/add-datastores/connector-logos/logo-ibm-db2.png){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 5. | [Dremio](../source-datastore/add-datastores/dremio.md) | ![Dremio](../assets/source-datastores/add-datastores/connector-logos/logo-dremio.svg){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 6. | [Fabric Analytics](../source-datastore/add-datastores/fabric-analytics.md) | ![Fabric](../assets/source-datastores/add-datastores/connector-logos/logo-microsoft-fabric.svg){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 7. | [Hive](../source-datastore/add-datastores/hive.md) | ![Hive](../assets/source-datastores/add-datastores/connector-logos/logo-hive.png){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 8. | [MariaDB](../source-datastore/add-datastores/maria-db.md#add-enrichment-datastore) | ![MariaDB](../assets/source-datastores/add-datastores/connector-logos/logo-mariadb.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 9. | [Microsoft SQL Server](../source-datastore/add-datastores/microsoft-sql-server.md#add-enrichment-datastore) | ![SQL Server](../assets/source-datastores/add-datastores/connector-logos/logo-ms-sql.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 10. | [MySQL](../source-datastore/add-datastores/mysql.md#add-enrichment-datastore) | ![MySQL](../assets/source-datastores/add-datastores/connector-logos/mysql-icon.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 11. | [Oracle](../source-datastore/add-datastores/oracle.md) | ![Oracle](../assets/source-datastores/add-datastores/connector-logos/oracle-icon.svg){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 12. | [PostgreSQL](../source-datastore/add-datastores/postgresql.md#add-enrichment-datastore) | ![PostgreSQL](../assets/source-datastores/add-datastores/connector-logos/logo-postgres.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 13. | [Presto](../source-datastore/add-datastores/presto.md) | ![Presto](../assets/source-datastores/add-datastores/connector-logos/presto.png){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 14. | [Redshift](../source-datastore/add-datastores/redshift.md#add-enrichment-datastore) | ![Redshift](../assets/source-datastores/add-datastores/connector-logos/logo-redshift.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 15. | [Snowflake](../source-datastore/add-datastores/snowflake.md#add-enrichment-datastore-connection) | ![Snowflake](../assets/source-datastores/add-datastores/connector-logos/snowflake.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 16. | [Synapse](../source-datastore/add-datastores/synapse.md#add-enrichment-datastore) | ![Synapse](../assets/source-datastores/add-datastores/connector-logos/logo-synapses.png){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 17. | [Teradata](../source-datastore/add-datastores/teradata.md) | ![Teradata](../assets/source-datastores/add-datastores/connector-logos/logo-teradata.png){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 18. | [TimescaleDB](../source-datastore/add-datastores/timescale-db.md) | ![Timescale](../assets/source-datastores/add-datastores/connector-logos/logo-timescaledb.png){ width="24" } | :material-close-circle-outline:{ .lg title="Not supported" } |
| 19. | [Trino](../source-datastore/add-datastores/trino.md) | ![Trino](../assets/source-datastores/add-datastores/connector-logos/trino.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |

<table>
<thead style="background-color:#f2f6fc;">
<tr>
<th style="color:#222;">No.</th>
<th style="color:#222;">Connector</th>
<th style="color:#222;">Enrichment Support</th>
</tr>
</thead>
<tbody>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">01.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/amazon-s3/#add-enrichment-datastore">Amazon S3</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">02.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/azure-datalake-storage/#add-enrichment-datastore">Azure Datalake Storage (ABFS)</a></td>
<td>✅</td>
</tr>
<tr style="background-color:#dff0d8;">
<td style="color:#222;">03.</td>
<td><a href="https://userguide.qualytics.io/source-datastore/add-datastores/google-cloud-storage/#add-enrichment-datastore">Google Cloud Storage (GCS)</a></td>
<td>✅</td>
</tr>
</tbody>
</table>
</div>

## DFS Connectors

DFS connectors that support enrichment store scan results and anomaly data as files (Parquet/JSON) in cloud object storage. **All 3 DFS connectors support enrichment**.

<div class="connector-table" markdown>

| No. | Connector | Logo | Enrichment Support |
| :--- | :--- | :---: | :---: |
| 1. | [Amazon S3](../source-datastore/add-datastores/amazon-s3.md#add-enrichment-datastore) | ![S3](../assets/source-datastores/add-datastores/connector-logos/logo-s3.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 2. | [Azure Datalake Storage (ABFS)](../source-datastore/add-datastores/azure-datalake-storage.md#add-enrichment-datastore) | ![ABFS](../assets/source-datastores/add-datastores/connector-logos/logo-abfs.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |
| 3. | [Google Cloud Storage (GCS)](../source-datastore/add-datastores/google-cloud-storage.md#add-enrichment-datastore) | ![GCS](../assets/source-datastores/add-datastores/connector-logos/logo-gcs.svg){ width="24" } | :material-check-circle:{ .lg title="Supported" } |

</div>
2 changes: 2 additions & 0 deletions docs/settings/security/team-permissions.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ Legend:
| Assign Tags in Flows | ❌ | ❌ | ✅ | ✅ | ✅ |
| View Datastore Groups | ✅ | ✅ | ✅ | ✅ | ✅ |
| Assign Datastore to Group | ❌ | ❌ | ❌ | ❌ | ✅ |
| Multi-Schema Datastore Creation | ❌ | ❌ | ❌ | ❌ | ❌ |
| View Field Status | ✅ | ✅ | ✅ | ✅ | ✅ |
| Mask / Unmask Field | ❌ | ❌ | ❌ | ❌ | ✅ |
| Exclude Field | ❌ | ❌ | ❌ | ❌ | ✅ |
Expand Down Expand Up @@ -88,6 +89,7 @@ Editor role allows users to manage datastore functions comprehensively. They can
| Feature | Operation | Can View/Can Run | Can Manage |
|:-----------------------|:----------------------------|:------------------|:-------------|
| **Datastores** | Add Datastore | ❌ | ✅ |
| | Multi-Schema Datastore Creation | ❌ | ✅ |
| | Edit Settings | ❌ | ✅ |
| **Enrichment** | Add Enrichment | ❌ | ✅ |
| | Edit Enrichment | ❌ | ✅ |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# How Multi-Schema Creation Works

## Overview

The multi-schema creation flow is a two-step wizard that guides you through discovering schemas from a connection and creating multiple source datastores at once.

<div style="display: flex; justify-content: center;">

```mermaid
graph TD
A["Select Connection"] --> B["Discover Catalogs/Schemas"]
B --> C["Select Schemas"]
C --> D["Configure Properties"]
D --> E["Validate & Create"]
E --> F["Link Enrichment (Optional)"]
```

</div>

## Step 1: Source Datastore Configuration

In the first step, you configure the connection and select which schemas to onboard.

### Connection Selection

You can either:

- **Create a new connection**: Provide all connection details (host, port, credentials, etc.) from scratch.
- **Use an existing connection**: Select a previously saved connection to reuse its credentials.

For more information about connections, refer to the [Connection Overview](../../connections/overview-of-a-connection.md) documentation.

### Catalog Discovery

For connectors that support a catalog hierarchy (e.g., databases in PostgreSQL, databases in Snowflake, projects in BigQuery), the first discovery step retrieves the list of available catalogs from the connection. You select which catalog to browse for schemas.

!!! tip
Click the **refresh** button next to the catalog dropdown to reload the available catalogs from your connection.

Not all connectors have a catalog level. For connectors like Oracle or DB2, schemas are discovered directly without a catalog selection step.

### Schema Discovery and Selection

Once a catalog is selected (or for connectors without a catalog level), the system discovers all available schemas and presents them in a multi-select dropdown.

Key behaviors:

- **Warning icons**: Schemas that already have an existing datastore display a warning icon with a tooltip showing which datastores are associated.
- **Search filtering**: You can filter schemas by name using the search field in the dropdown.
- **Selection count**: The dropdown shows how many schemas are selected (e.g., "3 selected").

!!! warning
You can select schemas that already have datastores, but this will create duplicate datastores for those schemas.

### Name Template

The **Name Template** field lets you define a naming pattern for the datastores that will be created. Use the `{{schema}}` placeholder, which will be replaced with each schema name.

For example, with the template `production_{{schema}}` and schemas `public`, `staging`, and `analytics`:

| Schema | Generated Datastore Name |
| :--- | :--- |
| `public` | `production_public` |
| `staging` | `production_staging` |
| `analytics` | `production_analytics` |

!!! tip
A preview of the generated names is displayed below the name template field, showing how the first few datastores will be named.

### Additional Properties

| Field | Required | Description |
| :--- | :--- | :--- |
| **Teams** | Yes | One or more teams to associate with all newly created datastores. |
| **Group** | No | Assign all datastores to a [datastore group](../../../managing-datastores/grouping/overview.md). |
| **Initiate Sync** | No | Automatically run a sync operation on each newly created datastore after creation. |

### Validation

Before creating the datastores, click **Test Connection** to validate connectivity for all selected schemas. The validation runs per-schema and reports individual results, so you can identify which schemas have issues before proceeding.

## Step 2: Enrichment Datastore Linking (Optional)

After configuring the source datastores, an optional second step lets you link all newly created datastores to a single enrichment datastore. The enrichment datastore stores analyzed results, anomalies, and additional metadata.

You can:

- **Use an existing enrichment datastore**: Select a previously created enrichment datastore.
- **Create a new enrichment datastore**: Configure a new enrichment datastore during the same flow.

!!! info
All source datastores created in the batch will be linked to the same enrichment datastore. The enrichment prefix is shared across all of them.

For step-by-step instructions on adding a datastore, refer to the [Add Source Datastore](../../overview-of-a-jdbc-datastore.md) documentation.

## What Happens After Creation

Once the creation process completes, Qualytics returns a summary showing:

- **Created datastores**: The list of successfully created datastore IDs.
- **Errors**: Any schemas that failed to create, with the corresponding error message.

If **Initiate Sync** was enabled, a sync operation will automatically start on each successfully created datastore, detecting containers and fields within each schema.
Loading
Loading