diff --git a/.typos.toml b/.typos.toml index 99ace14d05..b6a4417478 100644 --- a/.typos.toml +++ b/.typos.toml @@ -6,6 +6,7 @@ MyApp = "MyApp" OpenAPIv3 = "OpenAPIv3" AKS = "AKS" IST = "IST" +LGPD = "LGPD" CREATEIN = "CREATEIN" ALTERIN = "ALTERIN" diff --git a/docs/anomalies/source-record.md b/docs/anomalies/source-record.md index 69e7c4e859..fdf67752ae 100644 --- a/docs/anomalies/source-record.md +++ b/docs/anomalies/source-record.md @@ -29,7 +29,7 @@ User can download the source records that were captured for an anomaly during th ## Masked Fields in Source Records -If a container contains [masked fields](../fields/field-status/managing-field-status/mask-a-field.md), their values are obfuscated by default in source records. +If a container contains [masked fields](../fields/field-status/concepts/field-masking.md), their values are obfuscated by default in source records. Users with **Editor** permission can reveal masked values for an anomaly using the reveal toggle. Toggling reveal shows all source records attached to that anomaly at once — reveal is per-anomaly, not per-record. Every reveal action is recorded in the **masking audit log** with the user identity, timestamp, IP address, and the specific fields accessed — this log is reviewable by Administrators. diff --git a/docs/assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records-2.png b/docs/assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records-2.png new file mode 100644 index 0000000000..d89bc5b651 Binary files /dev/null and b/docs/assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records-2.png differ diff --git a/docs/assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records.png b/docs/assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records.png new file mode 100644 index 0000000000..69218af6be Binary files /dev/null and b/docs/assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records.png differ diff --git a/docs/assets/fields/field-status/concepts/field-masking/masked-data-preview.png b/docs/assets/fields/field-status/concepts/field-masking/masked-data-preview.png new file mode 100644 index 0000000000..c12d7b9796 Binary files /dev/null and b/docs/assets/fields/field-status/concepts/field-masking/masked-data-preview.png differ diff --git a/docs/assets/fields/field-status/concepts/field-masking/masked-field-detail.png b/docs/assets/fields/field-status/concepts/field-masking/masked-field-detail.png new file mode 100644 index 0000000000..cbaefec64f Binary files /dev/null and b/docs/assets/fields/field-status/concepts/field-masking/masked-field-detail.png differ diff --git a/docs/assets/fields/field-status/concepts/field-masking/masked-field-tree-view.png b/docs/assets/fields/field-status/concepts/field-masking/masked-field-tree-view.png new file mode 100644 index 0000000000..045c5b13ed Binary files /dev/null and b/docs/assets/fields/field-status/concepts/field-masking/masked-field-tree-view.png differ diff --git a/docs/assets/fields/field-status/concepts/field-masking/masked-fields-container-listing.png b/docs/assets/fields/field-status/concepts/field-masking/masked-fields-container-listing.png new file mode 100644 index 0000000000..560674fce9 Binary files /dev/null and b/docs/assets/fields/field-status/concepts/field-masking/masked-fields-container-listing.png differ diff --git a/docs/assets/fields/field-status/concepts/field-masking/masked-fields-explore-view.png b/docs/assets/fields/field-status/concepts/field-masking/masked-fields-explore-view.png new file mode 100644 index 0000000000..2f9d77c4da Binary files /dev/null and b/docs/assets/fields/field-status/concepts/field-masking/masked-fields-explore-view.png differ diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/masked-anomaly-source-records.png b/docs/assets/fields/field-status/managing-field-status/mask-a-field/masked-anomaly-source-records.png deleted file mode 100644 index 21d59ae2c6..0000000000 Binary files a/docs/assets/fields/field-status/managing-field-status/mask-a-field/masked-anomaly-source-records.png and /dev/null differ diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/masked-data-preview.png b/docs/assets/fields/field-status/managing-field-status/mask-a-field/masked-data-preview.png deleted file mode 100644 index 0c347a6de3..0000000000 Binary files a/docs/assets/fields/field-status/managing-field-status/mask-a-field/masked-data-preview.png and /dev/null differ diff --git a/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-1.png b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-1.png new file mode 100644 index 0000000000..9c3b696893 Binary files /dev/null and b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-1.png differ diff --git a/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-2.png b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-2.png new file mode 100644 index 0000000000..de24774b12 Binary files /dev/null and b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-2.png differ diff --git a/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-3.png b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-3.png new file mode 100644 index 0000000000..4700f481bc Binary files /dev/null and b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-3.png differ diff --git a/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-1.png b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-1.png new file mode 100644 index 0000000000..ed840c39f7 Binary files /dev/null and b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-1.png differ diff --git a/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-2.png b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-2.png new file mode 100644 index 0000000000..32a38e1651 Binary files /dev/null and b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-2.png differ diff --git a/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-3.png b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-3.png new file mode 100644 index 0000000000..3b4fe062b0 Binary files /dev/null and b/docs/assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-3.png differ diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-1.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-1.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-1.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-1.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-2.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-2.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-2.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-2.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-3.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-3.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-3.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-3.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-1.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-1.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-1.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-1.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-2.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-2.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-2.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-2.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-3.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-3.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-3.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-3.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-1.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-1.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-1.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-1.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-2.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-2.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-2.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-2.png diff --git a/docs/assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-3.png b/docs/assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-3.png similarity index 100% rename from docs/assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-3.png rename to docs/assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-3.png diff --git a/docs/container/data-preview.md b/docs/container/data-preview.md index c8bf575958..578cf1f0df 100644 --- a/docs/container/data-preview.md +++ b/docs/container/data-preview.md @@ -64,7 +64,7 @@ The **Download Records** feature in Qualytics allows users to easily export all ## Masked Fields in Data Preview -If a container contains [masked fields](../fields/field-status/managing-field-status/mask-a-field.md), their values are obfuscated in the data preview grid by default. +If a container contains [masked fields](../fields/field-status/concepts/field-masking.md), their values are obfuscated in the data preview grid by default. Users with **Editor** permission can reveal all masked values for the current view by clicking the **Show masked values** button that appears in the toolbar when masked fields are present. Every reveal action is recorded in the **masking audit log** with the user identity, timestamp, IP address, and the specific fields accessed — this log is reviewable by Administrators. diff --git a/docs/container/operations/export-operation.md b/docs/container/operations/export-operation.md index 9e968d74bb..c0ad573e7c 100644 --- a/docs/container/operations/export-operation.md +++ b/docs/container/operations/export-operation.md @@ -75,7 +75,7 @@ After clicking **Schedule**, a confirmation message appears stating **"Operation ## Field Masking and Export -If your datastore contains [masked fields](../../fields/field-status/managing-field-status/mask-a-field.md), masking is applied to Field Profile exports before the data is written to the enrichment datastore. +If your datastore contains [masked fields](../../fields/field-status/concepts/field-masking.md), masking is applied to Field Profile exports before the data is written to the enrichment datastore. Specifically, **histogram bucket values** for masked fields are obfuscated in the `__field_profiles_export` output file. This ensures that the statistical distribution of sensitive field values is not exposed in your enrichment datastore. diff --git a/docs/container/operations/materialize-operation.md b/docs/container/operations/materialize-operation.md index 73d45d44f3..184b8daab2 100644 --- a/docs/container/operations/materialize-operation.md +++ b/docs/container/operations/materialize-operation.md @@ -114,7 +114,7 @@ After clicking **Schedule**, a confirmation message appears stating **"Operation ## Field Masking and Materialize -If your datastore contains [masked fields](../../fields/field-status/managing-field-status/mask-a-field.md), masking is applied to the source data before it is written to the enrichment datastore during a Materialize operation. +If your datastore contains [masked fields](../../fields/field-status/concepts/field-masking.md), masking is applied to the source data before it is written to the enrichment datastore during a Materialize operation. Source record values for masked fields are obfuscated in every container snapshot written to the enrichment datastore. This applies to all containers included in the materialize run. diff --git a/docs/datastore-checks/dry-run.md b/docs/datastore-checks/dry-run.md index d0a0db5c62..f6947d0797 100644 --- a/docs/datastore-checks/dry-run.md +++ b/docs/datastore-checks/dry-run.md @@ -44,6 +44,9 @@ Highlights any violations detected during the dry run, such as constraint breach The **Source Records** section presents a detailed, tabular view of all records that were evaluated by the selected quality check. This section is designed to help users investigate the underlying data issues that may have led to anomalies, offering clear visibility into the records that failed to meet the defined constraint. +!!! info "Masked Fields in Dry Runs" + If a container contains [masked fields](../fields/field-status/concepts/field-masking.md), their values are automatically obfuscated in dry run source records. Masking in dry runs is **unconditional** — there is no inline reveal option. This ensures sensitive data is protected even during ad-hoc check testing. To investigate specific masked values, use the [Data Preview](../container/data-preview.md#masked-fields-in-data-preview) or [Anomaly Source Records](../anomalies/source-record.md#masked-fields-in-source-records) reveal controls instead. + ![source-records](../assets/datastore-checks/dry-run/source-records-light.png) ### Sort Options diff --git a/docs/fields/field-status/concepts/field-masking.md b/docs/fields/field-status/concepts/field-masking.md new file mode 100644 index 0000000000..e62f0db28e --- /dev/null +++ b/docs/fields/field-status/concepts/field-masking.md @@ -0,0 +1,197 @@ +# Field Masking + +Field masking is a data protection mechanism that hides the actual values of sensitive fields across the platform while keeping those fields fully operational for quality monitoring. When a field is masked, its values are replaced with the `***MASKED***` placeholder in all surfaces where data is displayed or exported. + +This is useful for fields that contain sensitive data (e.g., PII, financial data, health records) that should be protected but still monitored for quality. + +## Identifying Masked Fields + +Masked fields are visually identified across the platform so you can quickly tell which fields are protected. + +### In the Tree View + +The left-side tree view shows the hierarchy of datastores, containers, and fields. Masked fields display an amber :material-shield-lock-outline: icon next to their name in the tree view, making it easy to spot protected fields while navigating the data hierarchy. + +![masked-field-tree-view](../../../assets/fields/field-status/concepts/field-masking/masked-field-tree-view.png) + +### In the Container Field Listing + +When viewing a container's fields, masked fields are identified by the amber :material-shield-lock-outline: icon next to the field name. Switching to the **Masked** tab filters the listing to show only masked fields in that container. + +![masked-fields-container-listing](../../../assets/fields/field-status/concepts/field-masking/masked-fields-container-listing.png) + +### In the Explore View + +In the Explore view, you can filter fields across all containers by status. Selecting the **Masked** filter shows all masked fields across the platform, helping you get a global view of which fields are protected. + +![masked-fields-explore-view](../../../assets/fields/field-status/concepts/field-masking/masked-fields-explore-view.png) + +### In the Field Detail Page + +When you open a masked field's detail page, the field status is displayed as **Masked** with the amber :material-shield-lock-outline: indicator at the top of the page. + +![masked-field-detail](../../../assets/fields/field-status/concepts/field-masking/masked-field-detail.png) + +## How Masking Works + +Masking only affects **value visibility** — it does not change how the platform processes data. A masked field continues to be profiled, scanned, and evaluated by quality checks exactly like an active field. The only difference is that actual values are hidden by default across the platform. + +When you mask a field: + +- **Quality Checks**: Continue to run normally — masking does not affect quality monitoring +- **Profiling and Scanning**: Continue to run normally — all metrics are computed using actual source data +- **Values**: Replaced with `***MASKED***` across all surfaces listed below + +## Where Masking Is Applied + +Masked values are obfuscated at every point where field values are surfaced or written: + +| Surface | Operation that produces it | Reveal available? | +| :--- | :--- | :--- | +| Data Preview | Container read (live query) | Yes — "Show masked values" button | +| Anomaly Source Records | Scan / Dry Run | Yes — per-anomaly reveal toggle (all records revealed together) | +| Quality Check Dry Runs | Dry Run | No — unconditionally masked | +| Field Profile Histograms (UI) | Profile | Yes — via `include_masked` API parameter | +| Anomaly Assertion Context | Scan / Dry Run | No — unconditionally masked | +| Field Profiles Export | Export Operation | Yes — via `include_masked` API parameter (not available in the UI) | +| Materialized Snapshots | Materialize Operation | Yes — via `include_masked` API parameter (not available in the UI) | + +## Revealing Masked Values + +Users with **Editor** permission or above can temporarily reveal masked values in the surfaces that support inline reveal. + +### Data Preview + +In the container's data preview, masked fields display their values as hidden. A **Show masked values** button allows you to reveal the values for the current view. + +![masked-data-preview](../../../assets/fields/field-status/concepts/field-masking/masked-data-preview.png) + +### Anomaly Source Records + +In anomaly source records, masked field values are replaced with the `***MASKED***` placeholder. You can toggle the visibility of masked values using the reveal control — toggling it reveals all source records attached to that anomaly at once. + +![masked-anomaly-source-records](../../../assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records.png) + +When revealed, the actual values are displayed in place of the placeholder for all source records of that anomaly. + +!!! tip + To permanently restore a field's actual values without needing to reveal them each time, see [Unmask a Field](../managing-field-status/unmask-a-field.md){:target="_blank"}. + +![masked-anomaly-source-records-2](../../../assets/fields/field-status/concepts/field-masking/masked-anomaly-source-records-2.png) + +### Anomaly Assertion Context + +Masked field values that appear in anomaly check details and assertion context are **unconditionally masked**. There is no inline reveal for this surface — this is by design to ensure that sensitive values are never inadvertently exposed when reviewing anomaly descriptions or sharing anomaly details. + +### Export and Materialize Outputs + +Histogram bucket values in exported field profile files and source record values in materialized container snapshots are written with masking applied by default. To obtain revealed data in these outputs, pass `include_masked=true` when triggering the operation via the API. This parameter is not available in the UI. + +## Masking Audit Log + +Every time masked values are revealed, the action is recorded in the masking audit log for security and compliance purposes. The audit log captures the user identity, timestamp, IP address, fields accessed, and the resource where the reveal occurred. + +For step-by-step instructions on accessing and using the audit log, see [Masking Audit Log](../managing-field-status/masking-audit-log.md){:target="_blank"}. + +## Unmasking a Field + +Unmasking a field restores its actual values across the platform, making them visible without requiring explicit reveal actions. + +### What Happens When a Field is Unmasked? + +When you unmask a field, its status changes from **Masked** back to **Active**. The `***MASKED***` placeholder is removed and actual values become visible across every surface where they were previously hidden. + +#### Platform Behavior After Unmasking + +| Surface | Behavior after unmasking | +| :--- | :--- | +| **Data Preview** | Values are shown directly — no reveal action required | +| **Anomaly Source Records** | Values are visible by default — no reveal toggle needed | +| **Quality Check Dry Runs** | Values are visible in dry run source records | +| **Field Profile Histograms** | Chart values are shown normally in the UI | +| **Anomaly Assertion Context** | Values are visible in anomaly check details and descriptions | +| **Export and Materialize outputs** | Future runs write actual values — previously written files are **not** retroactively updated | + +!!! warning + Unmasking is immediate and affects all users with access to the container. There is no per-user unmasking — once a field is unmasked, its values are visible to everyone. + +#### Audit Log Considerations + +When a field is unmasked, no further reveal actions are recorded in the [masking audit log](../managing-field-status/masking-audit-log.md){:target="_blank"} because there is nothing to reveal — the values are always visible. If your compliance requirements depend on tracking who accesses sensitive values, unmasking removes that layer of visibility. + +### When to Unmask a Field + +Unmasking is appropriate when the protection provided by masking is no longer necessary or is actively hindering your workflow. Consider unmasking when: + +#### The data is no longer sensitive + +- The field was reclassified and no longer contains PII, financial data, or protected health information. +- The data has been anonymized or aggregated at the source, so the values in the platform are no longer identifiable. +- A retention policy has expired and the data is now in the public domain or approved for open access. + +#### Masking is blocking a legitimate workflow + +- Your team needs to review actual values during an incident investigation or root cause analysis and the inline reveal is too cumbersome for the volume of data involved. +- You need to share anomaly details (including assertion context) with stakeholders who require the actual values — assertion context is unconditionally masked and has no inline reveal. +- Export or materialize operations must include actual values and you prefer not to rely on the `include_masked` API parameter for every run. + +#### The field was masked by mistake + +- The wrong field was masked during a bulk masking operation. +- A field was masked during initial setup but later determined to not contain sensitive data. + +### When Not to Unmask a Field + +Keep a field masked when the protection it provides is still valuable. Avoid unmasking when: + +#### The data is still sensitive + +- The field contains PII (names, emails, phone numbers, national IDs), financial data (account numbers, balances), or protected health information that is subject to regulatory requirements (GDPR, HIPAA, LGPD, etc.). +- Your organization's data governance policy classifies the field as restricted or confidential. + +#### Compliance requires access tracking + +- Your compliance framework requires an audit trail of who accessed sensitive values and when. Masking with reveal logging provides this — unmasking does not. +- You are in a regulated industry where demonstrating controlled access to sensitive data is part of your audit obligations. + +#### Broad access to the container exists + +- Many users have access to the container, and not all of them should see the actual values. Masking ensures that only users who explicitly reveal values (and have Editor permission or above) can see them. +- The container is used in shared dashboards, reports, or integrations where masked values prevent accidental exposure. + +#### Temporary access is sufficient + +- If you only need to see actual values occasionally (e.g., during anomaly investigation), use the **inline reveal** in Data Preview or Anomaly Source Records instead of unmasking. This keeps the audit trail intact and the field protected for all other users. +- For export or materialize operations, use the `include_masked` API parameter to include actual values on a per-run basis rather than permanently unmasking. + +### Unmasking Best Practices + +1. **Prefer reveal over unmask** — Use the inline reveal controls in Data Preview and Anomaly Source Records for temporary access. This keeps the audit trail and protects the field for other users. + +2. **Check with your data governance team** — Before unmasking, confirm that the field's sensitivity classification has changed or that unmasking is approved under your organization's data policy. + +3. **Re-mask after temporary needs** — If you unmask a field for a specific task (e.g., an investigation), re-mask it as soon as the task is complete. This minimizes the window of exposure. + +4. **Review container access** — Before unmasking, check who has access to the container. Unmasking exposes values to all users with container access, not just the person performing the unmask. + +5. **Document the decision** — Record why a field was unmasked, who approved it, and when. This is especially important in regulated environments where you may need to justify the decision during an audit. + +6. **Consider export implications** — Remember that previously written export and materialize files are not retroactively updated. If you unmask a field, only future runs will include actual values. + +To learn how to unmask a field step by step, see [Unmask a Field](../managing-field-status/unmask-a-field.md){:target="_blank"}. + +## Fields That Cannot Be Masked + +The following fields cannot be masked: + +| Field Type | Reason | +| :--- | :--- | +| **Excluded** fields | Only active fields can be masked. Restore the field to active status first. | +| **Missing** fields | System-managed status — cannot be changed directly. | +| **Already masked** fields | The field is already masked. | +| **Incremental identifier** fields | Used to track scan boundaries for incremental processing. Masking would make incremental scans indistinguishable. | +| **Partition** fields | Used for data partitioning and segmentation. Masking would hide partition boundaries. | +| **Group-by** fields | Used for segment grouping. Masking would make segment identifiers indistinguishable. | + +!!! info + If you need to mask a field that is currently configured as a container identifier (incremental, partition, or group-by), you must first remove the identifier assignment from the container settings, and then mask the field. diff --git a/docs/fields/field-status/concepts/field-status-faq.md b/docs/fields/field-status/concepts/field-status-faq.md index 7dea751199..c21d410232 100644 --- a/docs/fields/field-status/concepts/field-status-faq.md +++ b/docs/fields/field-status/concepts/field-status-faq.md @@ -42,6 +42,7 @@ Masked field values are automatically hidden at every stage of data processing: - **Data Preview** — values are obfuscated; users with Editor permission can reveal them with the "Show masked values" button - **Anomaly Source Records** — values are obfuscated by default; users with Editor permission can reveal them per anomaly (all source records for the anomaly are revealed together) +- **Quality Check Dry Runs** — source record values are unconditionally obfuscated; there is no inline reveal - **Field Profile Histograms** — chart values are obfuscated for masked fields - **Anomaly Assertion Context** — values in anomaly check details are unconditionally obfuscated; there is no inline reveal - **Export Operation (Field Profiles)** — histogram bucket values are obfuscated in the `_field_profiles_export` file written to the enrichment datastore diff --git a/docs/fields/field-status/managing-field-status/mask-a-field.md b/docs/fields/field-status/managing-field-status/mask-a-field.md index bf2d7f7684..28dedd9faa 100644 --- a/docs/fields/field-status/managing-field-status/mask-a-field.md +++ b/docs/fields/field-status/managing-field-status/mask-a-field.md @@ -4,34 +4,8 @@ Masking a field hides its actual values across the platform while keeping the fi Only fields with **Active** status can be masked. This includes both regular fields and computed fields. -## What Happens When a Field is Masked? - -When you mask a field, its actual values are obfuscated everywhere they would otherwise appear. Quality checks, profiling, and scanning all continue to run normally — masking only affects value visibility. - -Specifically, obfuscated values appear in the following places: - -- **Data Preview**: Values are hidden in the container preview — users must explicitly reveal them -- **Anomaly Source Records**: Values are hidden by default — users can toggle reveal per anomaly (all source records for the anomaly are revealed together) -- **Field Profile Histograms**: Chart values are obfuscated for masked fields in the field profile view -- **Anomaly Assertion Context**: The values embedded in anomaly check details are unconditionally obfuscated — they cannot be revealed inline -- **Export Operation (Field Profiles)**: Histogram bucket values for masked fields are obfuscated in the `_field_profiles_export` file written to the enrichment datastore -- **Materialize Operation**: Source record values for masked fields are obfuscated in materialized container snapshots written to the enrichment datastore -- **Quality Checks**: Continue to run normally — masking does not affect quality monitoring -- **Profiling and Scanning**: Continue to run normally - -!!! info - Users with **Editor** permission or above can request to reveal values in the surfaces that support reveal (Data Preview and Anomaly Source Records). Every reveal action is **audit-logged** for security and compliance purposes. - -## Where Masking Is Applied - -| Surface | Operation that produces it | Reveal available? | -| :--- | :--- | :--- | -| Data Preview | Container read (live query) | Yes — "Show masked values" button | -| Anomaly Source Records | Scan / Dry Run | Yes — per-anomaly reveal toggle (all records revealed together) | -| Field Profile Histograms (UI) | Profile | Yes — via `include_masked` API parameter | -| Anomaly Assertion Context | Scan / Dry Run | No — unconditionally masked | -| Field Profiles Export | Export Operation | Yes — via `include_masked` API parameter (not available in the UI) | -| Materialized Snapshots | Materialize Operation | Yes — via `include_masked` API parameter (not available in the UI) | +!!! tip + For a detailed explanation of how masking works, where it is applied, audit logging, and which fields cannot be masked, see [Field Masking](../concepts/field-masking.md){:target="_blank"}. ## Mask a Field from the Container View @@ -82,102 +56,3 @@ You can mask multiple fields at once from the container's field listing. 4. Confirm the bulk masking in the dialog. ![bulk-mask-field-3](../../../assets/fields/field-status/managing-field-status/mask-a-field/bulk-mask-field-3.png) - -## Unmask a Field - -Unmasking a field restores its actual values across the platform, making them visible without requiring explicit reveal actions. - -### What Happens When a Field is Unmasked? - -When you unmask a field: - -- **Data Preview**: Values are shown directly without a reveal action -- **Anomaly Source Records**: Values are visible by default -- **Field Profile Histograms**: Chart values are shown normally -- **Anomaly Assertion Context**: Values are visible in anomaly check details -- **Future Export and Materialize runs**: Values will no longer be masked in new export and materialize outputs — previously written enrichment datastore files are not retroactively updated - -### Unmask from the Container View - -1. Navigate to the container's field listing. -2. Click the **Masked** tab to view masked fields. -3. Locate the field you want to unmask. -4. Click the vertical ellipsis menu (**⋮**) on the field row. - - ![row-unmask-field-1](../../../assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-1.png) - -5. Click the **Unmask** option from the menu. - - ![row-unmask-field-2](../../../assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-2.png) - -6. Confirm the unmasking in the dialog. - - ![row-unmask-field-3](../../../assets/fields/field-status/managing-field-status/mask-a-field/row-unmask-field-3.png) - -### Unmask from the Field View - -1. Navigate to the field's detail page by clicking on the field name in the container's field listing. -2. Click the settings icon (gear icon) in the top-right corner of the field page. - - ![field-context-unmask-field-1](../../../assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-1.png) - -3. Click the **Unmask** option from the dropdown menu. - - ![field-context-unmask-field-2](../../../assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-2.png) - -4. Confirm the unmasking in the dialog. - - ![field-context-unmask-field-3](../../../assets/fields/field-status/managing-field-status/mask-a-field/field-context-unmask-field-3.png) - -### Bulk Unmask - -You can unmask multiple fields at once from the container's field listing. - -1. Navigate to the container's field listing. -2. Click the **Masked** tab to view masked fields. -3. Select the fields you want to unmask by clicking the checkbox on each field row. - - ![bulk-unmask-field-1](../../../assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-1.png) - -4. Click the **Unmask** action in the selection toolbar that appears at the top. - - ![bulk-unmask-field-2](../../../assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-2.png) - -5. Confirm the bulk unmasking in the dialog. - - ![bulk-unmask-field-3](../../../assets/fields/field-status/managing-field-status/mask-a-field/bulk-unmask-field-3.png) - -## Revealing Masked Values - -Users with **Editor** permission can temporarily reveal masked values in the surfaces that support inline reveal. Not every surface supports reveal — see the [Where Masking Is Applied](#where-masking-is-applied) table above. - -### Data Preview - -In the container's data preview, masked fields display their values as hidden (`••••••••`). A **Show masked values** button allows you to reveal the values for the current view. - -![masked-data-preview](../../../assets/fields/field-status/managing-field-status/mask-a-field/masked-data-preview.png) - -### Anomaly Source Records - -In anomaly source records, masked field values are hidden by default. You can toggle the visibility of masked values using the reveal control — toggling it reveals all source records attached to that anomaly at once. - -![masked-anomaly-source-records](../../../assets/fields/field-status/managing-field-status/mask-a-field/masked-anomaly-source-records.png) - -### Anomaly Assertion Context - -Masked field values that appear in anomaly check details and assertion context are **unconditionally masked**. There is no inline reveal for this surface — this is by design to ensure that sensitive values are never inadvertently exposed when reviewing anomaly descriptions or sharing anomaly details. - -### Export and Materialize Outputs - -Histogram bucket values in exported field profile files and source record values in materialized container snapshots are written with masking applied by default. To obtain revealed data in these outputs, pass `include_masked=true` when triggering the operation via the API. This parameter is not available in the UI. - -!!! warning - Every time masked values are revealed, the action is recorded in the **masking audit log** with the user identity, timestamp, IP address, and the specific fields and resources accessed. Administrators can review these logs from the masking audit log page. - -!!! note - The following fields cannot be masked: - - - **Excluded** fields - - **Missing** fields - - **Already masked** fields - - **Container identifiers** — fields configured as the incremental field or partition field, which the platform uses to organize and track data processing diff --git a/docs/fields/field-status/managing-field-status/masking-audit-log.md b/docs/fields/field-status/managing-field-status/masking-audit-log.md new file mode 100644 index 0000000000..ab3a341fd1 --- /dev/null +++ b/docs/fields/field-status/managing-field-status/masking-audit-log.md @@ -0,0 +1,84 @@ +# Audit Masked Field Access + +The masking audit log records every instance where masked field values are revealed. Use it to track who accessed sensitive data, when, and from which surface. + +!!! tip + For background on how masking works and where values are obfuscated, see [Field Masking](../concepts/field-masking.md){:target="_blank"}. + +## Who Can Access the Audit Log? + +Only users with the **Admin** user role can view the masking audit log. + +## What Is Recorded? + +Every time a user reveals masked values — whether through the UI or the API — an entry is created with the following details: + +| Detail | Description | +| :--- | :--- | +| **User** | The name and email of the user who performed the reveal | +| **Action** | The type of action performed (e.g., revealed masked field values) | +| **Timestamp** | The date and time of the reveal action | +| **IP Address** | The IP address of the client that performed the reveal | +| **Fields Accessed** | The list of masked field names whose values were revealed | +| **Resource** | The resource type and ID where the reveal was performed (e.g., container, anomaly) | + +## Actions That Generate Audit Entries + +| Surface | Action | Audit entry created? | +| :--- | :--- | :--- | +| **Data Preview** | Clicking **Show masked values** | Yes | +| **Anomaly Source Records** | Toggling the reveal control | Yes | +| **Field Profile Histograms** | Using `include_masked` API parameter | Yes | +| **Export Operation** | Using `include_masked` API parameter | Yes | +| **Materialize Operation** | Using `include_masked` API parameter | Yes | +| **Quality Check Dry Runs** | N/A — unconditionally masked, no reveal available | No | +| **Anomaly Assertion Context** | N/A — unconditionally masked, no reveal available | No | + +!!! note + Unmasking a field (changing its status from Masked to Active) is a different action — it does not generate reveal entries because the values become permanently visible. For more on unmasking implications, see [Field Masking — Unmasking a Field](../concepts/field-masking.md#unmasking-a-field){:target="_blank"}. + +## Accessing the Audit Log from the UI + +You can open the masking audit log from the surfaces where reveal is available. + +### From Data Preview + +1. Select a container and click the **Data Preview** tab. + + ![audit-log-data-preview-1](../../../assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-1.png) + +2. Click the **Show audit log :material-history:** button next to the reveal control. + + ![audit-log-data-preview-2](../../../assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-2.png) + +3. The audit log side panel opens, showing a list of reveal events for this container. + + ![audit-log-data-preview-3](../../../assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-data-preview-3.png) + +### From Anomaly Source Records + +1. Navigate to the Anomaly Overview page and scroll down to the **Source Records** section. + + ![audit-log-anomaly-source-records-1](../../../assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-1.png) + +2. Click the **Show audit log :material-history:** button next to the reveal control. + + ![audit-log-anomaly-source-records-2](../../../assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-2.png) + +3. The audit log side panel opens, showing a list of reveal events for this anomaly. + + ![audit-log-anomaly-source-records-3](../../../assets/fields/field-status/managing-field-status/masking-audit-log/audit-log-anomaly-source-records-3.png) + +### The Audit Log Side Panel + +The audit log side panel displays a chronological list of reveal events. Each entry shows the user who performed the reveal. Clicking on an entry expands it to show the full details, including the action, timestamp, IP address, fields accessed, and the resource where the reveal occurred. + +## Accessing the Audit Log via API + +Administrators can also query the audit log programmatically. See the [Masking Audit Log API](../concepts/field-status-api.md#masking-audit-log){:target="_blank"} for the endpoint, query parameters, and example requests. + +## Related + +- [Field Masking](../concepts/field-masking.md){:target="_blank"} — How masking works and where it is applied. +- [Field Masking — Unmasking a Field](../concepts/field-masking.md#unmasking-a-field){:target="_blank"} — What happens when a field is unmasked and best practices. +- [Field Status API — Masking Audit Log](../concepts/field-status-api.md#masking-audit-log){:target="_blank"} — API endpoint for querying audit log entries. diff --git a/docs/fields/field-status/managing-field-status/restore-a-field.md b/docs/fields/field-status/managing-field-status/restore-a-field.md index b24994cc41..6ccd9b62ef 100644 --- a/docs/fields/field-status/managing-field-status/restore-a-field.md +++ b/docs/fields/field-status/managing-field-status/restore-a-field.md @@ -3,7 +3,7 @@ Restoring a field changes its status back to **Active**, making it available for profiling and scanning operations again. Only fields with **Excluded** status can be restored. !!! tip - To change a **Masked** field back to **Active**, use the [Unmask](mask-a-field.md#unmask-a-field){:target="_blank"} operation instead. + To change a **Masked** field back to **Active**, use the [Unmask](unmask-a-field.md){:target="_blank"} operation instead. ## What Happens When a Field is Restored? diff --git a/docs/fields/field-status/managing-field-status/unmask-a-field.md b/docs/fields/field-status/managing-field-status/unmask-a-field.md new file mode 100644 index 0000000000..5d43826cb3 --- /dev/null +++ b/docs/fields/field-status/managing-field-status/unmask-a-field.md @@ -0,0 +1,56 @@ +# Unmask a Field + +Unmasking a field restores its actual values across the platform, making them visible without requiring explicit reveal actions. + +!!! tip + For a detailed explanation of what happens when a field is unmasked, platform behavior changes, and best practices, see [Field Masking — Unmasking a Field](../concepts/field-masking.md#unmasking-a-field){:target="_blank"}. + +## Unmask from the Container View + +1. Navigate to the container's field listing. +2. Click the **Masked** tab to view masked fields. +3. Locate the field you want to unmask. +4. Click the **vertical ellipsis :material-dots-vertical:** on the field row. + + ![row-unmask-field-1](../../../assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-1.png) + +5. Click the **Unmask :material-shield-off-outline:** option from the menu. + + ![row-unmask-field-2](../../../assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-2.png) + +6. Click the **Unmask** button to confirm the unmasking in the dialog. + + ![row-unmask-field-3](../../../assets/fields/field-status/managing-field-status/unmask-a-field/row-unmask-field-3.png) + +## Unmask from the Field View + +1. Navigate to the field's detail page by clicking on the field name in the container's field listing. +2. Click the **:material-cog:** button in the top-right corner of the field page. + + ![field-context-unmask-field-1](../../../assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-1.png) + +3. Click the **Unmask :material-shield-off-outline:** option from the dropdown menu. + + ![field-context-unmask-field-2](../../../assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-2.png) + +4. Click the **Unmask** button to confirm the unmasking in the dialog. + + ![field-context-unmask-field-3](../../../assets/fields/field-status/managing-field-status/unmask-a-field/field-context-unmask-field-3.png) + +## Bulk Unmask + +You can unmask multiple fields at once from the container's field listing. + +1. Navigate to the container's field listing. +2. Click the **Masked** tab to view masked fields. +3. Select the fields you want to unmask by clicking the checkbox on each field row. + + ![bulk-unmask-field-1](../../../assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-1.png) + +4. Click the **Unmask :material-shield-off-outline:** action in the selection toolbar that appears at the top. + + ![bulk-unmask-field-2](../../../assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-2.png) + +5. Click the **Unmask** button to confirm the bulk unmasking in the dialog. + + ![bulk-unmask-field-3](../../../assets/fields/field-status/managing-field-status/unmask-a-field/bulk-unmask-field-3.png) diff --git a/docs/fields/field-status/overview.md b/docs/fields/field-status/overview.md index 6da1a4de8a..358c07e3d4 100644 --- a/docs/fields/field-status/overview.md +++ b/docs/fields/field-status/overview.md @@ -29,6 +29,7 @@ With Field Status, you can: | [Understanding Field Status](concepts/understanding-field-status.md) | Learn how field status works, its role in data quality, practical scenarios, and best practices. | | [Status Types](concepts/field-status-types.md) | Detailed reference for all four statuses, how they are assigned, and visual indicators. | | [Status Lifecycle](concepts/field-status-lifecycle.md) | Status transition diagram and details on what triggers each transition. | +| [Field Masking](concepts/field-masking.md) | How masking works, where it is applied, revealing values, unmasking behavior, audit logging, and field restrictions. | | [Merge Fields](concepts/merge-fields.md) | How to combine a missing field with an active field after a column rename, preserving all history. | ## Managing Field Status @@ -37,6 +38,8 @@ With Field Status, you can: | :--- | :--- | | [Filtering by Status](managing-field-status/filtering-by-status.md) | Use status tabs to find specific fields in Explore and Container views. | | [Mask a Field](managing-field-status/mask-a-field.md) | Protect sensitive field values while maintaining quality monitoring. | +| [Unmask a Field](managing-field-status/unmask-a-field.md) | Restore masked field values to full visibility across the platform. | +| [Masking Audit Log](managing-field-status/masking-audit-log.md) | View and filter the audit trail of masked value reveal actions. | | [Exclude a Field](managing-field-status/exclude-a-field.md) | Remove a field from quality monitoring, including computed field cascade behavior. | | [Restore a Field](managing-field-status/restore-a-field.md) | Bring an excluded field back to active monitoring. | | [Delete a Field](managing-field-status/delete-a-field.md) | Permanently remove a missing or computed field. | diff --git a/docs/glossary.md b/docs/glossary.md index 026ea71b05..1b15cb3fcd 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -116,7 +116,7 @@ After Compute Daemon gathers all the metadata generated by a profiling operation ### __Masked Field__ -A field that remains fully operational (profiled, scanned, and quality-checked) but whose actual values are hidden across the platform by default. Users with Editor permission can reveal masked values, and every access is recorded in the masking audit log. Learn more in [Mask a Field](fields/field-status/managing-field-status/mask-a-field.md). +A field that remains fully operational (profiled, scanned, and quality-checked) but whose actual values are hidden across the platform by default. Users with Editor permission can reveal masked values, and every access is recorded in the masking audit log. Learn more in [Field Masking](fields/field-status/concepts/field-masking.md). ### __Merge Fields__ diff --git a/docs/source-datastore/operations/profile.md b/docs/source-datastore/operations/profile.md index 2cab5b5755..3fcb6fe9cb 100644 --- a/docs/source-datastore/operations/profile.md +++ b/docs/source-datastore/operations/profile.md @@ -328,6 +328,18 @@ To define a custom schedule, enter the appropriate Cron expression in the **Cust !!! note You will receive a notification when the profile operation is completed. +## Field Masking and Profiling + +Profiling runs normally on [masked fields](../../fields/field-status/concepts/field-masking.md) — masking does not affect the collection of statistical metadata. The platform continues to compute all profile metrics (distinct values, min/max, mean, standard deviation, completeness, etc.) using the actual source data. + +However, when profile results are displayed or exported, masked field values are obfuscated: + +- **Field Profile Histograms** in the UI show obfuscated bucket values for masked fields +- **Exported Field Profiles** (`_field_profiles_export`) written to the enrichment datastore contain obfuscated histogram values for masked fields + +!!! info + To obtain revealed histogram data in exported field profiles, pass `include_masked=true` when triggering the Export operation via the API. This parameter is not available in the UI. See [Field Masking and Export](../../container/operations/export-operation.md#field-masking-and-export) for details. + ## Operation Insights When the profile operation is completed, you will receive the notification and can navigate to the Activity tab for the datastore on which you triggered the Profile Operation and learn about the operation results. diff --git a/docs/source-datastore/operations/scan.md b/docs/source-datastore/operations/scan.md index 24713278ad..08f214c6b3 100644 --- a/docs/source-datastore/operations/scan.md +++ b/docs/source-datastore/operations/scan.md @@ -151,6 +151,18 @@ If you need to **download more records**, increase this value **before running t ![record-limit](../../assets/source-datastores/scan/record-limit.png) +## Field Masking and Scanning + +Scan operations run normally on [masked fields](../../fields/field-status/concepts/field-masking.md) — masking does not affect anomaly detection or quality check execution. The platform evaluates all checks using the actual source data. + +However, when scan results are displayed, masked field values are obfuscated in the following surfaces: + +- **Anomaly Source Records** — values are hidden by default; users with Editor permission can reveal them per anomaly +- **Anomaly Assertion Context** — values embedded in anomaly check details are unconditionally obfuscated +- **Enrichment Datastore** — source record values written during a [Materialize operation](../../container/operations/materialize-operation.md#field-masking-and-materialize) are obfuscated for masked fields + +For more details, see [Masked Fields in Source Records](../../anomalies/source-record.md#masked-fields-in-source-records). + ## Run Instantly Click on the **Run Now** button to perform the scan operation immediately. diff --git a/mkdocs.yml b/mkdocs.yml index d7c1bae219..2cb6ae78aa 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -134,10 +134,13 @@ nav: - Introduction: fields/field-status/concepts/understanding-field-status.md - Status Types: fields/field-status/concepts/field-status-types.md - Lifecycle: fields/field-status/concepts/field-status-lifecycle.md + - Field Masking: fields/field-status/concepts/field-masking.md - Merge Fields: fields/field-status/concepts/merge-fields.md - Managing: - Filtering by Status: fields/field-status/managing-field-status/filtering-by-status.md - Mask a Field: fields/field-status/managing-field-status/mask-a-field.md + - Unmask a Field: fields/field-status/managing-field-status/unmask-a-field.md + - Audit Log: fields/field-status/managing-field-status/masking-audit-log.md - Exclude a Field: fields/field-status/managing-field-status/exclude-a-field.md - Restore a Field: fields/field-status/managing-field-status/restore-a-field.md - Delete a Field: fields/field-status/managing-field-status/delete-a-field.md