Merge pull request #1072 from Qualytics/qua-1518-update-the-field-status-structure-files

QUA-1518: Update the Field Status structure files

Author: RafaelOsiro

docs/fields/computed-fields/computed-fields-details.md

@@ -67,7 +67,7 @@ Deleting a computed field permanently removes both the **transformation definiti
 2. Click the **Delete** button in the top-right corner of the field page.
 3. Confirm the deletion in the dialog.
 
-### What Happens When a Computed Field is Deleted
+### What Happens When a Computed Field is Deleted?
 
 - The **transformation definition** is permanently removed
 - The **output field** is permanently removed

docs/fields/computed-fields/transformation-types.md

@@ -4,7 +4,7 @@
 
 The **Cast** transformation converts field values to a specified target data type using Spark SQL casting rules. This is commonly used when a field is ingested as a string but must behave as a numeric, date, timestamp, or boolean value for profiling, checks, or downstream computations.
 
-### When to Use Cast
+### When to Use Cast?
 
 Use **Cast** when:
 

docs/fields/field-profiles.md

@@ -2,7 +2,7 @@
 
 A Field Profile provides a detailed breakdown of a field’s data after a profiling operation. It helps you understand the structure, quality, and distribution of values for each field inside a container.
 
-## What Field Profiles Are Used For
+## What Field Profiles Are Used For?
 
 Field Profiles help you:
 
@@ -13,7 +13,7 @@ Field Profiles help you:
 
 These insights make it easier to detect data issues early and understand how a field behaves over time.
 
-## How Field Profiles Are Generated
+## How Field Profiles Are Generated?
 
 Field Profiles are automatically created when you run a [**Profile**](../source-datastore/profile.md){target="_blank"} operation on a container:
 

docs/fields/field-status/concepts/field-status-faq.md

@@ -30,7 +30,7 @@ Only **missing** fields and **computed fields** can be permanently deleted. Addi
 
 #### How do I know if a field has changed status?
 
-Fields display visual status indicators in the field listing. Masked fields show an amber shield icon, missing fields show a warning icon (yellow/orange), and excluded fields show a negative icon (red). You can also use the [status tabs](../managing-field-status/filtering-by-status.md) to filter and view fields by their current status.
+Fields display visual status indicators in the field listing. Masked fields show an amber shield icon, missing fields show a warning icon (yellow/orange), and excluded fields show a negative icon (red). You can also use the [status tabs](../managing-field-status/filtering-by-status.md){:target="_blank"} to filter and view fields by their current status.
 
 #### Will a missing field affect my data quality score?
 
@@ -42,7 +42,7 @@ The field is automatically restored to **Active** status during the next profile
 
 #### Can I manually restore a missing field?
 
-No, a missing field cannot be manually restored. It is automatically restored to **Active** when the field reappears in the source data during a subsequent profile operation. If the field will not reappear, you can use the [merge](merge-fields.md) operation if the field was renamed.
+No, a missing field cannot be manually restored — the platform will reject the request. It is automatically restored to **Active** when the field reappears in the source data during a subsequent profile operation. If the field will not reappear, you can use the [merge](merge-fields.md){:target="_blank"} operation if the field was renamed.
 
 #### Does excluding a field affect computed fields that depend on it?
 
@@ -58,7 +58,7 @@ No, the Missing status is automatically assigned by the system during profiling
 
 #### What happens when I merge two fields?
 
-The source field (old, with history) adopts the target field's name, and the target field record is removed. All historical field profiles, anomalies, and quality checks from both fields are preserved under the merged field. The merged field is set to **Active** status. For more details, see [Merge Fields](merge-fields.md).
+The source field (old, with history) adopts the target field's name, and the target field record is removed. All historical field profiles, anomalies, and quality checks from both fields are preserved under the merged field. The merged field is set to **Active** status. For more details, see [Merge Fields](merge-fields.md){:target="_blank"}.
 
 #### What happens to container identifiers when a field goes missing?
 
@@ -101,12 +101,16 @@ No. Masked field values that appear in anomaly check details and assertion conte
 
 #### Can I manage field status via API?
 
-Yes. All field status operations — masking, excluding, restoring, deleting, and merging — are available through the Qualytics API. You can also perform bulk operations to update multiple fields at once. See the [Field Status API](field-status-api.md) page for endpoints, parameters, and examples.
+Yes. All field status operations — masking, excluding, restoring, deleting, and merging — are available through the Qualytics API. You can also perform bulk operations to update multiple fields at once. See the [Field Status API](field-status-api.md){:target="_blank"} page for endpoints, parameters, and examples.
 
 #### Can I mask or exclude multiple fields at once?
 
-Yes. The platform supports bulk operations for updating, excluding, and restoring fields. You can change the status of multiple fields in a single API request using the bulk endpoints. See the [Field Status API](field-status-api.md) for details.
+Yes. The platform supports bulk operations for masking, unmasking, excluding, restoring, and deleting fields. You can perform bulk actions directly from the container's field listing by selecting multiple fields and choosing the desired action from the toolbar. Bulk operations are also available via the API — see the [Field Status API](field-status-api.md){:target="_blank"} for details.
 
 #### Can I rename a field that is excluded or missing?
 
 No. Only **active** and **masked** fields can be renamed. Excluded and missing fields must first be restored to active status before they can be renamed.
+
+#### Can I mask an excluded or missing field directly?
+
+No. Only **active** fields can be masked. If a field is excluded, you must first restore it to active status and then mask it. Missing fields cannot be masked — they must reappear in the source data first.

docs/fields/field-status/concepts/field-status-lifecycle.md

@@ -29,11 +29,13 @@ flowchart LR
     Start(( )) -- Field discovered during first profile --> Active
 
     Active -- Field not found in profile results --> Missing
+    Masked -- Field not found in profile results --> Missing
     Missing -- Field reappears in profile results --> Active
 
     Missing -- Permanently delete --> End(( ))
 
     style Active fill:#4CAF50,color:#fff,stroke:#388E3C
+    style Masked fill:#FFA726,color:#fff,stroke:#F57C00
     style Missing fill:#FFEE58,color:#000,stroke:#F9A825
     style Start fill:#333,stroke:#333
     style End fill:#333,stroke:#333
@@ -46,6 +48,7 @@ flowchart LR
 | **New → Active** | Field discovered during initial profile | Yes | None — this is the default state |
 | **Active → Masked** | User masks the field | No | Values hidden across the platform; no impact on quality checks |
 | **Active → Missing** | Field not found in subsequent profile | Yes | Dependent computed fields also marked as Missing; container identifiers referencing this field are cleared |
+| **Masked → Missing** | Field not found in subsequent profile | Yes | Same as Active → Missing |
 | **Active → Excluded** | User manually excludes the field | No | Quality checks archived (except Expected Schema); dependent computed fields excluded recursively |
 | **Masked → Active** | User unmasks the field | No | Values become visible again across the platform |
 | **Masked → Excluded** | User manually excludes the field | No | Same as Active → Excluded |

docs/fields/field-status/concepts/field-status-types.md

@@ -2,7 +2,7 @@
 
 Qualytics assigns one of four statuses to each field:
 
-| Status | Assigned By | Description |
+| Status | Assignment | Description |
 | :--- | :--- | :--- |
 | **Active** | Automatic | The field is fully operational. It will be included in profiling (collecting metadata and statistics), scan operations (detecting anomalies), and quality checks will run against it. This is the default status assigned by the system when a field is first discovered during a profile operation. |
 | **Masked** | Manual | The field is fully operational (profiled, scanned, quality-checked) but its actual values are hidden by default. Users with Editor permission can request to view the real values, and every reveal action is audit-logged. |
@@ -12,7 +12,7 @@ Qualytics assigns one of four statuses to each field:
 !!! info
     **Active** and **Masked** are both considered **operational** statuses. Everything you can do with an active field, you can also do with a masked field — the only difference is that masked field values are hidden by default.
 
-## How Statuses Are Assigned
+## How Statuses Are Assigned?
 
 ### Active (Automatic)
 
@@ -49,7 +49,7 @@ Key behaviors for missing fields:
 
 - **Automatic restoration**: If the field reappears in a subsequent profile, it is automatically restored to **Active**
 - **No check archival**: Quality checks are not archived for missing fields (unlike excluded fields), since the status is expected to be transient
-- **Cascading**: If a source field goes missing, any computed fields that depend on it are also marked as **Missing**
+- **Cascading**: When a profile marks a source field as Missing, any computed fields that depend on it are also marked as **Missing** within that same run
 
 ### Excluded (Manual)
 
@@ -70,9 +70,9 @@ Fields display visual status indicators to help you quickly identify their state
 
 | Ref. | Status | Color | Icon |
 | :--- | :--- | :--- | :--- |
-| 1 | **Active** | Default (neutral) | Standard field type icon |
+| 1 | **Active** | Default (neutral) | Check circle icon (`mdi-check-circle`) |
 | 2 | **Masked** | Amber/Warning | Shield lock icon (`mdi-shield-lock-outline`) |
-| 3 | **Missing** | Warning (yellow/orange) | Alert icon with "Missing field" tooltip |
-| 4 | **Excluded** | Negative (red) | Alert icon with "Excluded field" tooltip |
+| 3 | **Missing** | Warning (yellow/orange) | Alert circle icon (`mdi-alert-circle`) with "Missing field" tooltip |
+| 4 | **Excluded** | Negative (red) | Eye-off icon (`mdi-eye-off`) with "Excluded field" tooltip |
 
 ![status-indicators](../../../assets/fields/field-status/concepts/field-status-types/status-indicators.png)

docs/fields/field-status/concepts/merge-fields.md

@@ -2,7 +2,7 @@
 
 When a column is renamed in the source data, the next profile operation creates a new **Active** field with the updated name and marks the original field as **Missing**. The merge operation combines these two fields into one, preserving all historical data under the new column name.
 
-## Why Merge
+## Why Merge?
 
 Without merging, a column rename results in:
 
@@ -11,7 +11,7 @@ Without merging, a column rename results in:
 
 Merging transfers the historical context from the old field to the new one, ensuring continuity in your quality monitoring.
 
-## What Happens During a Merge
+## What Happens During a Merge?
 
 When you merge a **source field** (the old field with history) into a **target field** (the new field with the desired name):
 
@@ -23,7 +23,7 @@ When you merge a **source field** (the old field with history) into a **target f
 | 4 | All historical field profiles are renamed to match the new field name |
 | 5 | Quality checks from the target field are reassociated to the source field |
 
-## What Is Preserved
+## What Is Preserved?
 
 | Asset | Behavior |
 | :--- | :--- |

docs/fields/field-status/concepts/understanding-field-status.md

@@ -11,11 +11,11 @@ In any data quality platform, not every field in a dataset requires the same lev
 Rather than treating all fields equally, Qualytics uses field status to:
 
 - **Control the scope of quality operations** — Only operational fields (active and masked) are profiled and scanned, ensuring that resources are focused on relevant data.
-- **Track schema evolution** — When fields appear or disappear from source data, the system automatically updates their status, providing visibility into schema changes.
+- **Track schema evolution** — Each time a profile operation runs, Qualytics compares the source fields against the ones it already knows about and updates their statuses accordingly, giving you a clear picture of how your schema has changed.
 - **Manage the quality check lifecycle** — Field status transitions trigger automatic actions on associated quality checks, keeping your monitoring configuration consistent with the actual state of your data.
 - **Protect sensitive data** — Masked fields remain fully operational for quality monitoring while hiding raw values from API responses, with audit-logged access controls.
 
-## How Field Status Supports Data Quality
+## How Field Status Supports Data Quality?
 
 Field status plays a direct role in how you manage and maintain data quality across your datasets:
 
@@ -42,16 +42,19 @@ Users with Editor permission can request to reveal values in the surfaces that s
 
 ### Schema Change Detection
 
-When a field disappears from the source data, Qualytics automatically marks it as **Missing**. This serves as an early warning system for schema changes, allowing you to investigate whether the change was intentional or indicates a problem in your data pipeline.
+Each time a profile runs, Qualytics compares the fields it finds in the source against the ones it already knows about. If a previously active field is no longer present, it is marked as **Missing** — a clear signal that something has changed in your data pipeline or schema.
 
-Common scenarios that trigger the Missing status:
+!!! info
+    Qualytics does not monitor source schemas in real time. The Missing status is assigned during profile operations, whether those are scheduled or triggered manually. If your source changes between profiles, Qualytics won't know until the next profile runs.
+
+Common scenarios that lead to the Missing status include:
 
 - A column was renamed during a database migration
 - A table was restructured and certain fields were removed
-- A data source experienced an issue and is returning incomplete schemas
+- A data source experienced an issue and returned an incomplete schema
 - An upstream ETL process changed its output format
 
-In all these cases, the Missing status gives you immediate visibility without requiring manual monitoring of your data schemas.
+In each case, the next profile run reveals the change — without you having to manually compare schemas.
 
 ### Quality Check Lifecycle Management
 
@@ -61,15 +64,15 @@ Field status directly impacts the lifecycle of quality checks:
 - When a field goes **missing**, its quality checks remain intact, ready to resume when the field reappears. This is because missing is considered a transient state — the field is expected to return.
 - When a field is **active** or **masked**, all configured quality checks run normally during scan operations.
 
-This distinction between excluded and missing is intentional: archiving checks for excluded fields prevents unnecessary work, while keeping checks for missing fields avoids the overhead of reconfiguring them when the field returns.
+This distinction between excluded and missing is intentional: archiving checks for excluded fields prevents unnecessary work, while keeping checks for missing fields ensures scans surface failures when a referenced field is no longer present—acting as a signal to users that the schema has changed.
 
 ### Computed Field Dependencies
 
-Field status cascades through computed field dependencies. If a source field goes missing or is excluded, any computed fields that depend on it are automatically updated to reflect the same status, ensuring consistency across your data model.
+Field status cascades through computed field dependencies, though the timing depends on what triggered the change. When a user excludes a source field, its dependent computed fields are excluded immediately. When a profile operation marks a source field as Missing, those same computed fields are also marked as Missing within that same run.
 
 The cascade behavior differs by status:
 
-- **Missing cascade**: If a source field goes missing, its dependent computed fields are also marked as Missing. When the source field returns, the computed fields are restored automatically. Quality checks on computed fields are **not** archived during this transition.
+- **Missing cascade**: When a profile marks a source field as Missing, its dependent computed fields are marked as Missing in the same run. If the source field reappears in a later profile, both the source and its computed dependents are restored automatically. Quality checks on computed fields are **not** archived during this transition.
 - **Excluded cascade**: If a source field is excluded, its dependent computed fields are also excluded recursively. Quality checks on the computed fields are archived. The computed field definitions are preserved so they can be recovered when the source field is restored.
 
 !!! info
@@ -92,7 +95,7 @@ Your team has decided to stop using the `legacy_customer_id` column. Instead of
 
 1. The field's quality checks are archived — no more alerts about this column.
 2. Computed fields that depend on `legacy_customer_id` are automatically excluded.
-3. When your database team eventually drops the column, it does not trigger a Missing status because the field is already excluded.
+3. When your database team eventually drops the column, the next profile operation will not mark it as Missing because the field is already excluded.
 
 ### Scenario 2: Protecting sensitive data
 
@@ -112,18 +115,14 @@ After a routine profile operation, you notice that several fields are now marked
 3. Update your quality checks to reference the new column names.
 4. Exclude the old missing fields once the new columns are configured.
 
-### Scenario 4: Temporary data source outage
-
-A data source experiences a temporary outage, and the next profile operation returns an incomplete schema. Several fields are marked as **Missing**, but their quality checks remain intact. When the data source recovers and the next profile runs, all fields return to **Active** automatically and quality checks resume without any manual intervention.
-
 ## Best Practices
 
 | Practice | Rationale |
 | :--- | :--- |
 | **Mask sensitive fields** | Masking protects sensitive data while maintaining full quality monitoring. Use it for PII, financial data, or any field that should not be freely visible. |
 | **Exclude rather than delete** | Excluding preserves field history and allows restoration. Only hard-delete missing fields that were created by mistake and have no quality check references. |
 | **Investigate missing fields promptly** | Missing fields may indicate unplanned schema changes or data pipeline issues that require attention. |
-| **Use [merge](merge-fields.md) for column renames** | Merging preserves all historical profiles, anomalies, and quality checks under the new column name. |
+| **Use [merge](merge-fields.md){:target="_blank"} for column renames** | Merging preserves all historical profiles, anomalies, and quality checks under the new column name. |
 | **Review excluded fields periodically** | Fields that were excluded may become relevant again as business requirements change. |
 | **Re-enable quality checks after restoring** | Archived checks are not automatically restored — always verify that the appropriate checks are re-enabled after restoring a field. |
 | **Use status filters to monitor field health** | Regularly review the Missing and Excluded tabs to maintain awareness of your data model's current state. |