From 366033b813538dafda89b61fd6f5b90f6322d098 Mon Sep 17 00:00:00 2001 From: fmercadop Date: Fri, 3 Apr 2026 09:15:30 -0700 Subject: [PATCH 1/4] docs: add rollback strategy to reference/datacoves --- docs/reference/datacoves/rollback-strategy.md | 93 +++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 docs/reference/datacoves/rollback-strategy.md diff --git a/docs/reference/datacoves/rollback-strategy.md b/docs/reference/datacoves/rollback-strategy.md new file mode 100644 index 0000000..e805383 --- /dev/null +++ b/docs/reference/datacoves/rollback-strategy.md @@ -0,0 +1,93 @@ +--- +title: Rollback Strategy +sidebar_label: Rollback Strategy +sidebar_position: 132 +--- + +# Rollback Strategy + +Datacoves manages change across three distinct platform layers: infrastructure, application, and custom customer images. Each layer has different rollback considerations, summarized below. + +| Layer | Rollback Capability | Responsibility | +|-------|-------------------|----------------| +| Infrastructure | Not managed by Datacoves | Cloud provider / Customer infrastructure team | +| Datacoves Application | Forward-fix model (hotfix) | Datacoves Engineering | +| Custom Customer Image | Technically feasible but strongly discouraged | Customer-initiated, Datacoves-executed | + +## Infrastructure Layer + +The infrastructure layer includes cloud provider services and container orchestration systems (such as Kubernetes) that host the Datacoves platform. Changes at this layer, including version upgrades, configuration modifications, and scaling adjustments, are managed by the customer's infrastructure team or cloud provider. + +:::info + +Datacoves does not have control over infrastructure-level changes and cannot provide a rollback plan for this layer. For example, if the underlying Kubernetes cluster is upgraded from one version to another, that operation falls outside the scope of Datacoves platform management. Customers should consult their cloud provider's documentation for rollback procedures at this layer. + +::: + +## Datacoves Application Layer + +The Datacoves application follows a forward-coding development model. All changes, including new features, enhancements, and bug fixes, are implemented as forward-moving code changes. The platform does not support reverting to a prior application version. + +If a defect or regression is identified following a release, the Datacoves engineering team will produce and deploy a **hotfix** as a new forward release that corrects the identified problem, ensuring the codebase remains in a consistent, forward-progressing state. + +### Defect Resolution Process + +1. The customer reports the issue through the established support channel. +2. The Datacoves engineering team triages the issue and determines severity. +3. A hotfix is developed, tested, and deployed as a new forward release. +4. The customer is notified upon successful deployment and resolution. + +## Custom Customer Image Layer + +Custom customer images allow customers to specify particular versions of third-party packages and tools that are not included in the standard Datacoves distribution. For example, a customer may request a specific version of the `elementary` package to be included in their environment. + +### Cascading Impact Risk + +:::warning + +While version rollback is technically feasible at this layer, **it is strongly discouraged** due to the risk of cascading disruptions in multi-user environments. + +::: + +In enterprise environments where many users share the same Datacoves instance, rolling back a third-party package version can produce unintended effects across the organization. Consider the following scenario: + +1. A customer requests an upgrade to Package X (a third-party dependency). +2. After deployment, the new version introduces a breaking change that affects certain pipelines. +3. **User A** detects the issue early and modifies their code to accommodate the new version, restoring normal execution. +4. **User B**, also affected, requests a rollback of Package X to the prior version. +5. If the rollback is executed, **User A's code**, which was adapted to the newer version, now breaks under the restored older version. + +In an organization with dozens or hundreds of users on the platform concurrently, this type of cascading disruption can propagate rapidly and unpredictably. + +### Recommended Approach + +Rather than reverting a shared package version, the recommended course of action is for affected users to apply targeted fixes to their own code to accommodate the updated package. This preserves environmental consistency and avoids introducing new failures for users who have already adapted. A version rollback should be considered only as a **last resort**, after a thorough impact assessment has confirmed that no other users have made accommodations for the current version. + +### Rollback Process (Last Resort) + +If a rollback is determined to be necessary after impact assessment: + +1. The customer identifies the package and the target version to revert to. +2. The customer submits a rollback request through the support channel. +3. Datacoves validates the request and confirms target version availability. +4. Datacoves performs an impact assessment across the customer's user base. +5. The image configuration is updated and deployed to the customer's environment. +6. The customer validates the rollback and confirms resolution. + +## Summary of Responsibilities + +| Scenario | Datacoves | Customer | +|----------|-----------|----------| +| Infrastructure change requires rollback | Outside Datacoves scope | Coordinate with cloud provider or internal infrastructure team | +| Application defect after release | Develop and deploy a hotfix as a forward release | Report the issue through the support channel | +| Custom image package version issue | Advise forward-fix approach; execute rollback only as a last resort after impact assessment | Identify the target version and submit the request | + +## Additional Considerations + +- Custom image rollbacks are limited to packages and versions that were previously configured and validated. Rolling back to an untested version may require additional validation. + +:::tip + +Customers are encouraged to maintain their own change logs and version records for custom image configurations to facilitate efficient communication with Datacoves support. + +::: \ No newline at end of file From 276efd38ce2d263cd787e771bddda4ba973548d0 Mon Sep 17 00:00:00 2001 From: fmercadop Date: Fri, 3 Apr 2026 09:35:54 -0700 Subject: [PATCH 2/4] docs: add tabs for DAGs sync options in airflow initial setup --- .../airflow/{initial-setup.md => initial-setup.mdx} | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) rename docs/how-tos/airflow/{initial-setup.md => initial-setup.mdx} (94%) diff --git a/docs/how-tos/airflow/initial-setup.md b/docs/how-tos/airflow/initial-setup.mdx similarity index 94% rename from docs/how-tos/airflow/initial-setup.md rename to docs/how-tos/airflow/initial-setup.mdx index 84d7ecf..d0eebf7 100644 --- a/docs/how-tos/airflow/initial-setup.md +++ b/docs/how-tos/airflow/initial-setup.mdx @@ -2,6 +2,9 @@ title: Airflow - Initial setup sidebar_position: 1 --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + # How to configure Airflow in Datacoves ## Turn on Airflow @@ -35,7 +38,8 @@ Once you enabled Airflow, click on the `Services configuration > Airflow Setting There are **2 options** to choose from for your DAGs sync: **Git Sync** and **S3 Sync**. Each requires specific information to be provided during configuration. Our recommended default is Git Sync. -**Git Sync** + + - **Provider** Select `Git` - **Git branch name** The branch airflow will monitor for changes. If you have more than 1 environment (Development and Production), we suggest `airflow_development` for the development environment and `main` for the production environment. Note: You would need to create an `airflow_development` branch in your repo. If only have 1 environment, then the `Git branch name` should be `main`. @@ -44,7 +48,8 @@ Each requires specific information to be provided during configuration. Our reco We recommend combining your dbt transformations in the same project as your Airflow orchestration. However, you may wish to separate orchestration from transformation into different git projects. In Datacoves you can achieve this by having two projects. Each project will be associated with one git repo. Find out how to configure a [project](/docs/how-tos/datacoves/how_to_projects). ::: -**S3 Sync** + + You must create the s3 bucket and IAM user before this step. @@ -58,6 +63,9 @@ You must create the s3 bucket and IAM user before this step. Once configured, you will need to configure your CI/CD process to clone your project into the S3 bucket. + + + ### Logs Configuration - Private Deployment ONLY. Not applicable in SaaS. :::info From 84d7f3896960dc5e85ece33470d226e05b69366f Mon Sep 17 00:00:00 2001 From: fmercadop Date: Fri, 3 Apr 2026 10:43:42 -0700 Subject: [PATCH 3/4] docs: revise rollback strategy per review feedback --- docs/reference/datacoves/rollback-strategy.md | 69 ++++++++++--------- 1 file changed, 36 insertions(+), 33 deletions(-) diff --git a/docs/reference/datacoves/rollback-strategy.md b/docs/reference/datacoves/rollback-strategy.md index e805383..881372a 100644 --- a/docs/reference/datacoves/rollback-strategy.md +++ b/docs/reference/datacoves/rollback-strategy.md @@ -6,29 +6,29 @@ sidebar_position: 132 # Rollback Strategy -Datacoves manages change across three distinct platform layers: infrastructure, application, and custom customer images. Each layer has different rollback considerations, summarized below. +Datacoves manages change across three distinct platform layers: infrastructure, platform services, and customer image. Each layer has different rollback considerations, summarized below. | Layer | Rollback Capability | Responsibility | |-------|-------------------|----------------| -| Infrastructure | Not managed by Datacoves | Cloud provider / Customer infrastructure team | -| Datacoves Application | Forward-fix model (hotfix) | Datacoves Engineering | -| Custom Customer Image | Technically feasible but strongly discouraged | Customer-initiated, Datacoves-executed | +| Infrastructure | Cannot be undone once applied | Datacoves + Customer (collaborative upgrade, no rollback) | +| Platform Services (Airflow, Python, etc.) | Cannot be rolled back | Datacoves Engineering | +| Customer Image (dbt, packages, libraries) | Technically feasible but strongly discouraged | Customer-initiated, Datacoves-executed | ## Infrastructure Layer -The infrastructure layer includes cloud provider services and container orchestration systems (such as Kubernetes) that host the Datacoves platform. Changes at this layer, including version upgrades, configuration modifications, and scaling adjustments, are managed by the customer's infrastructure team or cloud provider. +The infrastructure layer includes cloud provider services and container orchestration systems (such as Kubernetes) that host the Datacoves platform. Datacoves works collaboratively with the customer's team to plan and execute infrastructure upgrades. Both parties participate in testing prior to applying changes. -:::info +:::warning -Datacoves does not have control over infrastructure-level changes and cannot provide a rollback plan for this layer. For example, if the underlying Kubernetes cluster is upgraded from one version to another, that operation falls outside the scope of Datacoves platform management. Customers should consult their cloud provider's documentation for rollback procedures at this layer. +Once an infrastructure upgrade is applied, it cannot be undone. Customers and Datacoves should ensure thorough testing before proceeding with any infrastructure-level change. ::: -## Datacoves Application Layer +## Platform Services Layer -The Datacoves application follows a forward-coding development model. All changes, including new features, enhancements, and bug fixes, are implemented as forward-moving code changes. The platform does not support reverting to a prior application version. +Platform-level services, such as Airflow versions, Python versions, and other core components managed by Datacoves, follow a forward-fix model. These versions cannot be rolled back once upgraded. -If a defect or regression is identified following a release, the Datacoves engineering team will produce and deploy a **hotfix** as a new forward release that corrects the identified problem, ensuring the codebase remains in a consistent, forward-progressing state. +If a defect or regression is identified following a platform release, the Datacoves engineering team will produce and deploy a **hotfix** as a new forward release that corrects the identified problem, ensuring the platform remains in a consistent, forward-progressing state. ### Defect Resolution Process @@ -37,9 +37,9 @@ If a defect or regression is identified following a release, the Datacoves engin 3. A hotfix is developed, tested, and deployed as a new forward release. 4. The customer is notified upon successful deployment and resolution. -## Custom Customer Image Layer +## Customer Image Layer -Custom customer images allow customers to specify particular versions of third-party packages and tools that are not included in the standard Datacoves distribution. For example, a customer may request a specific version of the `elementary` package to be included in their environment. +The customer image defines the specific versions of packages and libraries used in the customer's environment, including both standard components (such as dbt) and any additional tools. For example, if dbt is upgraded from version 1.10 to 1.11 and an issue is discovered, the image can be reverted to use dbt 1.10. ### Cascading Impact Risk @@ -49,7 +49,7 @@ While version rollback is technically feasible at this layer, **it is strongly d ::: -In enterprise environments where many users share the same Datacoves instance, rolling back a third-party package version can produce unintended effects across the organization. Consider the following scenario: +In enterprise environments where many users share the same Datacoves instance, rolling back a package version can produce unintended effects across the organization. Consider the following scenario: 1. A customer requests an upgrade to Package X (a third-party dependency). 2. After deployment, the new version introduces a breaking change that affects certain pipelines. @@ -57,37 +57,40 @@ In enterprise environments where many users share the same Datacoves instance, r 4. **User B**, also affected, requests a rollback of Package X to the prior version. 5. If the rollback is executed, **User A's code**, which was adapted to the newer version, now breaks under the restored older version. +Additionally, rolling back a single package may introduce dependency conflicts. For example, if both a Snowflake connector library and dbt were upgraded together because they share a common dependency, rolling back only dbt could create an incompatibility between the two libraries. + In an organization with dozens or hundreds of users on the platform concurrently, this type of cascading disruption can propagate rapidly and unpredictably. ### Recommended Approach -Rather than reverting a shared package version, the recommended course of action is for affected users to apply targeted fixes to their own code to accommodate the updated package. This preserves environmental consistency and avoids introducing new failures for users who have already adapted. A version rollback should be considered only as a **last resort**, after a thorough impact assessment has confirmed that no other users have made accommodations for the current version. +Rather than reverting a shared package version, the recommended course of action is: -### Rollback Process (Last Resort) +1. **Test before promoting to production.** Customers who maintain a separate testing cluster or environment can identify issues before they reach production users. +2. **Apply targeted fixes.** Affected users should update their own code to accommodate the new package version, preserving environmental consistency and avoiding new failures for users who have already adapted. -If a rollback is determined to be necessary after impact assessment: +A version rollback should be considered only as a **last resort**. -1. The customer identifies the package and the target version to revert to. -2. The customer submits a rollback request through the support channel. -3. Datacoves validates the request and confirms target version availability. -4. Datacoves performs an impact assessment across the customer's user base. -5. The image configuration is updated and deployed to the customer's environment. -6. The customer validates the rollback and confirms resolution. +### Rollback Process (Last Resort) {#pis-rollback-process} -## Summary of Responsibilities +If a rollback is determined to be necessary: -| Scenario | Datacoves | Customer | -|----------|-----------|----------| -| Infrastructure change requires rollback | Outside Datacoves scope | Coordinate with cloud provider or internal infrastructure team | -| Application defect after release | Develop and deploy a hotfix as a forward release | Report the issue through the support channel | -| Custom image package version issue | Advise forward-fix approach; execute rollback only as a last resort after impact assessment | Identify the target version and submit the request | +1. The customer identifies the package and the target version to revert to. +2. The customer submits a rollback request through the support channel. +3. Datacoves validates the request and confirms target version availability, including a review of potential dependency conflicts with other libraries in the image. +4. The image configuration is updated and deployed to the customer's environment. +5. The customer validates the rollback and confirms resolution. -## Additional Considerations +:::info -- Custom image rollbacks are limited to packages and versions that were previously configured and validated. Rolling back to an untested version may require additional validation. +If a rollback introduces new issues (for example, breaking other users' workflows or creating dependency conflicts), the customer assumes responsibility for resolving those downstream effects. -:::tip +::: -Customers are encouraged to maintain their own change logs and version records for custom image configurations to facilitate efficient communication with Datacoves support. +## Summary of Responsibilities -::: \ No newline at end of file +| Scenario | Datacoves | Customer | +|----------|-----------|----------| +| Infrastructure upgrade | Collaborate on planning and execution | Participate in testing before applying changes | +| Infrastructure rollback | Cannot be undone | Cannot be undone | +| Platform service defect after release | Develop and deploy a hotfix as a forward release | Report the issue through the support channel | +| Customer image package version issue | Advise forward-fix approach; execute rollback only as a last resort with dependency review | Identify the target version, submit the request, and accept responsibility for downstream effects | \ No newline at end of file From aaf880ae85bc58c394fd5bf43645babb87a36337 Mon Sep 17 00:00:00 2001 From: fmercadop Date: Mon, 6 Apr 2026 13:44:37 -0700 Subject: [PATCH 4/4] docs: warn against org prefix in Azure DevOps HTTPS URL --- .../how_to_configure_azure_DevOps.mdx | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_DevOps.mdx b/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_DevOps.mdx index d79127e..db6a533 100644 --- a/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_DevOps.mdx +++ b/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_DevOps.mdx @@ -109,6 +109,18 @@ Now that you have created your EntraID application and added it as a user in the Be sure to save all of these details in a safe notepad. Proceed to the next tab to configure **Azure DevOps authentication**. +:::warning +When copying the HTTPS URL, ensure it does **not** include an organization prefix before `dev.azure.com`. For example, use: + +`https://dev.azure.com/your-org/your-project/_git/your-repo` + +**Not:** + +`https://your-org@dev.azure.com/your-org/your-project/_git/your-repo` + +The `your-org@` prefix interferes with Datacoves Azure authentication and will cause URL validation to fail. +::: +