From 20a42295e3f5d1c612f1c43718b0232e99c47f04 Mon Sep 17 00:00:00 2001 From: Jvst Me Date: Thu, 3 Jul 2025 09:32:33 +0200 Subject: [PATCH 1/6] [Docs]: Service rolling deployments --- docs/docs/concepts/services.md | 46 ++++++++++++++++++++ src/dstack/_internal/server/services/runs.py | 1 + 2 files changed, 47 insertions(+) diff --git a/docs/docs/concepts/services.md b/docs/docs/concepts/services.md index 5dd92f19c3..0793bf795d 100644 --- a/docs/docs/concepts/services.md +++ b/docs/docs/concepts/services.md @@ -681,6 +681,52 @@ utilization_policy: --8<-- "docs/concepts/snippets/manage-runs.ext" +## Deployments + +To deploy a new version of the service, change its configuration or associated resources, such as files or repo contents. Then, re-run `dstack apply`. + +
+ +```shell +$ dstack apply -f my-service.dstack.yml + +Active run my-service already exists. Detected changes that can be updated in-place: +- Repo state (branch, commit, or other) +- File archives +- Configuration properties: + - env + - files + +Update the run? [y/n]: +``` + +
+ +If `dstack` can apply all the detected changes without a full service restart, it starts a rolling deployment. During the rolling deployment, `dstack` gradually updates the service replicas to the new version. First, it starts an extra replica. Then, it waits until this replica enters the `running` state. Finally, it terminates the old replica. + +During deployment, new and old replicas can coexist and handle requests simultaneously. If the service has multiple replicas, they are updated one by one — there can be at most one extra replica above the desired number of replicas, as determined by the `replicas` and `scaling` properties. + +You can track the deployment progress in the `dstack apply` output. The `deployment` number will be lower for old replicas and higher for new ones. + +```shell +$ dstack apply -f my-service.dstack.yml + +⠋ Launching my-service... + NAME BACKEND PRICE STATUS SUBMITTED + my-service deployment=1 running 11 mins ago + replica=0 job=0 deployment=0 aws (us-west-2) $0.0026 terminating 11 mins ago + replica=1 job=0 deployment=1 aws (us-west-2) $0.0026 running 1 min ago +``` + +A deployment stops either when all of the replicas are up to date or when the next deployment is submitted. + + + +Rolling deployments apply to the following configuration properties: `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`. They also apply to changes to [repo](repos.md) or [file](#files) contents. + +??? info "Forcing a redeployment" + Sometimes you may want to run a rolling deployment even if `dstack` detects no changes to the configuration or associated resources. This can be useful if you've changed the project [secrets](secrets.md) or if you just want to gradually restart all service replicas. Currently, this can be achieved by making an insignificant change to the configuration, such as adding a dummy [environment variable](#environment-variables). + !!! info "What's next?" 1. Read about [dev environments](dev-environments.md), [tasks](tasks.md), and [repos](repos.md) 2. Learn how to manage [fleets](fleets.md) diff --git a/src/dstack/_internal/server/services/runs.py b/src/dstack/_internal/server/services/runs.py index ce8325ce72..a8775ce7b6 100644 --- a/src/dstack/_internal/server/services/runs.py +++ b/src/dstack/_internal/server/services/runs.py @@ -930,6 +930,7 @@ def _validate_run_spec_and_set_defaults(run_spec: RunSpec): "replicas", "scaling", # rolling deployment + # NOTE: keep this list in sync with the "Deployments" section in services.md "resources", "volumes", "docker", From 17830627214f781badc97ff7577b3096acdd60f8 Mon Sep 17 00:00:00 2001 From: Jvst Me Date: Thu, 3 Jul 2025 15:19:58 +0200 Subject: [PATCH 2/6] Minor update --- docs/docs/concepts/services.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/docs/concepts/services.md b/docs/docs/concepts/services.md index 0793bf795d..7bcdaf613f 100644 --- a/docs/docs/concepts/services.md +++ b/docs/docs/concepts/services.md @@ -706,7 +706,12 @@ If `dstack` can apply all the detected changes without a full service restart, i During deployment, new and old replicas can coexist and handle requests simultaneously. If the service has multiple replicas, they are updated one by one — there can be at most one extra replica above the desired number of replicas, as determined by the `replicas` and `scaling` properties. -You can track the deployment progress in the `dstack apply` output. The `deployment` number will be lower for old replicas and higher for new ones. +You can track the deployment progress in the `dstack apply` output or in `dstack ps`. The `deployment` number will be lower for old replicas and higher for new ones. + + ```shell $ dstack apply -f my-service.dstack.yml From 6853688a22a0cf7d1d96a4d83a7d78a419149234 Mon Sep 17 00:00:00 2001 From: peterschmidt85 Date: Fri, 4 Jul 2025 10:29:05 +0200 Subject: [PATCH 3/6] [Docs]: Service rolling deployments Minor edits --- docs/docs/concepts/services.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/docs/docs/concepts/services.md b/docs/docs/concepts/services.md index 7bcdaf613f..ebf52811a5 100644 --- a/docs/docs/concepts/services.md +++ b/docs/docs/concepts/services.md @@ -679,11 +679,9 @@ utilization_policy: [`max_price`](../reference/dstack.yml/service.md#max_price), and among [others](../reference/dstack.yml/service.md). ---8<-- "docs/concepts/snippets/manage-runs.ext" - -## Deployments +## Rolling deployment -To deploy a new version of the service, change its configuration or associated resources, such as files or repo contents. Then, re-run `dstack apply`. +To deploy a new version of a service that is already `running`, use `dstack apply`. `dstack` will automatically detect changes and update replicas one by one. New and old replicas coexist and handle requests simultaneously until deployment finishes.
@@ -702,11 +700,8 @@ Update the run? [y/n]:
-If `dstack` can apply all the detected changes without a full service restart, it starts a rolling deployment. During the rolling deployment, `dstack` gradually updates the service replicas to the new version. First, it starts an extra replica. Then, it waits until this replica enters the `running` state. Finally, it terminates the old replica. - -During deployment, new and old replicas can coexist and handle requests simultaneously. If the service has multiple replicas, they are updated one by one — there can be at most one extra replica above the desired number of replicas, as determined by the `replicas` and `scaling` properties. - -You can track the deployment progress in the `dstack apply` output or in `dstack ps`. The `deployment` number will be lower for old replicas and higher for new ones. +You can track the deployment progress in both `dstack apply` or `dstack ps`. +Older replicas have lower `deployment` numbers; newer ones have higher. -Rolling deployments apply to the following configuration properties: `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`. They also apply to changes to [repo](repos.md) or [file](#files) contents. +??? info "Force a redeployment" + `dstack` automatically detects changes to `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`, and to [repo](repos.md) and [files](#files) contents. + + To trigger a rolling deployment when no properties have changed (e.g., after updating [secrets](secrets.md) or to restart all replicas), + make a minor config change, such as adding a dummy [environment variable](#environment-variables). -??? info "Forcing a redeployment" - Sometimes you may want to run a rolling deployment even if `dstack` detects no changes to the configuration or associated resources. This can be useful if you've changed the project [secrets](secrets.md) or if you just want to gradually restart all service replicas. Currently, this can be achieved by making an insignificant change to the configuration, such as adding a dummy [environment variable](#environment-variables). +--8<-- "docs/concepts/snippets/manage-runs.ext" !!! info "What's next?" 1. Read about [dev environments](dev-environments.md), [tasks](tasks.md), and [repos](repos.md) From 5a79e41702b5e24ad77b3e0c261b5cfe68abf330 Mon Sep 17 00:00:00 2001 From: Jvst Me Date: Mon, 7 Jul 2025 09:17:18 +0200 Subject: [PATCH 4/6] Fix --- docs/docs/concepts/services.md | 14 ++++++++++---- src/dstack/_internal/server/services/runs.py | 2 +- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/docs/concepts/services.md b/docs/docs/concepts/services.md index ebf52811a5..9d3d5bc87e 100644 --- a/docs/docs/concepts/services.md +++ b/docs/docs/concepts/services.md @@ -681,7 +681,7 @@ utilization_policy: ## Rolling deployment -To deploy a new version of a service that is already `running`, use `dstack apply`. `dstack` will automatically detect changes and update replicas one by one. New and old replicas coexist and handle requests simultaneously until deployment finishes. +To deploy a new version of a service that is already `running`, use `dstack apply`. `dstack` will automatically detect changes and suggest to update the run.
@@ -700,6 +700,8 @@ Update the run? [y/n]:
+To update a replica, `dstack` starts a new replica, then waits until it becomes `running`, then terminates the old replica. If the service has multiple replicas, they are updated one by one — `dstack` does not proceed to updating the next replica until the previous one was fully updated. New and old replicas can coexist and handle requests simultaneously until deployment finishes. + You can track the deployment progress in both `dstack apply` or `dstack ps`. Older replicas have lower `deployment` numbers; newer ones have higher. @@ -720,10 +722,14 @@ $ dstack apply -f my-service.dstack.yml The rolling deployment stops when all replicas are updated or when a new deployment is submitted. - +??? info "Updatable properties" + + + Rolling deployments apply to the following configuration properties: `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`. They also apply to changes to [repo](repos.md) or [file](#files) contents. + + Additionally, changes to `replicas` and `scaling` can be applied without redeploying replicas. -??? info "Force a redeployment" - `dstack` automatically detects changes to `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`, and to [repo](repos.md) and [files](#files) contents. + Changes to other properties will require a full service restart. To trigger a rolling deployment when no properties have changed (e.g., after updating [secrets](secrets.md) or to restart all replicas), make a minor config change, such as adding a dummy [environment variable](#environment-variables). diff --git a/src/dstack/_internal/server/services/runs.py b/src/dstack/_internal/server/services/runs.py index a8775ce7b6..629f9b2ecc 100644 --- a/src/dstack/_internal/server/services/runs.py +++ b/src/dstack/_internal/server/services/runs.py @@ -930,7 +930,7 @@ def _validate_run_spec_and_set_defaults(run_spec: RunSpec): "replicas", "scaling", # rolling deployment - # NOTE: keep this list in sync with the "Deployments" section in services.md + # NOTE: keep this list in sync with the "Rolling deployment" section in services.md "resources", "volumes", "docker", From 414d4dc7a03ea5622dc79a1d98443e82420be587 Mon Sep 17 00:00:00 2001 From: peterschmidt85 Date: Mon, 7 Jul 2025 12:37:30 +0200 Subject: [PATCH 5/6] [Docs]: Service rolling deployments Minor edits --- docs/docs/concepts/services.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/docs/concepts/services.md b/docs/docs/concepts/services.md index 9d3d5bc87e..2a50dc1f68 100644 --- a/docs/docs/concepts/services.md +++ b/docs/docs/concepts/services.md @@ -681,7 +681,7 @@ utilization_policy: ## Rolling deployment -To deploy a new version of a service that is already `running`, use `dstack apply`. `dstack` will automatically detect changes and suggest to update the run. +To deploy a new version of a service that is already `running`, use `dstack apply`. `dstack` will automatically detect changes and suggest a rolling deployment update.
@@ -700,9 +700,10 @@ Update the run? [y/n]:
-To update a replica, `dstack` starts a new replica, then waits until it becomes `running`, then terminates the old replica. If the service has multiple replicas, they are updated one by one — `dstack` does not proceed to updating the next replica until the previous one was fully updated. New and old replicas can coexist and handle requests simultaneously until deployment finishes. +If approved, `dstack` updates each replica one by one—waiting for each to reach the `running` state before updating the next. +New and old replicas coexist and handle requests in parallel. -You can track the deployment progress in both `dstack apply` or `dstack ps`. +You can track the progress of rolling deployment in both `dstack apply` or `dstack ps`. Older replicas have lower `deployment` numbers; newer ones have higher. - Rolling deployments apply to the following configuration properties: `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`. They also apply to changes to [repo](repos.md) or [file](#files) contents. + Rolling deployment supports changes to the following properties: `resources`, `volumes`, `docker`, `files`, `image`, `user`, `privileged`, `entrypoint`, `working_dir`, `python`, `nvcc`, `single_branch`, `env`, `shell`, `commands`, as well as changes to [repo](repos.md) or [file](#files) contents. - Additionally, changes to `replicas` and `scaling` can be applied without redeploying replicas. + Changes to `replicas` and `scaling` can be applied without redeploying replicas. - Changes to other properties will require a full service restart. + Changes to other properties require a full service restart. - To trigger a rolling deployment when no properties have changed (e.g., after updating [secrets](secrets.md) or to restart all replicas), + To trigger a rolling deployment when no properties have changed (e.g., after updating [secrets](secrets.md) or to restart all replicas), make a minor config change, such as adding a dummy [environment variable](#environment-variables). --8<-- "docs/concepts/snippets/manage-runs.ext" From 29d96ee1d1e7809297b094b3f68393f9fe504fde Mon Sep 17 00:00:00 2001 From: Jvst Me Date: Wed, 9 Jul 2025 23:37:24 +0200 Subject: [PATCH 6/6] Return the algorithm description --- docs/docs/concepts/services.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/docs/concepts/services.md b/docs/docs/concepts/services.md index 2a50dc1f68..b4b7288bca 100644 --- a/docs/docs/concepts/services.md +++ b/docs/docs/concepts/services.md @@ -700,8 +700,7 @@ Update the run? [y/n]: -If approved, `dstack` updates each replica one by one—waiting for each to reach the `running` state before updating the next. -New and old replicas coexist and handle requests in parallel. +If approved, `dstack` gradually updates the service replicas. To update a replica, `dstack` starts a new replica, waits for it to become `running`, then terminates the old replica. This process is repeated for each replica, one at a time. You can track the progress of rolling deployment in both `dstack apply` or `dstack ps`. Older replicas have lower `deployment` numbers; newer ones have higher.