Support processing more resources per replica (#2871)

* Process fleets in batches

* Increase offers_cache

* Cache redundant AWS API calls

* Set MIN_PROCESSING_INTERVAL

* Prevent concurrent quotas requests

* Increase MIN_PROCESSING_INTERVAL

* Make runs stopping async

* Introduce SERVER_BACKGROUND_PROCESSING_RATE

* Use dummy locking on Postgres

* Process submitted jobs in one steps if no instance lock

* Fix missing cache lock

* Configure custom executor

* Fix typo

* Increase process_submitted_jobs max_instances

* Implement offers cache warm up trick

* Replace DSTACK_SERVER_BACKGROUND_PROCESSING_RATE with DSTACK_SERVER_BACKGROUND_PROCESSING_FACTOR

* Document DSTACK_SERVER_BACKGROUND_PROCESSING_FACTOR

* Do not create clusters from burstable AWS instances

Fixes #2872

* Set MIN_PROCESSING_INTERVAL for process_fleets

* Add TODO

Author: r4victor

docs/docs/guides/server-deployment.md

@@ -381,8 +381,20 @@ A single `dstack` server replica can support:
 * Up to 150 active jobs.
 * Up to 150 active instances.
 
-Having more active resources can affect server performance.
+Having more active resources will work but can affect server performance.
 If you hit these limits, consider using Postgres with multiple server replicas.
+You can also increase processing rates of a replica by setting the `DSTACK_SERVER_BACKGROUND_PROCESSING_FACTOR` environment variable.
+You should also increase `DSTACK_DB_POOL_SIZE` and `DSTACK_DB_MAX_OVERFLOW` proportionally.
+For example, to increase processing rates 4 times, set:
+
+```
+export DSTACK_SERVER_BACKGROUND_PROCESSING_FACTOR=4
+export DSTACK_DB_POOL_SIZE=80
+export DSTACK_DB_MAX_OVERFLOW=80
+```
+
+You have to ensure your Postgres installation supports that many connections by
+configuring [`max_connections`](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-MAX-CONNECTIONS) and/or using connection pooler.
 
 ## FAQs
 

docs/docs/reference/environment-variables.md

@@ -116,10 +116,13 @@ For more details on the options below, refer to the [server deployment](../guide
 - `DSTACK_ENABLE_PROMETHEUS_METRICS`{ #DSTACK_ENABLE_PROMETHEUS_METRICS } — Enables Prometheus metrics collection and export.
 - `DSTACK_DEFAULT_SERVICE_CLIENT_MAX_BODY_SIZE`{ #DSTACK_DEFAULT_SERVICE_CLIENT_MAX_BODY_SIZE } – Request body size limit for services running with a gateway, in bytes. Defaults to 64 MiB.
 - `DSTACK_FORBID_SERVICES_WITHOUT_GATEWAY`{ #DSTACK_FORBID_SERVICES_WITHOUT_GATEWAY } – Forbids registering new services without a gateway if set to any value.
-- `DSTACK_SERVER_CODE_UPLOAD_LIMIT`{ #DSTACK_SERVER_CODE_UPLOAD_LIMIT } - The repo size limit when uploading diffs or local repos, in bytes. Set to 0 to disable size limits. Defaults to 2MiB.
+- `DSTACK_SERVER_CODE_UPLOAD_LIMIT`{ #DSTACK_SERVER_CODE_UPLOAD_LIMIT } - The repo size limit when uploading diffs or local repos, in bytes. Set to `0` to disable size limits. Defaults to `2MiB`.
 - `DSTACK_SERVER_S3_BUCKET`{ #DSTACK_SERVER_S3_BUCKET } - The bucket that repo diffs will be uploaded to if set. If unset, diffs are uploaded to the database.
 - `DSTACK_SERVER_S3_BUCKET_REGION`{ #DSTACK_SERVER_S3_BUCKET_REGION } - The region of the S3 Bucket.
-- `DSTACK_SERVER_GCS_BUCKET`{ #DSTACK_SERVER_GCD_BUCKET } - The bucket that repo diffs will be uploaded to if set. If unset, diffs are uploaded to the database.
+- `DSTACK_SERVER_GCS_BUCKET`{ #DSTACK_SERVER_GCS_BUCKET } - The bucket that repo diffs will be uploaded to if set. If unset, diffs are uploaded to the database.
+- `DSTACK_DB_POOL_SIZE`{ #DSTACK_DB_POOL_SIZE } - The client DB connections pool size. Defaults to `20`,
+- `DSTACK_DB_MAX_OVERFLOW`{ #DSTACK_DB_MAX_OVERFLOW } - The client DB connections pool allowed overflow. Defaults to `20`.
+- `DSTACK_SERVER_BACKGROUND_PROCESSING_FACTOR`{ #DSTACK_SERVER_BACKGROUND_PROCESSING_FACTOR } - The number of background jobs for processing server resources. Increase if you need to process more resources per server replica quickly. Defaults to `1`.
 
 ??? info "Internal environment variables"
      The following environment variables are intended for development purposes: