Add cloud acceptance test for the DMS read path - #6202
Draft
shreyas-goenka wants to merge 4 commits into
Draft
Conversation
Adds acceptance/cmd/bundle/dms-cloud-read, which runs the read-only bundle-deployments verbs against a real workspace instead of a stubbed testserver. DMS is placed on DEV and STAGING shards only, so this case is gated to the staging cloud env: on a prod control plane the routes 404 at the API proxy, which is why the existing aws-prod-ucws env cannot cover this service. The sibling dms-read-only test keeps the local stubbed coverage. list-deployments asserts only the response shape, since the deployment set and its server-assigned ids vary per workspace. The four not-found cases pin the error contract and prove the whole read path is reachable — DMS resolves the workspace and runs the permission check before looking up the row, so a clean NotFound exercises more than the front door. Verified against e2-dogfood (staging shard): passes on cloud, skips locally. Co-authored-by: Isaac
CLOUD_ENV cannot express "staging only": the staging and prod aws test envs both set CLOUD_ENV=aws, and the acceptance framework gates on cloud family rather than individual environment. Without a finer signal, adding aws-stg to the shared cloud matrix would also run this test on aws-prod-is and aws-prod-ucws-is, where the DMS routes 404 at the API proxy. Gate on the workspace host instead: staging control planes carry ".staging." in the domain. Skipping rather than failing keeps the prod matrix entries green. Remove the guard once DMS reaches prod; the test then becomes a plain aws cloud test with no host check. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: fc468b8
11 interesting tests: 4 SKIP, 3 FAIL, 3 RECOVERED, 1 KNOWN
|
Adds a RequiresClassic config flag. When a test sets it and the environment
exports DATABRICKS_TEST_NO_CLASSIC=true, the test is skipped rather than run.
The motivation is a new serverless-only staging test workspace. Of the 217
cloud-enabled acceptance tests, 76 deploy an all-purpose cluster, a job cluster,
or an instance pool; the other 141 need no classic compute. Without a flag those
76 fail at deploy time against a serverless-only workspace, so a genuine
regression is indistinguishable from missing infrastructure.
This is deliberately an environment-declared switch rather than one inferred from
TEST_DEFAULT_CLUSTER_ID being empty: inferring would silently skip these tests
anywhere that fixture happens to be unset, including local runs.
Annotations prefer parent directories where every cloud test underneath needs
classic compute (config-remote-sync, integration_whl, resources/{clusters,
pipelines,jobs}, invariant, deployment/bind/job), with a RequiresClassic = false
override on config-remote-sync/dashboard_etag, which syncs a warehouse-backed
dashboard rather than a job cluster.
Verified against a serverless-only staging workspace: the annotated tests skip
with "Disabled via RequiresClassic", dashboard_etag still runs and passes, and
cmd/fs plus resources/secret_scopes pass on the direct engine.
out.test.toml regeneration is still pending: it requires jq >= 1.7 and this
machine has 1.6, so `go test ./acceptance -run "^TestAccept$" -only-out-test-toml`
must run elsewhere before merge.
Co-authored-by: Isaac
Adds the missing "why" comment to the 19 RequiresClassic annotations that had none, naming the specific reason per test (a job cluster, a spark_jar_task, a pipeline, an all-purpose cluster) rather than repeating one generic line. Two comments were wrong on the first pass: apps/job_permissions and resources/permissions/jobs/delete_one were justified as "deploys jobs", which is not classic on its own. Their bundles use spark_python_task and notebook_task respectively, both of which run on a cluster, so the comments now cite that. Also picks up the out.test.toml snapshots the harness regenerated for RequiresClassic, now that materialized_config.go emits it alongside the sibling Requires* flags. Without that the inherited value is invisible on PRs. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The read-only
databricks bundle-deployments(DMS) commands are covered todayonly by
acceptance/cmd/bundle/dms-read-only, which stubs all eight endpointsagainst the local testserver. That proves the CLI renders responses correctly,
but never exercises a real server — so a bad DMS commit cannot be caught here.
What
Adds
acceptance/cmd/bundle/dms-cloud-read, which runs the same read verbsagainst a real workspace.
list-deploymentsasserts only the response shape: the deployment set growsas other tests and engineers deploy, and the ids are server-assigned. The four
not-found cases pin the error contract and prove the whole read path is
reachable — DMS resolves the workspace and runs the permission check before it
looks for the row, so a clean
NotFoundexercises more than the front door.The staging gate
DMS is placed on DEV and STAGING shards only (
environment_in: ["DEV","STAGING"]),so on a prod control plane every request 404s at the API proxy.
CLOUD_ENVcannot express that split — the staging and prod aws test envs both set
CLOUD_ENV=aws, and the acceptance framework gates on cloud family, notindividual environment.
So
scriptgates on the workspace host, since staging control planes carry.staging.in the domain, and skips elsewhere. Skipping rather than failingmeans this test can join the shared cloud matrix without red-lining the
existing prod entries.
Once DMS reaches prod, drop that guard and this becomes a plain aws cloud
test.
Testing
Verified against
e2-dogfood.staging.cloud.databricks.com(a staging shardwhere DMS serves):
dms-cloud-readpasses,dms-read-onlyskipsdms-read-onlypasses,dms-cloud-readskips-update(golden output is stable and workspace-independent)The first golden capture returned ~1,300 lines of real deployments from the
staging workspace, confirming the CLI reached live DMS rather than a stub.
This PR alone changes no CI behaviour: no cloud test env currently points at a
staging workspace, so the test skips everywhere until the companion
eng-dev-ecosystem change adds
aws-stgto the matrix.This pull request and its description were written by Isaac.