[tharunbhuk-db]

🥞 Stacked PR

Use this link to review incremental changes.

• stack/pipelines-test-pr1-command [Files changed] ← this PR
  • stack/pipelines-test-bundle-key [Files changed]
    • stack/pipelines-test-read-events [Files changed]
      • stack/pipelines-test-print-results [Files changed]
        • stack/pipelines-test-pr2-output [Files changed]
          • stack/pipelines-test-a4-run-control [Files changed]
            • stack/pipelines-test-pr3-junit [Files changed]
              • stack/pipelines-test-junit-flag [Files changed]

---

• Add databricks pipelines test --pipeline-id <id> [TEST_SELECTION...]: starts a test-only update and prints its IDs.
• Experimental: runs only with DATABRICKS_CLI_PIPELINES_TEST=true.
• Selections are workspace paths or paths relative to root_path, sent as typed. --pytest-arg accepts only -k and -m.

Tests: unit tests; acceptance in acceptance/pipelines/test/{arguments,pipeline-id,submit-error}.

This pull request and its description were written by Isaac.

[github-actions]

Commit: 9be85fdc8 | Workflow run

Regression test report

Selected 4 changed tests (limit=20, 0 not selected)

4 tests: regression=4 unreleased=0 coverage=0 failing=0

test	branch	main (688c228)	latest (v1.25.0)
TestAccept/pipelines/databricks-cli-help	✅	❌	❌
TestAccept/pipelines/test/arguments	✅	❌	❌
TestAccept/pipelines/test/pipeline-id	✅	❌	❌
TestAccept/pipelines/test/submit-error	✅	❌	❌

TestAccept/pipelines/databricks-cli-help: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help307591585/002/output.txt
        @@ -30,7 +30,6 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  *Preview* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/arguments: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments1763688048/002/output.txt
        @@ -1,61 +1,434 @@
         
         >>> [CLI] pipelines test --help
        -Run the tests of a pipeline in a test-only update.
        +The Lakeflow Spark Declarative Pipelines API allows you to create, edit,
        +  delete, start, and view details about pipelines.
         
        -This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +  Spark Declarative Pipelines is a framework for building reliable,
        +  maintainable, and testable data processing pipelines. You define the
        +  transformations to perform on your data, and Spark Declarative Pipelines
        +  manages task orchestration, cluster management, monitoring, data quality, and
        +  error handling.
         
        -Identify the pipeline with --pipeline-id.
        +  Instead of defining your data pipelines using a series of separate Apache
        +  Spark tasks, Spark Declarative Pipelines manages how your data is transformed
        +  based on a target schema you define for each processing step. You can also
        +  enforce data quality with Spark Declarative Pipelines expectations.
        +  Expectations allow you to define expected data quality and specify how to
        +  handle records that fail those expectations.
         
        -TEST_SELECTION is a test directory, file, or pytest test ID, for example
        -tests/test_orders.py::test_backfill. A relative path is relative to the
        -pipeline's root_path. Paths that start with / are workspace paths. Without a
        -selection, all of the pipeline's tests run.
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -Each --pytest-arg passes one argument to pytest, in the order given. Only -k
        -and -m are supported, each followed by its value in the next --pytest-arg, for
        -example:
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        -  databricks pipelines test --pipeline-id <id> --pytest-arg=-k --pytest-arg='orders and not slow'
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        -Usage:
        -  databricks pipelines test [flags] [TEST_SELECTION...]
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -Pipeline Test Flags:
        -      --pipeline-id string       ID of the pipeline to test.
        -      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -
         Flags:
        -  -h, --help   help for test
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
               --debug            enable debug logging
           -o, --output type      output type: text or json (default text)
           -p, --profile string   ~/.databrickscfg profile
           -t, --target string    bundle target to use (if applicable)
        -      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
         === The command is experimental and off by default
         >>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         === pytest arguments must be passed with --pytest-arg
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
        -Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -- -k orders
        -Error: the -- separator is not supported: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: unknown flag: --pipeline-id
... (266 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/pipeline-id: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/pipeline-id/output.txt
        +++ /tmp/TestAcceptpipelinestestpipeline-id1473081112/002/output.txt
        @@ -1,46 +1,55 @@
         
         === Relative test selections resolve against the pipeline root_path
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][0] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        +>>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID] tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/other/tests
        +Error: unknown flag: --pipeline-id
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_backfill",
        -        "/Workspace/Users/someone@example.com/other/tests"
        -      ]
        -    }
        -  }
        -}
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
         
        -=== Without a root_path, test selections must be workspace paths
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][2] tests
        -Error: cannot resolve test selection "tests": the pipeline has no root_path; use an absolute workspace path such as /Workspace/Users/someone@example.com/tests
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
         
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test --pipeline-id [UUID][2] /Workspace/Users/someone@example.com/tests/test_other.py::test_x
        -Pipeline: [UUID][2]
        -Update: [UUID][3]
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][2]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_other.py::test_x"
        -      ]
        -    }
        -  }
        -}
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
         
        -=== --pipeline-id is required
        ->>> musterr [CLI] pipelines test tests
        -Error: identify the pipeline with --pipeline-id
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

TestAccept/pipelines/test/submit-error: branch ✅ | main (688c228) ❌ | latest (v1.25.0) ❌

main (688c228):

acceptance_test.go:1345: Diff:
        --- pipelines/test/submit-error/output.txt
        +++ /tmp/TestAcceptpipelinestestsubmit-error2532180761/002/output.txt
        @@ -1,13 +1,58 @@
         
         === A submit error from the service is shown as is, and no update ID is printed
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][0]
        -Error: An active update '[UUID][1]' already exists for pipeline '[UUID][0]'.
        +>>> musterr [CLI] pipelines test --pipeline-id [UUID]
        +Error: unknown flag: --pipeline-id
         
        +Usage:
        +  databricks pipelines [flags]
        +  databricks pipelines [command]
        +
        +Main Commands
        +  deploy                Deploy pipelines
        +  destroy               Destroy a pipelines project
        +  dry-run               Validate correctness of the pipeline's graph
        +  generate              Generate pipeline configuration
        +  history               Retrieve past runs for a pipeline
        +  init                  Initialize a new pipelines project
        +  logs                  Retrieve events for a pipeline
        +  open                  Open a pipeline in the browser
        +  run                   Run a pipeline
        +  show                  Preview a table's columns and sample rows
        +  stop                  Stop a pipeline
        +
        +Management Commands
        +  apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        +  clone                 Clone a pipeline.
        +  create                Create a pipeline.
        +  delete                Delete a pipeline.
        +  get                   Get a pipeline.
        +  get-update            Get a pipeline update.
        +  list-pipeline-events  List pipeline events.
        +  list-pipelines        List pipelines.
        +  list-updates          List pipeline updates.
        +  start-update          Start a pipeline.
        +  update                Edit a pipeline.
        +
        +Permission Commands
        +  get-permission-levels Get pipeline permission levels.
        +  get-permissions       Get pipeline permissions.
        +  set-permissions       Set pipeline permissions.
        +  update-permissions    Update pipeline permissions.
        +
        +Flags:
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +
        +Use "databricks pipelines [command] --help" for more information about a command.
        +
        +
         >>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true
        -  }
        -}
        +File [OUTPUT_DIR]/out.requests.txt not found
        +
        +Exit code: 1

latest (v1.25.0): same output as main (688c228)

[eng-dev-ecosystem-bot]

✅ Integration test report

Commit: 62049a1

Run: 42725192482

	Env	✅​pass	🙈​skip	Time
✅​	aws linux-2core-8gb	271	21	4:03
✅​	aws-windows-latest-4core-16gb	273	19	3:17
✅​	azure linux-2core-8gb	270	21	4:03
✅​	azure-windows-latest-4core-16gb	272	19	3:13
✅​	gcp linux-2core-8gb	271	21	4:05
✅​	gcp-windows-latest-4core-16gb	273	19	3:13

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
3:12	aws-windows-latest-4core-16gb	TestAccept
3:08	azure-windows-latest-4core-16gb	TestAccept
3:03	gcp-windows-latest-4core-16gb	TestAccept

[pranshupand-db]

path.Clean quietly turns /Workspace/x/../y into a different path than the user typed, and path.Join in rootPathSelection does the same for relative selections. Could we pass selections through as typed instead? The service already rejects .. and wildcards with its own error, so dropping the normalization is less code and keeps the error about what the user actually wrote. The test at test_test.go:53 would then expect the input unchanged.

[tharunbhuk-db]

Done. Selections now pass through as typed, with no path.Clean or path.Join.

[pranshupand-db]

--pipeline-id goes into the URL path unescaped. Could we validate the ID format, or url.PathEscape it?

[tharunbhuk-db]

Done. --pipeline-id is checked with uuid.Parse, as pipelines history does.

[pranshupand-db]

Question: with --pipeline-id inside a project, the bundle isn't loaded, so the bundle's host and profile are ignored, and the command can talk to a different workspace than pipelines run would. Is that intended?

[tharunbhuk-db]

Yes, intended. It follows pipelines history <id>: with an ID and no --target, the bundle isn't loaded. Pass --target to use the bundle's workspace.

[pranshupand-db]

Public text:

• Short says "Preview" while Long and the gate say "experimental". Could we pick one label and name the env var in the help?
• Other strings:
  • "outside a project" → "Run this command from a bundle directory, or pass --pipeline-id."
  • "synchronized bundle files" → "%s is not in the files synced from %s; pass its /Workspace path."
  • The immutable error → "Relative test paths need synced local files, but workspace.sync_mode is immutable; pass the /Workspace path."

[tharunbhuk-db]

I'll do this in a follow-up.

[pranshupand-db]

Nit: --pytest-arg="" lands here and reports unsupported pytest argument "": pass test paths..., which is misleading for an empty value. Since this flag's rules are the CLI's own, an explicit empty-value message would help. Also nit: the Use string puts selections before flags, while sibling commands use run [flags] [KEY].

[tharunbhuk-db]

Done. An empty value now gets its own error, and the usage is test [flags] [TEST_SELECTION...].

[pranshupand-db]

In a one-pipeline project, pipelines test tests/test_orders.py takes the path as the KEY. The rule is fine, but the error could name the pipeline and show the working command, so an agent can self-correct in one step: "tests/test_orders.py" is not a pipeline in this bundle (pipelines: my_pipeline). Usage: databricks pipelines test my_pipeline tests/test_orders.py.

[tharunbhuk-db]

Done in #7431. The error lists the bundle's pipelines and shows the working command.

[pranshupand-db]

Agents and CI scripts learn the command from --help, which has one example. Could the Long text cover the common flows and the outcome contract?

• Examples: run all tests of a pipeline; run one file; re-run one failing test by ID; CI with --junit-xml; resume after --timeout.
• A short exit-code paragraph: when it exits 0, and that failures, incomplete runs and errors exit 1.
• Both env vars needed while the feature is experimental.

[tharunbhuk-db]

The exit codes and the env var are in the help from #7298. I'll add more examples in a follow-up.

[tharunbhuk-db]

Follow-ups from review:

• Use one label (Preview or experimental) in Short, Long and the gate error, and name the env var in the help. (thread)
• Reword the errors for: outside a project, a path outside the synced files, and workspace.sync_mode immutable. These are in Run a bundle pipeline's tests by its KEY #7431. (thread)
• Add --help examples: all tests of a pipeline, one file, one test ID, CI with --junit-xml, resume after --timeout. (thread)

[pranshupand-db]

Thanks Tharun for re-splitting the stack and working through every thread. The smaller PRs made this much easier to review. LGTM for the whole stack (#7297 → #7431 → #7432 → #7433 → #7298 → #7426 → #7299 → #7434).

What I checked:

• No false pass. Exit 0 requires all of these: a COMPLETED update, exactly one summary, one final result per case that matches the summary, at least one case, nothing failed or errored, and no fatal error from the update. The earlier blockers are fixed: fatalError on success, Ctrl+C during Start Update, atomic JUnit writes, [ in IDs, and --pipeline-id validation.
• Smoke run. I built #7434 (8b690b3) and ran it against dogfood staging with --pipeline-id. It handled pass/fail/skip/xfail/fixture/teardown/parametrized outcomes, a passing selection, zero collected, an unknown test ID, --timeout then --resume, --no-wait, and a second start while one is active. Every case gave the expected exit code and JUnit report. Example 1 passes 5/5.

Merge note: #7297 through #7433 return after Start Update without waiting, so let's land the stack together before a CLI release cuts between them. The gate covers it, but it's cleaner that way.

Follow-ups (not blocking, can you track them in a ticket?):

1. Headline user errors by their error class instead of "RUN INCOMPLETE". Today an unknown test ID (PIPELINE_TEST_RUN_REFUSED) or an import error in a test file reads as incomplete, so agents and CI retry instead of fixing the test. Still reproduces on 8b690b3.
2. Add next-step hints: the --resume command on INCOMPLETE and on a failed GetUpdate/events read, a next step for "an active update already exists" (--resume <id> or pipelines stop; pipelines run has --restart), and the selection/filters in "No tests were collected".
3. Agree on one rule with the pytest plugin (universe #2932592) for a test case pytest runs twice. The plugin sends a second final status, which incompleteReason rejects as "has more than one result" (test_events.go:284).
4. Help and public text: more examples in --help, one "Preview"/"experimental" label, and the deploy recommendation (it names pipelines deploy in bundles and prints on every run).
5. JUnit: stable classnames with sync_mode: immutable, one naming scheme for the synthetic entries, and skip/xfail reasons in <skipped message=…>.
6. The output is now dbt-style (1 of 11 PASSED …, Done. PASSED=4 FAILED=3 … TOTAL=11), while the design doc and the bugbash doc show 11 tests: 4 passed, 3 failed, …. Fine if that was intentional; let's update both docs to match.

The data-plane items from the bugbash (--assert plain, redacted paths, ANSI codes, captured stdout) are tracked in the findings doc, separate from this stack.

[eng-dev-ecosystem-bot]

❌ Integration test report

Commit: 4528983

Run: 42913787390

	Env	❌​FAIL	🔄​flaky	✅​pass	🙈​skip	Time
❌​	aws linux-2core-8gb	4		1527	1271	153:38
❌​	aws-windows-latest-4core-16gb	4		1446	1298	176:24
❌​	azure linux-2core-8gb	4		1507	1279	127:44
❌​	azure-windows-latest-4core-16gb	4		1426	1306	144:57
❌​	gcp linux-2core-8gb	4	4	1510	1277	137:17
❌​	gcp-windows-latest-4core-16gb	4	8	1425	1304	167:29

14 interesting tests: 10 flaky, 4 FAIL

	Test Name	aws linux-2core-8gb	aws-windows-latest-4core-16gb	azure linux-2core-8gb	azure-windows-latest-4core-16gb	gcp linux-2core-8gb	gcp-windows-latest-4core-16gb
❌​	TestAccept	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
🔄​	TestAccept/bundle/debug/fetch-repository-info/DMS=	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
🔄​	TestAccept/bundle/debug/fetch-repository-info/DMS=true	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
🔄​	TestAccept/bundle/dms/depends-on	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
🔄​	TestAccept/bundle/dms/depends-on/DMS=true	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
❌​	TestAccept/bundle/resources/alerts/basic	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/resources/alerts/basic/DMS=	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/resources/alerts/basic/DMS=true	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
🔄​	TestAccept/bundle/resources/clusters/deploy/update-after-create	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
🔄​	TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
🔄​	TestAccept/bundle/resources/postgres_synced_tables/basic	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
🔄​	TestAccept/bundle/resources/postgres_synced_tables/basic/DMS=	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p
🔄​	TestAccept/bundle/resources/postgres_synced_tables/recreate	✅​p	✅​p	✅​p	✅​p	🔄​f	🔄​f
🔄​	TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=	✅​p	✅​p	✅​p	✅​p	🔄​f	🔄​f

Top 50 slowest tests (at least 2 minutes):

duration	env	testname
21:36	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
18:18	aws linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
15:06	aws linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
14:23	aws linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
14:08	gcp linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
13:52	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
13:23	aws linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
12:42	gcp linux-2core-8gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=
12:39	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
12:25	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
12:04	gcp linux-2core-8gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
11:26	azure linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:08	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:01	aws linux-2core-8gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:27	aws linux-2core-8gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:12	aws-windows-latest-4core-16gb	TestAccept/bundle/deploy/spark-jar-task/DMS=
9:09	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:01	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:00	azure linux-2core-8gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:55	aws-windows-latest-4core-16gb	TestAccept/bundle/deploy/spark-jar-task/DMS=true
8:48	azure linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
8:39	azure-windows-latest-4core-16gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:36	azure-windows-latest-4core-16gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:58	gcp linux-2core-8gb	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:56	azure linux-2core-8gb	TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:56	azure linux-2core-8gb	TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=
7:42	aws-windows-latest-4core-16gb	TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
7:24	gcp linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:20	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:12	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
7:12	gcp linux-2core-8gb	TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
7:11	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:08	aws linux-2core-8gb	TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
7:06	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:06	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:02	gcp linux-2core-8gb	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:59	aws-windows-latest-4core-16gb	TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:56	azure linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:55	gcp linux-2core-8gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:49	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/deploy/simple/DMS=
6:49	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:41	gcp-windows-latest-4core-16gb	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
6:40	aws linux-2core-8gb	TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:35	aws linux-2core-8gb	TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:23	azure-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:21	aws linux-2core-8gb	TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:19	aws-windows-latest-4core-16gb	TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=
6:18	gcp linux-2core-8gb	TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:18	azure linux-2core-8gb	TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:56	azure linux-2core-8gb	TestAccept/bundle/config-remote-sync/multiple_resources/DMS=