Skip to content

Run pipelines test against the pipeline's testing.sources - #7429

Draft
tharunbhuk-db wants to merge 7 commits into
tharun/pipelines-test-changelogfrom
tharun/pipelines-test-j1-testing-sources
Draft

Run pipelines test against the pipeline's testing.sources#7429
tharunbhuk-db wants to merge 7 commits into
tharun/pipelines-test-changelogfrom
tharun/pipelines-test-j1-testing-sources

Conversation

@tharunbhuk-db

@tharunbhuk-db tharunbhuk-db commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🥞 Stacked PR

Review only the last commit (Run pipelines test against the pipeline's testing.sources). The merge commit before it brings in the pipelines test command stack (#7297 → #7434), so CI can run. When that stack merges, I rebase this PR and the merge goes away.

  • The pipelines test acceptance bundles declare testing, as the service requires testing.sources.
  • When Start Update fails with PIPELINE_TEST_NO_TEST_SOURCES and the bundle sets testing, the error says to deploy first.
  • A live test deploys a bundle with testing and a local test dependency after -e, and runs its tests. Locally it uses stubs; on a real workspace it runs only with TEST_PIPELINES_TESTING_ENABLED.
  • I'm not sure of the service's error shape for PIPELINE_TEST_NO_TEST_SOURCES, so the hint matches the class in the error code or the message.

Tests: TestWithDeployHint; acceptance pipelines/test/no-test-sources and pipelines/test/live.

This pull request and its description were written by Isaac.

This was referenced Nov 10, 2026
Merged
Draft
Open
Open
Open
Open
Draft
Draft
@github-actions

github-actions Bot commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: f5109d518 | Workflow run

Regression test report

Selected 19 changed tests (limit=20, 0 not selected)

23 tests: regression=23 unreleased=0 coverage=0 failing=0
test branch main (cec3f63) latest (v1.26.0)
TestAccept/bundle/environments/dependencies/DMS= ✅ ❌ ❌
TestAccept/bundle/environments/dependencies/DMS=true ✅ ❌ ❌
TestAccept/bundle/refschema/DMS= ✅ ❌ ❌
TestAccept/bundle/refschema/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/pipelines/testing/DMS=true ✅ ❌ ❌
TestAccept/pipelines/test/arguments ✅ ❌ ❌
TestAccept/pipelines/test/immutable ✅ ❌ ❌
TestAccept/pipelines/test/invalid-project ✅ ❌ ❌
TestAccept/pipelines/test/live ✅ ❌ ❌
TestAccept/pipelines/test/no-test-sources ✅ ❌ ❌
TestAccept/pipelines/test/pipeline-id ✅ ❌ ❌
TestAccept/pipelines/test/results/failed ✅ ❌ ❌
TestAccept/pipelines/test/results/incomplete ✅ ❌ ❌
TestAccept/pipelines/test/results/passed ✅ ❌ ❌
TestAccept/pipelines/test/run-control/canceled ✅ ❌ ❌
TestAccept/pipelines/test/run-control/not-test-update ✅ ❌ ❌
TestAccept/pipelines/test/run-control/resume ✅ ❌ ❌
TestAccept/pipelines/test/run-control/timeout ✅ ❌ ❌
TestAccept/pipelines/test/selection ✅ ❌ ❌
TestAccept/pipelines/test/submit-error ✅ ❌ ❌
TestAccept/bundle/environments/dependencies/DMS=: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/environments/dependencies/output.txt
        +++ /tmp/TestAcceptbundleenvironmentsdependenciesDMS=2114833029/002/output.txt
        @@ -1,150 +1,13 @@
         
         >>> [CLI] bundle validate
        +Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory
        +
         Name: dependencies
         Target: default
         Workspace:
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/dependencies/default
         
        -Validation OK!
        +Found 1 error
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files...
        -Created jobs.test_job
        -Created pipelines.test_pipeline
        -Files: 6 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> jq -s .[] | select(.path=="/api/2.2/jobs/create") | .body.environments [OUTPUT_DIR]/out.requests.txt
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> jq -s .[] | select(.path=="/api/2.0/pipelines") [OUTPUT_DIR]/out.requests.txt
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "body": {
        -    "channel": "CURRENT",
        -    "deployment": {
        -      "kind": "BUNDLE",
        -      "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/state/metadata.json"
        -    },
        -    "edition": "ADVANCED",
        -    "environment": {
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    },
        -    "name": "Test Pipeline"
        -  }
        -}
        -
        ->>> [CLI] bundle validate -o json
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "./dist/*.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> [CLI] bundle validate -o json
        -{
        -  "dependencies": [
        -    "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -    "-i http://myindexurl.com",
        -    "--index-url http://myindexurl.com",
        -    "./dist/*.whl",
        -    "/Workspace/Users/test@databricks.com/test-package.whl",
        -    "test_package",
        -    "test_package==2.0.1",
        -    "test_package>=2.0.1",
        -    "test_package==2.0.*",
        -    "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -    "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -    "--find-links=/Workspace/Users/test"
        -  ]
        -}
        +Exit code: 1

latest (v1.26.0): same output as main (cec3f63)

TestAccept/bundle/environments/dependencies/DMS=true: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/environments/dependencies/output.txt
        +++ /tmp/TestAcceptbundleenvironmentsdependenciesDMS=true930280815/002/output.txt
        @@ -1,150 +1,13 @@
         
         >>> [CLI] bundle validate
        +Error: unable to determine if [TEST_TMP_DIR]/project is not a notebook: read [TEST_TMP_DIR]/project: is a directory
        +
         Name: dependencies
         Target: default
         Workspace:
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/dependencies/default
         
        -Validation OK!
        +Found 1 error
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files...
        -Created jobs.test_job
        -Created pipelines.test_pipeline
        -Files: 6 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> jq -s .[] | select(.path=="/api/2.2/jobs/create") | .body.environments [OUTPUT_DIR]/out.requests.txt
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> jq -s .[] | select(.path=="/api/2.0/pipelines") [OUTPUT_DIR]/out.requests.txt
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "body": {
        -    "channel": "CURRENT",
        -    "deployment": {
        -      "kind": "BUNDLE",
        -      "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/state/metadata.json"
        -    },
        -    "edition": "ADVANCED",
        -    "environment": {
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "/Workspace/Users/[USERNAME]/.bundle/dependencies/default/artifacts/.internal/test.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    },
        -    "name": "Test Pipeline"
        -  }
        -}
        -
        ->>> [CLI] bundle validate -o json
        -[
        -  {
        -    "environment_key": "test_env",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -        "-r \"/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt\"",
        -        "-r '/Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/Path With Spaces/requirements.txt'",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -        "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -        "-i http://myindexurl.com",
        -        "--index-url http://myindexurl.com",
        -        "test_package",
        -        "test_package==2.0.1",
        -        "test_package>=2.0.1",
        -        "test_package==2.0.*",
        -        "./dist/*.whl",
        -        "/Workspace/Users/test@databricks.com/test-package.whl",
        -        "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -        "beautifulsoup4[security, tests] ~= 4.12.3",
        -        "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -        "--find-links=/Workspace/Users/test"
        -      ]
        -    }
        -  },
        -  {
        -    "environment_key": "test_env_2",
        -    "spec": {
        -      "client": "1",
        -      "dependencies": [
        -        "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt"
        -      ]
        -    }
        -  }
        -]
        -
        ->>> [CLI] bundle validate -o json
        -{
        -  "dependencies": [
        -    "-r /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/requirements.txt",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/file.py",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project",
        -    "-e /Workspace/Users/[USERNAME]/.bundle/dependencies/default/files/project[test]",
        -    "-i http://myindexurl.com",
        -    "--index-url http://myindexurl.com",
        -    "./dist/*.whl",
        -    "/Workspace/Users/test@databricks.com/test-package.whl",
        -    "test_package",
        -    "test_package==2.0.1",
        -    "test_package>=2.0.1",
        -    "test_package==2.0.*",
        -    "beautifulsoup4>=1.0.0,~=1.2.0,<2.0.0",
        -    "requests[security] @ https://github.com/psf/requests/archive/refs/heads/main.zip",
        -    "--find-links=/Workspace/Users/test"
        -  ]
        -}
        +Exit code: 1

latest (v1.26.0): same output as main (cec3f63)

TestAccept/bundle/refschema/DMS=: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=1515413618/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	ALL
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	ALL
        -resources.pipelines.*.testing.environment.dependencies	[]string	ALL
        -resources.pipelines.*.testing.environment.dependencies[*]	string	ALL
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	ALL
        -resources.pipelines.*.testing.sources[*].glob.include	string	ALL
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=2604451586/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	ALL
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	ALL
        -resources.pipelines.*.testing.environment.dependencies	[]string	ALL
        -resources.pipelines.*.testing.environment.dependencies[*]	string	ALL
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	ALL
        -resources.pipelines.*.testing.sources[*].glob.include	string	ALL
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL
        
    acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/catalogs.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=2604451586/001/output/catalogs.txt
        @@ -12,7 +12,7 @@
         resources.catalogs.*.enable_predictive_optimization	catalog.EnablePredictiveOptimization	REMOTE
         resources.catalogs.*.full_name	string	REMOTE
         resources.catalogs.*.id	string	INPUT
        -resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	ALL
        +resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	REMOTE
         resources.catalogs.*.lifecycle	resources.Lifecycle	INPUT
         resources.catalogs.*.lifecycle.depends_on	[]resources.DependsOn	INPUT
         resources.catalogs.*.lifecycle.depends_on[*]	resources.DependsOn	INPUT
TestAccept/bundle/refschema/DMS=true: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=true2601358047/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	ALL
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	ALL
        -resources.pipelines.*.testing.environment.dependencies	[]string	ALL
        -resources.pipelines.*.testing.environment.dependencies[*]	string	ALL
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	ALL
        -resources.pipelines.*.testing.sources[*].glob.include	string	ALL
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL
        
FAIL

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/catalogs.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=true3672271687/001/output/catalogs.txt
        @@ -12,7 +12,7 @@
         resources.catalogs.*.enable_predictive_optimization	catalog.EnablePredictiveOptimization	REMOTE
         resources.catalogs.*.full_name	string	REMOTE
         resources.catalogs.*.id	string	INPUT
        -resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	ALL
        +resources.catalogs.*.isolation_mode	catalog.CatalogIsolationMode	REMOTE
         resources.catalogs.*.lifecycle	resources.Lifecycle	INPUT
         resources.catalogs.*.lifecycle.depends_on	[]resources.DependsOn	INPUT
         resources.catalogs.*.lifecycle.depends_on[*]	resources.DependsOn	INPUT
        
    acceptance_test.go:1345: Diff:
        --- bundle/refschema/output/pipelines.txt
        +++ /tmp/TestAcceptbundlerefschemaDMS=true3672271687/001/output/pipelines.txt
        @@ -867,14 +867,6 @@
         resources.pipelines.*.tags	map[string]string	ALL
         resources.pipelines.*.tags.*	string	ALL
         resources.pipelines.*.target	string	ALL
        -resources.pipelines.*.testing	*resources.PipelineTesting	ALL
        -resources.pipelines.*.testing.environment	*resources.PipelineTestEnvironment	ALL
        -resources.pipelines.*.testing.environment.dependencies	[]string	ALL
        -resources.pipelines.*.testing.environment.dependencies[*]	string	ALL
        -resources.pipelines.*.testing.sources	[]resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*]	resources.PipelineTestSource	ALL
        -resources.pipelines.*.testing.sources[*].glob	*pipelines.PathPattern	ALL
        -resources.pipelines.*.testing.sources[*].glob.include	string	ALL
         resources.pipelines.*.trigger	*pipelines.PipelineTrigger	ALL
         resources.pipelines.*.trigger.cron	*pipelines.CronTrigger	ALL
         resources.pipelines.*.trigger.cron.quartz_cron_schedule	string	ALL
        
FAIL
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie596283554/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/8359fc3806db7812.json

latest (v1.26.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie2115273741/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/2c558cd376fb9202.json
TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie1062551565/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/4d5e7d7a7b660731.json

latest (v1.26.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> musterr [CLI] bundle deploy\nError: Duplicate local library names: test_helpers-0.1-py3-none-any.whl\n  at resources.pipelines.orders.environment.dependencies[0]\n     resources.pipelines.orders.testing.environment.dependencies[0]\n  in databricks.yml:13:13\n     databricks.yml:21:15\n\nLocal library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl\n\n"
        	            	actual  : "\n>>> musterr [CLI] bundle deploy\nWarning: unknown field: testing\n  at resources.pipelines.orders\n  in databricks.yml:14:7\n\nUploading pipeline/test_helpers-0.1-py3-none-any.whl...\nUploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...\nCreated pipelines.orders\nFiles: 5 uploaded, 0 deleted\nResources: 1 created, 0 changed, 0 deleted, 0 unchanged\n\nUnexpected success\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,10 +2,15 @@
        	            	 >>> musterr [CLI] bundle deploy
        	            	-Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
        	            	-  at resources.pipelines.orders.environment.dependencies[0]
        	            	-     resources.pipelines.orders.testing.environment.dependencies[0]
        	            	-  in databricks.yml:13:13
        	            	-     databricks.yml:21:15
        	            	+Warning: unknown field: testing
        	            	+  at resources.pipelines.orders
        	            	+  in databricks.yml:14:7
        	            	 
        	            	-Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl
        	            	+Uploading pipeline/test_helpers-0.1-py3-none-any.whl...
        	            	+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-same-name-libraries/default/files...
        	            	+Created pipelines.orders
        	            	+Files: 5 uploaded, 0 deleted
        	            	+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        	            	 
        	            	+Unexpected success
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/bundle/resources/pipelines/testing-same-name-libraries/DMS=true
        	Messages:   	bundle/resources/pipelines/testing-same-name-libraries/output.txt vs /tmp/TestAcceptbundleresourcespipelinestesting-same-name-librarie1051557164/002/output.txt
    acceptance_test.go:1135: Test produced unexpected files:
        .databricks/.gitignore
        .databricks/bundle/default/deployment.json
        .databricks/bundle/default/resources.json
        .databricks/bundle/default/sync-snapshots/1124ebe8c2834f25.json
TestAccept/bundle/resources/pipelines/testing/DMS=: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/resources/pipelines/testing/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinestestingDMS=108683231/002/output.txt
        @@ -2,29 +2,18 @@
         === Relative test paths resolve to the deployed workspace paths
         
         >>> [CLI] bundle validate -o json
        -{
        -  "environment": {
        -    "dependencies": [
        -      "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -      "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -      "./dist/*.whl"
        -    ]
        -  },
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -      }
        -    }
        -  ]
        -}
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
        +null
        +
         === Without pipeline testing, the property is an error
         
         >>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate
        -Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
        -  at resources.pipelines.orders.testing
        -  in databricks.yml:12:9
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
         Name: pipeline-testing
         Target: default
        @@ -32,116 +17,8 @@
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
         
        -Found 1 error
        +Found 1 warning
         
        -=== Deploy sends testing, with the local wheel uploaded
        +Unexpected success
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Created pipelines.orders
        -Files: 6 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "testing": {
        -    "sources": [
        -      {
        -        "glob": {
        -          "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -        }
        -      }
        -    ],
        -    "environment": {
        -      "dependencies": [
        -        "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -        "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -        "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/artifacts/.internal/test_helpers-0.1-py3-none-any.whl"
        -      ]
        -    }
        -  }
        -}
        -
        -=== No changes after the deploy
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== A remote edit that drops testing shows in the plan
        -
        ->>> [CLI] bundle plan -o json
        -"update"
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== Changing testing updates the pipeline
        -
        ->>> update_file.py databricks.yml ./tests/** ./tests/unit/**
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/unit/**"
        -      }
        -    }
        -  ]
        -}
        -
        -=== Removing testing sends an update without it
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "testing": null
        -}
        -
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.pipelines.orders
        -
        -This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -  delete resources.pipelines.orders
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
        -
        -Destroy: 1 deleted
        +Exit code: 1

latest (v1.26.0): same output as main (cec3f63)

TestAccept/bundle/resources/pipelines/testing/DMS=true: branch ✅ | main (cec3f63) ❌ | latest (v1.26.0) ❌

main (cec3f63):

acceptance_test.go:1345: Diff:
        --- bundle/resources/pipelines/testing/output.txt
        +++ /tmp/TestAcceptbundleresourcespipelinestestingDMS=true3977155895/002/output.txt
        @@ -2,29 +2,18 @@
         === Relative test paths resolve to the deployed workspace paths
         
         >>> [CLI] bundle validate -o json
        -{
        -  "environment": {
        -    "dependencies": [
        -      "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -      "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -      "./dist/*.whl"
        -    ]
        -  },
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -      }
        -    }
        -  ]
        -}
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
        +null
        +
         === Without pipeline testing, the property is an error
         
         >>> musterr env -u DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING [CLI] bundle validate
        -Error: the testing property of pipelines is experimental. Set DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_PIPELINE_TESTING=true to enable it
        -  at resources.pipelines.orders.testing
        -  in databricks.yml:12:9
        +Warning: unknown field: testing
        +  at resources.pipelines.orders
        +  in databricks.yml:11:7
         
         Name: pipeline-testing
         Target: default
        @@ -32,116 +17,8 @@
           User: [USERNAME]
           Path: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
         
        -Found 1 error
        +Found 1 warning
         
        -=== Deploy sends testing, with the local wheel uploaded
        +Unexpected success
         
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Created pipelines.orders
        -Files: 6 uploaded, 0 deleted
        -Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines",
        -  "testing": {
        -    "sources": [
        -      {
        -        "glob": {
        -          "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
        -        }
        -      }
        -    ],
        -    "environment": {
        -      "dependencies": [
        -        "-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
        -        "-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
        -        "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/artifacts/.internal/test_helpers-0.1-py3-none-any.whl"
        -      ]
        -    }
        -  }
        -}
        -
        -=== No changes after the deploy
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== A remote edit that drops testing shows in the plan
        -
        ->>> [CLI] bundle plan -o json
        -"update"
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
        -
        -=== Changing testing updates the pipeline
        -
        ->>> update_file.py databricks.yml ./tests/** ./tests/unit/**
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading dist/test_helpers-0.1-py3-none-any.whl...
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "sources": [
        -    {
        -      "glob": {
        -        "include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/unit/**"
        -      }
        -    }
        -  ]
        -}
        -
        -=== Removing testing sends an update without it
        -
        ->>> [CLI] bundle plan
        -update pipelines.orders
        -
        -Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged
        -
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
        -Updated pipelines.orders
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        ->>> print_requests.py //api/2.0/pipelines --nostamp
        -{
        -  "method": "PUT",
        -  "path": "/api/2.0/pipelines/[ORDERS_ID]",
        -  "testing": null
        -}
        -
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.pipelines.orders
        -
        -This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -  delete resources.pipelines.orders
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default
        -
        -Destroy: 1 deleted
        +Exit code: 1

latest (v1.26.0): same output as main (cec3f63)

15 more failure details omitted, see the full report in the run.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: 453fa56

Run: 43092531796

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from ed5750c to 75f2713 Compare November 10, 2026 01:16
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-j1-base to tharun/pipelines-test-b3-dependency-checks November 10, 2026 01:16
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 75f2713 to ccdd807 Compare November 10, 2026 10:07
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-b3-dependency-checks to tharun/pipelines-test-b4-testing-deploy November 10, 2026 10:07
This was referenced Nov 10, 2026
Draft
Draft
Draft
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b4-testing-deploy branch from 1fd5e11 to 3af4a58 Compare November 10, 2026 10:46
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch 2 times, most recently from 2e9b82e to a149708 Compare November 10, 2026 11:11
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b4-testing-deploy branch from 3af4a58 to 7498e1b Compare November 10, 2026 16:49
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch 2 times, most recently from 490849a to 6c31046 Compare November 10, 2026 17:14
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-b4-testing-deploy branch from 7498e1b to e79333b Compare November 10, 2026 17:22
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 6c31046 to bbb4adb Compare November 10, 2026 17:22
@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 10, 2026 18:11
@tharunbhuk-db
tharunbhuk-db requested review from a team as code owners November 10, 2026 18:11
@tharunbhuk-db
tharunbhuk-db removed request for a team and andrewnester November 10, 2026 18:12
pranshupand-db
pranshupand-db approved these changes Nov 10, 2026
View reviewed changes

@pranshupand-db pranshupand-db left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Merge-order note: this PR still carries Stack A's pipelines test code (3,246 lines), and its own change is the roughly 49-line deploy hint for PIPELINE_TEST_NO_TEST_SOURCES. Let's rebase it once #7431 → #7434 land, so the command doesn't land twice or conflict. The summary and notes for the stack are on #7300: #7300

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from bbb4adb to e6963d5 Compare November 11, 2026 00:21
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-b4-testing-deploy to tharun/pipelines-test-changelog November 11, 2026 00:21
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from e6963d5 to 453fa56 Compare November 11, 2026 19:58
@tharunbhuk-db
tharunbhuk-db requested a review from a team as a code owner November 11, 2026 19:58
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharun/pipelines-test-changelog to tharunbhuk-db/stack/pipelines-test-changelog November 11, 2026 19:58
@tharunbhuk-db
tharunbhuk-db changed the base branch from tharunbhuk-db/stack/pipelines-test-changelog to tharun/pipelines-test-changelog November 11, 2026 20:00
@tharunbhuk-db
tharunbhuk-db marked this pull request as draft November 11, 2026 20:01
tharunbhuk-db and others added 7 commits November 11, 2026 21:58
@tharunbhuk-db Isaac
Read the test events of a pipeline update
e178511 
readTestEvents reads every page of an update's events and keeps the test fields
that the Go SDK event type does not have. It sends the SDK's
ListPipelineEventsRequest and uses the filter of buildPipelineEventFilter.
fetchAllPipelineEvents, which pipelines logs and run use, is not changed.
testRun collects one result per test case and reports whether the results are
complete. The command does not use them yet.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Print test progress and results
45a257b 
Each finished case prints "<i> of <n> <STATUS> <test ID>", padded with dots to
80 columns, then "[<STATUS> in <s>s]", so the statuses line up while each case
prints as it ends. After the run, the details of every failed or errored case
follow, then the reason for an incomplete run, and the run ends with
"Done. PASSED=… FAILED=… SKIPPED=… ERROR=… TOTAL=…". The command does not use
the printer yet.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Wait for and print the results of databricks pipelines test
6c6bbaa 
The command waits for the test-only update, prints each test case as it
finishes, then the failure details and a summary line. It exits with code 0
only for a complete run where at least one test ran and none failed or errored,
and a completed update that reports a fatal error does not pass. Test IDs are
shown relative to the project folder. With the no-wait flag, the command returns
after the update starts.

The wait takes the steps of the bundle pipeline runner's wait loop. The
acceptance tests stub the service per case, as the other pipelines tests do.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Add --resume and --timeout to databricks pipelines test
c7fc800 
--timeout and Ctrl+C end only the local wait and print the command that reads
the results later; the update keeps running. That command keeps the KEY or
--pipeline-id, --target, --profile and every --var. --resume <update-id> reads
the results of an existing test update instead of starting one, and rejects an
update that the service reports is not a test update.

The signal handler is installed before the update starts, but only the wait
listens to it, so a Ctrl+C during Start Update still prints the update ID. The
first signal releases the handler, so a second one ends the process.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Build a JUnit XML report from the results of a test run
b88ef55 
The report has one testcase per collected test case, named as pytest's JUnit
report names it, with its outcome and duration. A case without a final outcome
is an error, and a run problem, such as an incomplete run, adds a failing entry.
A bracket in a folder name stays part of the path, and the file is replaced
atomically so that a reader never sees a partly written report. The command
does not write the report yet.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Write a JUnit XML report from databricks pipelines test
80497f2 
--junit-xml writes one testcase per collected test case, with its outcome and
duration; an incomplete run, a run without tests, or a timeout adds a failing
entry that names the cause. The report replaces the file on every exit after
the flags parse, including the disabled-command error and project errors, so
an older report never survives; a flag that cannot be parsed ends the command
before it runs. --junit-xml cannot be combined with the no-wait flag, and the
printed resume command keeps --junit-xml so that it replaces a partial report.
The report line prints before the summary, so the summary stays the last line.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db Isaac
Run pipelines test against the pipeline's testing.sources
f5109d5 
The pipelines test acceptance bundles declare testing, as the service requires
testing.sources for a test-only update.

When Start Update fails with PIPELINE_TEST_NO_TEST_SOURCES and the bundle sets
testing for the pipeline, the error says to deploy first: the bundle changed
after its last deploy.

Add a live acceptance test that deploys a bundle with testing, including a local
test dependency after -e, and runs its tests. Locally it uses stubs for the
test run. Against a real workspace it runs only when
TEST_PIPELINES_TESTING_ENABLED is set.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-changelog branch from ffee694 to adc99fc Compare November 11, 2026 22:00
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-j1-testing-sources branch from 453fa56 to f5109d5 Compare November 11, 2026 22:00
@tharunbhuk-db
tharunbhuk-db removed request for a team and Sankalp-Mittal November 11, 2026 22:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pranshupand-db pranshupand-db pranshupand-db approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tharunbhuk-db @eng-dev-ecosystem-bot @pranshupand-db