Skip to content

[draft] Allow target overrides for complex-variable job tasks - #7347

Open
lennartkats-db wants to merge 2 commits into
mainfrom
factory/dabs/issue-2436-task-variable-overrides
Open

[draft] Allow target overrides for complex-variable job tasks#7347
lennartkats-db wants to merge 2 commits into
mainfrom
factory/dabs/issue-2436-task-variable-overrides

Conversation

@lennartkats-db

@lennartkats-db lennartkats-db commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Changes

Defer merging target job task overrides until complex variable values are assigned. Keep the target tasks visible to configuration rewrites before that merge. Add regression coverage for shared jobs, variable-file task lists, and workspace paths, and update the issue #2436 acceptance fixture.

Why

Issue #2436 reports cannot merge string with sequence when a target overrides a task in a job whose tasks come from a complex variable. The target merge sees the unresolved variable string. This change lets validation apply the target task override while retaining the other tasks. It covers direct complex-variable task sequences; transitive variable references remain outside this change.

Tests

  • go test ./bundle/config/... -count=1 passed at commit 9d9f63a.
  • go test ./acceptance -run TestAccept/bundle/variables/issue_2436 -tail -test.v passed both local variants.
  • The new TestSelectTargetMergesComplexVariableTaskOverride compiles and fails on base 3aa5c96 with the reported merge error, then passes at 9d9f63a.
  • ./task checks passed. Full Go lint reported 0 issues. Python formatting and linting could not fetch Ruff because PyPI DNS lookup failed.
  • Full ./task test is red locally in libs/databrickscfg (/dev/databrickscfg exists) and libs/patchwheel (uv system-config warnings are treated as test errors). The changed bundle/config/... packages pass separately.

This PR was written by Codex.

@github-actions github-actions Bot added the DABs DABs related issues label Nov 4, 2026
@lennartkats-db lennartkats-db added the software-factory PR produced by the software factories task label Nov 4, 2026
@lennartkats-db
lennartkats-db marked this pull request as ready for review November 4, 2026 14:36
@lennartkats-db
lennartkats-db requested review from a team as code owners November 4, 2026 14:36
@lennartkats-db
lennartkats-db requested review from Sankalp-Mittal and rclarey and removed request for Sankalp-Mittal and rclarey November 4, 2026 14:36
@github-actions

github-actions Bot commented Nov 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 98256b344 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (eb1328d) latest (v1.24.0)
TestAccept/bundle/variables/issue_2436/DMS= ✅ ❌ ❌
TestAccept/bundle/variables/issue_2436/DMS=true ✅ ❌ ❌
TestAccept/bundle/variables/issue_2436/DMS=: branch ✅ | main (eb1328d) ❌ | latest (v1.24.0) ❌

main (eb1328d):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,24 +4,7 @@
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	+
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=1375562813/002/output.txt

latest (v1.24.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,26 +2,7 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	-Warning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	 
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=4000496220/002/output.txt
TestAccept/bundle/variables/issue_2436/DMS=true: branch ✅ | main (eb1328d) ❌ | latest (v1.24.0) ❌

main (eb1328d):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,24 +4,7 @@
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	+
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=true
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=true1397646324/002/output.txt
FAIL

latest (v1.24.0):

acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] bundle validate -o json\nWarning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]\n\n{\n  \"job_one\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Medium\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ],\n  \"job_two\": [\n    {\n      \"task_key\": \"Task_One\",\n      \"job_cluster_key\": \"Small\"\n    },\n    {\n      \"task_key\": \"Task_Two\",\n      \"job_cluster_key\": \"Small\"\n    }\n  ]\n}\n"
        	            	actual  : "\n>>> [CLI] bundle validate -o json\nError: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence\n\njq: error (at <stdin>:98): Cannot iterate over string (\"${var.job_...)\n\nExit code: 5\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -2,26 +2,7 @@
        	            	 >>> [CLI] bundle validate -o json
        	            	-Warning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.229.0, current: [CLI_VERSION]
        	            	+Error: failed to perform target override for target=dev: failed to merge target=dev field=resources: cannot merge string with sequence
        	            	 
        	            	-{
        	            	-  "job_one": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Medium"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ],
        	            	-  "job_two": [
        	            	-    {
        	            	-      "task_key": "Task_One",
        	            	-      "job_cluster_key": "Small"
        	            	-    },
        	            	-    {
        	            	-      "task_key": "Task_Two",
        	            	-      "job_cluster_key": "Small"
        	            	-    }
        	            	-  ]
        	            	-}
        	            	+jq: error (at <stdin>:98): Cannot iterate over string ("${var.job_...)
        	            	 
        	            	+Exit code: 5
        	            	+
        	Test:       	TestAccept/bundle/variables/issue_2436/DMS=true
        	Messages:   	bundle/variables/issue_2436/output.txt vs /tmp/TestAcceptbundlevariablesissue_2436DMS=true1273844601/002/output.txt
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 98256b3

Run: 42040757537

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 16 5:15
✅​ aws-windows-latest-4core-16gb 273 14 4:32
🔄​ azure linux-2core-8gb 2 268 16 5:37
✅​ azure-windows-latest-4core-16gb 272 14 3:56
✅​ gcp linux-2core-8gb 271 16 5:14
✅​ gcp-windows-latest-4core-16gb 273 14 4:31
Test Name azure linux-2core-8gb
🔄​ TestFilerReadDir 🔄​f
🔄​ TestFilerReadDir/files 🔄​f
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:08 aws-windows-latest-4core-16gb TestAccept
3:02 gcp-windows-latest-4core-16gb TestAccept
3:01 azure-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

DABs DABs related issues software-factory PR produced by the software factories task

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lennartkats-db @eng-dev-ecosystem-bot