Skip to content

bundle: read and edit the typed config in five mutators - #7145

Merged
denik merged 14 commits into
mainfrom
denik/typed-mutators-simple
Oct 28, 2026
Merged

denik merged 14 commits into
mainfrom
denik/typed-mutators-simple

Conversation

@denik

@denik denik commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Four mutators now read and edit the typed config directly instead of going through b.Config.Mutate:

  • PrependWorkspacePrefix: the workspace.*_path fields.
  • ApplyBundlePermissions: the permissions field of each supported resource.
  • ProcessStaticResources: lists resource keys from the typed Resources.
  • ProcessRootIncludes: writes the expanded include list after the included files are merged, because merging reloads the typed config from the dynamic tree.

Split out of #7098.

Why

This is a step toward removing dyn from the mutator pipeline. Mutators that edit typed config are simpler and don't need to convert between the two forms.

Intentional behavior notes

  • (a) Invalid-permission-level warnings now show their source location (in databricks.yml:L:C). Before, ApplyBundlePermissions rebuilt the permissions without locations. One golden changed because of this.
  • (b) ApplyBundlePermissions now walks resource names in sorted order. This only changes the order of overlap warnings.
  • (c) Expanded include items now get the location of the raw include pattern at the same index. Before, they had no location.

ExpandPipelineGlobPaths stays on the dynamic tree: with a typed edit the libraries that a glob expands to lose its location, because locations are attached by index when the typed config is converted back.

Tests

  • New unit tests for the edge cases: empty and unset permissions, includes of included files, and resource keys for every resource type. They pass on both the old and the new code.
  • The full TestAccept/bundle suite ran locally. The only golden diff is the one in validate/secret_scope_invalid_permission_level.

This pull request and its description were written by Isaac.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 9092e610f | Workflow run

Regression test report

Selected 8 changed tests (limit=20, 0 not selected)

16 tests: regression=2 unreleased=0 coverage=14 failing=0

test branch main (35ed095) latest (v1.22.1)
TestAccept/bundle/override/cluster_id_clears_new_cluster_reference/DMS= ✅ ✅ ✅
TestAccept/bundle/override/cluster_id_clears_new_cluster_reference/DMS=true ✅ ✅ ✅
TestAccept/bundle/override/preset_clears_forced_schedule_reference/DMS= ✅ ✅ ✅
TestAccept/bundle/override/preset_clears_forced_schedule_reference/DMS=true ✅ ✅ ✅
TestAccept/bundle/presets/name_prefix_reference_to_name/DMS= ✅ ✅ ✅
TestAccept/bundle/presets/name_prefix_reference_to_name/DMS=true ✅ ✅ ✅
TestAccept/bundle/resource_deps/preset_tags_not_applied_to_whole_map_reference/DMS= ✅ ✅ ✅
TestAccept/bundle/resource_deps/preset_tags_not_applied_to_whole_map_reference/DMS=true ✅ ✅ ✅
TestAccept/bundle/validate/references/job_schedule_whole_reference/DMS= ✅ ✅ ✅
TestAccept/bundle/validate/references/job_schedule_whole_reference/DMS=true ✅ ✅ ✅
TestAccept/bundle/validate/references/job_trigger_whole_reference/DMS= ✅ ✅ ✅
TestAccept/bundle/validate/references/job_trigger_whole_reference/DMS=true ✅ ✅ ✅
TestAccept/bundle/validate/required_struct_reference/DMS= ✅ ✅ ✅
TestAccept/bundle/validate/required_struct_reference/DMS=true ✅ ✅ ✅
TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS= ✅ ❌ ❌
TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=true ✅ ❌ ❌
TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=: branch ✅ | main (35ed095) ❌ | latest (v1.22.1) ❌

main (35ed095):

acceptance_test.go:1341: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level2217844303/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default

latest (v1.22.1): same output as main (35ed095)

TestAccept/bundle/validate/secret_scope_invalid_permission_level/DMS=true: branch ✅ | main (35ed095) ❌ | latest (v1.22.1) ❌

main (35ed095):

acceptance_test.go:1341: Diff:
        --- bundle/validate/secret_scope_invalid_permission_level/output.txt
        +++ /tmp/TestAcceptbundlevalidatesecret_scope_invalid_permission_level1812665118/002/output.txt
        @@ -2,11 +2,9 @@
         >>> [CLI] bundle validate
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default
        @@ -19,11 +14,9 @@
         >>> [CLI] bundle validate --strict
         Warning: invalid value "12" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[0].level
        -  in databricks.yml:10:18
         
         Warning: invalid value "fuzz_badlevel" for enum field. Valid values are [READ WRITE MANAGE]
           at resources.secret_scopes.bad_levels.permissions[1].level
        -  in databricks.yml:12:18
         
         Name: test-bundle
         Target: default

latest (v1.22.1): same output as main (35ed095)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 9092e61

Run: 40697783468

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 23 6:30
✅​ aws-windows-latest-4core-16gb 283 21 5:25
✅​ azure linux-2core-8gb 280 23 5:39
✅​ azure-windows-latest-4core-16gb 282 21 5:01
✅​ gcp linux-2core-8gb 281 23 6:00
✅​ gcp-windows-latest-4core-16gb 283 21 5:10
Top 6 slowest tests (at least 2 minutes):
duration env testname
5:17 aws-windows-latest-4core-16gb TestAccept
5:02 gcp-windows-latest-4core-16gb TestAccept
4:54 azure-windows-latest-4core-16gb TestAccept
3:58 aws linux-2core-8gb TestAccept
3:47 azure linux-2core-8gb TestAccept
3:39 gcp linux-2core-8gb TestAccept

@denik
denik force-pushed the denik/typed-mutators-simple branch from 6218b4b to bd00f2a Compare October 21, 2026 15:50
@denik
denik changed the base branch from main to denik/acc-record-main-behaviour October 21, 2026 15:50
@denik
denik added this pull request to stack #7150 October 21, 2026 15:50
@denik
denik force-pushed the denik/typed-mutators-simple branch 2 times, most recently from cd53363 to 6e0b1a9 Compare October 22, 2026 09:40
@denik
denik force-pushed the denik/typed-mutators-simple branch from 6e0b1a9 to 26b7cc4 Compare October 22, 2026 09:54
@denik
denik force-pushed the denik/typed-mutators-simple branch 3 times, most recently from 911fb1b to a279053 Compare October 27, 2026 12:12
@denik
denik force-pushed the denik/typed-mutators-simple branch from a279053 to 9a36648 Compare October 27, 2026 13:00
@denik
denik force-pushed the denik/typed-mutators-simple branch from 9a36648 to 5a264c3 Compare October 27, 2026 17:17
@denik
denik force-pushed the denik/typed-mutators-simple branch 2 times, most recently from 213f547 to 083e679 Compare October 27, 2026 20:35
@denik
denik force-pushed the denik/typed-mutators-simple branch 2 times, most recently from 4863c47 to 284e826 Compare October 28, 2026 13:14
@denik
denik force-pushed the denik/typed-mutators-simple branch from 284e826 to 9092e61 Compare October 28, 2026 13:34
@denik
denik marked this pull request as ready for review October 28, 2026 14:14
@denik
denik requested review from a team as code owners October 28, 2026 14:14
@denik
denik requested a review from shreyas-goenka October 28, 2026 14:14
@denik
denik enabled stacked auto-merge October 28, 2026 14:14
// the dynamic tree, which still holds the patterns, followed by the include sections
// of included files (merging concatenates sequences). Those are kept as they are.
// Include is omitempty, so an empty list is absent from the configuration.
b.Config.Include = append(files, b.Config.Include[len(patterns):]...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not b.Config.Include = files? Seems like a noops to append b.Config.Include[len(patterns):]

}

vs = append(vs, v...)
expanded = append(expanded, withPath(filepath.ToSlash(m)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small regression in that we do loose the location metadata for these fields?

@denik
denik added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit d3e5dde Oct 28, 2026
27 checks passed
Base automatically changed from denik/acc-record-main-behaviour to main October 28, 2026 15:00
@denik
denik deleted the denik/typed-mutators-simple branch October 28, 2026 15:00
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: d3e5dde

Run: 40711948829

Env 🔄​flaky ✅​pass 🙈​skip Time
🔄​ aws linux-2core-8gb 3 1496 1227 138:15
✅​ aws-windows-latest-4core-16gb 1424 1252 152:31
🔄​ azure linux-2core-8gb 3 1343 1279 99:04
✅​ azure-windows-latest-4core-16gb 1271 1304 115:30
✅​ gcp linux-2core-8gb 1333 1284 104:22
✅​ gcp-windows-latest-4core-16gb 1258 1309 122:31
Test Name aws linux-2core-8gb azure linux-2core-8gb
🔄​ TestAccept 🔄​f 🔄​f
🔄​ TestAccept/bundle/invariant/no_drift 🔄​f 🔄​f
🔄​ TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN= 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN= ✅​p 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
14:42 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
13:37 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:45 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
8:39 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
8:37 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:19 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
8:12 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
8:08 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
7:54 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:46 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:46 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:28 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:27 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:26 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:18 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:17 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
7:14 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:08 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:07 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:07 aws linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
7:06 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
7:04 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:00 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:58 aws linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:57 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:56 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:54 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:53 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
6:50 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:50 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
6:50 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
6:48 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:47 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
6:45 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:45 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:44 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:40 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:34 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:20 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:19 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/simple/DMS=true
6:11 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
5:59 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:58 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:57 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
5:56 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:43 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:37 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:37 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
5:20 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:18 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants