Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
62464fc
Add PLAN.md for bundle bitmap
denik Jul 29, 2026
119091c
Add bundle bitmap for per-field telemetry
denik Jul 29, 2026
3fd01b4
Remove PLAN.md
denik Jul 29, 2026
bc5fb1a
Add bool value bits, decode round-trip, and template recording
denik Jul 29, 2026
a7842e9
Move boolean telemetry into a typed Telemetry struct
denik Jul 30, 2026
1481143
Regenerate goldens after rebase; ignore .databricks in bitmap test
denik Jul 30, 2026
9157dfc
Fix bitmap test: executable decode script and deterministic cache
denik Jul 30, 2026
c1b2a46
Strip CR when parsing embedded bitmap schema
denik Jul 30, 2026
f41d92e
Regenerate bitmap schema from scratch; refresh goldens in generate task
denik Jul 30, 2026
2d7575e
Make Telemetry additive: mirror bool metrics into a struct on Bundle
denik Jul 31, 2026
dadb5eb
Record DMS undeclared-writer flags as True/False pairs in bitmap
denik Jul 31, 2026
fa701ca
Revert DMS flags to single bits: false is not lost here
denik Aug 3, 2026
17dbecd
Fix bitmap encoding every telemetry bit as set
denik Aug 3, 2026
b4221d1
Regenerate bitmap schema and drop removed Local test option
denik Aug 11, 2026
fc536c8
Regenerate bitmap schema and goldens after rebase
denik Aug 17, 2026
cd3054c
Port bitmap telemetry mirror to current main
denik Nov 3, 2026
745e0f7
Regenerate bitmap schema after rebase (data_quality_monitors)
denik Nov 3, 2026
0de94d5
Bitmap: record user-authored config; fix presence detection; simplify…
denik Nov 4, 2026
941b6da
Add a dedicated bitmap test that records every schema line
denik Nov 4, 2026
76de89b
Make Telemetry the source of bool_values; encode bool values; add decode
denik Nov 4, 2026
50b4ead
Add a self-contained bitmap decoder to databricks-bundles
denik Nov 4, 2026
13aba43
Regenerate bitmap schema after rebase (variables.*.sensitive)
denik Nov 4, 2026
b45b487
Regenerate bitmap schema after rebase (materialization_sets)
denik Nov 4, 2026
1e2f4a9
Keep the bitmap decoder out of pydabs codegen cleanup and docs
denik Nov 4, 2026
7b6315d
no_drift: log bitmap stderr instead of diffing it
denik Nov 4, 2026
85e5142
Add a local-only invariant test for validate and the bitmap round trip
denik Nov 4, 2026
6b08e4f
Update selftest/selection for the new validate_bitmap invariant test
denik Nov 4, 2026
9a2a6ca
bitmap: one bitmap command with -o compressed|text|json
denik Nov 5, 2026
fca7de9
bitmap/full: split the schema golden per resource type
denik Nov 5, 2026
179a950
bitmap/presence: lifecycle.started true on one cluster and false on a…
denik Nov 5, 2026
cfa5c13
Report each lifecycle.started value seen instead of a random resource's
denik Nov 5, 2026
f07900f
Regenerate bitmap schema after rebase (postgres_data_apis)
denik Nov 5, 2026
375d8ba
Telemetry: BoolSet.SetBool instead of appending to []bool
denik Nov 6, 2026
db4aa2b
bitmap: compute bits early and fill in variable references later
denik Nov 9, 2026
671d4d0
Adapt to optional config bools being *bool
denik Nov 9, 2026
4229844
Regenerate bitmap schema after rebase (feature_specs, materialization…
denik Nov 11, 2026
148248c
Telemetry: record ai_runtime_task_scheduled/multitask per job
denik Nov 17, 2026
18ffc41
databricks-bundles bitmap decoder: add -o json
denik Nov 17, 2026
16b5bc8
Regenerate bitmap schema and goldens after rebase
denik Nov 17, 2026
0e320c0
bitmap: record the config at load, before command-line flags set fields
denik Nov 17, 2026
fcdc0cf
bitmap: WalkSchema returns each path once; test a list-element variable
denik Nov 17, 2026
55fb8a6
bitmap/full: put the bit first, right-aligned
denik Nov 18, 2026
da3f1b9
bitmap JSON: key a list's elements as tasks[*] instead of tasks.[*]
denik Nov 18, 2026
2a8f866
bitmap text: one "PTF <field>" line per field
denik Nov 18, 2026
5c49cf3
python mutator test: expect the PTF text lines
denik Nov 18, 2026
d000b8d
Keep the deploy-ai-runtime-task fixture as on main
denik Nov 18, 2026
9a3907c
telemetry/deploy-app-lifecycle-started: print the bitmap
denik Nov 18, 2026
9ed2920
bitmap: record the fields a Python mutator sets on existing resources
denik Nov 20, 2026
d1de1a9
bitmap/presence: keep the encoded bitmap as out.bitmap.b64; trace the…
denik Nov 20, 2026
84684bd
bitmap: decode a newer CLI's bitmap, naming its extra fields "future #N"
denik Nov 20, 2026
4e323b3
telemetry tests: named fields, one per line
denik Nov 20, 2026
dbc2d28
Revert the bitmap step in telemetry/deploy-app-lifecycle-started
denik Nov 20, 2026
f9ecc38
bitmap tests: named fields, one per line
denik Nov 20, 2026
f7a1190
bitmap: store the schema once, compressed (schema.deflate)
denik Nov 20, 2026
5c84d1c
Send the bundle bitmap with deploy telemetry (bundle_deploy_event.bun…
denik Nov 20, 2026
8320d43
Adapt to main: direct_migrate_state_missing on Telemetry; regenerate …
denik Nov 20, 2026
ccdeadf
templates/default-python: check the bitmap a deploy sends against bun…
denik Nov 20, 2026
ac883c1
telemetry/deploy-load-error: fail loading with a bad include glob
denik Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/rules/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The deploy payload is declared twice: the proto in the universe repo (`proto/log

Other things worth knowing:

- A one-off boolean needs no proto change: add a key constant to `bundle/metrics/metrics.go` (snake_case, matching the constant name) and call `b.Metrics.SetBoolValue(...)`. It lands in `experimental.bool_values`.
- A one-off boolean needs no proto change: add a field to `bundle.Telemetry` (`bundle/telemetry.go`) with a snake_case `metric:"..."` tag and set it. Pick the type by what is measured: `bool` for a flag reported only when true, `*bool` for a measured value (nil when not measured), `BoolSet` for a measurement taken several times (e.g. once per artifact; record each with `SetBool`, and each value taken is reported once). It lands in `experimental.bool_values` and in the bundle bitmap (`bundle_deploy_event.bundle_bitmap`); run `./task generate-bitmap-schema` to append it to the bitmap schema.
- Metrics that may be removed later belong in `BundleDeployExperimental`, which carries no compatibility guarantees.
- Positional encodings such as the `upload_file_sizes` histogram are frozen once the metric has adoption: changing a bucket bound silently changes the meaning of every previously reported entry.
- Per-resource state sizes come from the direct engine's state in `resources.json`.
Expand Down
33 changes: 33 additions & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -859,6 +859,7 @@ tasks:
# Refreshes acceptance/bundle/refschema/output/*.txt, which feeds
# generate-direct-apitypes and generate-direct-resources below.
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand All @@ -877,6 +878,7 @@ tasks:
cmds:
- task: generate-cligen
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand Down Expand Up @@ -1065,6 +1067,37 @@ tasks:
cmds:
- go test ./acceptance -run TestAccept/bundle/refschema -update

# Appends any newly added bundle config fields to the bitmap schema (see
# cmd/bundle/bitmap). The schema is append-only, so update-schema reads the
# currently embedded schema and only appends. genschema compresses it into
# bundle/bitmap/schema.deflate, which the CLI embeds, and an identical copy in
# the databricks-bundles decoder (python/databricks/bundles/_bitmap). The
# compressed files are not reviewable; the listing in
# acceptance/bundle/bitmap/full shows every schema line with its bit.
generate-bitmap-schema:
desc: Regenerate the bitmap schema and the acceptance goldens that depend on it
sources:
- bundle/**/*.go
- exclude: bundle/**/*_test.go
- acceptance/bundle/bitmap/full/script
- acceptance/bundle/bitmap/full/split.py
- acceptance/bundle/bitmap/presence/script
- acceptance/bundle/bitmap/presence/databricks.yml
- go.mod
- go.sum
- "{{.EMBED_SOURCES}}"
generates:
- bundle/bitmap/schema.deflate
- acceptance/bundle/bitmap/full/output/*.txt
- acceptance/bundle/bitmap/presence/out.bitmap.b64
- python/databricks/bundles/_bitmap/schema.deflate
cmds:
- go run . bundle bitmap update-schema > bundle/bitmap/schema.txt.tmp
- go run ./bundle/bitmap/genschema bundle/bitmap/schema.txt.tmp bundle/bitmap/schema.deflate python/databricks/bundles/_bitmap/schema.deflate
- rm bundle/bitmap/schema.txt.tmp
# The encoded bitmap in presence carries the schema size.
- go test ./acceptance -run 'TestAccept/bundle/bitmap/(full|presence)' -update

# Upstream field documentation comes from the checked-in .codegen/cli.json;
# bundle/internal/schema/annotations.yml carries the CLI-owned docs and
# overrides and is rewritten in place (synced with the config structure).
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/bitmap/full/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = []
Empty file.
75 changes: 75 additions & 0 deletions acceptance/bundle/bitmap/full/output/alerts.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
4295 resources.alerts
4296 resources.alerts.*
4297 resources.alerts.*.url
4298 resources.alerts.*.modified_status
4299 resources.alerts.*.lifecycle
4300 resources.alerts.*.lifecycle.prevent_destroy
4301 resources.alerts.*.lifecycle.prevent_destroy=true
4302 resources.alerts.*.lifecycle.prevent_destroy=false
4303 resources.alerts.*.lifecycle.depends_on
4304 resources.alerts.*.lifecycle.depends_on[*]
4305 resources.alerts.*.lifecycle.depends_on[*].resource
4306 resources.alerts.*.create_time
4307 resources.alerts.*.custom_description
4308 resources.alerts.*.custom_summary
4309 resources.alerts.*.custom_template_format
4310 resources.alerts.*.display_name
4311 resources.alerts.*.effective_run_as
4312 resources.alerts.*.effective_run_as.service_principal_name
4313 resources.alerts.*.effective_run_as.user_name
4314 resources.alerts.*.evaluation
4315 resources.alerts.*.evaluation.comparison_operator
4316 resources.alerts.*.evaluation.empty_result_state
4317 resources.alerts.*.evaluation.last_evaluated_at
4318 resources.alerts.*.evaluation.notification
4319 resources.alerts.*.evaluation.notification.notify_on_ok
4320 resources.alerts.*.evaluation.notification.notify_on_ok=true
4321 resources.alerts.*.evaluation.notification.notify_on_ok=false
4322 resources.alerts.*.evaluation.notification.retrigger_seconds
4323 resources.alerts.*.evaluation.notification.subscriptions
4324 resources.alerts.*.evaluation.notification.subscriptions[*]
4325 resources.alerts.*.evaluation.notification.subscriptions[*].destination_id
4326 resources.alerts.*.evaluation.notification.subscriptions[*].user_email
4327 resources.alerts.*.evaluation.source
4328 resources.alerts.*.evaluation.source.aggregation
4329 resources.alerts.*.evaluation.source.display
4330 resources.alerts.*.evaluation.source.name
4331 resources.alerts.*.evaluation.state
4332 resources.alerts.*.evaluation.threshold
4333 resources.alerts.*.evaluation.threshold.column
4334 resources.alerts.*.evaluation.threshold.column.aggregation
4335 resources.alerts.*.evaluation.threshold.column.display
4336 resources.alerts.*.evaluation.threshold.column.name
4337 resources.alerts.*.evaluation.threshold.value
4338 resources.alerts.*.evaluation.threshold.value.bool_value
4339 resources.alerts.*.evaluation.threshold.value.bool_value=true
4340 resources.alerts.*.evaluation.threshold.value.bool_value=false
4341 resources.alerts.*.evaluation.threshold.value.double_value
4342 resources.alerts.*.evaluation.threshold.value.string_value
4343 resources.alerts.*.id
4344 resources.alerts.*.lifecycle_state
4345 resources.alerts.*.owner_user_name
4346 resources.alerts.*.parameters
4347 resources.alerts.*.parameters[*]
4348 resources.alerts.*.parameters[*].name
4349 resources.alerts.*.parameters[*].type
4350 resources.alerts.*.parameters[*].value
4351 resources.alerts.*.parent_path
4352 resources.alerts.*.query_text
4353 resources.alerts.*.run_as
4354 resources.alerts.*.run_as.service_principal_name
4355 resources.alerts.*.run_as.user_name
4356 resources.alerts.*.run_as_user_name
4357 resources.alerts.*.schedule
4358 resources.alerts.*.schedule.pause_status
4359 resources.alerts.*.schedule.quartz_cron_schedule
4360 resources.alerts.*.schedule.timezone_id
4361 resources.alerts.*.update_time
4362 resources.alerts.*.warehouse_id
4363 resources.alerts.*.permissions
4364 resources.alerts.*.permissions[*]
4365 resources.alerts.*.permissions[*].level
4366 resources.alerts.*.permissions[*].user_name
4367 resources.alerts.*.permissions[*].service_principal_name
4368 resources.alerts.*.permissions[*].group_name
4369 resources.alerts.*.file_path
Loading
Loading