Repository navigation
Support Feature Store's materialization sets in bundles - #7164
Merged
Merged
Conversation
A materialization set groups the materialized features that are created, updated and deleted together, so a bundle can declare a feature's offline and online materializations as one resource and reconcile them against their desired membership. Members are addressed by content rather than position: KeyedSlices pairs a configured member with a stored one the way UpdateMaterializationSet does, by feature name, online store and whether the member is streaming. Because a whole-member change is a shorter path than the per-field rules under materialized_features[*], and the matcher truncates a pattern longer than the path, such a change would otherwise classify as recreate and destroy every sibling the config left alone; OverrideChangeDesc keeps it on the update path. This branch holds only the hand-written half of the work. It does not compile on its own: ml.MaterializationSet reaches the SDK when the proto stages are promoted to PRIVATE_PREVIEW and a release carries them. Every generated artifact (cli.json, command stubs, *.generated.yml, refschema, jsonschema) is deliberately absent and is re-derived at that point. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…eturns A member's destination fields are input-only: the server accepts them on write and strips them from every read. The generated spec:input_only rule under ignore_remote_changes exists to absorb that, but it cannot fire here. It only fires when the persisted value equals the configured one, and the planner recovers the persisted value by resolving the change path against stored state -- which means matching this resource's composite key label against a real string field of the element, so the lookup finds nothing. With no persisted value the rule is skipped and the change falls through to the immutable-field rule, re-planning every set with an online member as a recreate on every deploy. OverrideChangeDesc is the one place that needs no persisted value, so the suppression goes there. It is gated on the persisted value being absent rather than merely empty: a repoint the config actually asked for carries that value in from the local diff, and so still recreates, which is the right plan for a destination the server cannot alter in place. The fake now redacts the same fields on store. Echoing them back hid this entirely -- a local read matched the config where a real one does not, so the acceptance goldens agreed with a server that behaves differently. Co-authored-by: Isaac <no-reply@databricks.com>
The unit tests prove the member-key trap exists and that the pattern matcher behaves as the override assumes; they do not prove the override fires in a real plan. These cases close that gap, and each one pins a claim about the API rather than about the fake: the fake mirrors UpdateMaterializationSet's reconcile semantics and returns the backend's own message text, so turning on Cloud = true checks every golden here against a workspace. Six cases: a create and a redeploy that plans nothing; membership reconciliation routed to update rather than recreate; the schedule state applying in place without cascading; a write-once field caught by the planner before the request is built; the delete cascade scoped per feature; and the destination fields the read never returns staying quiet while a repoint of one still recreates. README.md records what each case pins, the three fixture changes a cloud run needs, and the two areas still uncovered. Co-authored-by: Isaac <no-reply@databricks.com>
…le cannot The doc comment described only the membership half and buried the reasoning for it, and the suppression comment said the persisted value "reads as unset" without saying that it is recorded in full and merely unreachable. Name both jobs on the function, then put each one's reasoning on the branch that does it. Co-authored-by: Isaac <no-reply@databricks.com>
…st the server
Neither shape was covered. Sawtooth is the one a user is most likely to get wrong,
because the offline-and-online batch pair looks complete and is not: a set stamps its
id on every member before the backend reaches the normalization that would expand the
pair, and that normalization is skipped whenever the id is set, so the third
materialization has to be declared. The fake now resolves a member's feature and
applies the same three-member rule, which is what lets one golden carry the claim.
Both new cases reference an existing Stream, since a Stream is not a bundle resource,
and both reach the feature through ${resources.features.*.full_name} rather than
repeating its name. That reference is load-bearing: without it the engine has no edge
between the feature and the set, and the deploy order decides whether the set create
sees a feature at all.
Running the pair against a workspace also showed the online pipeline_schedule_state
refusal had drifted. The server says the change "would be written on an online
materialized feature" and returns BAD_REQUEST; the fake claimed a different sentence
and INVALID_PARAMETER_VALUE, so schedule_state had been asserting a message no server
produces. Corrected to the observed text, which moves that golden too.
Co-authored-by: Isaac <no-reply@databricks.com>
Sawtooth moves out of "not covered yet" and bind takes its place. Also writes down the two auth facts a local cloud run trips over, since neither is visible from the fixture: the profile environment variable does not reach the CLI subprocess, and the harness authenticates on its own before any script runs. Co-authored-by: Isaac <no-reply@databricks.com>
…loud run prints The batch shape had no case that could run against a workspace, which left the two claims a user is most likely to care about untested there: that repointing a destination is a recreate rather than a silent no-op, and that an update mixing an acceptable change with an unacceptable one is refused whole. Both now run live. The destination repoint plans recreate on the two immutable destination fields with their persisted values intact. The mixed update -- pausing the offline member, which the endpoint accepts, together with pausing the online one, which it does not -- is refused, and a read afterwards shows neither member moved, so the endpoint validates before it writes. Pausing the offline member alone is then applied and does not cascade. Each phase is its own template rather than a string edit on the last one, so what a phase deploys is readable without replaying the ones before it. Two replacements had to be added for the cloud run: a real set id is 32 undashed hex characters where the fake's are dashed, and the workspace host is absent from a local run entirely. Without them the goldens differ between the two runs for no reason beyond formatting. Co-authored-by: Isaac <no-reply@databricks.com>
The deployment-history leg cannot reach a workspace at all: the bundle deployments API keeps an allowlist of resource-type prefixes and neither features nor materialization_sets is on it, so a deploy under that leg fails before any feature-engineering call is made. Worth writing down next to the run instructions, because the local fake accepts the prefix and hides it. Co-authored-by: Isaac <no-reply@databricks.com>
…e it exposed Four things ran against a workspace without a case of their own: a mixed update whose halves are both legal, the same update with either half rejected, out-of-band creation, and bind/unbind. Each needs a multi-feature set, and a multi-feature set cannot be built the way the local cases build one. That is the rule the run exposed. Every feature in a set must name the same catalog, schema and table name prefix; differing only in the prefix is refused. The fake does not check it, so membership and removal_scope hand each feature its own prefix and would both be refused against a real workspace. Noted with the other fixture deltas rather than changed, since those two cases are local-only and their claims are about reconciliation, not destinations. Bind also settles the question left open earlier: the first plan after a bind reports skip, with the offline destination suppressed the same way as on a bundle-created set. Co-authored-by: Isaac <no-reply@databricks.com>
…uard the field list Two gaps that only show up late. CaptureUCDependencies rewrites a feature's full_name so a catalog or schema declared in the same bundle becomes an explicit dependency, and a set's four destination fields got none of that: a set pointing at a bundle-declared catalog had no edge to it, so nothing ordered the catalog first. Same shape as the feature-before-set ordering that only surfaced against a real workspace. The second is that nothing kept the hand-written recreate list exhaustive. TestNoUpdateResourcesCoverAllFields stops at HasDoUpdate and this resource has one, so a member field added to the proto later would be classified as a plain update and the endpoint would accept the request and write nothing. The new coverage test walks the SDK types instead, and pins the destination fields the change override suppresses -- both verified by deleting a rule and watching them fail. Making the fake redact input-only fields had also broken TestAll's round-trip check, which claims to skip fields configured in ignore_remote_changes and could not: it compares a pattern written foo[*].bar against a walked path foo[0].bar by string equality. Every other fake echoes input-only values back, so this resource is the first to reach it. Now matched with the pattern matcher; TestAll stays green for every resource. The offline blocks in the local fixtures were empty, which the regenerated required field validation correctly rejects and the real API rejects too. Filled in, which also gives every feature in those sets one shared destination -- the other thing the API requires. Co-authored-by: Isaac <no-reply@databricks.com>
…peration
The create now returns databricks.longrunning.Operation. The generated SDK wraps that
in a typed operation whose Wait polls GetOperation and unmarshals the response, so
DoCreate submits and then waits, and reads the created set out of the response rather
than out of the submit. A failed wait names the set from the operation metadata, which
carries the id precisely so a half-finished creation can still be identified.
The fake records the operation instead of answering with the set, and serves the poll
at the path the client builds, /api/2.0/feature-engineering/operations/{id}. It records
the operation already terminal, matching a backend whose execution is still inline.
basic now prints that GET, so the goldens pin that the poll happens at all rather than
passing because the request filter hides it.
Co-authored-by: Isaac <no-reply@databricks.com>
The create's long-running operation now carries the materialization set as its response rather than a wrapper holding one, so Wait hands back the set itself. The guard moves with it: the wrapper's set field was optional, where now an operation that settles without an id is the failure worth naming, because the id is what the engine records the resource under. Dropping the wrapper also removes the reason the generated lifecycle config had to be produced from a doctored spec. Nothing references the set's type from an output-only field any more, so generate_resources.py no longer infers that the whole type is read-only, and the config comes out correct from the spec as built. Co-authored-by: Isaac <no-reply@databricks.com>
The set update endpoint no longer has any mutable member field: the schedule state joined the cron schedule trigger on the server's immutable list, so the endpoint reconciles membership and nothing else. A declared schedule-state change therefore has to be caught at plan time, because the in-place update the planner used to produce is now refused. The update mask is unchanged. It names only the member list, the handler never reads it, and the state is nested inside the list, so nothing about the request shape can narrow what the server inspects. Dropping the coverage test's exemption for the field exposed that a backend_defaults entry was being counted as classifying it, which it does not: a backend default only absorbs a field the config leaves unset. The collector now keeps the two apart, so removing the recreate rule fails the test. Co-authored-by: Isaac <no-reply@databricks.com>
The two membership cases pinned one rule between them: removal_scope asserted the per-feature cascade scope and the refusal, which membership already asserted from the other direction. One fixture with a feature contributing a pair and two contributing a single member covers both refusal directions, the singleton removal and the pair removal, with one deploy cycle instead of two. The fixture also gives every feature the same destination, which the real create endpoint requires and both old fixtures violated. Bind had no case at all, only a live check. Unbinding stages the same shape as a set created out of band, so one case covers both: the plan reports create while the bundle holds no state, and binding settles the first plan to skip with the destination suppressed as on a bundle-created set. Two fake messages did not match the backend. The empty-member-list refusal now uses materializationSetRequiresMembers verbatim; the unknown-operation one is marked as the single message in the file not taken from the backend, since that is resolved in the entity-store client. Co-authored-by: Isaac <no-reply@databricks.com>
Three cases now carry Cloud = true alongside batch, sawtooth and streaming. immutable and membership each held the only copy of a server refusal message no workspace had ever seen -- the member-addition refusal and the per-feature delete cascade -- and both are now confirmed against the real API, as is batch's recreate carrying PAUSED on every member, which was the last claim in the goldens a workspace had not checked. Converting them surfaced two create-path rules the fake did not model, so fixtures that the real server rejects were passing locally. Every feature in a set must ask for an identical (offline, online) destination pair, which a set mixing a feature materialized to both stores with one materialized offline only cannot satisfy; and a feature's offline and online destinations must differ in catalog, schema or prefix. Both are now mirrored in the fake, which immediately caught two fixtures in this suite. The first rule also reverses an earlier merge. membership and removal_scope were folded together on the grounds that they pinned one rule between them, but the pair rule makes their two shapes mutually exclusive within a single set, so the cascade refusal and the single-member removal cannot be one case. They are split again, each with a fixture the server accepts. Co-authored-by: Isaac <no-reply@databricks.com>
Update and delete now return a long-running operation like create already did, so DoUpdate and DoDelete wait for theirs. The operation metadata type is shared across the three verbs, so the helper naming what an operation was changing is shared too. All three verbs also take an idempotency token the server reserves an operation record against, and the adapter supplies one per call. The fake's update and delete return operations to match, and the delete gets a handler of its own because the generic map delete returns an empty body. basic and removal_scope now pin the delete's and the update's operation round trip, which no golden asserted before. Co-authored-by: Isaac <no-reply@databricks.com>
The set update's validations moved into MaterializationSetUtils and the cascade refusal was reworded to name the removed members of the offending feature, so the fake groups removed members by feature and renders the ids the way the server does. materializedFeatureFieldCannotBeChanged follows the server's new wording too, though no case reaches it. Verified against the daniel-bai-fs2680 swap at universe 8438b2287fb03, where the update no longer refuses a member whose pipeline_schedule_state the config leaves unset. Co-authored-by: Isaac <no-reply@databricks.com>
…ture Co-authored-by: Isaac <no-reply@databricks.com>
… error The update and delete endpoints name the members they could not reach in a MaterializationSet*Progress error detail and leave the message deferring to it. libs/diag/sdk_error.go renders only error_code and message, and nothing in the bundle path reads an error detail, so those ids never reached the user. Read from APIError.ErrorDetails().UnknownDetails, which is where the SDK puts a detail type it does not model. Applied to the wait as well as the call, so it holds whether the server raises the partial failure or settles the operation with it. Co-authored-by: Isaac <no-reply@databricks.com>
The adapter's member key and MaterializationSetMemberKey.keyOf derive the same three components in the same order, and the server pairs a requested member with a stored one by it, so the two have to move together. The comment named neither the class nor that coupling. Co-authored-by: Isaac <no-reply@databricks.com>
… was missing The backend now has a message for each direction of the deletion cascade, picked by whether the removal names the member a delete is applied to, and refuses a set that mixes a batch-path feature with a streaming-path one. The fake carried one older cascade message for both directions and did not model the mixing rule at all. The cases now walk the removal matrix rather than sampling it: sawtooth covers all three strict subsets of its one cascade group, batch covers each half of its pair, streaming gains a second feature so a lone online member can be removed as an update, and immutable covers the destroy and redeploy pair those refusals leave as the only route. Co-authored-by: Isaac <no-reply@databricks.com>
Every message the new arms assert is now the server's own text rather than a reading of its source, including the claim that two stream features materialized to one destination can share a set. Co-authored-by: Isaac <no-reply@databricks.com>
…tails Once the backend has reserved an operation record it no longer raises: all three verbs settle the record FAILED and return it with HTTP 200, with the error code and message on the operation and the per-member outcome on the metadata's progress arm. The adapter was reading error details off an APIError, which that shape never produces, so a partial change surfaced as a flat string with the members it left alone dropped. The adapter now re-reads the settled operation, recovers the code structurally, appends the failed members, and returns an APIError so the engine classifies and renders it like any other failure. The status is the 200 the call returned, which maps to no sentinel on purpose: a member that could not be torn down must not reach the destroy step's permission-denied leniency. The fake can produce the shape too, keyed off a marker in a feature name, since a real teardown fails on infrastructure no request can ask for. The new settled_failure case pins what the adapter makes of it, and pins the engine gap behind it: a destroy forgives a delete error matching none of its known cases, so a partly deleted set exits zero with its state dropped. Co-authored-by: Isaac <no-reply@databricks.com>
Three messages had been reworded, and two create-path refusals had moved from an HTTP 400 to a settled FAILED operation: the batch-and-streaming mixing rule and the sawtooth triple rule both live inside the batch create the handler calls after reserving its operation record, as do the two destination rules. Only an empty member list, a colliding set id and the wire-level schedule-state hook are reached early enough to raise. The fake now splits its create rules around the reservation and returns a refusal value the raise and settle paths each report their own way. Four goldens move with it, every one of them corrected from what the live run returned rather than from reading the handler. All six Cloud = true cases pass against universe cd7a68312ca3, run one at a time. Co-authored-by: Isaac <no-reply@databricks.com>
The summary line ends in the 200 the call returned, which read as a contradiction beside an error code on its own. Saying the operation failed makes the status mean what it is: the call succeeded, the operation it returned did not. Co-authored-by: Isaac <no-reply@databricks.com>
…LI fans out A sawtooth-window feature needs three members and a feature materialized to both stores needs two, so the member list has exactly one legal shape per feature and the author repeats the feature name and the destinations to reach it. Two authoring fields now declare one feature's materialization once, and a mutator fans each entry out into the members it is made of before the plan. The mutator judges nothing: it moves named fields across and lets the service refuse a malformation, which it already does by name. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
pietern
reviewed
Nov 3, 2026
pietern
changed the title
Co-authored-by: Isaac <no-reply@databricks.com>
The SDK version main moved to adds notifications to the set itself and marks the member-level field output only. Both the create and update endpoints refuse the set field with NOT_IMPLEMENTED, so it is classified as immutable like the member field. The generated schema and Python models are regenerated against the new spec, which also deprecates the member's pipeline_schedule_state and so drops it from the Python models. Co-authored-by: Isaac <no-reply@databricks.com>
…error A FAILED operation is the outcome of a call that succeeded, so dressing it up as an APIError with HTTP 200 printed an endpoint and a status that describe nothing. The adapter now wraps the wait's own error, which already carries the operation's error code and message, and appends the members the change left unchanged. That removes the second GetOperation call the APIError needed for a structured error code. A plain error is not transient and matches no missing-resource sentinel, as the 200 status did not, so retries and the delete step treat the failure as before. Co-authored-by: Isaac <no-reply@databricks.com>
…ferences The README under the materialization set acceptance cases read as an implementation log, and code comments named backend source files and classes, none of which belong in a public repository. The README is removed, the comments now describe the service's behaviour without naming its source, and the fixture defaults that named one developer's catalog, schema, online store and Stream use neutral names. The goldens change only by those names. Co-authored-by: Isaac <no-reply@databricks.com>
…le_state The SDK version main moved to marks MaterializedFeature.PipelineScheduleState deprecated, and staticcheck flags the test server's four reads of it. The service still validates the field on create and on a set update, so the fake keeps mirroring those rules and the reads are suppressed. Co-authored-by: Isaac <no-reply@databricks.com>
The update endpoint can remove members but not add one, so a member the deployed set lacked planned as an update that the deploy then failed on. That includes a member whose feature name, online store or streaming trigger changed, because those fields make up the member key. Such a member now plans a recreate of the set, with a warning that every member is dropped and recomputed. Removing a member is still an update. Co-authored-by: Isaac <no-reply@databricks.com>
A Feature View recreate or delete never prompted. Feature Views now join the deploy and destroy approval groups, with a message that a view a set still materializes cannot be recreated, and that FeatureSpecs using a view are not checked. Co-authored-by: Isaac <no-reply@databricks.com>
| @@ -0,0 +1 @@ | |||
| * `bundle deploy` now asks for confirmation before recreating or deleting a Feature View, and `bundle deploy` and `bundle destroy` warn that FeatureSpecs using a Feature View are not checked. ([#7164](https://github.com/databricks/cli/pull/7164)) | |||
Member
There was a problem hiding this comment.
can we split this (& the associated changes to acceptance test and bundle/phases/ out of the materialized features PR)?
Member
Author
There was a problem hiding this comment.
Thanks for raising this!
The change itself is small and self-contained: features joins the deploy and destroy approval groups, plus its messages and a changelog entry. It's needed because of the new materialization set resource. A set depends on the Feature Views it materializes, which makes recreating or deleting a Feature View riskier. Our field engineers found that without this warning, users can easily recreate or delete a Feature View without realizing the impact, so shipping the proper warning message with the same release would be ideal. Given that, would it be possible to keep it in this PR? Thanks!
Moves MaterializationSets and Operations into their own block and notes that Operations belongs to MaterializationSets. Co-authored-by: Isaac <no-reply@databricks.com>
Replaces TestCaptureUCDependenciesMaterializationSet with implicit_deps_materialization_set, the same shape as the other implicit_deps_* cases, and stops listing resource types in the mutator's comment. TestCaptureUCDependenciesNilResources still covers a nil set. Co-authored-by: Isaac <no-reply@databricks.com>
…elog entry Fills the three schema placeholders the resource added (the resource map, lifecycle, and the deprecated cron_schedule, which the service rejects) and regenerates the JSON schema and the PyDABs model docstring. Co-authored-by: Isaac <no-reply@databricks.com>
The deploy message now says what to do when a set still materializes the Feature View, and both messages state the consequence for a FeatureSpec that serves it. Co-authored-by: Isaac <no-reply@databricks.com>
The SDK bump moved notifications to the set and made the member field output only, which the generated config already classifies. The hand-written immutable rule predates that and would only recreate the set for a field the service ignores. Co-authored-by: Isaac <no-reply@databricks.com>
… fake pipeline_schedule_state is deprecated, so the fake no longer reproduces the service's two rules on it (batch members agree on one state; a set update cannot change it), and the schedule_state case that exercised them is removed. The fake still stores the default ACTIVE state, which the backend default rule depends on. Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
approved these changes
Nov 4, 2026
janniklasrose
left a comment
janniklasrose
left a comment
Member
There was a problem hiding this comment.
approving with reservations due to urgency. since this resource is still in PrPr, we can iterate on the implementation in the next week
| "markdown_description": |- | ||
| The materialization set definitions for the bundle, where each key is the name of the materialization set. | ||
| "$fields": | ||
| "lifecycle": |
Member
There was a problem hiding this comment.
note to self: for centrally managed fields like this, we should have a central annotation
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Nov 4, 2026
## Release v1.24.0 ### Notable Changes * A variable used as the value of a secret is now sensitive: referencing it from any other field, such as a job parameter or a secret's `comment`, is an error because the value would be used in plain text. Mark other secret-holding variables with `sensitive: true` to get the same protection. ([#7288](#7288)) ### CLI * Fix `databricks auth login` and `databricks configure` overwriting the parent profile's host and token when saving a profile with a dotted name such as `team.prod` next to an existing `[team]`. ([#7266](#7266)) * `databricks auth login` and `databricks auth token` now reject the name of an existing profile after "Create a new profile" instead of logging in to that profile. ([#7196](#7196)) * `databricks auth login` and `databricks auth token` now reject a host that is not a URL with a hostname, such as `/`, `?o=123` or `https://`, instead of falling back to `DATABRICKS_HOST` or failing later. An empty host after "Enter a host URL manually" in the login profile picker no longer starts discovery login. ([#7195](#7195)) * Prompts that validate their input, such as the `databricks configure` host prompt, no longer open with a red cross before anything is typed. ([#7314](#7314)) * `databricks ssh connect` now documents `--environment-version` for choosing the serverless environment version, starts a separate default-named server for each version, and rejects the flag together with `--cluster`. Default connection names also include the environment version in their hash, so the default name when `--environment-version` is not specified changes with this release. Configuration associated with the old connection name (secret scope, SSH keys and known hosts in your local `.databricks` folder) will no longer be used and can be safely removed. ([#7310](#7310)) ### AI Runtime * Show workspace-supported accelerator types in `databricks air run -h config.compute.accelerator_type`. ([#7275](#7275)) * `air list` now reports `No active runs found.` for an empty active-only result. ([#7276](#7276)) * Stop printing a retry notice on every `air run --watch`; retries are still reported when they happen. ([#7270](#7270)) ### Bundles * Fix `bundle plan` reporting an update to an app's `config.env` values on every run, which also redeployed the app on every `bundle deploy`. The Apps API returns environment variable names only, so `value` and `value_from` are no longer compared against the remote deployment. ([#7326](#7326)) * Add the `data_quality_monitors` resource for data quality monitors on Unity Catalog schemas and tables (direct engine only). ([#7094](#7094)) * `bundle deploy` now asks for confirmation before recreating or deleting a Feature View, and `bundle deploy` and `bundle destroy` warn that FeatureSpecs using a Feature View are not checked. ([#7164](#7164)) * Accept grant `privileges` values that the SDK enum does not list yet: `bundle validate` no longer warns (or errors under `--strict`) on them, and the PyDABs `Privilege` enum accepts unknown strings instead of raising. The JSON schema and PyDABs still advertise the known values for completion. ([#7315](#7315)) * Add the `materialization_sets` resource, which materializes Feature Store Feature Views to offline and online stores. ([#7164](#7164)) * Detect notebooks with the same rule as the workspace: a `.py`, `.sql`, `.scala` or `.r` file is a notebook when its first line contains `Databricks notebook source`, for example after a byte order mark. Previously such a file was tracked as a plain file, so removing it locally left the remote notebook behind. ([#7290](#7290)) * Add `sensitive: true` for bundle variables. A variable used as the value of a secret is sensitive by default. `bundle validate -o json` redacts only sensitive variables instead of every string variable, and a sensitive variable can only be referenced from a sensitive resource field (such as a secret's `value`) or from another sensitive variable. ([#7288](#7288)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7305](#7305)) * Bump dependencies with known vulnerabilities. ([#7327](#7327)) * Update the Go SDK to [v0.191.1](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.191.1). ([#7277](#7277)) * Update the Go SDK to [v0.193.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.193.0). ([#7303](#7303))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What did you change, and why?
Change: Onboards
MaterializationSetas a bundle resource on the direct engine.A materialization set is a list of materialized features. The list has no meaningful order, so the resource declares
materialized_featuresas a keyed slice and the engine diffs it by key rather than by position. The key is a composite of three values, each read off a materialized feature's own fields:Because that key is the materialization set member's identity, the mutator
NormalizeMaterializationSetMaterializedFeatureMembersenforces that it is unique across a set's materialized features and raises otherwise.A few changes worth highlighting:
Synchronous wait in the adapter
The create, update and delete endpoints for a materialization set all return a long-running operation, so each adapter method waits for it to settle and reads the result from the operation's response. A failed operation is restated as an API error.
This follows the predominant precedent in the direct engine: the Postgres resources wait on their operations the same way, inside the adapter.
Overriding two change classifications
OverrideChangeDesccorrects the generic classification in two places.A removed member is an update; a member the deployed set lacks is a recreate. A whole member's short path, materialized_features[], still matches an immutable pattern like materialized_features[*].field1.field2, because the matcher truncates a pattern longer than the path, so every membership change would plan a recreate. The update endpoint can remove members but not add one, so the override plans a removal as an update, and a member missing from the set the read returned as a recreate, with a warning that every member is dropped and recomputed. Changing a member's feature name, online store or streaming trigger changes its key, so it recreates too.
The input-only suppression cannot fire on this resource. ignore_remote_changes applies only when the value in persisted state equals the one in config. But the persisted-value lookup resolves the composite key we use against the field names of a materialization set member. A synthetic label never matches a real field name, so the lookup comes back empty and the suppression never applies. The override skips those fields directly.
How do you know it works?
acceptance/bundle/resources/materialization_sets/— create, update, delete, and the refusalsacceptance/bundle/resources/features/duration/— extended and mixed duration units