Record bundle behavior with faults for every DMS API

acceptance/bundle/dms_fault/README.md

@@ -0,0 +1,54 @@
+Each API case runs the same linear workflow with a job and a pipeline. DMS records
+deployment history while files remain the source of truth. All bundle commands
+must succeed with the selected API configured to return HTTP 500.
+
+| Step | Command or action | Assertion |
+| --- | --- | --- |
+| 1 | Deploy with DMS healthy. | Establish an existing deployment and its DMS deployment ID. |
+| 2 | Enable the API fault, rename the bundle and resources with the workspace root pinned, then deploy. | Both resources exist with their updated names. State retains file-backed history recording. The bundle rename exercises `UpdateDeployment`. |
+| 3 | Run `bundle summary`. | Output contains both resource IDs. |
+| 4 | Run `bundle generate` into the export directory. | The resource YAML exists; pipeline SQL is downloaded with its original content. |
+| 5 | Run `bundle deployment unbind`. | The resource ID is removed from bundle state. |
+| 6 | Run `bundle generate --bind` into the bundle's resource directory. | Generation and binding succeed for the original resource ID, proving unbind preserved the remote resource. Pipeline SQL is downloaded with its original content. |
+| 7 | Run `bundle validate`, then deploy again. | Generated configuration is valid, both resource IDs stay unchanged, and the pipeline references the newly downloaded SQL path. |
+| 8 | Disable history and deploy, then enable it and deploy. | Both IDs stay unchanged. The file-backed history marker follows the setting, and disabling history makes no DMS calls. Each toggle changes the job description to exercise an actual resource update. |
+| 9 | Run `bundle destroy` with history enabled. | Both remote resources and remote state are gone. |
+| 10 | Disable history, deploy fresh resources, then destroy. | The resources exist after deploy and are gone after destroy. Neither command calls DMS. |
+
+Steps 4–6 run first for the job, then for the pipeline. The fault stays enabled
+through step 10 and is cleared in `finally`. For `CreateDeployment`, skip the healthy
+setup deployment so the fault applies to the first deployment.
+
+History is toggled with `experimental.deployment_history: true/false` in YAML.
+That explicit setting takes priority over the suite's environment override.
+
+Each case registers one fault rule. A version wildcard such as
+`POST .../versions/*/complete` covers every version the workflow accesses or creates.
+Each `*` stays within one path segment, so a version read fault cannot also match
+nested operation endpoints.
+`GetDeployment` also uses a deployment-ID wildcard so its fault stays active if a
+failed lookup triggers a replacement deployment record.
+
+Each snapshot records the command invocations and output, including warnings and
+deployment-version links, then reports whether the injected endpoint was called.
+An unused endpoint records `DMS API called: no`.
+
+Fault cases cover all DMS APIs:
+
+- `CompleteVersion`
+- `CreateDeployment`
+- `CreateVersion`
+- `DeleteDeployment`
+- `GetDeployment`
+- `GetOperation`
+- `GetResource`
+- `GetVersion`
+- `Heartbeat`
+- `ListDeployments`
+- `ListOperations`
+- `ListResources`
+- `ListVersions`
+- `UpdateDeployment`
+- `UpdateOperation`
+
+Missing APIs: none.

acceptance/bundle/dms_fault/complete_version/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["true"]

acceptance/bundle/dms_fault/complete_version/output.txt

@@ -0,0 +1,148 @@
+
+=== commands: setup
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
+  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=1
+  Created jobs.foo
+  Created pipelines.bar
+  Files: 4 uploaded, 0 deleted
+  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+=== commands: complete_version fault enabled
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
+  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=2
+  Updated jobs.foo
+  Updated pipelines.bar
+  Files: 3 uploaded, 0 deleted
+  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
+  Warn: Failed to complete deployment history version: Fault injected by test.
+
+>>> bundle summary
+  Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
+  Target: default
+  Workspace:
+    User: [USERNAME]
+    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
+  Resources:
+    Jobs:
+      foo:
+        Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
+        URL:  [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID]
+    Pipelines:
+      bar:
+        Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
+        URL:  [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID]
+
+>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force
+  Job configuration successfully saved to exported/foo.job.yml
+  Warning: Generated configuration is not included in the bundle
+
+  The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
+  so it will not be deployed. Add a matching entry to the 'include' section, for example:
+
+  include:
+    - exported/*.yml
+
+
+>>> bundle deployment unbind foo
+
+>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind
+  Job configuration successfully saved to resources/foo.job.yml
+  Successfully bound job with an id '[FOO_ID]'
+
+>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force
+  File successfully saved to exported-src/pipeline.sql
+  Pipeline configuration successfully saved to exported/bar.pipeline.yml
+  Warning: Generated configuration is not included in the bundle
+
+  The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
+  so it will not be deployed. Add a matching entry to the 'include' section, for example:
+
+  include:
+    - exported/*.yml
+
+
+>>> bundle deployment unbind bar
+
+>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind
+  File successfully saved to src/pipeline.sql
+  Pipeline configuration successfully saved to resources/bar.pipeline.yml
+  Successfully bound pipeline with an id '[BAR_ID]'
+
+>>> bundle validate
+  Name: updated-dms-fault-complete_version-commands-[UNIQUE_NAME]
+  Target: default
+  Workspace:
+    User: [USERNAME]
+    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
+
+  Validation OK!
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
+  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=9
+  Updated pipelines.bar
+  Files: 6 uploaded, 0 deleted
+  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
+  Warn: Failed to complete deployment history version: Fault injected by test.
+
+=== deployment_history: false
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
+  Updated jobs.foo
+  Updated pipelines.bar
+  Files: 2 uploaded, 0 deleted
+  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
+
+=== deployment_history: true
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
+  Current Deployment Version: [DATABRICKS_URL]/deployments/[DEPLOYMENT_ID]?version=11
+  Updated jobs.foo
+  Updated pipelines.bar
+  Files: 2 uploaded, 0 deleted
+  Resources: 0 created, 2 changed, 0 deleted, 0 unchanged
+  Warn: Failed to complete deployment history version: Fault injected by test.
+
+>>> bundle destroy --auto-approve
+  The following resources will be deleted:
+    delete resources.jobs.foo
+    delete resources.pipelines.bar
+
+  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
+  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
+    delete resources.pipelines.bar
+
+  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
+
+  Warn: Failed to complete deployment history version: Fault injected by test.
+  Destroy: 2 deleted
+
+=== deployment_history: false
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]/files...
+  Created jobs.foo
+  Created pipelines.bar
+  Files: 8 uploaded, 0 deleted
+  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+>>> bundle destroy --auto-approve
+  The following resources will be deleted:
+    delete resources.jobs.foo
+    delete resources.pipelines.bar
+
+  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
+  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
+    delete resources.pipelines.bar
+
+  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-complete_version-commands-[UNIQUE_NAME]
+
+  Destroy: 2 deleted
+
+DMS API called: yes

acceptance/bundle/dms_fault/complete_version/script

@@ -0,0 +1 @@
+python3 "$TESTDIR/../run.py" complete_version

acceptance/bundle/dms_fault/create_deployment/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["true"]

acceptance/bundle/dms_fault/create_deployment/output.txt

@@ -0,0 +1,135 @@
+
+=== commands: setup
+
+=== commands: create_deployment fault enabled
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
+  Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test.
+  Created jobs.foo
+  Created pipelines.bar
+  Files: 4 uploaded, 0 deleted
+  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+>>> bundle summary
+  Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
+  Target: default
+  Workspace:
+    User: [USERNAME]
+    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
+  Resources:
+    Jobs:
+      foo:
+        Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
+        URL:  [DATABRICKS_URL]/jobs/[FOO_ID]?w=[WORKSPACE_ID]
+    Pipelines:
+      bar:
+        Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
+        URL:  [DATABRICKS_URL]/pipelines/[BAR_ID]?w=[WORKSPACE_ID]
+
+>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir exported --source-dir exported-src --force
+  Job configuration successfully saved to exported/foo.job.yml
+  Warning: Generated configuration is not included in the bundle
+
+  The file exported/foo.job.yml is not matched by any pattern in the 'include' section of databricks.yml,
+  so it will not be deployed. Add a matching entry to the 'include' section, for example:
+
+  include:
+    - exported/*.yml
+
+
+>>> bundle deployment unbind foo
+
+>>> bundle generate job --existing-job-id [FOO_ID] --key foo --config-dir resources --source-dir src --force --bind
+  Job configuration successfully saved to resources/foo.job.yml
+  Successfully bound job with an id '[FOO_ID]'
+
+>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir exported --source-dir exported-src --force
+  File successfully saved to exported-src/pipeline.sql
+  Pipeline configuration successfully saved to exported/bar.pipeline.yml
+  Warning: Generated configuration is not included in the bundle
+
+  The file exported/bar.pipeline.yml is not matched by any pattern in the 'include' section of databricks.yml,
+  so it will not be deployed. Add a matching entry to the 'include' section, for example:
+
+  include:
+    - exported/*.yml
+
+
+>>> bundle deployment unbind bar
+
+>>> bundle generate pipeline --existing-pipeline-id [BAR_ID] --key bar --config-dir resources --source-dir src --force --bind
+  File successfully saved to src/pipeline.sql
+  Pipeline configuration successfully saved to resources/bar.pipeline.yml
+  Successfully bound pipeline with an id '[BAR_ID]'
+
+>>> bundle validate
+  Name: updated-dms-fault-create_deployment-commands-[UNIQUE_NAME]
+  Target: default
+  Workspace:
+    User: [USERNAME]
+    Path: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
+
+  Validation OK!
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
+  Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test.
+  Updated pipelines.bar
+  Files: 6 uploaded, 0 deleted
+  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
+
+=== deployment_history: false
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
+  Updated jobs.foo
+  Files: 2 uploaded, 0 deleted
+  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
+
+=== deployment_history: true
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
+  Warn: Could not enable deployment history for this deploy: failed to create deployment: Fault injected by test.
+  Updated jobs.foo
+  Files: 2 uploaded, 0 deleted
+  Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
+
+>>> bundle destroy --auto-approve
+  The following resources will be deleted:
+    delete resources.jobs.foo
+    delete resources.pipelines.bar
+
+  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
+  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
+    delete resources.pipelines.bar
+
+  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
+
+  Warn: Could not record deployment history for this destroy: failed to create deployment: Fault injected by test.
+  Destroy: 2 deleted
+
+=== deployment_history: false
+
+>>> bundle deploy --auto-approve
+  Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]/files...
+  Created jobs.foo
+  Created pipelines.bar
+  Files: 8 uploaded, 0 deleted
+  Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+>>> bundle destroy --auto-approve
+  The following resources will be deleted:
+    delete resources.jobs.foo
+    delete resources.pipelines.bar
+
+  This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
+  Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
+    delete resources.pipelines.bar
+
+  All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-fault-create_deployment-commands-[UNIQUE_NAME]
+
+  Destroy: 2 deleted
+
+DMS API called: yes

acceptance/bundle/dms_fault/create_deployment/script

@@ -0,0 +1 @@
+python3 "$TESTDIR/../run.py" create_deployment

acceptance/bundle/dms_fault/create_version/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["true"]