Skip to content

Record bundle behavior with faults for every DMS API - #7231

Open
shreyas-goenka wants to merge 10 commits into
mainfrom
shreyas-goenka_data/shreyas/test-bundle-dms-fail-open
Open

shreyas-goenka wants to merge 10 commits into
mainfrom
shreyas-goenka_data/shreyas/test-bundle-dms-fail-open

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Procedurally generate and run tests for all DMS APIs to ensure that the deployment history feature is truly best effort and DMS going down will not cause deployments to start failing.

This tests also ensure we display adequate warnings when the errors actually happen.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 0bb4e7486 | Workflow run

Regression test report

Selected 15 changed tests (limit=20, 0 not selected)

15 tests: regression=0 unreleased=0 coverage=15 failing=0

test branch main (aed6702) latest (v1.23.0)
TestAccept/bundle/dms_fault/complete_version/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/create_deployment/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/create_version/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/delete_deployment/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/get_deployment/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/get_operation/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/get_resource/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/get_version/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/heartbeat/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/list_deployments/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/list_operations/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/list_resources/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/list_versions/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/update_deployment/DMS=true ✅ ✅ ✅
TestAccept/bundle/dms_fault/update_operation/DMS=true ✅ ✅ ✅

@shreyas-goenka shreyas-goenka changed the title Test bundle commands with individual DMS API failures Record bundle behavior with faults for every DMS API Oct 28, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 0bb4e74

Run: 40877726924

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 30 5:31
✅​ aws-windows-latest-4core-16gb 283 28 3:27
✅​ azure linux-2core-8gb 280 30 5:59
✅​ azure-windows-latest-4core-16gb 282 28 3:30
✅​ gcp linux-2core-8gb 281 30 5:45
✅​ gcp-windows-latest-4core-16gb 283 28 3:21
Top 6 slowest tests (at least 2 minutes):
duration env testname
3:49 aws linux-2core-8gb TestAccept
3:48 azure linux-2core-8gb TestAccept
3:44 gcp linux-2core-8gb TestAccept
3:24 azure-windows-latest-4core-16gb TestAccept
3:20 aws-windows-latest-4core-16gb TestAccept
3:10 gcp-windows-latest-4core-16gb TestAccept

@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 5dbdc5d to 9b95f43 Compare October 28, 2026 16:06
@shreyas-goenka
shreyas-goenka changed the base branch from shreyas-goenka_data/shreyas/generate-omit-deployment-metadata to shreyas-goenka_data/shreyas/test-fault-glob-segments October 28, 2026 16:06
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 9b95f43 to c769fef Compare October 28, 2026 16:15
Base automatically changed from shreyas-goenka_data/shreyas/test-fault-glob-segments to shreyas-goenka_data/shreyas/generate-omit-deployment-metadata October 28, 2026 16:50
shreyas-goenka added a commit that referenced this pull request Oct 28, 2026
## Changes
Support `*` within a path segment and `**` across path segments in
`fault.py` patterns. Both match zero or more characters; all other
characters are literal. Compile each pattern once when its rule is
registered..

## Why

useful for tests being added in
#7231.
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from c769fef to 15d2a7e Compare October 28, 2026 16:50
@shreyas-goenka
shreyas-goenka removed this pull request from stack #7239 October 28, 2026 16:56
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 15d2a7e to e1c4557 Compare October 28, 2026 16:56
@shreyas-goenka
shreyas-goenka changed the base branch from shreyas-goenka_data/shreyas/generate-omit-deployment-metadata to shreyas-goenka_data/shreyas/test-fault-glob-main October 28, 2026 16:56
@github-actions github-actions Bot added the DABs DABs related issues label Oct 28, 2026
Base automatically changed from shreyas-goenka_data/shreyas/test-fault-glob-main to main October 28, 2026 17:34
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 9e2109d to 1e09ad4 Compare October 28, 2026 17:34
=== commands: get_deployment fault enabled

>>> bundle deploy --auto-approve
Warn: retrying after 500 Internal Server Error from GET /api/2.0/bundle/deployments/[DEPLOYMENT_ID]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these repeated warnings can be made prettier, we can iterate on them separately. It is good enough for now tbh.


Destroy: 2 deleted

DMS API called: no

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an important signal, shows whether the API is called or not (should we thus expect a warning in the output or not).

@shreyas-goenka
shreyas-goenka marked this pull request as ready for review October 28, 2026 22:53
@shreyas-goenka
shreyas-goenka requested review from a team as code owners October 28, 2026 22:53

@ilyakuz-db ilyakuz-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach is fine, but maybe it will be simpler instead of making a suite for each API, do suite for each CLI command, and force all DMS APIs to return 500 there (and maybe also add some other Go error to ensure that any bugs won't break the direct engine flow)

@denik WDYT?

@@ -0,0 +1,148 @@

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add init / config-remote-sync? I assume they don't interact directly in DMS, but is it possible through some shared code?

@shreyas-goenka

Copy link
Copy Markdown
Contributor Author

@ilyakuz-db if we make all APIs return 500, that'll not exercise all code paths since later flows can depend upon earlier ones succeeding. This is the most robust way to ensure nothing can break for customers.

@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas/test-bundle-dms-fail-open branch from 260fb4e to 0bb4e74 Compare October 29, 2026 12:57

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants