Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
bundle:
name: pipeline-testing-dynamic-version

artifacts:
other_test_code:
type: whl
files:
- source: dist/other_test_code-0.0.1-py3-none-any.whl
dynamic_version: true

resources:
pipelines:
orders:
name: orders
libraries:
- glob:
include: ./transformations/**
testing:
sources:
- glob:
include: ./tests/**
environment:
dependencies:
- ./dist/other_test_code-0.0.1-py3-none-any.whl
- ./dist/other_test_code-0.0.1-py3-none-any.whl[test]
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["", "true"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@

=== Test dependencies use the patched wheel, with extras kept

>>> [CLI] bundle deploy
Uploading .databricks/bundle/default/patched_wheels/other_test_code_other_test_code/other_test_code-0.0.1+[UNIX_TIME_NANOS]-py3-none-any.whl...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-dynamic-version/default/files...
Created pipelines.orders
Files: 4 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

>>> print_requests.py //api/2.0/pipelines --nostamp
[
"/Workspace/Users/[USERNAME]/.bundle/pipeline-testing-dynamic-version/default/artifacts/.internal/other_test_code-0.0.1+[UNIX_TIME_NANOS]-py3-none-any.whl",
"/Workspace/Users/[USERNAME]/.bundle/pipeline-testing-dynamic-version/default/artifacts/.internal/other_test_code-0.0.1+[UNIX_TIME_NANOS]-py3-none-any.whl[test]"
]

>>> [CLI] bundle destroy --auto-approve
The following resources will be deleted:
delete resources.pipelines.orders

This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
delete resources.pipelines.orders

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing-dynamic-version/default

Destroy: 1 deleted
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
trap "rm -fr dist" EXIT
mkdir dist
cp $TESTDIR/../../../artifacts/whl_prebuilt_multiple/dist/lib/other_test_code-0.0.1-py3-none-any.whl dist

title "Test dependencies use the patched wheel, with extras kept"
trace $CLI bundle deploy
trace print_requests.py //api/2.0/pipelines --nostamp | jq '.body.testing.environment.dependencies'
trace $CLI bundle destroy --auto-approve
rm "$OUT_REQUESTS"
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Runs only against the local test server.
Cloud = false
RecordRequests = true
Ignore = [".databricks"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
def test_orders():
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from pyspark import pipelines as dp
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
bundle:
name: pipeline-testing-same-name-libraries

resources:
pipelines:
orders:
name: orders
libraries:
- file:
path: ./transformations/orders.py
environment:
dependencies:
- ./pipeline/test_helpers-0.1-py3-none-any.whl
testing:
sources:
- glob:
include: ./tests/**
environment:
dependencies:
# A different wheel with the same file name would be uploaded to the same path.
- ./tests/test_helpers-0.1-py3-none-any.whl
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["", "true"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@

>>> musterr [CLI] bundle deploy
Error: Duplicate local library names: test_helpers-0.1-py3-none-any.whl
at resources.pipelines.orders.environment.dependencies[0]
resources.pipelines.orders.testing.environment.dependencies[0]
in databricks.yml:13:13
databricks.yml:21:15

Local library names must be unique but found libraries with the same name: pipeline/test_helpers-0.1-py3-none-any.whl, tests/test_helpers-0.1-py3-none-any.whl

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
trace musterr $CLI bundle deploy
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Runs only against the local test server.
Cloud = false

[[Repls]]
Old = '\\'
New = '/'
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
def test_orders():
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
x = 1
19 changes: 19 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/databricks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
bundle:
name: pipeline-testing

resources:
pipelines:
orders:
name: orders
libraries:
- glob:
include: ./transformations/**
testing:
sources:
- glob:
include: ./tests/**
environment:
dependencies:
- -e .[test]
- -r ./tests/requirements.txt
- ./dist/*.whl
2 changes: 2 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["", "true"]
132 changes: 132 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@

=== Relative test paths resolve to the deployed workspace paths

>>> [CLI] bundle validate -o json
{
"environment": {
"dependencies": [
"-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
"-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
"./dist/*.whl"
]
},
"sources": [
{
"glob": {
"include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
}
}
]
}

=== Deploy sends testing, with the local wheel uploaded

>>> [CLI] bundle deploy
Uploading dist/test_helpers-0.1-py3-none-any.whl...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
Created pipelines.orders
Files: 6 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

>>> print_requests.py //api/2.0/pipelines --nostamp
{
"method": "POST",
"path": "/api/2.0/pipelines",
"testing": {
"environment": {
"dependencies": [
"-e /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files[test]",
"-r /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/requirements.txt",
"/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/artifacts/.internal/test_helpers-0.1-py3-none-any.whl"
]
},
"sources": [
{
"glob": {
"include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/**"
}
}
]
}
}

=== No changes after the deploy

>>> [CLI] bundle plan
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged

=== A remote edit that drops testing shows in the plan

>>> [CLI] bundle plan -o json
"update"

>>> [CLI] bundle deploy
Uploading dist/test_helpers-0.1-py3-none-any.whl...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
Updated pipelines.orders
Files: 1 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> [CLI] bundle plan
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged

=== Changing testing updates the pipeline

>>> update_file.py databricks.yml ./tests/** ./tests/unit/**

>>> [CLI] bundle plan
update pipelines.orders

Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged

>>> [CLI] bundle deploy
Uploading dist/test_helpers-0.1-py3-none-any.whl...
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
Updated pipelines.orders
Files: 1 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> print_requests.py //api/2.0/pipelines --nostamp
{
"method": "PUT",
"path": "/api/2.0/pipelines/[ORDERS_ID]",
"sources": [
{
"glob": {
"include": "/Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files/tests/unit/**"
}
}
]
}

=== Removing testing sends an update without it

>>> [CLI] bundle plan
update pipelines.orders

Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged

>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default/files...
Updated pipelines.orders
Files: 1 uploaded, 0 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> print_requests.py //api/2.0/pipelines --nostamp
{
"method": "PUT",
"path": "/api/2.0/pipelines/[ORDERS_ID]",
"has_testing": false
}

>>> [CLI] bundle destroy --auto-approve
The following resources will be deleted:
delete resources.pipelines.orders

This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
delete resources.pipelines.orders

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/pipeline-testing/default

Destroy: 1 deleted
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[project]
name = "orders"
version = "0.1"

[project.optional-dependencies]
test = ["faker"]
35 changes: 35 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
title "Relative test paths resolve to the deployed workspace paths"
trace $CLI bundle validate -o json | jq .resources.pipelines.orders.testing

title "Deploy sends testing, with the local wheel uploaded"
trace $CLI bundle deploy
trace print_requests.py //api/2.0/pipelines --nostamp | jq '{method, path, testing: .body.testing}'

title "No changes after the deploy"
trace $CLI bundle plan
rm "$OUT_REQUESTS"

title "A remote edit that drops testing shows in the plan"
PIPELINE_ID=$(read_id.py orders)
MSYS_NO_PATHCONV=1 $CLI api get "/api/2.0/pipelines/$PIPELINE_ID" | jq '.spec | del(.testing)' > edit.json
MSYS_NO_PATHCONV=1 $CLI api put "/api/2.0/pipelines/$PIPELINE_ID" --json @edit.json &> LOG.edit
rm edit.json
trace $CLI bundle plan -o json | jq '.plan["resources.pipelines.orders"].changes.testing.action'
trace $CLI bundle deploy
trace $CLI bundle plan
rm "$OUT_REQUESTS"

title "Changing testing updates the pipeline"
trace update_file.py databricks.yml './tests/**' './tests/unit/**'
trace $CLI bundle plan
trace $CLI bundle deploy
trace print_requests.py //api/2.0/pipelines --nostamp | jq '{method, path, sources: .body.testing.sources}'

title "Removing testing sends an update without it"
update_file.py databricks.yml "$(sed -n '/^ testing:/,$p' databricks.yml)" ""
trace $CLI bundle plan
trace $CLI bundle deploy
trace print_requests.py //api/2.0/pipelines --nostamp | jq '{method, path, has_testing: (.body | has("testing"))}'

trace $CLI bundle destroy --auto-approve
rm "$OUT_REQUESTS"
4 changes: 4 additions & 0 deletions acceptance/bundle/resources/pipelines/testing/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Runs only against the local test server.
Cloud = false
RecordRequests = true
Ignore = [".databricks"]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
faker
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
def test_orders():
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from pyspark import pipelines as dp
Loading
Loading