test: run each test bundle in its own runtime with JUnit discovery by joaodinissf · Pull Request #1599 · dsldevkit/dsl-devkit

joaodinissf · 2026-11-03T18:33:33Z

Summary

Each test bundle now runs its own tests in its own Eclipse runtime. A new test class needs no registration, and the hand-maintained *TestSuite classes and AllTests are gone.

tycho-surefire-plugin is configured once in ddk-parent/pom.xml. Individual test poms carry no test configuration.
Surefire finds test classes by name within the bundle: *Test, *Tests, Test*, *TestCase. Classes run in alphabetical order within a module, and the 30-minute timeout applies per module.
The 15 *TestSuite classes, the two AllTests classes, their launch configurations and the JUnit suite dependencies of the test bundles are removed. ddk-target/ddk.target keeps its two JUnit suite units, as Tycho's JUnit 6 provider imports those packages. The UI test bundles keep a launch configuration (CheckUiTests.launch, XtextUiTests.launch, DdkTestUiTests.launch) which runs all tests of the project via JUnit discovery in Eclipse. These launches and the Maven test runs use the org.eclipse.platform.ide product, as ddk-target/ddk.target has no org.eclipse.sdk bundle.
com.avaloq.tools.ddk.checkcfg.ui.test only held an empty suite, so the bundle is removed. com.avaloq.tools.ddk.xtext.test no longer exports its now empty com.avaloq.tools.ddk.xtext package.
AGENTS.md describes the new setup.

Changes the per-module runs need

com.avaloq.tools.ddk.check.test.runtime.tests is now packaged as eclipse-test-plugin. As eclipse-plugin, its tests would not run.
com.avaloq.tools.ddk.checkcfg.core.test now declares com.avaloq.tools.ddk.check.test.runtime.ui and com.avaloq.tools.ddk.check.ui. Six of its tests failed without them, as they relied on bundles the shared runtime happened to contain.

CI

check-surefire-reports.sh now requires a report for every concrete class declaring tests, instead of any single report. This catches a test bundle with the wrong packaging, a class name Surefire does not include and a module the reactor skipped. Classes which only inherit their tests are not covered.
Failed builds now archive .metadata/.log of every test module. The log of xtext.test was never archived, as upload-artifact ignores hidden files by default (include-hidden-files is now set).

Fixes surfaced by the switch

Order-dependent Check tests. The generated CheckInjectorProvider restores the global EMF registries to a copy taken on its first use. If that first use precedes the activation of the Check UI plug-in, every later standalone Check test wipes the Check validator registered since. The new RegistryIsolatingCheckInjectorProvider restores the state from before each test, and the standalone Check tests use it.
BugAig1084, BugAig830, BugAig1314, BugDsl27 are renamed to …Test, as Surefire would not include them otherwise.
com.avaloq.tools.ddk.test.ui.test was never referenced by AllTests, and its three test classes had rotted. They now run and pass:
- ErrorLogListenerTest ignored a location named after the test's old class name, and checked the log before the job manager logged the failure.
- DeChKeyboardLayoutTest referenced the layout under a package which no longer exists. It now checks the layout's keystrokes directly, as typing with it would change SWTBot's global keyboard layout for all later tests.
- SwtBotRadioTest opened the preferences via the Window menu, which does not exist on macOS, and changed a workbench preference. It now uses a shell of its own.

Test set before and after

	Before	After
Tests	451	455
Test classes	98	101
Failures / errors	0	0
Skipped	2	2 (same)

Both columns are local clean verify runs on macOS with Temurin 21, compared test by test: "before" is master at 55b37d0e8, "after" is this branch, which also ran all quality checks. Linux, which CI uses, was not run locally.

No longer run: none.
Renamed: BugAig1084, BugAig830, BugAig1314, BugDsl27 → …Test (same tests).
Newly run: ErrorLogListenerTest (1), DeChKeyboardLayoutTest (1), SwtBotRadioTest (2).

Cost and behaviour

Measured on an earlier base, one runtime per module costs about a minute of Surefire time in a serial local build (155 s against 92 s).
With -T, test runtimes run one at a time through reactorConcurrencyLevel, as they never overlapped with the single runtime before. CheckQuickfixTest failed intermittently on macOS when they did, and passed in all serial runs. Everything else stays parallel, and CI builds without -T.
A local build with all quality checks took 459 s serially, and 248 s and 275 s in two -T 3C runs. Without the lock, -T 3C took 188 s and 210 s, with up to 4 test runtimes at once.
With --fail-at-end, a failure in check.core.test skips the 36 tests of check.ui.test, check.test.runtime.tests and sample.helloworld.ui.test, which depend on it. They run once the failure is fixed.

🤖 Generated with Claude Code

rubenporras

Thanks a lot!

rubenporras · 2026-11-04T06:00:58Z

@@ -1,43 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>


I think that for the UI tests we still need the launch configurations, which can be adapted to use JUnit discovery. For the core tests, I agree we can close them

Kept as CheckUiTests.launch, XtextUiTests.launch and DdkTestUiTests.launch, each running all tests of its project via JUnit discovery. AllTests DDK.launch is gone.

Posted by Claude Code on behalf of @joaodinissf.

rubenporras · 2026-11-04T06:03:06Z

+ * JUnit's default scanner only sees the classes of the test bundle itself, so package selectors such as the one in {@link AllTests} would miss the
+ * tests of all other bundles in the test runtime. This scanner is registered via {@code META-INF/services}.
+ */
+public class OsgiClasspathScanner implements ClasspathScanner {


I think this is weird. We should just have different junit runs for each plugin, and not have a single entry point. It is true that is a bit more expensive but not worth the complexity, let us just do plain Junit, once test execution per module

Removed. Each test bundle now has its own plain JUnit run, configured once in ddk-parent.

Posted by Claude Code on behalf of @joaodinissf.

rubenporras · 2026-11-04T06:03:28Z

-  DdkTestSuite.class
-})
-// @Format-On
+@SelectPackages("com.avaloq.tools.ddk")


And completely remove this class

Removed.

Posted by Claude Code on behalf of @joaodinissf.

rubenporras · 2026-11-06T08:07:38Z

I know that you can run a build in parallel with -T 3 and still tell maven surefire (or maybe Tycho, I am not sure of the details) that UI tests execution must be serialized. Could you look into that?

joaodinissf · 2026-11-06T10:40:23Z

Yes, Tycho's reactorConcurrencyLevel does this, and it's now set to 1 in ddk-parent. Only the forked test runs take turns; compilation and quality checks stay parallel.

In a full local -T 3C build on macOS, the most test runtimes running at once drops from 4 to 1, and the build takes 248-275 s instead of 188-210 s (two runs each), against 459 s without -T. CI builds without -T, so the lock doesn't change it. A follow-up PR will run the modules that don't need a workbench headless, which wins back part of that time (213-233 s in local runs). Those modules also skip the lock, though that adds no measurable saving today.

Posted by Claude Code on behalf of @joaodinissf.

…ests The generated CheckInjectorProvider restores the global EMF registries to a copy taken on its first use, not to the state before each test. When that first use precedes the activation of the Check UI plug-in, every later standalone Check test wipes the Check validator the UI plug-in has registered since. Tests using the UI injector, such as CheckApiAccessValidationsTest and CheckValidationTest, then validate without any Check validator and miss the expected issues. The hand-maintained suites only avoided this by their order. Add RegistryIsolatingCheckInjectorProvider, which restores the state from before each test, and use it in the standalone Check tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

These tests were never part of the aggregated test run and had rotted: - ErrorLogListenerTest ignored a location named after the old class name of the test, and checked the log before the job manager had logged the failure. Wait for the logged exception, and also assert that it is ignored. - DeChKeyboardLayoutTest referenced the layout under a package which no longer exists. SWTBot fixes its keyboard layout globally on first use, so typing with DE_CH would also break all later SWTBot tests; check the layout's keystrokes directly instead. - SwtBotRadioTest opened the preferences via the Window menu, which does not exist on macOS, and changed a workbench preference. Use a shell of its own with a radio group and look the buttons up within it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BugAig1084, BugAig830, BugAig1314 and BugDsl27 are the only test classes whose names do not end in Test or Tests, which automatic test discovery by class name relies on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@suite

All tests ran in the single OSGi runtime of com.avaloq.tools.ddk.xtext.test, through AllTests and one hand-maintained @suite per test bundle. A test class not listed in a suite never ran, and the tests of com.avaloq.tools.ddk.test.ui.test were not referenced at all. Each eclipse-test-plugin bundle now runs its own tests in its own runtime. tycho-surefire-plugin is configured once in ddk-parent and finds the test classes by name, so a new test class needs no registration. Classes run in alphabetical order within a module, and the timeout applies per module. AllTests, the suites, their launch configurations and the JUnit suite dependencies are removed. The UI test bundles keep a launch configuration each, which runs all tests of the project via JUnit discovery in Eclipse. These launches and the Maven test runs use the org.eclipse.platform.ide product, as the target platform has no org.eclipse.sdk bundle. com.avaloq.tools.ddk.checkcfg.ui.test only held an empty suite, so the bundle is removed. With -T, test runtimes run one at a time through reactorConcurrencyLevel, as they never overlapped with the single runtime before, and CheckQuickfixTest failed intermittently on macOS when they did. Everything else stays parallel, and a build without -T is unaffected. Two bundles relied on the shared runtime. The packaging of com.avaloq.tools.ddk.check.test.runtime.tests is now eclipse-test-plugin, as its tests would not run otherwise. com.avaloq.tools.ddk.checkcfg.core.test now declares its runtime dependencies on com.avaloq.tools.ddk.check.test.runtime.ui and com.avaloq.tools.ddk.check.ui. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t logs With one test run per module, a single surefire report no longer shows that the tests ran. The check now requires a report for every concrete class declaring tests, which also catches a test bundle not packaged as eclipse-test-plugin, a class name Surefire does not include and a module the reactor skipped. Failed builds now archive the workspace log of every test module. The log of xtext.test was never archived: upload-artifact ignores hidden files by default, so the step found no files for .metadata/.log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

joaodinissf force-pushed the test-discovery branch from 6854e70 to 9d1e10e Compare November 3, 2026 18:43

joaodinissf mentioned this pull request Nov 3, 2026

fix(test): invert results only for not-fixed @Issue tests #1600

Merged

2 of 3 tasks

rubenporras requested changes Nov 4, 2026

View reviewed changes

joaodinissf force-pushed the test-discovery branch 2 times, most recently from 103d89e to 77265e5 Compare November 4, 2026 12:04

joaodinissf changed the title ~~test: discover tests automatically instead of hand-maintained suites~~ test: run each test bundle in its own runtime with JUnit discovery Nov 4, 2026

joaodinissf force-pushed the test-discovery branch 2 times, most recently from 9d1ae7f to 3e90c8f Compare November 5, 2026 23:15

joaodinissf force-pushed the test-discovery branch from 3e90c8f to 0786bf7 Compare November 6, 2026 10:38

joaodinissf mentioned this pull request Nov 6, 2026

test: run test modules headless unless they need a workbench #1622

Draft

joaodinissf added this pull request to stack #1623 November 6, 2026 10:43

joaodinissf force-pushed the test-discovery branch from 0786bf7 to 7b80500 Compare November 12, 2026 19:52

joaodinissf and others added 5 commits November 12, 2026 21:57

joaodinissf force-pushed the test-discovery branch from 7b80500 to 4d5163b Compare November 12, 2026 20:57

Navigation Menu

Repository navigation

test: run each test bundle in its own runtime with JUnit discovery - #1599

test: run each test bundle in its own runtime with JUnit discovery#1599
joaodinissf wants to merge 5 commits into
masterfrom
test-discovery

joaodinissf commented Nov 3, 2026 •

edited

Loading

Uh oh!

rubenporras left a comment

Uh oh!

rubenporras Nov 4, 2026

Uh oh!

joaodinissf Nov 6, 2026

Uh oh!

rubenporras Nov 4, 2026

Uh oh!

joaodinissf Nov 6, 2026

Uh oh!

rubenporras Nov 4, 2026

Uh oh!

joaodinissf Nov 6, 2026

Uh oh!

rubenporras commented Nov 6, 2026

Uh oh!

joaodinissf commented Nov 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1,43 +0,0 @@
		<?xml version="1.0" encoding="UTF-8" standalone="no"?>

Conversation

joaodinissf commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes the per-module runs need

CI

Fixes surfaced by the switch

Test set before and after

Cost and behaviour

Uh oh!

rubenporras left a comment

Choose a reason for hiding this comment

Uh oh!

rubenporras Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

joaodinissf Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

rubenporras Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

joaodinissf Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

rubenporras Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

joaodinissf Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

rubenporras commented Nov 6, 2026

Uh oh!

joaodinissf commented Nov 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joaodinissf commented Nov 3, 2026 •

edited

Loading