Skip to content

[tests] Remove build timing tests and verify fast deployment - #13676

Open
simonrozsival wants to merge 3 commits into
mainfrom
simonrozsival-remove-superseded-perf-tests
Open

simonrozsival wants to merge 3 commits into
mainfrom
simonrozsival-remove-superseded-perf-tests

Conversation

@simonrozsival

@simonrozsival simonrozsival commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Fixes: #13636
Fixes: #13641

Context: #13429
Context: #13646
Context: #13446

Problem

MSBuildDeviceIntegration contained a timing-based performance fixture and a flaky incremental deployment test:

Measuring performance inside functional test runs is unreliable. These tests run on shared CI agents and emulators where CPU contention, disk and emulator I/O, cold caches, and the previous test's state can change timings by more than the regressions they are meant to catch. Loose budgets can miss regressions, while tighter budgets cause flaky failures (for example Build_XAML_Change, #13636). The results are often ignored and cannot reliably surface performance regressions at the PR level. #13646 explored disabling the flaky cases; this change replaces that approach.

Changes

Remove the timing tests. Delete the whole PerformanceTest fixture rather than marking cases [Ignore]:

  • Build_No_Changes, Build_CSharp_Change, Build_AndroidResource_Change, Build_AndroidAsset_Change, Build_AndroidManifest_Change, Build_XAML_Change
  • Install_CSharp_Change, Install_CSharp_FromClean
  • DesignTimeBuild_CSharp_From_Clean

Its budget file, tests/msbuild-times-reference/MSBuildDeviceIntegration.csv, is deleted too; nothing else reads it. Separate, more robust build-time and runtime benchmarks will replace this performance coverage as part of the .NET 12 work.

Keep IncrementalFastDeployment functional and timing-independent. The test is kept for APK and AAB packaging and verifies incremental fast deployment correctness and skipped work without build-speed assertions. #13446 had already removed its elapsed-time comparisons; this change also removes the Xamarin.Forms dependency and its XAML scenario. It uses a plain Android app that logs values from two project-reference libraries, Library1 (an Android library) and Library2 (netstandard2.0):

  1. Build and install the app, run it, and verify that logcat contains INCREMENTAL_FAST_DEPLOY:500:40.
  2. Install again with no changes. Verify selected build and deployment work is skipped:
    • Skipped targets: CoreCompile, _CompileToDalvik, _BuildApkFastDev, _Upload (AAB: _BuildApkSet, _DeployAppBundle). _GenerateTrimmableTypeMap may be skipped or absent.
    • The install binlog runs none of these tasks: Csc, GenerateTrimmableTypeMap, D8, R8, BuildApk, BuildAppBundle, BuildApkSet, InstallApkSet, FastDeploy.
  3. Change Library1.Class1.GetValue () from 500 to 501. The test verifies that the assembly contents change but its size does not. This guards against change detection based only on file size. Explicitly rebuild that library, because Xamarin.ProjectTools builds with BuildingInsideVisualStudio=true, which does not rebuild project references automatically; verify that Csc ran.
  4. Install the app again. Verify:
    • FastDeploy runs, but _CompileToDalvik and _BuildApkFastDev are skipped (AAB: _BuildApkSet and _DeployAppBundle are also skipped). The binlog runs none of Csc, D8, R8, BuildApk, BuildAppBundle, BuildApkSet, or InstallApkSet.
    • The unchanged Library2.dll is not uploaded again (NotifySync SkipCopyFile), and no redundant files are removed (Remove redundant file).
    • The running app logs INCREMENTAL_FAST_DEPLOY:501:40, proving the updated assembly was deployed and loaded.
  5. Repeat the no-change install checks from step 2, then uninstall the app.

No step asserts build or install duration. The only time limit is a timeout while waiting for the app's logcat output, so the test fails instead of hanging if the app never starts.

The other functional deployment tests (AppBundleDeploymentInvalidation, AdbTargetChangesAppBundle, ReInstallIfUserUninstalled) are unchanged. There are no product code changes.

Validation

  • make leeroy CONFIGURATION=Debug: passed.
  • ./dotnet-local.sh build tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj -c Debug --no-restore -v:quiet: passed, with 21 existing Repository has no commit warnings.
  • ADB_TARGET='-s emulator-5554' ./dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net11.0/MSBuildDeviceIntegration.dll --filter 'Name~IncrementalFastDeployment' --logger 'console;verbosity=minimal': passed, with both APK and AAB cases passing on an API 35 arm64 emulator.
  • ./dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net11.0/MSBuildDeviceIntegration.dll --list-tests: IncrementalFastDeployment cases discovered; no PerformanceTest cases discovered.
  • Searching the repository found no remaining references to PerformanceTest or the deleted budget file.

Copilot AI balanced review requested due to automatic review settings November 20, 2026 13:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the removal of the flaky integration scenarios and their unused timing budgets.

No open issues 🎉

What changed in this PR

Removes two historically flaky scenarios from .NET for Android’s device integration suite, with build-performance measurement planned separately through broader MAUI benchmarking. This PR adds no replacement benchmarks or product behavior changes.

Changes:

  • Deletes Build_XAML_Change build/install cases and IncrementalFastDeployment APK/AAB cases, including the latter’s functional assertions.
  • Removes the two unused XAML timing budgets and the orphaned structured-logger import.
  • Leaves the remaining performance tests and separate functional deployment tests unchanged.
File Description
tests/​MSBuildDeviceIntegration/​Tests/​PerformanceTest.cs Removes the XAML-change performance scenario.
tests/​MSBuildDeviceIntegration/​Tests/​InstallTests.cs Removes incremental fast-deployment cases and their unused import.
tests/​msbuild-times-reference/​MSBuildDeviceIntegration.csv Removes build/install XAML timing budgets.

Review effort: Balanced

@simonrozsival simonrozsival changed the title [tests] Remove superseded build-performance scenarios [tests] Remove build-performance timing tests Nov 20, 2026
@simonrozsival simonrozsival changed the title [tests] Remove build-performance timing tests [tests] Remove build timing tests and verify fast deployment Nov 20, 2026
@simonrozsival
simonrozsival enabled auto-merge (squash) November 20, 2026 23:01
simonrozsival and others added 3 commits November 21, 2026 08:03
Context: #13646, #13636, #13641, #13446

Remove Build_XAML_Change and IncrementalFastDeployment instead of disabling them. Build-performance measurement will be handled separately from the general integration suite. IncrementalFastDeployment already uses zero-work assertions rather than elapsed-time comparisons; its removal is intentional.

Remove the orphaned XAML timing budgets and structured-logger import. Preserve the remaining PerformanceTest fixture and separate functional deployment tests.

Validation: MSBuildDeviceIntegration compilation passed with no warnings or errors, test discovery passed with the removed cases absent, and git diff --check passed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Delete the rest of PerformanceTest and its MSBuildDeviceIntegration.csv
budgets. Every remaining case asserted build/install wall-clock time
against hard-coded budgets; their results are not used, and build
performance will be measured with dedicated benchmarks instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run APK and app-bundle deployments before and after a same-size project-reference update. Check that the changed assembly is deployed and unnecessary work is skipped, without elapsed-time thresholds.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@simonrozsival
simonrozsival force-pushed the simonrozsival-remove-superseded-perf-tests branch from 0179deb to 38a72b5 Compare November 21, 2026 07:03
@simonrozsival simonrozsival added the ready-for-review Ready for review label Nov 21, 2026
@simonrozsival

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #13676

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code assessment: no blocking findings. Reviewed all three changed files, including the deleted fixture and timing-budget file, and traced the project-update, build, fast-deployment logging, and activity/logcat helpers. Findings: 0 errors, 0 warnings, 1 non-blocking suggestion posted inline.

The replacement checks observable behavior rather than elapsed time: unchanged installs skip selected tasks across projects, the modified library retains its assembly size while changing contents, and a force-restarted app reports the new library value. Explicitly rebuilding the library is consistent with the harness's BuildingInsideVisualStudio=true behavior. Removing Xamarin.Forms isolates this scenario from XAML compilation, and no remaining code/configuration references to the deleted fixture or budget file were found.

CI verification is incomplete: every returned GitHub check reports success, including the aggregate dotnet-android check for Azure build 1693458. Direct Azure build/timeline queries were blocked by Azure CLI setup/authentication errors, so I could not independently inspect the Azure result or non-gating test failures. No local device tests were rerun for this review.

Generated by Android PR Reviewer for #13676 · copilot · gpt61 · 26.9 AIC · ⌖ 7.89 AIC · ⊞ 20.3K
Comment /review to run again

CollectionAssert.Contains (tasks, "FastDeploy", "The updated assembly should be fast deployed.");
AssertTasksNotExecuted (tasks, "Csc", "D8", "R8", "BuildApk", "BuildAppBundle", "BuildApkSet", "InstallApkSet");
Assert.IsFalse (builder.LastBuildOutput.Any (line => line.Contains ("Remove redundant file")), "No files should be deleted.");
Assert.IsTrue (builder.LastBuildOutput.Any (line => line.Contains ("NotifySync SkipCopyFile") && line.Contains ("Library2.dll")),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 💡 Testing — Consider also asserting that the unchanged app assembly (app.AssemblyName + ".dll") has no NotifySync CopyFile entry during this incremental install. Only Library1 changes now, and the test already verifies that Csc does not run, so the app assembly should remain unchanged too. The current Library2.dll assertion would still pass if deployment unnecessarily re-uploaded the app assembly whenever an Android project reference changed. This would extend the selective-upload coverage without reintroducing timing assertions.

{Rule: Test edge cases; test assertions must be specific}

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Flaky test] InstallTests.IncrementalFastDeployment [Flaky test] PerformanceTest.Build_XAML_Change

2 participants