Repository navigation
[tests] Remove build timing tests and verify fast deployment - #13676
simonrozsival wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the removal of the flaky integration scenarios and their unused timing budgets.
No open issues 🎉
What changed in this PR
Removes two historically flaky scenarios from .NET for Android’s device integration suite, with build-performance measurement planned separately through broader MAUI benchmarking. This PR adds no replacement benchmarks or product behavior changes.
Changes:
- Deletes
Build_XAML_Changebuild/install cases andIncrementalFastDeploymentAPK/AAB cases, including the latter’s functional assertions. - Removes the two unused XAML timing budgets and the orphaned structured-logger import.
- Leaves the remaining performance tests and separate functional deployment tests unchanged.
| File | Description |
|---|---|
tests/MSBuildDeviceIntegration/Tests/PerformanceTest.cs |
Removes the XAML-change performance scenario. |
tests/MSBuildDeviceIntegration/Tests/InstallTests.cs |
Removes incremental fast-deployment cases and their unused import. |
tests/msbuild-times-reference/MSBuildDeviceIntegration.csv |
Removes build/install XAML timing budgets. |
Review effort: Balanced
Context: #13646, #13636, #13641, #13446 Remove Build_XAML_Change and IncrementalFastDeployment instead of disabling them. Build-performance measurement will be handled separately from the general integration suite. IncrementalFastDeployment already uses zero-work assertions rather than elapsed-time comparisons; its removal is intentional. Remove the orphaned XAML timing budgets and structured-logger import. Preserve the remaining PerformanceTest fixture and separate functional deployment tests. Validation: MSBuildDeviceIntegration compilation passed with no warnings or errors, test discovery passed with the removed cases absent, and git diff --check passed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Delete the rest of PerformanceTest and its MSBuildDeviceIntegration.csv budgets. Every remaining case asserted build/install wall-clock time against hard-coded budgets; their results are not used, and build performance will be measured with dedicated benchmarks instead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Run APK and app-bundle deployments before and after a same-size project-reference update. Check that the changed assembly is deployed and unnecessary work is skipped, without elapsed-time thresholds. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
0179deb to
38a72b5
Compare
|
/review |
|
✅ Android PR Reviewer completed successfully!
|
There was a problem hiding this comment.
Code assessment: no blocking findings. Reviewed all three changed files, including the deleted fixture and timing-budget file, and traced the project-update, build, fast-deployment logging, and activity/logcat helpers. Findings: 0 errors, 0 warnings, 1 non-blocking suggestion posted inline.
The replacement checks observable behavior rather than elapsed time: unchanged installs skip selected tasks across projects, the modified library retains its assembly size while changing contents, and a force-restarted app reports the new library value. Explicitly rebuilding the library is consistent with the harness's BuildingInsideVisualStudio=true behavior. Removing Xamarin.Forms isolates this scenario from XAML compilation, and no remaining code/configuration references to the deleted fixture or budget file were found.
CI verification is incomplete: every returned GitHub check reports success, including the aggregate dotnet-android check for Azure build 1693458. Direct Azure build/timeline queries were blocked by Azure CLI setup/authentication errors, so I could not independently inspect the Azure result or non-gating test failures. No local device tests were rerun for this review.
Generated by Android PR Reviewer for #13676 · copilot · gpt61 · 26.9 AIC · ⌖ 7.89 AIC · ⊞ 20.3K
Comment /review to run again
| CollectionAssert.Contains (tasks, "FastDeploy", "The updated assembly should be fast deployed."); | ||
| AssertTasksNotExecuted (tasks, "Csc", "D8", "R8", "BuildApk", "BuildAppBundle", "BuildApkSet", "InstallApkSet"); | ||
| Assert.IsFalse (builder.LastBuildOutput.Any (line => line.Contains ("Remove redundant file")), "No files should be deleted."); | ||
| Assert.IsTrue (builder.LastBuildOutput.Any (line => line.Contains ("NotifySync SkipCopyFile") && line.Contains ("Library2.dll")), |
There was a problem hiding this comment.
🤖 💡 Testing — Consider also asserting that the unchanged app assembly (app.AssemblyName + ".dll") has no NotifySync CopyFile entry during this incremental install. Only Library1 changes now, and the test already verifies that Csc does not run, so the app assembly should remain unchanged too. The current Library2.dll assertion would still pass if deployment unnecessarily re-uploaded the app assembly whenever an Android project reference changed. This would extend the selective-upload coverage without reintroducing timing assertions.
{Rule: Test edge cases; test assertions must be specific}
Fixes: #13636
Fixes: #13641
Context: #13429
Context: #13646
Context: #13446
Problem
MSBuildDeviceIntegrationcontained a timing-based performance fixture and a flaky incremental deployment test:PerformanceTestcompared wall-clock build, install, and design-time-build durations with hard-coded budgets intests/msbuild-times-reference/MSBuildDeviceIntegration.csv.InstallTests.IncrementalFastDeploymentused a Xamarin.Forms XAML scenario and was flaky in CI ([Flaky test] InstallTests.IncrementalFastDeployment #13641). Its elapsed-time comparisons had already been replaced with no-work assertions in [Xamarin.Android.Build.Tasks] Skip unchanged AAB installs #13446.Measuring performance inside functional test runs is unreliable. These tests run on shared CI agents and emulators where CPU contention, disk and emulator I/O, cold caches, and the previous test's state can change timings by more than the regressions they are meant to catch. Loose budgets can miss regressions, while tighter budgets cause flaky failures (for example
Build_XAML_Change, #13636). The results are often ignored and cannot reliably surface performance regressions at the PR level. #13646 explored disabling the flaky cases; this change replaces that approach.Changes
Remove the timing tests. Delete the whole
PerformanceTestfixture rather than marking cases[Ignore]:Build_No_Changes,Build_CSharp_Change,Build_AndroidResource_Change,Build_AndroidAsset_Change,Build_AndroidManifest_Change,Build_XAML_ChangeInstall_CSharp_Change,Install_CSharp_FromCleanDesignTimeBuild_CSharp_From_CleanIts budget file,
tests/msbuild-times-reference/MSBuildDeviceIntegration.csv, is deleted too; nothing else reads it. Separate, more robust build-time and runtime benchmarks will replace this performance coverage as part of the .NET 12 work.Keep
IncrementalFastDeploymentfunctional and timing-independent. The test is kept for APK and AAB packaging and verifies incremental fast deployment correctness and skipped work without build-speed assertions. #13446 had already removed its elapsed-time comparisons; this change also removes the Xamarin.Forms dependency and its XAML scenario. It uses a plain Android app that logs values from two project-reference libraries,Library1(an Android library) andLibrary2(netstandard2.0):INCREMENTAL_FAST_DEPLOY:500:40.CoreCompile,_CompileToDalvik,_BuildApkFastDev,_Upload(AAB:_BuildApkSet,_DeployAppBundle)._GenerateTrimmableTypeMapmay be skipped or absent.Csc,GenerateTrimmableTypeMap,D8,R8,BuildApk,BuildAppBundle,BuildApkSet,InstallApkSet,FastDeploy.Library1.Class1.GetValue ()from500to501. The test verifies that the assembly contents change but its size does not. This guards against change detection based only on file size. Explicitly rebuild that library, becauseXamarin.ProjectToolsbuilds withBuildingInsideVisualStudio=true, which does not rebuild project references automatically; verify thatCscran.FastDeployruns, but_CompileToDalvikand_BuildApkFastDevare skipped (AAB:_BuildApkSetand_DeployAppBundleare also skipped). The binlog runs none ofCsc,D8,R8,BuildApk,BuildAppBundle,BuildApkSet, orInstallApkSet.Library2.dllis not uploaded again (NotifySync SkipCopyFile), and no redundant files are removed (Remove redundant file).INCREMENTAL_FAST_DEPLOY:501:40, proving the updated assembly was deployed and loaded.No step asserts build or install duration. The only time limit is a timeout while waiting for the app's logcat output, so the test fails instead of hanging if the app never starts.
The other functional deployment tests (
AppBundleDeploymentInvalidation,AdbTargetChangesAppBundle,ReInstallIfUserUninstalled) are unchanged. There are no product code changes.Validation
make leeroy CONFIGURATION=Debug: passed../dotnet-local.sh build tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.csproj -c Debug --no-restore -v:quiet: passed, with 21 existingRepository has no commitwarnings.ADB_TARGET='-s emulator-5554' ./dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net11.0/MSBuildDeviceIntegration.dll --filter 'Name~IncrementalFastDeployment' --logger 'console;verbosity=minimal': passed, with both APK and AAB cases passing on an API 35 arm64 emulator../dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net11.0/MSBuildDeviceIntegration.dll --list-tests:IncrementalFastDeploymentcases discovered; noPerformanceTestcases discovered.PerformanceTestor the deleted budget file.