Skip to content

[tests] Migrate remaining test suites to MSTest/MTP - #13487

Open
simonrozsival wants to merge 9 commits into
mainfrom
simonrozsival-mstest-migrate-remaining-suites
Open

simonrozsival wants to merge 9 commits into
mainfrom
simonrozsival-mstest-migrate-remaining-suites

Conversation

@simonrozsival

@simonrozsival simonrozsival commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Context: #13434

#13434 (merged) moved the on-device suites (Mono.Android.NET-Tests, Java.Interop-Tests, Xamarin.Android.JcwGen-Tests) and Xamarin.SourceWriter-Tests to MSTest on Microsoft.Testing.Platform (MTP). Every other test project was still NUnit, so the repo had to carry two runners, TestRunner.Core, the NUnit console scripts, dotnet-test-slicer, and per-project global.json files to select MTP.

This PR migrates all remaining NUnit test projects, which means one framework and one runner for the whole repository.

Projects

  • Java tooling: generator-Tests, Java.Interop.Tools.{Generator,JavaCallableWrappers,JavaSource,JavaTypeSystem,Maven}-Tests, Xamarin.Android.Tools.{Bytecode,ApiXmlAdjuster}-Tests
  • Android tooling: Xamarin.Android.Tools.{Aidl,AndroidSdk}-Tests, Microsoft.Android.Build.BaseTasks-Tests, Microsoft.Android.Sdk.Analysis.Tests
  • Build tasks: Microsoft.Android.Build.Tasks.Tests, Xamarin.Android.Build.Tests and its shared Xamarin.ProjectTools library
  • Device integration: MSBuildDeviceIntegration

build-tools/scripts/NUnitReferences.projitems is replaced by MSTestReferences.projitems. It makes each project an MTP executable (EnableMSTestRunner) and references MSTest.* and Microsoft.Testing.Extensions.TrxReport with versions from eng/Versions.props.

MSTest.SourceGeneration is used everywhere except Xamarin.Android.Build.Tests and MSBuildDeviceIntegration. Those two see Xamarin.Android.Build.Tasks' internal DynamicallyAccessedMemberTypes polyfill through InternalsVisibleTo, which conflicts with the generated code (CS0433), so they use reflection discovery (UseMSTestSourceGeneration=false).

The root global.json now selects MTP ("test": { "runner": "Microsoft.Testing.Platform" }). The DOTNET_TEST_RUNNER environment variable is removed. The per-project global.json files for the on-device suites (Mono.Android.NET-Tests, Xamarin.Android.JcwGen-Tests) are kept: without them, the _MTPBuild MSBuild process that dotnet test spawns in apk-instrumentation.yaml resolved the 11.0.100-rc.1 Android workload manifest and failed with NETSDK1139.

Removed: tests/TestRunner.Core, build-tools/scripts/nunit3-console*, the NUnitVersion/NUnitConsoleVersion properties, and the dead run-nunit-tests/list-nunit-tests make targets (RunTests.targets no longer exists).

Conversion notes

  • [TestCase]/[TestCaseSource] → [DataRow]/[DynamicData]. TestCaseData.SetName() → TestDataRow<T>.DisplayName. [Values] → [CombinatorialData]/[CombinatorialValues]. [ValueSource (…ValidRuntimeConfigurations)] → a small [ValidRuntimeConfigurations] values provider.
  • [Category] → [TestCategory], [Platform] → [OSCondition], [Explicit] → [Ignore]. NUnit [Retry (n)] (n total attempts) → MSTest [Retry (n - 1)] (n - 1 retries).
  • Parallelism matches NUnit. Xamarin.Android.Build.Tests and Microsoft.Android.Build.Tasks.Tests run method-level parallel. Classes that were not [Parallelizable (Children/All)] are [DoNotParallelize]. MSBuildDeviceIntegration stays assembly-wide [DoNotParallelize].
  • BaseTest uses the MSTest TestContext. TestName comes from TestDisplayName, so every data row still gets its own directory. Outcome checks use CurrentTestOutcome, and attachments use AddResultFile(). [OneTimeSetUp]/[OneTimeTearDown] → [ClassInitialize]/[ClassCleanup] with InheritanceBehavior.BeforeEachDerivedClass.
  • NUnit fixture parameters have no MSTest equivalent:
    • TimeZoneInfoTests and LocalizationTests become abstract bases with …CoreCLRTests/…NativeAOTTests subclasses. Each subclass installs its app lazily, once per runtime.
    • BundleToolTests and BundleToolNoAbiSplitTests take the runtime as a [DataRow].

CI

run-nunit-tests.yaml and run-sliced-nunit-tests.yaml are replaced by run-mstest-tests.yaml. It runs the test app directly (dotnet <dll> --report-trx) and publishes its TRX. A failing run marks the step SucceededWithIssues, and fail-on-issue.yaml then fails the job, the same as before.

  • Slicing: --list-tests json lists the selected tests. The job takes every $(System.TotalJobsInPhase)th test, sorted by UID, and passes those UIDs in a --filter-uid response file. This replaces dotnet-test-slicer, which only understands NUnit. It is the same count-based round-robin split the slicer did: our pipelines never passed it a --balance-file, so it weighed every test equally, and neither approach balances by test duration, so slice sizes are unchanged.
  • Retry: MSBuildDeviceIntegration references Microsoft.Testing.Extensions.Retry, new in eng/Versions.props and eng/Version.Details.xml. The lanes that used the slicer's auto-retry now pass --retry-failed-tests 1 and publish every attempt from Retries/.
  • Filters:
    • Name = BuildTest → ClassName=Xamarin.Android.Build.Tests.BuildTest (also for PackagingTest, XASdkTests, AndroidDependenciesTests and SystemApplicationTests).
    • method == X → Name~X.
    • cat != A & … → TestCategory!=A&… (ExcludedNightlyNUnitCategories is renamed to ExcludedNightlyTestCategories).
  • The Android tools stage uses dotnet test --project … --report-trx, because DotNetCoreCLI@2 injects --logger trx, which MTP rejects. The run-dotnet-preview.yaml callers switch to --report-trx.

Results

Rebased onto main (bfda00d). All 17 migrated projects build with -p:_AndroidTreatWarningsAsErrors=true.

Standalone suites, MTP run on this branch (passed/failed/skipped):

Suite MSTest/MTP
Xamarin.Android.Tools.Aidl-Tests 17/0/0
Microsoft.Android.Build.BaseTasks-Tests 166/0/4
Xamarin.Android.Tools.AndroidSdk-Tests 444/4/11
Microsoft.Android.Sdk.Analysis.Tests 14/0/1
Java.Interop.Tools.Generator-Tests 18/0/0
Java.Interop.Tools.JavaCallableWrappers-Tests 21/0/0
Java.Interop.Tools.JavaSource-Tests 41/0/0
Java.Interop.Tools.JavaTypeSystem-Tests 17/0/0
Java.Interop.Tools.Maven-Tests 106/0/0
Xamarin.Android.Tools.ApiXmlAdjuster-Tests 13/0/0
Xamarin.Android.Tools.Bytecode-Tests 132/0/2
generator-Tests 478/0/0
Microsoft.Android.Build.Tasks.Tests 1071/0/16

The four AndroidSdk-Tests failures (Constructor_SetValuesFromPath, CreateProcessStartInfo_*) are environmental on my machine and fail the same way under NUnit.

Comparison with main's NUnit suites:

  • Main's NUnit BaseTasks-Tests and AndroidSdk-Tests were run in a clean worktree. Totals are 170 and 459 (NUnit --list-tests), the same as MTP, with identical per-method case counts.
  • For every test source file, the NUnit test-case count on main was compared with the MSTest count on this branch. They match except for the intentional expansions listed below: fixture parameters and [Values] that become rows (AndroidUpdateResourcesTest +3, TrimmableTypeMapBuildTests +1, ExtractTypeMapKeysFromNativeAotObjectCoverageTests +2, BundleTool*Tests +1 each, DebuggingTest +1).
  • MTP discovery: Xamarin.Android.Build.Tests 1853, MSBuildDeviceIntegration 4533.

Device runs before the rebase, with a local Release SDK on an API 35 arm64 emulator:

Check MSTest/MTP
Xamarin.Android.Build.Tests TestCategory=SmokeTests 35/0/2 (the 2 are Windows-only Assert.Inconclusive)
UncaughtExceptionTests, BundleToolTests, WebViewJavascriptInterfaceTests (with --retry-failed-tests 1) 8/0/0
LocalizationCoreCLRTests.CheckLocalizationIsCorrectNode15 (then persist.sys.locale is back to en-US) 30/1/0 (ar-KM: the app didn't start within the logcat window after the reboot, all 3 attempts)
2× TimeZone CoreCLR, 2× TimeZone NativeAOT, 2× Localization CoreCLR, 4× InstallTests (UID slice) 10/0/0

Discovery differs from NUnit only in naming. Rows that NUnit named with TestName =/SetName() now appear under their method name with a DisplayName, and fixture-parameterized classes appear under their new subclass names.

The run-mstest-tests.yaml script was exercised locally with pwsh: slicing Xamarin.Android.Build.Tests 3 ways (8/8/8 of 24), plus a 4-way dry run over the 1182 TimeZone tests. All pipeline YAML parses.

Notable conversions

  • AndroidSdk-Tests' StreamingTests (added on main) built an in-process MTP TestApplication with --report-trx. Under the MTP 2.6 that MSTest brings in, --report-trx always makes the app a test-host controller, which relaunches the entry assembly (the MSTest test app itself) and recurses. They now run the built Microsoft.Android.Run under dotnet test --test-modules, which is how Android test projects run it, with a fake adb replaying an am instrument transcript. This also tests the MTP 2.2.1 that Microsoft.Android.Run ships with. Cancellation is driven by the fake adb sending SIGINT to Microsoft.Android.Run. The missing-adb case uses a non-executable adb, because Microsoft.Android.Run rejects a missing path before MTP starts. The two parser/state tests stay in process.
  • LocalizationTests restore persist.sys.locale to en-US in a [ClassCleanup]. The NUnit version never restored it, so later suites on the same device ran in the last tested locale.
  • AndroidComputeResPathsTests and ResolveSdksTaskTests dropped main's no-op AfterAllTests overrides, because AfterAllTests is now a static [ClassCleanup]. The only effect is an extra adb kill-server on Windows after those classes.
  • UncaughtExceptionTests' Assert.Multiple (no MSTest equivalent) collects the property-restore failures and asserts once.

Known issues

  • The full Xamarin.Android.Build.Tests and MSBuildDeviceIntegration suites were not run locally; they take many hours. CI will be the first full run.
  • AndroidUpdateResourcesTest.LightlyModifyLayout drops NUnit's [Repeat (3)]; it runs once. ZipArchiveExTests' [Repeat (100)] becomes an explicit loop.
  • [Explicit] tests become [Ignore]. MSTest has no opt-in-only equivalent, so they no longer run when selected explicitly by name.

@simonrozsival
simonrozsival force-pushed the simonrozsival-mstest-migrate-remaining-suites branch from 567ebc4 to d6fe575 Compare November 11, 2026 12:20
@simonrozsival
simonrozsival force-pushed the simonrozsival-mstest-migrate-remaining-suites branch from d6fe575 to 64f0dfd Compare November 11, 2026 17:13
Base automatically changed from simonrozsival-mstest-delivery-host-ci to main November 17, 2026 14:35
@simonrozsival
simonrozsival force-pushed the simonrozsival-mstest-migrate-remaining-suites branch from c727bc5 to 9ee1bd8 Compare November 17, 2026 15:10
@simonrozsival
simonrozsival marked this pull request as ready for review November 17, 2026 15:10
Copilot AI balanced review requested due to automatic review settings November 17, 2026 15:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The APK size scripts can copy stale .apkdesc files because positional DLL arguments no longer run BuildReleaseArm64, and the same invocation needs correction in the test documentation.

1 open issue
What changed in this PR

Completes the test-framework migration started in #13434, unifying .NET for Android’s remaining test suites on MSTest and Microsoft.Testing.Platform (MTP).

Changes:

  • Converts test attributes, assertions, data sources, lifecycle hooks, and parallelism settings.
  • Centralizes MTP configuration and updates CI test execution, filtering, and retry settings.
  • Removes NUnit infrastructure and refreshes testing documentation.
File Description
tests/​Xamarin.SourceWriter-Tests/​Xamarin.SourceWriter-Tests.csproj Uses shared MSTest references.
tests/​Xamarin.SourceWriter-Tests/​TypeReferenceWriterTests.cs Converts test attributes and null assertion.
tests/​Xamarin.SourceWriter-Tests/​README.md Documents MTP host-test execution.
tests/​Xamarin.SourceWriter-Tests/​PropertyWriterTests.cs Converts tests to MSTest.
tests/​Xamarin.SourceWriter-Tests/​MethodWriterTests.cs Converts tests to MSTest.
tests/​Xamarin.SourceWriter-Tests/​InterfaceWriterTests.cs Converts tests to MSTest.
tests/​Xamarin.SourceWriter-Tests/​FieldWriterTests.cs Converts tests to MSTest.
tests/​Xamarin.SourceWriter-Tests/​EventWriterTests.cs Converts tests to MSTest.
tests/​Xamarin.SourceWriter-Tests/​DelegateWriterTests.cs Converts tests to MSTest.
tests/​Xamarin.SourceWriter-Tests/​ConstructorWriterTests.cs Converts tests to MSTest.
tests/​Xamarin.SourceWriter-Tests/​ClassWriterTests.cs Converts tests and disables assembly parallelism.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​Xamarin.Android.Tools.Bytecode-Tests.csproj Uses shared MSTest references.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​NullableAnnotationTests.cs Converts attributes and null assertions.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​NonGenericGlobalTypeTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​ModuleInfoTests.cs Converts attributes and assertions.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JvmOverloadsConstructorTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaTypeTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaTypeNoParametersTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaType.RNCTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaType.RNC.RPNCTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaType.PSCTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaType.ASCTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaType.1Tests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaType.1MyStringListTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaEnumTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​JavaAnnotationTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​IJavaInterfaceTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​ExpectedTypeDeclaration.cs Switches assertion alias to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​ExpectedMethodDeclaration.cs Switches assertion alias to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​ExpectedInnerClassInfo.cs Switches assertion alias to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​ExpectedFieldDeclaration.cs Switches assertion alias to MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​ClassFileTests.cs Preserves exact exception assertions in MSTest.
tests/​Xamarin.Android.Tools.Bytecode-Tests/​ClassFileFixture.cs Converts setup, cleanup, and assertions.
tests/​Xamarin.Android.Tools.ApiXmlAdjuster-Tests/​Xamarin.Android.Tools.ApiXmlAdjuster-Tests.csproj Uses shared MSTest references.
tests/​Xamarin.Android.Tools.ApiXmlAdjuster-Tests/​TypeResolverTest.cs Converts tests and class initialization.
tests/​Xamarin.Android.Tools.ApiXmlAdjuster-Tests/​OverrideMarkerTest.cs Converts tests and class initialization.
tests/​Xamarin.Android.Tools.ApiXmlAdjuster-Tests/​JavaApiTest.cs Converts tests and class initialization.
tests/​Xamarin.Android.Tools.ApiXmlAdjuster-Tests/​GenericInheritanceMappingTest.cs Converts tests and class initialization.
tests/​Xamarin.Android.Tools.AndroidSdk-Tests/​Xamarin.Android.Tools.AndroidSdk-Tests.csproj Adopts MSTest and references the streaming-test tool.
tests/​Xamarin.Android.Tools.AndroidSdk-Tests/​JdkVersionInfoTests.cs Converts tests to MSTest.
tests/​Xamarin.Android.Tools.AndroidSdk-Tests/​AndroidSdkWindowsTests.cs Converts the test-class attribute.
tests/​Xamarin.Android.Tools.Aidl-Tests/​Xamarin.Android.Tools.Aidl-Tests.csproj Uses shared MSTest references.
tests/​Xamarin.Android.Tools.Aidl-Tests/​AidlCompilerTests.cs Converts data rows and assertions.
tests/​Xamarin.Android.Tools.Aidl-Tests/​AidlCompilerTestBase.cs Converts shared assertions and skip reporting.
tests/​TestRunner.Core/​TestRunner.Core.NET.csproj Removes the NUnit runner project.
tests/​MSBuildDeviceIntegration/​Tests/​WebViewJavascriptInterfaceTests.cs Converts runtime cases, assertions, and cleanup logging.
tests/​MSBuildDeviceIntegration/​Tests/​SystemApplicationTests.cs Converts categories and combinatorial runtime cases.
tests/​MSBuildDeviceIntegration/​Tests/​ScopedTypeMapR8Tests.cs Converts attributes and string assertions.
tests/​MSBuildDeviceIntegration/​Tests/​R8ObfuscationTests.cs Converts runtime cases and cleanup logging.
tests/​MSBuildDeviceIntegration/​Tests/​InterfaceCollectionTests.cs Converts attributes and assertions.
tests/​MSBuildDeviceIntegration/​MSBuildDeviceIntegration.csproj Configures reflection discovery and MTP retries.
tests/​MSBuildDeviceIntegration/​AssemblyInfo.cs Preserves assembly-wide sequential execution.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Xamarin.Android.Net/​AndroidMessageHandlerCancellationTests.cs Removes an outdated NUnit teardown comment.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Java.Lang/​ObjectTest.cs Removes NUnit frames from a comment.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Java.Interop/​JnienvTest.cs Makes diagnostic comments framework-neutral.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Java.Interop/​JavaPeerProxyTests.cs Removes the NUnit test fixture.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​global.json Removes redundant project-local runner selection.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Android.Widget/​AdapterTests.cs Removes NUnit frames from a comment.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Android.Runtime/​PInvokeOverrideTests.cs Removes the NUnit test fixture.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Android.OS/​AsyncTaskTests.cs Removes the NUnit test fixture.
tests/​Mono.Android-Tests/​Mono.Android-Tests/​Android.Content/​ContentValuesTests.cs Removes the NUnit test fixture.
tests/​Mono.Android-Tests/​Java.Interop-Tests/​Java.Interop/​JniNativeInterfaceTests.cs Removes the excluded NUnit fixture.
tests/​Mono.Android-Tests/​Java.Interop-Tests/​Java.Interop-Tests.NET.csproj Removes the deleted fixture’s compile exclusion.
tests/​Microsoft.Android.Build.BaseTasks-Tests/​Microsoft.Android.Build.BaseTasks-Tests.csproj Uses shared MSTest references.
tests/​Microsoft.Android.Build.BaseTasks-Tests/​MemoryStreamPoolTests.cs Converts lifecycle hooks and typed assertions.
tests/​Microsoft.Android.Build.BaseTasks-Tests/​AsyncTaskTests.cs Converts attributes and test-output handling.
tests/​Microsoft.Android.Build.BaseTasks-Tests/​AsyncTaskExtensionsTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.Maven-Tests/​PropertySubstitutionTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.Maven-Tests/​ProjectResolverTests.cs Converts tests and exact exception assertions.
tests/​Java.Interop.Tools.Maven-Tests/​MavenVersionTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.Maven-Tests/​MavenVersionRangeTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.Maven-Tests/​Java.Interop.Tools.Maven-Tests.csproj Shares MSTest packages and global imports.
tests/​Java.Interop.Tools.JavaTypeSystem-Tests/​JavaTypeReferenceTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.JavaTypeSystem-Tests/​JavaTypeNameTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.JavaTypeSystem-Tests/​JavaTypeModelsTests.cs Converts tests and class initialization.
tests/​Java.Interop.Tools.JavaTypeSystem-Tests/​Java.Interop.Tools.JavaTypeSystem-Tests.csproj Uses shared MSTest references.
tests/​Java.Interop.Tools.JavaTypeSystem-Tests/​GenericInheritanceMappingTests.cs Converts tests and class initialization.
tests/​Java.Interop.Tools.JavaTypeSystem-Tests/​BaseMethodTests.cs Converts tests and class initialization.
tests/​Java.Interop.Tools.JavaSource-Tests/​SourceJavadocToXmldocParserTests.cs Converts parser cases to dynamic data.
tests/​Java.Interop.Tools.JavaSource-Tests/​SourceJavadocToXmldocGrammarFixture.cs Converts the shared fixture to MSTest.
tests/​Java.Interop.Tools.JavaSource-Tests/​SourceJavadocToXmldocGrammar.HtmlBnfTermsTests.cs Converts grammar tests to MSTest.
tests/​Java.Interop.Tools.JavaSource-Tests/​JavaStubParserTests.cs Converts attributes and null assertions.
tests/​Java.Interop.Tools.JavaSource-Tests/​Java.Interop.Tools.JavaSource-Tests.csproj Uses shared MSTest references.
tests/​Java.Interop.Tools.JavaCallableWrappers-Tests/​Java.Interop.Tools.JavaCallableWrappers/​InstrumentationTests.cs Converts attributes and materializes collection assertions.
tests/​Java.Interop.Tools.JavaCallableWrappers-Tests/​Java.Interop.Tools.JavaCallableWrappers/​IdentifierValidatorTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.JavaCallableWrappers-Tests/​Java.Interop.Tools.JavaCallableWrappers/​Crc64Tests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.JavaCallableWrappers-Tests/​Java.Interop.Tools.JavaCallableWrappers/​ActivityTests.cs Converts attributes and materializes collection assertions.
tests/​Java.Interop.Tools.JavaCallableWrappers-Tests/​Java.Interop.Tools.JavaCallableWrappers-Tests.csproj Uses shared MSTest references.
tests/​Java.Interop.Tools.Generator-Tests/​Utilities/​NamingConverterTests.cs Converts tests and exact exception assertions.
tests/​Java.Interop.Tools.Generator-Tests/​Utilities/​AndroidSdkVersionTests.cs Converts attributes and renames a test method.
tests/​Java.Interop.Tools.Generator-Tests/​Metadata/​NamespaceTransformTests.cs Converts tests to MSTest.
tests/​Java.Interop.Tools.Generator-Tests/​Java.Interop.Tools.Generator-Tests.csproj Uses shared MSTest references.
tests/​generator-Tests/​Unit-Tests/​TypeNameUtilitiesTests.cs Converts dynamic data and preserves culture restoration.
tests/​generator-Tests/​Unit-Tests/​SymbolTableTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​SourceWriters/​WeakImplementorFieldTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​SourceWriters/​SourceWritersTestBase.cs Converts the shared test-class attribute.
tests/​generator-Tests/​Unit-Tests/​SourceWriters/​PeerMembersFieldTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​SourceWriters/​MethodExtensionAsyncWrapperTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​SourceWriters/​MethodExplicitInterfaceImplementationTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​SealedProtectedFixupsTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​ReportTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​NRTJavaInteropCodeGeneratorTests.cs Converts the inherited test-class attribute.
tests/​generator-Tests/​Unit-Tests/​ManagedExtensionsTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​InterfaceConstantsTests.cs Converts tests to MSTest.
tests/​generator-Tests/​Unit-Tests/​CodeGeneratorTestBase.cs Converts shared initialization and cleanup hooks.
tests/​generator-Tests/​Unit-Tests/​CodeGeneratorContextTests.cs Converts tests to MSTest.
tests/​generator-Tests/​README.md Updates test-framework terminology.
tests/​generator-Tests/​Integration-Tests/​Streams.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​StaticProperties.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​StaticMethods.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​StaticFields.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​PamareterXPath.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​NormalProperties.cs Converts attributes and string assertions.
tests/​generator-Tests/​Integration-Tests/​NormalMethods.cs Converts attributes and string assertions.
tests/​generator-Tests/​Integration-Tests/​NonStaticFields.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​NestedTypes.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Java_Util_List.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Java_Lang_Object.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Java_Lang_Enum.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Interfaces.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​InterfaceMethodsConflict.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​GenericArguments.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​CSharpKeywords.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Core_Jar2Xml.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Core_ClassParse.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Constructors.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Compiler.cs Switches compiler helpers to MSTest assertions.
tests/​generator-Tests/​Integration-Tests/​BaseGeneratorTest.cs Converts shared test initialization.
tests/​generator-Tests/​Integration-Tests/​Arrays.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Android_Graphics_Color.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​Adapters.cs Converts tests to MSTest.
tests/​generator-Tests/​Integration-Tests/​AccessModifiers.cs Converts tests to MSTest.
tests/​generator-Tests/​generator-Tests.csproj Uses shared MSTest references.
tests/​generator-Tests/​AssemblyInfo.cs Enables discovery of internal test classes.
tests/​CodeGen-Binding/​Xamarin.Android.JcwGen-Tests/​Xamarin.Android.JcwGen-Tests.csproj Drops timing exclusion and updates a runtime comment.
tests/​CodeGen-Binding/​Xamarin.Android.JcwGen-Tests/​global.json Removes redundant project-local runner selection.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.ProjectTools/​Xamarin.ProjectTools.csproj Replaces NUnit dependencies with MSTest assertions.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.ProjectTools/​Utilities/​ProcessExtensions.cs Switches process helpers to MSTest assertions.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.ProjectTools/​Common/​DownloadedCache.cs Converts framework import and diagnostic output.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.ProjectTools/​Common/​Builder.cs Removes NUnit-specific MSBuild argument handling.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Xamarin.Android.Build.Tests.csproj Adopts shared MSTest packages with reflection discovery.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​WearTests.cs Converts runtime combinations and parallelism settings.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Utilities/​ValidRuntimeConfigurationsAttribute.cs Supplies runtime configurations to combinatorial tests.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Utilities/​BuildHelper.cs Switches build helpers to MSTest assertions.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Utilities/​AssertionExtensions.cs Switches assertion helpers to MSTest.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Tasks/​ZipAlignTests.cs Converts lifecycle hooks, assertions, and parallelism.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Tasks/​ProcessNativeLibrariesTests.cs Converts attributes and test-output handling.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Tasks/​JsonExtensionsTests.cs Converts parameterized cases to data rows.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Tasks/​JavaToolTaskTests.cs Converts parameterized cases to data rows.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Tasks/​GenerateResourceCaseMapTests.cs Converts attributes, assertions, and output handling.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Tasks/​BundleToolTaskTests.cs Converts attributes and preserves sequential execution.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​PackagingUtilsTests.cs Converts attributes and preserves sequential execution.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​ManifestTest.TestCaseSource.cs Switches the test-framework import.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​DeferredBuildTest.cs Converts runtime parameters to combinatorial data.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​BuildTest.TestCaseSource.cs Types test-source data as argument rows.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​AssemblyInfo.cs Enables method-level test parallelism.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​AidlTest.cs Converts runtime parameters to combinatorial data.
src/​Microsoft.Android.Sdk.Analysis/​Tests/​Microsoft.Android.Sdk.Analysis.Tests.csproj Adopts shared MSTest packages; disables packing.
src/​Microsoft.Android.Sdk.Analysis/​Tests/​DNAS0001Tests.cs Converts attributes while retaining the ignored test.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​TypeMapProguardNativeObject.cs Converts unavailable-tool reporting to inconclusive.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​TrimmableTypeMapRidCallbackTests.cs Converts attributes and exception assertions.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​TrimmableTypeMapIncrementalTests.cs Converts attributes and parameterized cases.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​TransferredReferenceActivationTests.cs Converts parameterized cases to data rows.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​TemporaryDirectoryTargetsTests.cs Converts data rows and string assertions.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​Tasks/​EnvironmentBuilderTests.cs Converts attributes and direct no-throw execution.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​ResourcesTests.cs Converts localization cases to data rows.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​NativeToolTestHelper.cs Converts filesystem assertions to MSTest.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​NativeAotBootstrapTestTools.cs Uses the application directory for repository discovery.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​Microsoft.Android.Build.Tasks.Tests.csproj Uses shared MSTest references.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​GenerateCoreClrRuntimeProviderSourcesTests.cs Converts attributes, assertions, and output handling.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​FastDeployTypeMapTests.cs Converts parameterized cases to data rows.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​DesignTimeResourceTargetsTests.cs Converts combinations to MSTest combinatorial data.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​CreateAssemblyStoreTests.cs Converts data rows, assertions, and output handling.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​BaseTest.cs Uses MSTest context, display names, and cleanup.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​AssemblyInfo.cs Enables method-level test parallelism.
src/​Microsoft.Android.Build.Tasks/​Tests/​Microsoft.Android.Build.Tasks.Tests/​AssemblyCompressionTargetsTests.cs Converts parameterized cases to data rows.
Microsoft.Android-Tests.slnx Removes the NUnit runner project.
Makefile Removes obsolete NUnit test targets.
global.json Selects MTP repository-wide.
eng/​Versions.props Adds the MTP retry-extension version.
eng/​Version.Details.xml Tracks the retry-extension dependency.
Documentation/​workflow/​DevelopmentTips.md Updates device-test framework terminology.
Documentation/​guides/​OneDotNet.md Removes NUnit entries from sample template output.
Documentation/​building/​windows/​instructions.md Updates host and device testing instructions.
Directory.Build.props Updates a comment and removes the runner warning exemption.
Configuration.props Removes NUnit versions and legacy analyzer exclusion.
build-tools/​scripts/​UpdateApkSizeReference.sh Updates the test assembly path and filter syntax.
build-tools/​scripts/​UpdateApkSizeReference.ps1 Replaces legacy execution with the built test assembly.
build-tools/​scripts/​TimingDefinitions.txt Removes the NUnit result marker.
build-tools/​scripts/​NUnitReferences.projitems Removes shared NUnit dependencies.
build-tools/​scripts/​nunit3-console.cmd Removes the Windows NUnit console launcher.
build-tools/​scripts/​nunit3-console Removes the Unix NUnit console launcher.
build-tools/​scripts/​MSTestReferences.projitems Centralizes MSTest packages and MTP configuration.
build-tools/​automation/​yaml-templates/​variables.yaml Removes NUnit variables and converts category filters.
build-tools/​automation/​yaml-templates/​stage-package-tests.yaml Uses the MSTest execution template.
build-tools/​automation/​yaml-templates/​stage-msbuild-emulator-tests.yaml Switches to MTP slicing and retries.
build-tools/​automation/​yaml-templates/​setup-test-environment.yaml Removes test-slicer configuration forwarding.
build-tools/​automation/​yaml-templates/​setup-test-environment-steps.yaml Removes NUnit test-slicer installation.
build-tools/​automation/​yaml-templates/​setup-test-environment-public.yaml Removes test-slicer configuration forwarding.
build-tools/​automation/​yaml-templates/​run-nunit-tests.yaml Removes the NUnit execution template.
build-tools/​automation/​yaml-templates/​run-msbuild-tests.yaml Uses MSTest execution and slicing.
build-tools/​automation/​yaml-templates/​kill-processes.yaml Removes NUnit process cleanup.
build-tools/​automation/​yaml-templates/​diagnose-hanging-processes.yaml Removes NUnit-specific process diagnostics.
build-tools/​automation/​yaml-templates/​build-windows-steps.yaml Converts test-template calls and TRX arguments.
.github/​skills/​tests/​SKILL.md Updates the skill description to MTP terminology.
.github/​skills/​code-review/​references/​testing-rules.md Replaces NUnit guidance with MSTest guidance.
.github/​copilot-instructions.md Updates the test-framework description.

Review effort: Balanced

simonrozsival and others added 9 commits November 21, 2026 14:07
Context: #13434

MSTest on Microsoft.Testing.Platform (MTP).  Every other test project
was still NUnit, so the repo carried two frameworks, two runners,
TestRunner.Core, the NUnit console scripts, and dotnet-test-slicer.

Migrate all remaining NUnit test projects to MSTest/MTP:

  * Java tooling: generator-Tests,
    Java.Interop.Tools.{Generator,JavaCallableWrappers,JavaSource,JavaTypeSystem,Maven}-Tests,
    Xamarin.Android.Tools.{Bytecode,ApiXmlAdjuster}-Tests
  * Android tooling: Xamarin.Android.Tools.{Aidl,AndroidSdk}-Tests,
    Microsoft.Android.Build.BaseTasks-Tests,
    Microsoft.Android.Sdk.Analysis.Tests
  * Build tasks: Microsoft.Android.Build.Tasks.Tests,
    Xamarin.Android.Build.Tests, Xamarin.ProjectTools
  * Device integration: MSBuildDeviceIntegration

`build-tools/scripts/NUnitReferences.projitems` is replaced by
`MSTestReferences.projitems`; versions come from `eng/Versions.props`.
The root `global.json` selects the MTP runner.

CI: `run-nunit-tests.yaml` and `run-sliced-nunit-tests.yaml` are
replaced by `run-mstest-tests.yaml`, which runs the test app directly
with `--report-trx`.  Slicing uses `--list-tests` and `--filter-uid`
with the same count-based round-robin split as dotnet-test-slicer,
retries use `Microsoft.Testing.Extensions.Retry`, and NUnit filters
are translated to MSTest filter syntax.

StreamingTests now run the built Microsoft.Android.Run out of process
under `dotnet test --test-modules`: with MTP 2.6, an in-process
`TestApplication` using `--report-trx` relaunches the MSTest test app
itself and recurses.

LocalizationTests now restore `persist.sys.locale` to `en-US`.

Remove the now unused TestRunner.Core, nunit3-console scripts, NUnit
version properties, dead NUnit test sources and stale references.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…erts

Restore the project-local global.json files for Mono.Android.NET-Tests
and Xamarin.Android.JcwGen-Tests. Without them, the `_MTPBuild` MSBuild
process that `dotnet test` spawns in apk-instrumentation.yaml resolved
the 11.0.100-rc.1 Android workload manifest and failed with NETSDK1139.

InstallWithoutSharedRuntime: NUnit's Assert.IsNotNull on the
RuntimeInfo struct always passed. The MSTest conversion asserted on
struct fields instead, which turned the no-op into a real check that
fails on CI. Remove both always-true asserts to keep main's behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
CallerCancellationDuringVerificationPropagates: NUnit's CatchAsync
accepts derived exceptions, but the conversion used ThrowsExactlyAsync,
which rejects the TaskCanceledException that HttpClient throws. Use
ThrowsAsync to keep the original semantics.

DebuggerAttribute: the build directory did not include the runtime, so
the CoreCLR and NativeAOT rows shared it. NUnit ran the rows one at a
time; MSTest's method-level parallelism ran them concurrently and the
builds collided. Add the runtime to the directory name.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Main added or moved NUnit tests in Aapt2Tests, LinkerTests,
ManagedResourceParserTests, CollectNonEmptyDirectoriesTests,
AndroidCreateDebugKeyTests, FilesTests and KotlinFixupsTests. Convert
them to MSTest: add [TestClass]/[DoNotParallelize] to fixtures that
had no attributes, use TestId/AppContext.BaseDirectory instead of
NUnit's TestContext members, AddResultFile for attachments, and
.ToArray () for CollectionAssert.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ExtractTypeMapKeysFromNativeAotObject: #13515 changed RunObjdumpAsync
to Process.RunAndCaptureTextAsync, which buffers stdout until the
process exits. ObjdumpStreamsOutputBeforeProcessExit (added in #12988)
checks that llvm-objdump output is streamed, so it failed on macOS and
Linux. CI did not notice because Microsoft.Android.Build.Tasks.Tests
only runs on Windows, where the test is skipped. Restore the
ProcessUtils.StartProcess call that #12988 used.

BindingBuildTest: the AndroidMavenLibrary_* tests all download
androidx.core:core 1.9.0 into the same Maven cache. When they ran
concurrently, one failed with "The process cannot access the file".
Mark them [DoNotParallelize], like AndroidMavenLibrary.

JavaMarshalGCBridgeTests: ManagedPeerCycleSurvivesJavaRootAndIsCollectedAfterRelease
returned from WaitForBridgeRound on the first completed bridge round.
A round that started before the cycle existed could complete first,
leaving no GlobalToWeak events for the cycle (tracked in #13429). Wait
for a round that emitted them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Locally, Xamarin.ProjectTools creates test projects under bin/Test*,
inside the git checkout, so SourceLink adds /sourcelink and embedded
untracked files. With DebugType="" no PDB is emitted and csc fails with
CS2044/CS2045 (ManifestTest.DebuggerAttribute). On CI the projects live
outside the repo and SourceLink is inactive. Set
EnableSourceControlManagerQueries=false in the generated
Directory.Build.props so local and CI builds behave the same.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@simonrozsival
simonrozsival force-pushed the simonrozsival-mstest-migrate-remaining-suites branch from 630256a to 71d859e Compare November 21, 2026 13:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants