From 9e4923668d990d88453743e84133d1c989a3804f Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Thu, 17 Jun 2021 05:00:53 -0700 Subject: [PATCH 01/12] Add warning on unmaintained testing doc page --- docs/workflow/testing/libraries/testing.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index fe4d76255a7913..bb7acf052b6def 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -1,5 +1,8 @@ # Testing Libraries +**WARNING** Instructions on this page appear to be unmaintained and do not work as written. See [here](https://github.com/dotnet/runtime/issues/54323) for details. + + We use the OSS testing framework [xunit](https://github.com/xunit/xunit). To build the tests and run them you can call the libraries build script. For libraries tests to work, you must have built the coreclr or mono runtime for them to run on. From 8b116050be5735be8c94dcb8b0cd7554cab1e8dc Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Thu, 17 Jun 2021 23:49:04 -0700 Subject: [PATCH 02/12] Update testing.md Some example text that seems more clear to me, but only offered as a suggestion. Feel free to adjust it or if you want to use it as-is please double check what I wrote is accurate : ) I think the useful elements are: 1. Being explicit about what workflow steps need to happen in total 2. Being explicit about which commands are covering the entire workflow and which ones are only covering a part of it 3. Show the simple "do-it-all" options first before showing more complex partial options. Glancing at the first example and blindly copying it should land in the pit of success. --- docs/workflow/testing/libraries/testing.md | 50 ++++++++++++++-------- 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index bb7acf052b6def..feee1b373735ec 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -1,50 +1,62 @@ # Testing Libraries -**WARNING** Instructions on this page appear to be unmaintained and do not work as written. See [here](https://github.com/dotnet/runtime/issues/54323) for details. +## Pre-requisites +Before tests can run the following steps must happen: +1. Build a runtime (either Mono or coreclr) +2. Build the libraries +3. Testhost deployment: Binaries must be arranged into a directory layout that tools/tests expect +4. Build the tests +Using build.cmd/sh these steps can either be accomplished independently or the testing and all the pre-requisites can be automated in a single command. See the [Building instructions](../../building/libraries/README.md) for more options on how to build pieces independently. -We use the OSS testing framework [xunit](https://github.com/xunit/xunit). +## All-in-one Examples -To build the tests and run them you can call the libraries build script. For libraries tests to work, you must have built the coreclr or mono runtime for them to run on. +These examples automate all pre-requisite steps and the test run in a single command from a clean enlistment. -**Examples** -- The following shows how to build only the tests but not run them: +- Run all tests using clr: ``` -build.cmd/sh -subset libs.tests +build.cmd/sh -subset clr+libs -test ``` -- The following builds and runs all tests using clr: +- Run all tests using mono: ``` -build.cmd/sh -subset clr+libs.tests -test +build.cmd/sh -subset mono+libs -test ``` -- The following builds and runs all tests using mono: +- Run all tests - Builds clr in release, libs+tests in debug: ``` -build.cmd/sh -subset mono+libs.tests -test +build.cmd/sh -subset clr+libs -test -rc Release ``` -- The following builds and runs all tests in release configuration: +- Run all tests - Build mono and libs for x86 architecture: ``` -build.cmd/sh -subset libs.tests -test -c Release +build.cmd/sh -subset mono+libs -test -arch x86 ``` -- The following builds clr in release, libs in debug and runs all tests: +## Partial workflow examples + +These examples allow portions of the workflow to be isolated and done individually: + +- Build only the tests but not run them: ``` -build.cmd/sh -subset clr+libs+libs.tests -test -rc Release +build.cmd/sh -subset libs.tests ``` -- The following builds mono and libs for x86 architecture and runs all tests: +- Builds and run all tests in release configuration. +_Pre-requisite: Build runtime+libs and deploy testhost directory_: ``` -build.cmd/sh -subset mono+libs+libs.tests -test -arch x86 +build.cmd/sh -subset libs.tests -test -c Release ``` -- The following example shows how to pass extra msbuild properties to ignore tests ignored in CI: +- The following example shows how to pass extra msbuild properties to ignore tests ignored in CI. +_Pre-requisite: Build runtime+libs and deploy testhost directory_: ``` build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI ``` -Unless you specifiy `-testnobuild`, test assemblies are implicitly built when invoking the `Test` action. -- The following shows how to only test the libraries without building them +- Unless you specifiy `-testnobuild`, test assemblies are implicitly built when invoking the `Test` action. +The following shows how to only test the libraries without building them. +_Pre-requisite: Build runtime+libs, deploy testhost directory, build tests_: ``` build.cmd/sh -subset libs.tests -test -testnobuild ``` From 6eaa4bc10d965d2867afb5bdf6a8da5ba7f73547 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Fri, 18 Jun 2021 02:32:30 -0700 Subject: [PATCH 03/12] Apply suggestions from code review Co-authored-by: Viktor Hofer --- docs/workflow/testing/libraries/testing.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index feee1b373735ec..07d4fb7d62b623 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -13,24 +13,24 @@ Using build.cmd/sh these steps can either be accomplished independently or the t These examples automate all pre-requisite steps and the test run in a single command from a clean enlistment. -- Run all tests using clr: +- Build all pre-requisites (clr + libs) and tests and run them using CoreCLR: ``` -build.cmd/sh -subset clr+libs -test +build.cmd/sh -subset clr+libs+libs.tests -test ``` -- Run all tests using mono: +- Build all pre-requisites (mono + libs) and tests and run them using mono: ``` -build.cmd/sh -subset mono+libs -test +build.cmd/sh -subset mono+libs+libs.tests -test ``` - Run all tests - Builds clr in release, libs+tests in debug: ``` -build.cmd/sh -subset clr+libs -test -rc Release +build.cmd/sh -subset clr+libs+libs.tests -test -rc Release ``` - Run all tests - Build mono and libs for x86 architecture: ``` -build.cmd/sh -subset mono+libs -test -arch x86 +build.cmd/sh -subset mono+libs+libs.tests -test -arch x86 ``` ## Partial workflow examples From cacba7dfdc97be3d1acdde0af31872d498b6d527 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Fri, 18 Jun 2021 03:42:49 -0700 Subject: [PATCH 04/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 70 +++++++++++++--------- 1 file changed, 41 insertions(+), 29 deletions(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index 07d4fb7d62b623..bf3dd44c32c12c 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -1,24 +1,15 @@ # Testing Libraries -## Pre-requisites -Before tests can run the following steps must happen: -1. Build a runtime (either Mono or coreclr) -2. Build the libraries -3. Testhost deployment: Binaries must be arranged into a directory layout that tools/tests expect -4. Build the tests +## Full Build and Test Run -Using build.cmd/sh these steps can either be accomplished independently or the testing and all the pre-requisites can be automated in a single command. See the [Building instructions](../../building/libraries/README.md) for more options on how to build pieces independently. - -## All-in-one Examples - -These examples automate all pre-requisite steps and the test run in a single command from a clean enlistment. +These example commands automate the test run and all pre-requisite build steps in a single command from a clean enlistment. - Build all pre-requisites (clr + libs) and tests and run them using CoreCLR: ``` build.cmd/sh -subset clr+libs+libs.tests -test ``` -- Build all pre-requisites (mono + libs) and tests and run them using mono: +- Build all pre-requisites (mono + libs) and tests and run them using Mono: ``` build.cmd/sh -subset mono+libs+libs.tests -test ``` @@ -33,37 +24,58 @@ build.cmd/sh -subset clr+libs+libs.tests -test -rc Release build.cmd/sh -subset mono+libs+libs.tests -test -arch x86 ``` -## Partial workflow examples +## Partial Build and Test Runs -These examples allow portions of the workflow to be isolated and done individually: +Doing full build and test runs takes a long time and is very inefficient if you need to iterate on a change. +For greater control and efficiency individual parts of the build + testing workflow can be run in isolation. +See the [Building instructions](../../building/libraries/README.md) for more info on build options. -- Build only the tests but not run them: +### Test Run Pre-requisites +Before any tests can run we need a complete build to run them on. Specifically: +1. Build a runtime (clr or mono) ``` -build.cmd/sh -subset libs.tests +build.cmd/sh -subset clr +``` +OR +``` +build.cmd/sh -subset mono +``` +2. Build all the libraries +``` +build.cmd/sh -subset libs +``` + +Building the `libs` subset automatically copies all product binaries into a testhost folder in the bin directory. +This is where the tests will load the binaries from during the run. If you later [rebuild any of product binaries](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) +using the individual library build projects you must also build the `libs.pretest` subset to copy the new binaries to testhost. ``` +build.cmd/sh -subset libs.pretest +``` +TODO: Is this a special case only for S.P.C or it is required for all libraries? + +### Running tests for all libraries -- Builds and run all tests in release configuration. -_Pre-requisite: Build runtime+libs and deploy testhost directory_: +- Build and run all tests in release configuration. ``` build.cmd/sh -subset libs.tests -test -c Release ``` -- The following example shows how to pass extra msbuild properties to ignore tests ignored in CI. -_Pre-requisite: Build runtime+libs and deploy testhost directory_: +- Build the tests without running them ``` -build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI +build.cmd/sh -subset libs.tests ``` -- Unless you specifiy `-testnobuild`, test assemblies are implicitly built when invoking the `Test` action. -The following shows how to only test the libraries without building them. -_Pre-requisite: Build runtime+libs, deploy testhost directory, build tests_: +- Run the tests without building them ``` build.cmd/sh -subset libs.tests -test -testnobuild ``` -## Running tests on the command line +- The following example shows how to pass extra msbuild properties to ignore tests ignored in CI. +``` +build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI +``` -To build tests you need to specify the `test` subset when invoking build.cmd/sh: `build.cmd/sh -subset libs.tests`. +### Running tests for a single library The easiest (and recommended) way to build and run the tests for a specific library, is to invoke the `Test` target on that library: ```cmd @@ -83,21 +95,21 @@ dotnet build /t:Test /p:TargetArchitecture=x86 There may be multiple projects in some directories so you may need to specify the path to a specific test project to get it to build and run the tests. -#### Running a single test on the command line +### Running a single test on the command line To quickly run or debug a single test from the command line, set the XunitMethodName property, e.g.: ```cmd dotnet build /t:Test /p:XunitMethodName={FullyQualifiedNamespace}.{ClassName}.{MethodName} ``` -#### Running outer loop tests +### Running outer loop tests To run all tests, including "outer loop" tests (which are typically slower and in some test suites less reliable, but which are more comprehensive): ```cmd dotnet build /t:Test /p:Outerloop=true ``` -#### Running tests on a different target framework +### Running tests on a different target framework Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks. By default we will build and run only the default build target framework which is `net5.0`. The rest of the `TargetFrameworks` will need to be built and ran by specifying the `BuildTargetFramework` option, e.g.: ```cmd From d32514d8154b9a2ab967c965d0f75b9d7c5fbd60 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Fri, 18 Jun 2021 04:05:17 -0700 Subject: [PATCH 05/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index bf3dd44c32c12c..4dc5f39b06c4b1 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -31,18 +31,17 @@ For greater control and efficiency individual parts of the build + testing workf See the [Building instructions](../../building/libraries/README.md) for more info on build options. ### Test Run Pre-requisites -Before any tests can run we need a complete build to run them on. Specifically: -1. Build a runtime (clr or mono) -``` -build.cmd/sh -subset clr -``` -OR +Before any tests can run we need a complete build to run them on. This requires building (1) a runtime, and +(2) all the libraries. Examples: + +- Build release clr + debug libraries ``` -build.cmd/sh -subset mono +build.cmd/sh -subset clr+libs -rc Release ``` -2. Build all the libraries + +- Build release mono + debug libraries ``` -build.cmd/sh -subset libs +build.cmd/sh -subset clr+libs -rc Release ``` Building the `libs` subset automatically copies all product binaries into a testhost folder in the bin directory. From 52ec4403f1b9a49b7be7befb119149361f70c6ba Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Fri, 18 Jun 2021 04:05:49 -0700 Subject: [PATCH 06/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index 4dc5f39b06c4b1..59a1b60a70c4b6 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -27,7 +27,7 @@ build.cmd/sh -subset mono+libs+libs.tests -test -arch x86 ## Partial Build and Test Runs Doing full build and test runs takes a long time and is very inefficient if you need to iterate on a change. -For greater control and efficiency individual parts of the build + testing workflow can be run in isolation. +For greater control and efficiency individual parts of the build + testing workflow can be run in isolation. See the [Building instructions](../../building/libraries/README.md) for more info on build options. ### Test Run Pre-requisites From aafbfd7fae7fab247ab73208130918dc559cb6e1 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Sun, 20 Jun 2021 15:44:06 -0700 Subject: [PATCH 07/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index 59a1b60a70c4b6..fee88fb147d5d5 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -41,7 +41,7 @@ build.cmd/sh -subset clr+libs -rc Release - Build release mono + debug libraries ``` -build.cmd/sh -subset clr+libs -rc Release +build.cmd/sh -subset mono+libs -rc Release ``` Building the `libs` subset automatically copies all product binaries into a testhost folder in the bin directory. From 91dc0b39d67e029a19823ded09eb49c02c3e9b6c Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Tue, 22 Jun 2021 01:57:17 -0700 Subject: [PATCH 08/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index fee88fb147d5d5..3d89b867d6bb23 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -44,13 +44,9 @@ build.cmd/sh -subset clr+libs -rc Release build.cmd/sh -subset mono+libs -rc Release ``` -Building the `libs` subset automatically copies all product binaries into a testhost folder in the bin directory. -This is where the tests will load the binaries from during the run. If you later [rebuild any of product binaries](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) -using the individual library build projects you must also build the `libs.pretest` subset to copy the new binaries to testhost. -``` -build.cmd/sh -subset libs.pretest -``` -TODO: Is this a special case only for S.P.C or it is required for all libraries? +Building the `libs` subset or any of individual library projects automatically copies product binaries into the testhost folder +in the bin directory. This is where the tests will load the binaries from during the run. However System.Private.CorLib is an +exception. If you [rebuild System.Private.CoreLib](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) you must also build the `libs.pretest` subset to ensure it is copied to the testhost folder before running tests. ### Running tests for all libraries From 72701980fa31105a4f062c6c1ed77e254b24db22 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Tue, 22 Jun 2021 02:02:49 -0700 Subject: [PATCH 09/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index 3d89b867d6bb23..de79f8e849ce7f 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -46,7 +46,7 @@ build.cmd/sh -subset mono+libs -rc Release Building the `libs` subset or any of individual library projects automatically copies product binaries into the testhost folder in the bin directory. This is where the tests will load the binaries from during the run. However System.Private.CorLib is an -exception. If you [rebuild System.Private.CoreLib](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) you must also build the `libs.pretest` subset to ensure it is copied to the testhost folder before running tests. +exception - the build does not automatically copy it to the testhost folder. If you [rebuild System.Private.CoreLib](https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/README.md#iterating-on-systemprivatecorelib-changes) you must also build the `libs.pretest` subset to ensure S.P.C is copied before running tests. ### Running tests for all libraries From 7a9c6c65c99c3c949c02d3c5b78ba60d956ef88e Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Tue, 22 Jun 2021 02:05:14 -0700 Subject: [PATCH 10/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index de79f8e849ce7f..824fdcb36fb62e 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -50,7 +50,7 @@ exception - the build does not automatically copy it to the testhost folder. If ### Running tests for all libraries -- Build and run all tests in release configuration. +- Build and run all tests in release configuration. ``` build.cmd/sh -subset libs.tests -test -c Release ``` From c96fd6e2226fa272f14ee81256e4c1ee23016ec7 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Tue, 22 Jun 2021 15:19:19 -0700 Subject: [PATCH 11/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index 824fdcb36fb62e..bd3b08cb0dccbb 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -4,22 +4,17 @@ These example commands automate the test run and all pre-requisite build steps in a single command from a clean enlistment. -- Build all pre-requisites (clr + libs) and tests and run them using CoreCLR: -``` -build.cmd/sh -subset clr+libs+libs.tests -test -``` - -- Build all pre-requisites (mono + libs) and tests and run them using Mono: +- Run all tests - Builds clr in release, libs+tests in debug: ``` -build.cmd/sh -subset mono+libs+libs.tests -test +build.cmd/sh -subset clr+libs+libs.tests -test -rc Release ``` -- Run all tests - Builds clr in release, libs+tests in debug: +- Run all tests - Builds Mono in release, libs+tests in debug: ``` -build.cmd/sh -subset clr+libs+libs.tests -test -rc Release +build.cmd/sh -subset mono+libs+libs.tests -test -rc Release ``` -- Run all tests - Build mono and libs for x86 architecture: +- Run all tests - Build Mono and libs for x86 architecture in debug: ``` build.cmd/sh -subset mono+libs+libs.tests -test -arch x86 ``` From 25394d7b808b3119e6a484680eabc09ddf00dfd4 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Tue, 22 Jun 2021 15:21:07 -0700 Subject: [PATCH 12/12] Update testing.md --- docs/workflow/testing/libraries/testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing.md b/docs/workflow/testing/libraries/testing.md index bd3b08cb0dccbb..a78a3d0409868d 100644 --- a/docs/workflow/testing/libraries/testing.md +++ b/docs/workflow/testing/libraries/testing.md @@ -14,7 +14,7 @@ build.cmd/sh -subset clr+libs+libs.tests -test -rc Release build.cmd/sh -subset mono+libs+libs.tests -test -rc Release ``` -- Run all tests - Build Mono and libs for x86 architecture in debug: +- Run all tests - Build Mono and libs for x86 architecture in debug (choosing debug for runtime will run very slowly): ``` build.cmd/sh -subset mono+libs+libs.tests -test -arch x86 ```