Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve generated dump debugging instructions #46493

Merged
merged 11 commits into from
Jan 5, 2021
2 changes: 1 addition & 1 deletion docs/workflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ For Linux and macOS
```

For Windows:
```bat
```cmd
build.cmd
```

Expand Down
4 changes: 2 additions & 2 deletions docs/workflow/building/libraries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Here is one example of a daily workflow for a developer working mainly on the libraries, in this case using Windows:

```bat
```cmd
:: From root:
git clean -xdf
git pull upstream master & git push origin master
Expand Down Expand Up @@ -60,7 +60,7 @@ For Linux:
```

For Windows:
```bat
```cmd
./build.cmd -rc Release
```

Expand Down
6 changes: 3 additions & 3 deletions docs/workflow/building/mono/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ To build a complete runtime environment, you need to build both the Mono runtime
./build.sh --subset mono+libs
```
or on Windows,
```bat
```cmd
build.cmd -subset mono+libs
```
Note that the debug configuration is the default option. It generates a 'debug' output and that includes asserts, fewer code optimizations, and is easier for debugging. If you want to make performance measurements, or just want tests to execute more quickly, you can also build the 'release' version which does not have these checks by adding the flag `-configuration release` (or `-c release`).
Expand All @@ -28,7 +28,7 @@ Once you've built the complete runtime and assuming you want to work with just m
./build.sh --subset mono
```
or on Windows,
```bat
```cmd
build.cmd -subset mono
```
When the build completes, product binaries will be dropped in the `artifacts\bin\mono\<OS>.<arch>.<flavor>` folder.
Expand Down Expand Up @@ -69,7 +69,7 @@ To generate nuget packages:
./build.sh --subset mono -pack (with optional release configuration)
```
or on Windows,
```bat
```cmd
build.cmd -subset mono -pack (with optional release configuration)
```

Expand Down
8 changes: 4 additions & 4 deletions docs/workflow/testing/using-corerun.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Consider that you already have a .NET application DLL called HelloWorld.dll and
(You could make such a DLL by using 'dotnet new' 'dotnet restore' 'dotnet build' in a 'HelloWorld' directory).

If you execute the following
```bat
```cmd
set PATH=%PATH%;%CoreCLR%\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root\
set CORE_LIBRARIES=%ProgramFiles%\dotnet\shared\Microsoft.NETCore.App\1.0.0

Expand Down Expand Up @@ -64,20 +64,20 @@ your new runtime.
When you execute 'runtime/src/tests/build.cmd' one of the things that it does is set up a directory where it
gathers the CoreCLR that has just been built with the pieces of the class library that tests need.
It places this runtime in the directory
```bat
```cmd
runtime\artifacts\tests\coreclr\<OS>.<Arch>.<BuildType>\Tests\Core_Root
```
off the CoreCLR Repository. The way the tests are expected to work is that you set the environment
variable CORE_ROOT to this directory
(you don't have to set CORE_LIBRARIES) and you can run any tests. For example after building the tests
(running src\tests\build from the repository base) and running 'src\tests\run') you can do the following

```bat
```cmd
set PATH=%PATH%;%CoreCLR%\artifacts\Product\windows.x64.Debug
set CORE_ROOT=%CoreCLR%\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root
```
sets you up so that corerun can run any of the test. For example
```bat
```cmd
corerun artifacts\tests\coreclr\windows.X64.Debug\GC\Features\Finalizer\finalizeio\finalizeio\finalizeio.exe
```
runs the finalizerio test.
8 changes: 4 additions & 4 deletions docs/workflow/testing/using-your-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ For another small walkthrough see [Dogfooding .NET SDK](https://github.com/dotne

At this point, you can create a new 'Hello World' program in the standard way.

```bat
```cmd
mkdir HelloWorld
cd HelloWorld
dotnet new console
Expand All @@ -57,7 +57,7 @@ For Windows you will want `win-x64`, for macOS `osx-x64` and `linux-x64` for Lin
Now is the time to publish. The publish step will trigger restore and build. You can iterate on build by calling `dotnet build` as
needed.

```bat
```cmd
dotnet publish
```

Expand Down Expand Up @@ -106,11 +106,11 @@ deploy your new bits. In a lot of cases it is easiest to just copy everything fr

You can build just the .NET Library part of the build by doing (debug, for release add 'release' qualifier)
(on Linux / OSX us ./build.sh)
```bat
```cmd
.\build skiptests skipnative
```
Which builds System.Private.CoreLib.dll if you modify C# code. If you wish to only compile the coreclr.dll you can do
```bat
```cmd
.\build skiptests skipmscorlib
```
Note that this technique does not work on .NET Apps that have not been published (that is you have not created
Expand Down
8 changes: 4 additions & 4 deletions docs/workflow/using-dotnet-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ static void Main(string[] args)
Now is the time to publish. The publish step will trigger restore and build. You can iterate on build by calling `dotnet build` as
needed.

```bat
```cmd
dotnet publish
```

Expand Down Expand Up @@ -179,21 +179,21 @@ So the steps are:

### 1. Build the package again

```bat
```cmd
build.cmd clr+libs+host+packs -c release
```

If you only changed libraries, `build.cmd libs+host+packs -c release` is a little faster; if you only changed clr, then `build.cmd clr+host+packs -c release`

### 2. Delete your local package cache

```bat
```cmd
rd /s /q c:\localcache
```

### 3. Publish again

```bat
```cmd
dotnet publish
```

Expand Down
195 changes: 108 additions & 87 deletions eng/testing/debug-dump-template.md
Original file line number Diff line number Diff line change
@@ -1,127 +1,148 @@
# Debugging a CI dump
# Get the dump

This document describes how to debug a CI/PR test dump by downloading assets from helix, using a dotnet tool called `runfo`.
Click the link to the dump on the `Helix Test Logs` tab in Azure DevOps. This is the same place you got these instructions from.

## What is runfo?
# Get the Helix payload

Runfo is a dotnet global tool that helps get information about helix test runs and azure devops builds. For more information see [this](https://github.com/jaredpar/runfo/tree/master/runfo#runfo)

### How do I install it?

You just need to run:

```script
[Runfo](https://github.com/jaredpar/runfo/tree/master/runfo#runfo) helps get information about helix test runs and azure devops builds. We will use it to download the payload and symbols:
```sh
dotnet tool install --global runfo
```

If you already have it installed, make sure you have at least version `0.6.1` installed, which contains support to download helix payloads. If you don't have the latest version just run:

```script
dotnet tool update --global runfo
```

## Download helix payload containing symbols:

You can just achieve this by running:

```script
runfo get-helix-payload -j %JOBID% -w %WORKITEM% -o <out-dir>
If prompted, open a new command prompt to pick up the updated PATH.
```sh
# On Windows
# assumes %WOUTDIR% does not exist
runfo get-helix-payload -j %JOBID% -w %WORKITEM% -o %WOUTDIR%
# On Linux and macOS
# assumes %LOUTDIR% does not exist
runfo get-helix-payload -j %JOBID% -w %WORKITEM% -o %LOUTDIR%
```

> NOTE: if the helix job is an internal job, you need to pass down a [helix authentication token](https://helix.dot.net/Account/Tokens) using the `--helix-token` argument.

This will download the workitem contents under `<out-dir>\workitems\` and the correlation payload under: `<out-dir>\correlation-payload\`.
Now extract the files:

> The correlation payload is usually the testhost or core root, which contain the runtime and dotnet host that we use to run tests.
```sh
# On Windows
for /f %i in ('dir /s/b %WOUTDIR%\*zip') do tar -xf %i -C %WOUTDIR%

Once you have those assets, you will need to extract the testhost or core root. Then extract the workitem assets into the same location where coreclr binary is.
# On Linux and macOS
# obtain `unzip` if necessary; eg `sudo apt-get install unzip` or `sudo dnf install unzip`
find %LOUTDIR% -name '*zip' -exec unzip -d %LOUTDIR% {} \;
```

## Windows dump on Windows
Now use the [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos) to install the SOS debugging extension.
```sh
dotnet tool install --global dotnet-sos
dotnet tool update --global dotnet-sos
```
If prompted, open a new command prompt to pick up the updated PATH.
```sh
# Install only one: the one matching your dump
dotnet sos install --architecture Arm
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last one will override the others, at least on Linux (not sure about windows, haven't seen if it is conditional on the bitness of the extension host, but given the errors I've seen I'd say no). Users should only install the one they are going to use.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh - I didn't notice that because I was testing with x64, which is last!

Could we imagine installing them all, and picking the right one - eliminate another decision point? cc @mikem8361

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix the text meantime

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it defaults silently to the bitness of the dotnet.exe you're running it with (?)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it does default to the architecture of the dotnet.exe you are running (the most common case).

As far as installing all the of the architectures, there would have to be a rid named subdirectory to separate them so the decision point would be postponed a little to the .load command in windbg.

This manual dotnet-sos install/load on Windows was really meant as a fallback while we wait for the public debugger to include SOS (all architectures) like the internal one does now. The internal Windows debugger automatically loads the correct architecture and latest version of SOS from the extension gallery.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh perfect then it will go away.

dotnet sos install --architecture Arm64
dotnet sos install --architecture x86
dotnet sos install --architecture x64
```

### Debug with WinDbg
# Now choose a section below based on your OS.

1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos).
2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios).
3. Load the dump with a recent WinDbg version for it to load sos automatically (if not you can run `.update sos`). It is important that bitness of WinDbg matches the bitness of the dump.
4. Then run the following commands:
## If it's a Windows dump on Windows...

```script
!setclrpath <path to core root or testhost where coreclr is>
.sympath+ <directory with symbols (for library tests these are in testhost dir)>
```
### Analyze with dotnet-dump
## ... and you want to debug with WinDbg

1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump).
2. Run: `dotnet-dump analyze <path-to-dump>`
3. Then run the following commands:
Install or update WinDbg if necessary ([external](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools), [internal](https://osgwiki.com/wiki/Installing_WinDbg)). If you don't have a recent WinDbg you may have to do `.update sos`.

```script
setclrpath (To verify an incorrect DAC hasn't been loaded).
setclrpath <path to core root or testhost where coreclr is>
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)>
Open WinDbg and open the dump with `File>Open Dump`.

```
!setclrpath %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0
.sympath+ %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0
```

## Linux dumps on Windows
Now you can use regular SOS commands like `!dumpstack`, `!pe`, etc.

In order to debug a Linux dump on Windows, you will have to first go to the PR/CI build
that sent the test run and download the cross DAC.
## ... and you want to debug with Visual Studio

Download the [`CoreCLRCrossDacArtifacts`](https://dev.azure.com/dnceng/public/_apis/build/builds/%BUILDID%/artifacts?artifactName=CoreCLRCrossDacArtifacts&api-version=6.0&%24format=zip), then extract it, and copy the matching flavor of the DAC with your dump and extract it in the same location where coreclr binary is.
Currently this is not possible because mscordbi.dll is not signed.

### Debug with WinDbg
## ... and you want to debug with dotnet-dump

1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos).
2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios).
3. Load the dump with a recent WinDbg version for it to load sos automatically (if not you can run `.update sos`). It is important that bitness of WinDbg matches the bitness of the dump.
4. Then run the following commands:
Install the [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump).
```sh
dotnet tool install --global dotnet-dump
dotnet tool update --global dotnet-dump
```
If prompted, open a new command prompt to pick up the updated PATH.
```sh
dotnet-dump analyze <path-to-dump>
```
Within dotnet-dump:
```sh
setclrpath %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0
setsymbolserver -directory %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0
```

```script
!setclrpath <path to core root or testhost where coreclr is>
.sympath+ <directory with symbols (for library tests these are in testhost dir)>
Now you can use regular SOS commands like `dumpstack`, `pe`, etc.
If you are debugging a 32 bit dump using 64 bit dotnet, you will get an error `SOS does not support the current target architecture`. In that case replace dotnet-dump with the 32 bit version:
```sh
dotnet tool uninstall --global dotnet-dump
"C:\Program Files (x86)\dotnet\dotnet.exe" tool install --global dotnet-dump
```
### Analyze with dotnet-dump
---
## If it's a Linux dump on Windows...

1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump).
2. Run: `dotnet-dump analyze <path-to-dump>`
3. Then run the following commands:
Download the [Cross DAC Binaries](https://dev.azure.com/dnceng/public/_apis/build/builds/%BUILDID%/artifacts?artifactName=CoreCLRCrossDacArtifacts&api-version=6.0&%24format=zip), open it and choose the flavor that matches the dump you are to debug, and copy those files to `%WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0`.

```script
setclrpath (To verify an incorrect DAC hasn't been loaded).
setclrpath <path to core root or testhost where coreclr is>
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)>
```
Now you can debug with WinDbg or `dotnet-dump` as if it was a Windows dump. See above.

## Linux dumps on Linux
---
## If it's a Linux dump on Linux...

### Debug with LLDB
## ... and you want to debug with LLDB

1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos).
2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios).
3. Load the dump by running `lldb -c <path-to-dmp> <host binary used (found in testhost)>`
4. Run the following commands:
Install or update LLDB if necessary ([instructions here](https://github.com/dotnet/diagnostics/blob/master/documentation/lldb/linux-instructions.md))

```script
setclrpath <path to core root or testhost where coreclr is>
sethostruntime '<path to your local dotnet runtime or testhost where coreclr is>'
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)>
loadsymbols (if you want to resolve native symbols)
Load the dump:
```sh
lldb --core <path-to-dmp> %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0/dotnet
```

### Analyze with dotnet-dump
Within lldb:
```
setclrpath %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0
sethostruntime /usr/bin/dotnet
setsymbolserver -directory %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0
```
If you want to load native symbols
```
loadsymbols
```

1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump).
2. Run: `dotnet-dump analyze <path-to-dump>`
3. Then run the following commands:
## ... and you want to debug with dotnet-dump

```script
setclrpath (To verify an incorrect DAC hasn't been loaded).
setclrpath <path to core root or testhost where coreclr is>
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)>
Install the [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump).
```sh
dotnet tool install --global dotnet-dump
dotnet tool update --global dotnet-dump
```
If prompted, open a new command prompt to pick up the updated PATH.
```sh
dotnet-dump analyze <path-to-dump>
```
Within dotnet-dump:
```sh
setclrpath %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0
setsymbolserver -directory %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0
```

---
## If it's a macOS dump

## MacOS dumps
Instructions for debugging dumps on macOS are essentially the same as [Linux](#If-it's-a-Linux-dump-on-Linux...) with one exception: `dotnet-dump` cannot analyze macOS system dumps: you must use `lldb` for those. `dotnet-dump` can only analyze dumps created by `dotnet-dump` or `createdump`, by the runtime on crashes when the appropriate environment variables are set, or the [`blame-hang` setting of `dotnet test`](https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test).

Instructions for debugging dumps on MacOS the same as [Linux](#linux-dumps-on-linux); however there are a couple of caveats.
---
# Other Helpful Information

1. It's only supported to debug them in `dotnet-dump` if it's a runtime generated dump. This includes hang dumps and dumps generated by `createdump`, `dotnet-dump` and the runtime itself.
2. If it's a system dump, then only `lldb` works.
* [How to debug a Linux core dump with SOS](https://github.com/dotnet/diagnostics/blob/master/documentation/debugging-coredump.md)
Loading