-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve generated dump debugging instructions #46493
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
e6bf847
Improve generated dump debugging instructions
danmoseley b3e190b
Update eng/testing/debug-dump-template.md
danmoseley c80b627
Update eng/testing/debug-dump-template.md
danmoseley f6dcf2b
Update eng/testing/debug-dump-template.md
danmoseley cc8f129
Update debug-dump-template.md
danmoseley 0db8d32
Fix ```
danmoseley 81ed479
fix some ```cmd
danmoseley 4b0b4ba
Reverse .cmd to .bat
danmoseley e9f8779
Open only for reading
danmoseley 5a0463f
Improve foldername
danmoseley 9c1f7b5
Install only one SOS
danmoseley File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,7 +24,7 @@ For Linux and macOS | |
``` | ||
|
||
For Windows: | ||
```bat | ||
```cmd | ||
build.cmd | ||
``` | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,127 +1,148 @@ | ||
# Debugging a CI dump | ||
# Get the dump | ||
|
||
This document describes how to debug a CI/PR test dump by downloading assets from helix, using a dotnet tool called `runfo`. | ||
Click the link to the dump on the `Helix Test Logs` tab in Azure DevOps. This is the same place you got these instructions from. | ||
|
||
## What is runfo? | ||
# Get the Helix payload | ||
|
||
Runfo is a dotnet global tool that helps get information about helix test runs and azure devops builds. For more information see [this](https://github.com/jaredpar/runfo/tree/master/runfo#runfo) | ||
|
||
### How do I install it? | ||
|
||
You just need to run: | ||
|
||
```script | ||
[Runfo](https://github.com/jaredpar/runfo/tree/master/runfo#runfo) helps get information about helix test runs and azure devops builds. We will use it to download the payload and symbols: | ||
```sh | ||
dotnet tool install --global runfo | ||
``` | ||
|
||
If you already have it installed, make sure you have at least version `0.6.1` installed, which contains support to download helix payloads. If you don't have the latest version just run: | ||
|
||
```script | ||
dotnet tool update --global runfo | ||
``` | ||
|
||
## Download helix payload containing symbols: | ||
|
||
You can just achieve this by running: | ||
|
||
```script | ||
runfo get-helix-payload -j %JOBID% -w %WORKITEM% -o <out-dir> | ||
If prompted, open a new command prompt to pick up the updated PATH. | ||
```sh | ||
# On Windows | ||
# assumes %WOUTDIR% does not exist | ||
runfo get-helix-payload -j %JOBID% -w %WORKITEM% -o %WOUTDIR% | ||
# On Linux and macOS | ||
# assumes %LOUTDIR% does not exist | ||
runfo get-helix-payload -j %JOBID% -w %WORKITEM% -o %LOUTDIR% | ||
``` | ||
|
||
> NOTE: if the helix job is an internal job, you need to pass down a [helix authentication token](https://helix.dot.net/Account/Tokens) using the `--helix-token` argument. | ||
|
||
This will download the workitem contents under `<out-dir>\workitems\` and the correlation payload under: `<out-dir>\correlation-payload\`. | ||
Now extract the files: | ||
|
||
> The correlation payload is usually the testhost or core root, which contain the runtime and dotnet host that we use to run tests. | ||
```sh | ||
# On Windows | ||
for /f %i in ('dir /s/b %WOUTDIR%\*zip') do tar -xf %i -C %WOUTDIR% | ||
|
||
Once you have those assets, you will need to extract the testhost or core root. Then extract the workitem assets into the same location where coreclr binary is. | ||
# On Linux and macOS | ||
# obtain `unzip` if necessary; eg `sudo apt-get install unzip` or `sudo dnf install unzip` | ||
find %LOUTDIR% -name '*zip' -exec unzip -d %LOUTDIR% {} \; | ||
``` | ||
|
||
## Windows dump on Windows | ||
Now use the [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos) to install the SOS debugging extension. | ||
```sh | ||
dotnet tool install --global dotnet-sos | ||
dotnet tool update --global dotnet-sos | ||
``` | ||
If prompted, open a new command prompt to pick up the updated PATH. | ||
```sh | ||
# Install only one: the one matching your dump | ||
dotnet sos install --architecture Arm | ||
dotnet sos install --architecture Arm64 | ||
dotnet sos install --architecture x86 | ||
dotnet sos install --architecture x64 | ||
``` | ||
|
||
### Debug with WinDbg | ||
# Now choose a section below based on your OS. | ||
|
||
1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos). | ||
2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios). | ||
3. Load the dump with a recent WinDbg version for it to load sos automatically (if not you can run `.update sos`). It is important that bitness of WinDbg matches the bitness of the dump. | ||
4. Then run the following commands: | ||
## If it's a Windows dump on Windows... | ||
|
||
```script | ||
!setclrpath <path to core root or testhost where coreclr is> | ||
.sympath+ <directory with symbols (for library tests these are in testhost dir)> | ||
``` | ||
### Analyze with dotnet-dump | ||
## ... and you want to debug with WinDbg | ||
|
||
1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). | ||
2. Run: `dotnet-dump analyze <path-to-dump>` | ||
3. Then run the following commands: | ||
Install or update WinDbg if necessary ([external](https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools), [internal](https://osgwiki.com/wiki/Installing_WinDbg)). If you don't have a recent WinDbg you may have to do `.update sos`. | ||
danmoseley marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```script | ||
setclrpath (To verify an incorrect DAC hasn't been loaded). | ||
setclrpath <path to core root or testhost where coreclr is> | ||
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)> | ||
Open WinDbg and open the dump with `File>Open Dump`. | ||
|
||
``` | ||
!setclrpath %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0 | ||
.sympath+ %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0 | ||
``` | ||
|
||
## Linux dumps on Windows | ||
Now you can use regular SOS commands like `!dumpstack`, `!pe`, etc. | ||
|
||
In order to debug a Linux dump on Windows, you will have to first go to the PR/CI build | ||
that sent the test run and download the cross DAC. | ||
## ... and you want to debug with Visual Studio | ||
|
||
Download the [`CoreCLRCrossDacArtifacts`](https://dev.azure.com/dnceng/public/_apis/build/builds/%BUILDID%/artifacts?artifactName=CoreCLRCrossDacArtifacts&api-version=6.0&%24format=zip), then extract it, and copy the matching flavor of the DAC with your dump and extract it in the same location where coreclr binary is. | ||
Currently this is not possible because mscordbi.dll is not signed. | ||
|
||
### Debug with WinDbg | ||
## ... and you want to debug with dotnet-dump | ||
|
||
1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos). | ||
2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios). | ||
3. Load the dump with a recent WinDbg version for it to load sos automatically (if not you can run `.update sos`). It is important that bitness of WinDbg matches the bitness of the dump. | ||
4. Then run the following commands: | ||
Install the [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). | ||
```sh | ||
dotnet tool install --global dotnet-dump | ||
dotnet tool update --global dotnet-dump | ||
``` | ||
If prompted, open a new command prompt to pick up the updated PATH. | ||
```sh | ||
dotnet-dump analyze <path-to-dump> | ||
``` | ||
Within dotnet-dump: | ||
```sh | ||
setclrpath %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0 | ||
setsymbolserver -directory %WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0 | ||
``` | ||
|
||
```script | ||
!setclrpath <path to core root or testhost where coreclr is> | ||
.sympath+ <directory with symbols (for library tests these are in testhost dir)> | ||
Now you can use regular SOS commands like `dumpstack`, `pe`, etc. | ||
If you are debugging a 32 bit dump using 64 bit dotnet, you will get an error `SOS does not support the current target architecture`. In that case replace dotnet-dump with the 32 bit version: | ||
```sh | ||
dotnet tool uninstall --global dotnet-dump | ||
"C:\Program Files (x86)\dotnet\dotnet.exe" tool install --global dotnet-dump | ||
``` | ||
### Analyze with dotnet-dump | ||
--- | ||
## If it's a Linux dump on Windows... | ||
|
||
1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). | ||
2. Run: `dotnet-dump analyze <path-to-dump>` | ||
3. Then run the following commands: | ||
Download the [Cross DAC Binaries](https://dev.azure.com/dnceng/public/_apis/build/builds/%BUILDID%/artifacts?artifactName=CoreCLRCrossDacArtifacts&api-version=6.0&%24format=zip), open it and choose the flavor that matches the dump you are to debug, and copy those files to `%WOUTDIR%\shared\Microsoft.NETCore.App\6.0.0`. | ||
|
||
```script | ||
setclrpath (To verify an incorrect DAC hasn't been loaded). | ||
setclrpath <path to core root or testhost where coreclr is> | ||
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)> | ||
``` | ||
Now you can debug with WinDbg or `dotnet-dump` as if it was a Windows dump. See above. | ||
|
||
## Linux dumps on Linux | ||
--- | ||
## If it's a Linux dump on Linux... | ||
|
||
### Debug with LLDB | ||
## ... and you want to debug with LLDB | ||
|
||
1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos). | ||
2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios). | ||
3. Load the dump by running `lldb -c <path-to-dmp> <host binary used (found in testhost)>` | ||
4. Run the following commands: | ||
Install or update LLDB if necessary ([instructions here](https://github.com/dotnet/diagnostics/blob/master/documentation/lldb/linux-instructions.md)) | ||
|
||
```script | ||
setclrpath <path to core root or testhost where coreclr is> | ||
sethostruntime '<path to your local dotnet runtime or testhost where coreclr is>' | ||
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)> | ||
loadsymbols (if you want to resolve native symbols) | ||
Load the dump: | ||
```sh | ||
lldb --core <path-to-dmp> %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0/dotnet | ||
``` | ||
|
||
### Analyze with dotnet-dump | ||
Within lldb: | ||
``` | ||
setclrpath %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0 | ||
sethostruntime /usr/bin/dotnet | ||
setsymbolserver -directory %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0 | ||
``` | ||
If you want to load native symbols | ||
``` | ||
loadsymbols | ||
``` | ||
|
||
1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). | ||
2. Run: `dotnet-dump analyze <path-to-dump>` | ||
3. Then run the following commands: | ||
## ... and you want to debug with dotnet-dump | ||
|
||
```script | ||
setclrpath (To verify an incorrect DAC hasn't been loaded). | ||
setclrpath <path to core root or testhost where coreclr is> | ||
setsymbolserver -directory <directory with symbols (for library tests these are in testhost dir)> | ||
Install the [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). | ||
```sh | ||
dotnet tool install --global dotnet-dump | ||
dotnet tool update --global dotnet-dump | ||
``` | ||
If prompted, open a new command prompt to pick up the updated PATH. | ||
```sh | ||
dotnet-dump analyze <path-to-dump> | ||
``` | ||
Within dotnet-dump: | ||
```sh | ||
setclrpath %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0 | ||
setsymbolserver -directory %LOUTDIR%/shared/Microsoft.NETCore.App/6.0.0 | ||
``` | ||
|
||
--- | ||
## If it's a macOS dump | ||
|
||
## MacOS dumps | ||
Instructions for debugging dumps on macOS are essentially the same as [Linux](#If-it's-a-Linux-dump-on-Linux...) with one exception: `dotnet-dump` cannot analyze macOS system dumps: you must use `lldb` for those. `dotnet-dump` can only analyze dumps created by `dotnet-dump` or `createdump`, by the runtime on crashes when the appropriate environment variables are set, or the [`blame-hang` setting of `dotnet test`](https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test). | ||
|
||
Instructions for debugging dumps on MacOS the same as [Linux](#linux-dumps-on-linux); however there are a couple of caveats. | ||
--- | ||
# Other Helpful Information | ||
|
||
1. It's only supported to debug them in `dotnet-dump` if it's a runtime generated dump. This includes hang dumps and dumps generated by `createdump`, `dotnet-dump` and the runtime itself. | ||
2. If it's a system dump, then only `lldb` works. | ||
* [How to debug a Linux core dump with SOS](https://github.com/dotnet/diagnostics/blob/master/documentation/debugging-coredump.md) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The last one will override the others, at least on Linux (not sure about windows, haven't seen if it is conditional on the bitness of the extension host, but given the errors I've seen I'd say no). Users should only install the one they are going to use.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh - I didn't notice that because I was testing with x64, which is last!
Could we imagine installing them all, and picking the right one - eliminate another decision point? cc @mikem8361
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll fix the text meantime
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like it defaults silently to the bitness of the dotnet.exe you're running it with (?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it does default to the architecture of the dotnet.exe you are running (the most common case).
As far as installing all the of the architectures, there would have to be a rid named subdirectory to separate them so the decision point would be postponed a little to the .load command in windbg.
This manual dotnet-sos install/load on Windows was really meant as a fallback while we wait for the public debugger to include SOS (all architectures) like the internal one does now. The internal Windows debugger automatically loads the correct architecture and latest version of SOS from the extension gallery.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh perfect then it will go away.