From 965a20dad2a28b160f3f235cb93af4957a09d99d Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Wed, 3 Nov 2021 13:15:50 -0700 Subject: [PATCH 01/23] Create an RFC to add support for embedding Natvis debugger views in Rust. --- text/0000-natvis.md | 415 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 415 insertions(+) create mode 100644 text/0000-natvis.md diff --git a/text/0000-natvis.md b/text/0000-natvis.md new file mode 100644 index 00000000000..70c0d9f9185 --- /dev/null +++ b/text/0000-natvis.md @@ -0,0 +1,415 @@ +- Feature Name: `natvis` +- Start Date: 2021-11-01 +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) + +# Summary +[summary]: #summary + +This RFC aims to improve the debugging experience for Rust developers, by +enabling Rust developers to package debugger visualization scripts with their +crates. + +# Motivation +[motivation]: #motivation + +Most, if not all, Rust developers will at some point have to debug an issue +in their crate. Trying to view types as they are laid out in memory is not +always the most telling. Furthermore when viewing types from external crates, +the information is even harder to interpret. + +Many languages and debuggers enable developers to control how a type is +displayed in a debugger. These are called "debugger visualizations" or "debugger +views". Debugger views are merely a convenience for some types, such as +`Vec`, but are essential for types such as `HashMap`, where non-trivial +logic is needed in order to correctly display the contents of a type. + +Currently, Rust provides visualizations for a handful of types defined in its +standard library via `.natvis` files or python scripts. However, this support +is inflexible; updating it requires modifying the Rust toolchain itself, +and either using a local build of the toolchain or waiting for a new upstream +build of the toolchain. It is not feasible for developers of ordinary crates +to update the Rust toolchain, solely to add visualizations for their crates. + +The expected outcome of this RFC is to design a way for developers to +seamlessly integrate debugger visualizations with their crates. This would mean: + +* Any developer can add debugger visualizations to their crate. +* If a Rust developer uses a crate that has debugger visualizations in it, + then the visualizations of those external crates will "just work" when viewed + under a debugger without the need of any manual configuration. +* Supports existing debugging visualization systems. We do not propose to + define a new debugger visualization system; that would be a tremendous + undertaking, and would ignore the value of existing systems. +* No impact on code quality or size. +* No impact on crates that do not use debugger visualizations. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +This RFC explores making Natvis debugger visualizations extensible, in Rust. +The scenario that we want to enable is: + +* Alice publishes a crate, say, `cool_stuff`. Alice wrote debugger + visualizations for `cool_stuff`, and included them in the crate. +* Bob is writing a new Rust application. Deep in the crate dependency graph of + Bob's application, some crate uses `cool_stuff`. + (Bob is not even aware of the existence of debugger visualizations.) +* While Bob is debugging the application, and examining data structures, + he comes across an instance of `cool_stuff::CoolType` in the debugger. + Because Rust and the debugger know about the visualizations that Alice wrote, + the `CoolType` value is displayed using its defined debugger view in the debugger. + Bob did not need any knowledge, a priori, of how debugger visualizations + worked or that Alice had written any debugger visualizations. From Bob's + point of view, debugging `CoolType` "just worked". + +## An example: The `regex` crate + +To make this less hypothetical, let's consider an important community crate, +one which would benefit from debugger visualizations, such as the `regex` +crate. Carol is writing an app that uses `regex` to scan over large input files. +The app code looks something like: + +```rust +// search for "#define FOO nnn" +fn find_c_defines(input: &str) { + let rx = Regex::new(r#"^#define\s+(\w+)\s+([0-9]+)\s*(//(.*))?"#).unwrap(); + for captures in rx.captures_iter(input) { + let my_match: Match = captures.get(1).unwrap(); + do_some_work(my_match.as_str()); + + } +} +``` + +Let's say that Carol is debugging the app, there's a problem in +`do_some_work()`. (Perhaps some code path has triggered a panic.) Carol wants +to look at the state of the app, inside the `find_c_defines` function, +and she specifically wants to see what the state of `captures` is. So she +selects the `find_c_defines` call frame and looks at the local variables +window. + +Unfortunately, the debugger's view of the `captures` variable does not give +her any useful information at all. It shows only something like: + +```text +> Variables + > captures: {...} + > text: "...the entire input text..." + > locs: {...} + > __0: (4) vec![None, None, None, None] + > named_groups: (refs:2) size=0, capacity=1 + > [raw]: alloc::sync::Arc> + > ptr: {pointer:0xNNNNNNNN} + > pointer: {...} + > strong: {...} + > weak: {...} + > data: size=0, capacity=1 + > base: {...} + > hash_builder: {...} + ... +``` + +The debugger shows the structure of the data, not its meaning. It is useless. +Even the implementor of `regex` would have a hard time knowing how to decode +this. In reality, when trying to understand the state of the captures +variable there are several methods defined for a `Captures` type that paint +the actual picture in terms of the information a Rust developer would like to +extract from this variable. In order to meaningfully understand what the `Captures` +type is truly trying to tell us, it would be very helpful to visualize this data +differently in the debugger. + +What we _want_ is something like this: + +```text +> Variables: + > captures: {...} + > $1: "SOME_CONSTANT" + > $2: "42" + > $3: "// some developer comment" +``` + +This RFC will describe how to support adding Natvis visualizations which is supported by: + +* The Windows Debugger (WinDbg) +* Visual Studio Debugger. + +It should be easy for crate developers to add debugger visualizations to their +crates. + +## Supporting Natvis + +This section describes how Microsoft's Natvis is supported in Rust. + +To use Natvis, developers write XML documents that describe how debugger types +should be displayed. (See: https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects?view=vs-2019) +The Natvis XML files provide patterns, which match type names, and for matching +types, a description of how to display those types. This allows for some +limited support for generic types. + +When writing Natvis files for C++, developers write a standalone XML document +and add it to their project. The build system (such as Visual Studio) knows +how to package the Natvis file into the debug data (the PDB file) for the +project, and the debugger knows how to find the Natvis XML in each PDB file. + +Developers can add one or more standalone Natvis XML files to their crate. +The `Cargo.toml` file specifies the name of these Natvis files or the Natvis +files can be specified via a command line option. This is the easiest way to +add Natvis support to a project. + +The advantage of a standalone XML document is that this process is already +well-understood by many developers. This will help C++ developers move from +C++ to Rust. It also avoids any need to modify Rust source code. If a code +base uses code generation (such as `bindgen` or proc-macros), then standalone +Natvis XML files would be the only way to provide visualizations for those +types. + +### Standalone Natvis XML files + +To provide standalone Natvis XML files, developers create a file with the +`.natvis` file extension. These Natvis files are then specified in the +`Cargo.toml` file via a new key or via the command line using the `-Z natvis` +option. + +As an example, consider a crate with this directory structure: + +```text +/Cargo.toml + +-- src + +-- main.rs + +-- main.natvis +``` + +Where `main.rs` contains: + +```rust +/// A rectangle in first quadrant +struct FancyRect { + x: f32, + y: f32, + dx: f32, + dy: f32, +} +``` + +and `main.natvis` contains: + +```xml + + + + ({x},{y}) + ({dx}, {dy}) + + + ({x}, {y}) + + + ({x}, {y + dy}) + + + ({x + dx}, {y + dy}) + + + ({x + dx}, {y}) + + + + +``` + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +Cargo would add a `-Z natvis={comma-separated list of .natvis files}` flag, +and forward this flag to rustc. + +`Cargo.toml` would add a new syntax for specifying the list of Natvis files to +be added to the crate. The new manifest key, `natvis` would be added to the +`[package]` section. This would be in control of setting the `-Z natvis` flag +that would be passed on to rustc. + +We would also add a `-Z natvis={comma-separated list of .natvis files}` flag +to rustc, which instructs the compiler to take the set of .natvis files for +a given crate and store them in the metadata. When running the linker, using +the `MSVC` toolchain, the `/NATVIS` linker option would be set and passed the +total set of .natvis files from all crate dependencies, if any exist, as well +as the current crate and embed them into the pdb. Since the `MSVC` linker is +the only one that supports embedding natvis files into a pdb, this feature +would be specific to the `MSVC` toolchain only. + +# Drawbacks +[drawbacks]: #drawbacks + +One drawback here is that a lot of types implement the Debug trait which +already specifies how a type should be viewed when debugging. Implementing +this RFC would mean a Rust developer would have to manually specify the +Natvis for a type that may already have implemented the Debug trait which +would be redundant. Currently, running the Debug trait in the debugger directly +is not possible and so a manual definition would be required to have a debugger view. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +This design provides a simple mechanism for cargo to collect the list of +.natvis files specified for a given crate and embed them in the resulting +pdb. It does not need any manual intervention by a Rust developer who is +consuming such a crate to get the debugging experience to work when it is +viewed under a debugger that supports the Natvis Framework. + +This design does not break any existing usage of cargo or rustc. This new feature +would be strictly opt-in. The Natvis syntax may not be familiar to many Rust +developers which may lead to a period of learning the syntax. Since this feature +would be optional, a consumer of a crate that has natvis definitions for types +would not need to go through this learning curve. + +Not doing this would keep the existing debugging experience for external Rust crates. +Most Rust types, outside of the standard library, do not have any debugger views +defined for them by default which makes them difficult to interpret when viewed +under a debugger. + +# Prior art +[prior-art]: #prior-art + +Many debuggers and languages already address this problem. Some do so in a way +that is more flexible than others. + +Briefly, we cover some of the known systems for debugger views: + +* Microsoft Natvis (Native Visualizers) +* Microsoft `[DebuggerDisplay]` in .NET + +## Microsoft Natvis + +Natvis is a framework that customizes how native types appear when viewed +under a debugger. The Visual Studio Natvis framework is supported out of the +box on the Windows Debugger(WinDBG) and the VS debugger. Natvis files are +essentially XML files that use the Natvis syntax to describe how to visualize +types to the debugger. This allows users to more easily interpret the data that +any given type holds. + +Taking a look at the previous Natvis example for the `FancyRect` type, the +resulting debugger view of this would be: + +```text +> Variables: + > fancy_rect: (10, 10) + (5, 5) + > LowerLeft: (10, 10) + > UpperLeft: (10, 15) + > UpperRight: (15, 15) + > LowerRight: (15, 10) +``` + +The MSVC linker supports embedding debugger visualizations defined in a Natvis file +(`.natvis`) into a PDB generated by LINK through the use of the `/NATVIS` linker flag. + +## Microsoft `[DebuggerDisplay]` and `ToString()` in .NET + +The .NET `[DebuggerDisplay]` attribute controls how objects, properties or fields +are to be displayed in the debugger. The `[DebuggerDisplay]` attribute +takes a single argument, the string to be displayed in the debugger. +Text within a pair of braces (`{``}`) is evaluated as a field, property, or method. + +If a class has an overridden `ToString()` method, then the debugger displays the +results of the `ToString()` and a `[DebuggerDisplay]` attribute is not required. +One setback here is that the debugger is unable to display the result of the +`ToString()` when viewing a crash dump. + +```csharp +/// A rectangle in first quadrant +[DebuggerDisplay("({x},{y}) + ({dx}, {dy})")] +public class FancyRect { + double x; + double y; + double dx; + double dy; +} +``` + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +Is the `[package]` section of the `Cargo.toml` manifest the best place to add this new syntax? + +# Future possibilities +[future-possibilities]: #future-possibilities + +## Inline Natvis XML fragments via an attribute + +Natvis support for Rust could be improved upon by adding support for natvis in source via an attribute. Example: + +```rust +/// A rectangle in first quadrant +#[dbgvis( + natvis(r#" + ({x},{y}) + ({dx}, {dy}) + ({x}, {y}) + ({x}, {y + dy}) + ({x + dx}, {y + dy}) + ({x + dx}, {y}) + "#))] +struct FancyRect { + x: f32, + y: f32, + dx: f32, + dy: f32, +} +``` + +## Inline Natvis XML fragments via a macro + +We may want to allow developers to provide Natvis descriptions using a +pseudo macro-call syntax, rather than an attribute. One disadvantage of +using attributes is that, lexically, attributes must be specified at the +definition of a type. Since Natvis descriptions could be quite large, this +would make it hard to read or edit the type definition while also seeing the +rustdoc comments. + +To solve this, we could define a `natvis!` macro, and use it like so: + +```rust +use std::dbgvis::natvis; + +/// A rectangle in first quadrant +struct FancyRect { + x: f32, + y: f32, + dx: f32, + dy: f32, +} + +natvis!(FancyRect, r#" + ({x},{y}) + ({dx}, {dy}) + ({x}, {y}) + ({x}, {y + dy}) + ({x + dx}, {y + dy}) + ({x + dx}, {y}) +"#); +``` + +The `natvis!` call would specify the name of the type the visualization applies +to, along with the XML fragment. This would give developers the freedom to +place visualizations anywhere in their crate, rather than at the definition +of each type. + +## Auto-discover Natvis XML files + +We may want to auto-discover Natvis files by searching specific directories +for .natvis files. For example, developers create a file with the +`.natvis` file extension, and place it within the `dbgvis/natvis` subdirectory +of their crate. The `dbgvis` directory is reserved for debugger visualizations, +and the `natvis` subdirectory is reserved for Natvis visualizations. (The +name `dbgvis` was chosen to avoid conflicts with `Debug` directories created +by build systems or IDEs; often, `.gitignore` files ignore `Debug` directories.) + +Cargo automatically scans for `dbgvis/natvis/*.natvis` files. This behavior +can be overridden by specifying manifest keys. + +# References + +* Natvis + + [Create custom views of C++ objects in the debugger using the Natvis framework](https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects) + + [Visual Studio native debug visualization (natvis) for C++/WinRT](https://docs.microsoft.com/en-us/windows/uwp/cpp-and-winrt-apis/natvis) + + https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/native-debugger-objects-in-natvis + +* .NET `[DebuggerDisplay]` + + https://docs.microsoft.com/en-us/visualstudio/debugger/using-the-debuggerdisplay-attribute \ No newline at end of file From 99f0d49fc033660be9d0776ef73bf375e75c9eae Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Wed, 3 Nov 2021 14:00:15 -0700 Subject: [PATCH 02/23] Minor cleanups. --- text/0000-natvis.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 70c0d9f9185..e858cb353de 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -59,8 +59,8 @@ The scenario that we want to enable is: he comes across an instance of `cool_stuff::CoolType` in the debugger. Because Rust and the debugger know about the visualizations that Alice wrote, the `CoolType` value is displayed using its defined debugger view in the debugger. - Bob did not need any knowledge, a priori, of how debugger visualizations - worked or that Alice had written any debugger visualizations. From Bob's + Bob did not need any knowledge, a priori, of how debugger visualizations + worked or that Alice had written any debugger visualizations. From Bob's point of view, debugging `CoolType` "just worked". ## An example: The `regex` crate @@ -77,7 +77,6 @@ fn find_c_defines(input: &str) { for captures in rx.captures_iter(input) { let my_match: Match = captures.get(1).unwrap(); do_some_work(my_match.as_str()); - } } ``` @@ -310,7 +309,7 @@ takes a single argument, the string to be displayed in the debugger. Text within a pair of braces (`{``}`) is evaluated as a field, property, or method. If a class has an overridden `ToString()` method, then the debugger displays the -results of the `ToString()` and a `[DebuggerDisplay]` attribute is not required. +results of the `ToString()` method and a `[DebuggerDisplay]` attribute is not required. One setback here is that the debugger is unable to display the result of the `ToString()` when viewing a crash dump. From 56d0cafb631e1568ab6ee041e58a276115ab999e Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Fri, 5 Nov 2021 17:58:09 -0700 Subject: [PATCH 03/23] Cleanup RFC to no longer include a new -Z cli option for Cargo. --- text/0000-natvis.md | 56 ++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 21 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index e858cb353de..74af06b6763 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -152,9 +152,8 @@ how to package the Natvis file into the debug data (the PDB file) for the project, and the debugger knows how to find the Natvis XML in each PDB file. Developers can add one or more standalone Natvis XML files to their crate. -The `Cargo.toml` file specifies the name of these Natvis files or the Natvis -files can be specified via a command line option. This is the easiest way to -add Natvis support to a project. +The `Cargo.toml` file specifies the name of these Natvis files. This is the +easiest way to add Natvis support to a project. The advantage of a standalone XML document is that this process is already well-understood by many developers. This will help C++ developers move from @@ -167,8 +166,7 @@ types. To provide standalone Natvis XML files, developers create a file with the `.natvis` file extension. These Natvis files are then specified in the -`Cargo.toml` file via a new key or via the command line using the `-Z natvis` -option. +`Cargo.toml` file via a new manifest key. As an example, consider a crate with this directory structure: @@ -219,22 +217,38 @@ and `main.natvis` contains: # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -Cargo would add a `-Z natvis={comma-separated list of .natvis files}` flag, -and forward this flag to rustc. - -`Cargo.toml` would add a new syntax for specifying the list of Natvis files to -be added to the crate. The new manifest key, `natvis` would be added to the -`[package]` section. This would be in control of setting the `-Z natvis` flag -that would be passed on to rustc. - -We would also add a `-Z natvis={comma-separated list of .natvis files}` flag -to rustc, which instructs the compiler to take the set of .natvis files for -a given crate and store them in the metadata. When running the linker, using -the `MSVC` toolchain, the `/NATVIS` linker option would be set and passed the -total set of .natvis files from all crate dependencies, if any exist, as well -as the current crate and embed them into the pdb. Since the `MSVC` linker is -the only one that supports embedding natvis files into a pdb, this feature -would be specific to the `MSVC` toolchain only. +In rustc, a new `-Z natvis={comma-separated list of .natvis files}` flag +will be added which instructs the compiler to take the set of .natvis +files for a given crate and store them in the crate metadata. When running the +linker, using the `MSVC` toolchain, a `/NATVIS` linker option would be set +for each .natvis file. This includes .natvis files from all crate dependencies, +if any exist, as well as the current crate and embed them into the pdb. +Since the `MSVC` linker is the only one that supports embedding natvis files +into a pdb, this feature would be specific to the `MSVC` toolchain only. + +Cargo would add new syntax in `Cargo.toml` for specifying the list of Natvis +files to be added to the crate. The new manifest key, `natvis` would be added +to the `[package]` section. This would be in control of setting the `-Z natvis` +flag in rustc. + +For example: + +`Cargo.toml`: + +```toml +cargo-features = ["natvis"] + +[package] +name = "natvis" +version = "0.1.0" +edition = "2018" +natvis = ["src/a.natvis", "src/b.natvis"] +``` + +This would generate a call to rustc similar to the following, +(for simplicity purposes, most of the rustc command line has been removed): + +`rustc -Z natvis=path/to/file/a.natvis,path/to/file/b.natvis` # Drawbacks [drawbacks]: #drawbacks From bc219af1da2c7dded5fcc7e77ab5013e256250b9 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Wed, 17 Nov 2021 08:47:44 -0800 Subject: [PATCH 04/23] Respond to PR feedback. --- text/0000-natvis.md | 240 +++++++++++++++++++++++++++----------------- 1 file changed, 149 insertions(+), 91 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 74af06b6763..001e7fc2453 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -24,6 +24,49 @@ views". Debugger views are merely a convenience for some types, such as `Vec`, but are essential for types such as `HashMap`, where non-trivial logic is needed in order to correctly display the contents of a type. +Since Natvis is supported via WinDbg and the VS Debugger, this support is +specific to Windows and the MSVC toolchain. + +For example, given the following instance of `HashMap`: +```rust +fn main() { + let mut map = HashMap::new(); + map.insert(1, 1); + map.insert(2, 2); + map.insert(3, 3); +} +``` + +Viewed under the Windows Debugger (WinDbg), the following is shown: +```text +> Variables + > map: [Type: std::collections::hash::map::HashMap] + > [+0x000] base: [Type: hashbrown::map::HashMap] + > [+0x000] hash_builder: [Type: std::collections::hash::map::RandomState] + > [+0x010] table: [Type: hashbrown::raw::RawTable,alloc::alloc::Global>] + > [+0x000] table: [Type: hashbrown::raw::RawTableInner] + > [+0x000] bucket_mask: 0x3 [Type: unsigned __int64] + > [+0x008] ctrl [Type: core::ptr::non_null::NonNull] + > [+0x010] growth_left: 0x0 [Type: unsigned __int64] + > [+0x018] items: 0x3 [Type: unsigned __int64] + > [+0x000] alloc: [Type: alloc::alloc::Global] + > [+0x000] marker: [Type: core::marker::PhantomData >] + ... +``` + +With Natvis applied, WinDbg results in the following: +```text +> Variables + > map: { len=0x1 } [Type: std::collections::hash::map::HashMap] + > [] [Type: std::collections::hash::map::HashMap] + > [len]: 0x1 [Type: unsigned __int64] + > [capacity]: 0x3 + > [state]: [Type: std::collections::hash::map::RandomState] + > ["1"]: 1 [Type: int] + > ["2"]: 2 [Type: int] + > ["3"]: 3 [Type: int] +``` + Currently, Rust provides visualizations for a handful of types defined in its standard library via `.natvis` files or python scripts. However, this support is inflexible; updating it requires modifying the Rust toolchain itself, @@ -31,8 +74,8 @@ and either using a local build of the toolchain or waiting for a new upstream build of the toolchain. It is not feasible for developers of ordinary crates to update the Rust toolchain, solely to add visualizations for their crates. -The expected outcome of this RFC is to design a way for developers to -seamlessly integrate debugger visualizations with their crates. This would mean: +The expected outcome of this RFC is to design a way for developers to seamlessly +integrate Natvis debugger visualizations with their crates. This would mean: * Any developer can add debugger visualizations to their crate. * If a Rust developer uses a crate that has debugger visualizations in it, @@ -57,11 +100,12 @@ The scenario that we want to enable is: (Bob is not even aware of the existence of debugger visualizations.) * While Bob is debugging the application, and examining data structures, he comes across an instance of `cool_stuff::CoolType` in the debugger. - Because Rust and the debugger know about the visualizations that Alice wrote, - the `CoolType` value is displayed using its defined debugger view in the debugger. - Bob did not need any knowledge, a priori, of how debugger visualizations - worked or that Alice had written any debugger visualizations. From Bob's - point of view, debugging `CoolType` "just worked". + Since the Rust compiler has embedded the Natvis visualizations that Alice wrote + into the debuginfo for the binary and the debugger is able to load up and serve the + Natvis visualizations, the `CoolType` value is displayed using its defined debugger + view in the debugger. Bob did not need any knowledge, a priori, of how debugger + visualizations worked or that Alice had written any debugger visualizations. + From Bob's point of view, debugging `CoolType` "just worked". ## An example: The `regex` crate @@ -109,9 +153,9 @@ her any useful information at all. It shows only something like: ... ``` -The debugger shows the structure of the data, not its meaning. It is useless. -Even the implementor of `regex` would have a hard time knowing how to decode -this. In reality, when trying to understand the state of the captures +The debugger shows the structure of the data, not its meaning. It is not very +useful for Carol. Even the implementor of `regex` would have a hard time knowing +how to decode this. In reality, when trying to understand the state of the captures variable there are several methods defined for a `Captures` type that paint the actual picture in terms of the information a Rust developer would like to extract from this variable. In order to meaningfully understand what the `Captures` @@ -123,9 +167,9 @@ What we _want_ is something like this: ```text > Variables: > captures: {...} - > $1: "SOME_CONSTANT" - > $2: "42" - > $3: "// some developer comment" + > 1: "SOME_CONSTANT" + > 2: "42" + > 3: "// some developer comment" ``` This RFC will describe how to support adding Natvis visualizations which is supported by: @@ -133,7 +177,7 @@ This RFC will describe how to support adding Natvis visualizations which is supp * The Windows Debugger (WinDbg) * Visual Studio Debugger. -It should be easy for crate developers to add debugger visualizations to their +It should be easy for Rust developers to add debugger visualizations to their crates. ## Supporting Natvis @@ -141,40 +185,30 @@ crates. This section describes how Microsoft's Natvis is supported in Rust. To use Natvis, developers write XML documents that describe how debugger types -should be displayed. (See: https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects?view=vs-2019) -The Natvis XML files provide patterns, which match type names, and for matching +should be displayed using the natvis schema. (See: https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects?view=vs-2019) +The Natvis files provide patterns, which match type names, and for matching types, a description of how to display those types. This allows for some limited support for generic types. -When writing Natvis files for C++, developers write a standalone XML document -and add it to their project. The build system (such as Visual Studio) knows -how to package the Natvis file into the debug data (the PDB file) for the -project, and the debugger knows how to find the Natvis XML in each PDB file. +Rust developers can add one or more `.natvis` files to their crate. The +`Cargo.toml` file specifies the path of these Natvis files via a new +manifest key. Cargo would then pass the set of `.natvis` files that were +specified in `Cargo.toml` and pass it to the rustc invocation for the crate +via a new `-Z` flag. See the below example for how rustc would embed these +Natvis files in the debuginfo for a binary as well as new compiler options +to be added to rustc. -Developers can add one or more standalone Natvis XML files to their crate. -The `Cargo.toml` file specifies the name of these Natvis files. This is the -easiest way to add Natvis support to a project. +To provide Natvis files, developers create a file with the `.natvis` file +extension and specify this Natvis file in the `Cargo.toml` under a new section +that will be added as part of this RFC. -The advantage of a standalone XML document is that this process is already -well-understood by many developers. This will help C++ developers move from -C++ to Rust. It also avoids any need to modify Rust source code. If a code -base uses code generation (such as `bindgen` or proc-macros), then standalone -Natvis XML files would be the only way to provide visualizations for those -types. - -### Standalone Natvis XML files - -To provide standalone Natvis XML files, developers create a file with the -`.natvis` file extension. These Natvis files are then specified in the -`Cargo.toml` file via a new manifest key. - -As an example, consider a crate with this directory structure: +As an example, consider a crate `foo` with this directory structure: ```text /Cargo.toml +/Foo.natvis (Note: the .natvis file does not have to match the name of the crate.) +-- src +-- main.rs - +-- main.natvis ``` Where `main.rs` contains: @@ -182,19 +216,24 @@ Where `main.rs` contains: ```rust /// A rectangle in first quadrant struct FancyRect { - x: f32, - y: f32, - dx: f32, - dy: f32, + pub x: f32, + pub y: f32, + pub dx: f32, + pub dy: f32, +} + +fn main() { + let mut fancy_rect = FancyRect::new(10.0, 10.0, 5.0, 5.0); + println!("FancyRect: {:?}", fancy_rect); } ``` -and `main.natvis` contains: +and `Foo.natvis` contains: ```xml - + ({x},{y}) + ({dx}, {dy}) @@ -214,6 +253,17 @@ and `main.natvis` contains: ``` +When viewed under WinDbg, the `fancy_rect` variable would be shown as follows: + +```text +> Variables: + > fancy_rect: (10, 10) + (5, 5) + > LowerLeft: (10, 10) + > UpperLeft: (10, 15) + > UpperRight: (15, 15) + > LowerRight: (15, 10) +``` + # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -221,14 +271,17 @@ In rustc, a new `-Z natvis={comma-separated list of .natvis files}` flag will be added which instructs the compiler to take the set of .natvis files for a given crate and store them in the crate metadata. When running the linker, using the `MSVC` toolchain, a `/NATVIS` linker option would be set -for each .natvis file. This includes .natvis files from all crate dependencies, -if any exist, as well as the current crate and embed them into the pdb. -Since the `MSVC` linker is the only one that supports embedding natvis files -into a pdb, this feature would be specific to the `MSVC` toolchain only. +for each `.natvis` file. This includes `.natvis` files from all crate dependencies, +if any exist, as well as the current crate and embed them into the PDB. + +The MSVC linker supports embedding debugger visualizations defined in a Natvis file +(`.natvis`) into a PDB generated by LINK through the use of the `/NATVIS` linker flag. +Since the `MSVC` linker is the only one that supports embedding Natvis files +into a PDB, this feature would be specific to the `MSVC` toolchain only. Cargo would add new syntax in `Cargo.toml` for specifying the list of Natvis files to be added to the crate. The new manifest key, `natvis` would be added -to the `[package]` section. This would be in control of setting the `-Z natvis` +to a new `[dbgvis]` section. This would be in control of setting the `-Z natvis` flag in rustc. For example: @@ -242,7 +295,9 @@ cargo-features = ["natvis"] name = "natvis" version = "0.1.0" edition = "2018" -natvis = ["src/a.natvis", "src/b.natvis"] + +[dbgvis] +natvis = ["a.natvis", "b.natvis"] ``` This would generate a call to rustc similar to the following, @@ -250,6 +305,16 @@ This would generate a call to rustc similar to the following, `rustc -Z natvis=path/to/file/a.natvis,path/to/file/b.natvis` +The `CrateRoot` type would also need to be updated to account for `.natvis` +files for crates within the dependency graph. To reflect this a new field, +`natvis_files: Lazy<[String]>,` would be added. This will store the list of +`.natvis` files that were passed to the invocation of rustc for the specific crate. + +Another change that would need to be made here is to add a new field to the +`CrateInfo` type, `pub natvis_files: Vec`. This will allow the `MsvcLinker` +type to query the list of Natvis files that exist within the crate dependency graph +and add the `/NATVIS` linker arg for each `.natvis` file. + # Drawbacks [drawbacks]: #drawbacks @@ -288,55 +353,48 @@ that is more flexible than others. Briefly, we cover some of the known systems for debugger views: -* Microsoft Natvis (Native Visualizers) -* Microsoft `[DebuggerDisplay]` in .NET +* Windows Debugger (WinDbg) +* Visual Studio Debugger (VS Debugger) +* GDB/LLDB -## Microsoft Natvis +## Windows Debugger (WinDbg) -Natvis is a framework that customizes how native types appear when viewed -under a debugger. The Visual Studio Natvis framework is supported out of the -box on the Windows Debugger(WinDBG) and the VS debugger. Natvis files are -essentially XML files that use the Natvis syntax to describe how to visualize -types to the debugger. This allows users to more easily interpret the data that -any given type holds. +Natvis is a framework that customizes how native types appear when viewed under +a debugger. The Visual Studio Natvis framework is supported out of the box on +WinDbg. The debugger has the ability to load `.natvis` files via the `.nvload` +command and directly apply them to types within loaded modules. WinDbg is also +able to load `.natvis` files that have been embedded in the PDB for a binary and +serve up the resulting views after applying those visualizations as well. This +allows for a very smooth debugging experience which would not depend on any manual +loading of Natvis files. -Taking a look at the previous Natvis example for the `FancyRect` type, the -resulting debugger view of this would be: +## Visual Studio Debugger (VS Debugger) -```text -> Variables: - > fancy_rect: (10, 10) + (5, 5) - > LowerLeft: (10, 10) - > UpperLeft: (10, 15) - > UpperRight: (15, 15) - > LowerRight: (15, 10) -``` +The Visual Studio Debugger also supports Natvis. Similar to WinDbg, the VS Debugger +is also able to apply Natvis on the fly by loading user-specified `.natvis` files. +As with WinDbg, it also supports loading `.natvis` files that were embedded in the +PDB for a binary and automatically applying the Natvis visualizations to types from +that binary. -The MSVC linker supports embedding debugger visualizations defined in a Natvis file -(`.natvis`) into a PDB generated by LINK through the use of the `/NATVIS` linker flag. +When using Visual Studio to build a C++ project, a developer can add a Natvis file +via the `.vcxproj` file. To add a Natvis file to a project the following can be +added to the `.vcxproj` file: -## Microsoft `[DebuggerDisplay]` and `ToString()` in .NET - -The .NET `[DebuggerDisplay]` attribute controls how objects, properties or fields -are to be displayed in the debugger. The `[DebuggerDisplay]` attribute -takes a single argument, the string to be displayed in the debugger. -Text within a pair of braces (`{``}`) is evaluated as a field, property, or method. +```text + + + +``` -If a class has an overridden `ToString()` method, then the debugger displays the -results of the `ToString()` method and a `[DebuggerDisplay]` attribute is not required. -One setback here is that the debugger is unable to display the result of the -`ToString()` when viewing a crash dump. +## GDB/LLDB -```csharp -/// A rectangle in first quadrant -[DebuggerDisplay("({x},{y}) + ({dx}, {dy})")] -public class FancyRect { - double x; - double y; - double dx; - double dy; -} -``` +GDB and LLDB also support debugger views but in a different way than WinDbg and the +VS debugger. Natvis is not supported by either GDB or LLDB but they do support pretty +printers. Pretty printers work in the similar manner as Natvis in which they tell +the debugger to serve up a specific visualization when viewing a type in the debugger. +Pretty printers are written as python scripts and then have to be imported in to the +debugger. When a type is viewed under the debugger that has a pretty printer, that view +is automatically shown. # Unresolved questions [unresolved-questions]: #unresolved-questions From e30a6151730cc2ada9d2ff8f38b99e648443414e Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Wed, 17 Nov 2021 17:17:22 -0800 Subject: [PATCH 05/23] Update the name of the new section to be added to Cargo.toml. --- text/0000-natvis.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 001e7fc2453..053ce492edb 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -281,8 +281,8 @@ into a PDB, this feature would be specific to the `MSVC` toolchain only. Cargo would add new syntax in `Cargo.toml` for specifying the list of Natvis files to be added to the crate. The new manifest key, `natvis` would be added -to a new `[dbgvis]` section. This would be in control of setting the `-Z natvis` -flag in rustc. +to a new `[debug-visualizations]` section. This would be in control of setting +the `-Z natvis` flag in rustc. For example: @@ -296,7 +296,7 @@ name = "natvis" version = "0.1.0" edition = "2018" -[dbgvis] +[debug-visualizations] natvis = ["a.natvis", "b.natvis"] ``` From 29d5cf72921ff77f7114c419c0f23bf843661830 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Wed, 24 Nov 2021 15:00:53 -0800 Subject: [PATCH 06/23] Respond to PR comments. --- text/0000-natvis.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 053ce492edb..5cf0fbc27b2 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -194,7 +194,8 @@ Rust developers can add one or more `.natvis` files to their crate. The `Cargo.toml` file specifies the path of these Natvis files via a new manifest key. Cargo would then pass the set of `.natvis` files that were specified in `Cargo.toml` and pass it to the rustc invocation for the crate -via a new `-Z` flag. See the below example for how rustc would embed these +via a new `-C` flag and enabling unstable options in Rust via the +`-Z unstable-options`. See the below example for how rustc would embed these Natvis files in the debuginfo for a binary as well as new compiler options to be added to rustc. @@ -267,12 +268,14 @@ When viewed under WinDbg, the `fancy_rect` variable would be shown as follows: # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -In rustc, a new `-Z natvis={comma-separated list of .natvis files}` flag +In rustc, a new `-C natvis={comma-separated list of .natvis files}` flag will be added which instructs the compiler to take the set of .natvis -files for a given crate and store them in the crate metadata. When running the -linker, using the `MSVC` toolchain, a `/NATVIS` linker option would be set -for each `.natvis` file. This includes `.natvis` files from all crate dependencies, -if any exist, as well as the current crate and embed them into the PDB. +files for a given crate and store them in the crate metadata. This will be gated +by the existing `-Z unstable-options` flag to ensure it is only used in unstable +Rust. When running the linker, using the `MSVC` toolchain, a `/NATVIS` linker +option would be set for each `.natvis` file. This includes `.natvis` files from +all crate dependencies, if any exist, as well as the current crate and embed them +into the PDB. The MSVC linker supports embedding debugger visualizations defined in a Natvis file (`.natvis`) into a PDB generated by LINK through the use of the `/NATVIS` linker flag. @@ -282,7 +285,7 @@ into a PDB, this feature would be specific to the `MSVC` toolchain only. Cargo would add new syntax in `Cargo.toml` for specifying the list of Natvis files to be added to the crate. The new manifest key, `natvis` would be added to a new `[debug-visualizations]` section. This would be in control of setting -the `-Z natvis` flag in rustc. +the `-C natvis` flag in rustc as well as `-Z unstable-options` to enable it. For example: @@ -303,7 +306,12 @@ natvis = ["a.natvis", "b.natvis"] This would generate a call to rustc similar to the following, (for simplicity purposes, most of the rustc command line has been removed): -`rustc -Z natvis=path/to/file/a.natvis,path/to/file/b.natvis` +`rustc -C natvis=path/to/file/a.natvis,path/to/file/b.natvis -Z unstable-options` + +`.natvis` files that contain spaces within the path will be quoted to ensure +the path remains valid when passing command line options to rustc. Since the `-Z` +option is a comma-separated list, a comma would not be valid within the path +for a Natvis file. The `CrateRoot` type would also need to be updated to account for `.natvis` files for crates within the dependency graph. To reflect this a new field, @@ -480,7 +488,4 @@ can be overridden by specifying manifest keys. * Natvis + [Create custom views of C++ objects in the debugger using the Natvis framework](https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects) + [Visual Studio native debug visualization (natvis) for C++/WinRT](https://docs.microsoft.com/en-us/windows/uwp/cpp-and-winrt-apis/natvis) - + https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/native-debugger-objects-in-natvis - -* .NET `[DebuggerDisplay]` - + https://docs.microsoft.com/en-us/visualstudio/debugger/using-the-debuggerdisplay-attribute \ No newline at end of file + + https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/native-debugger-objects-in-natvis \ No newline at end of file From 90138abbff607b79f6720f772c20599da017d54e Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Thu, 9 Dec 2021 12:56:25 -0800 Subject: [PATCH 07/23] Respond to PR comments. Add clarity that rustc will not enforce the .natvis extension on files passed via the `-C natvis` flag. --- text/0000-natvis.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 5cf0fbc27b2..df5554a9ab9 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -270,12 +270,14 @@ When viewed under WinDbg, the `fancy_rect` variable would be shown as follows: In rustc, a new `-C natvis={comma-separated list of .natvis files}` flag will be added which instructs the compiler to take the set of .natvis -files for a given crate and store them in the crate metadata. This will be gated -by the existing `-Z unstable-options` flag to ensure it is only used in unstable -Rust. When running the linker, using the `MSVC` toolchain, a `/NATVIS` linker -option would be set for each `.natvis` file. This includes `.natvis` files from -all crate dependencies, if any exist, as well as the current crate and embed them -into the PDB. +files for a given crate and store them in the crate metadata. Rustc will not +have any enforcement that the files specified via the new flag have the `.natvis` +extension, but this is required in order for the Natvis to be loaded by both +the VS debugger and WinDbg. This will be gated by the existing `-Z unstable-options` +flag to ensure it is only used in unstable Rust. When running the linker, +using the `MSVC` toolchain, a `/NATVIS` linker option would be set for each +`.natvis` file. This includes `.natvis` files from all crate dependencies, +if any exist, as well as the current crate and embed them into the PDB. The MSVC linker supports embedding debugger visualizations defined in a Natvis file (`.natvis`) into a PDB generated by LINK through the use of the `/NATVIS` linker flag. @@ -284,7 +286,7 @@ into a PDB, this feature would be specific to the `MSVC` toolchain only. Cargo would add new syntax in `Cargo.toml` for specifying the list of Natvis files to be added to the crate. The new manifest key, `natvis` would be added -to a new `[debug-visualizations]` section. This would be in control of setting +to a new `[debugger-visualizations]` section. This would be in control of setting the `-C natvis` flag in rustc as well as `-Z unstable-options` to enable it. For example: @@ -299,7 +301,7 @@ name = "natvis" version = "0.1.0" edition = "2018" -[debug-visualizations] +[debugger-visualizations] natvis = ["a.natvis", "b.natvis"] ``` @@ -407,7 +409,7 @@ is automatically shown. # Unresolved questions [unresolved-questions]: #unresolved-questions -Is the `[package]` section of the `Cargo.toml` manifest the best place to add this new syntax? +None. # Future possibilities [future-possibilities]: #future-possibilities From 901f528fdc4341ee44a4d1b9dca7c54c97142689 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Thu, 9 Dec 2021 15:51:46 -0800 Subject: [PATCH 08/23] Update the RFC to include new Cargo warning that would be emitted if the natvis files specified via the manifest key do not have the `.natvis` extension. --- text/0000-natvis.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index df5554a9ab9..c9a6558676f 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -286,8 +286,10 @@ into a PDB, this feature would be specific to the `MSVC` toolchain only. Cargo would add new syntax in `Cargo.toml` for specifying the list of Natvis files to be added to the crate. The new manifest key, `natvis` would be added -to a new `[debugger-visualizations]` section. This would be in control of setting -the `-C natvis` flag in rustc as well as `-Z unstable-options` to enable it. +to a new `[debugger-visualizations]` section. Cargo would emit a warning if +any of the specified Natvis files do not have the `.natvis` extension. This +would be in control of setting the `-C natvis` flag in rustc as well as +`-Z unstable-options` to enable it. For example: @@ -317,11 +319,11 @@ for a Natvis file. The `CrateRoot` type would also need to be updated to account for `.natvis` files for crates within the dependency graph. To reflect this a new field, -`natvis_files: Lazy<[String]>,` would be added. This will store the list of +`natvis_files: Lazy<[PathBuf]>,` would be added. This will store the list of `.natvis` files that were passed to the invocation of rustc for the specific crate. Another change that would need to be made here is to add a new field to the -`CrateInfo` type, `pub natvis_files: Vec`. This will allow the `MsvcLinker` +`CrateInfo` type, `pub natvis_files: Option>`. This will allow the `MsvcLinker` type to query the list of Natvis files that exist within the crate dependency graph and add the `/NATVIS` linker arg for each `.natvis` file. From a28ae0038ab4991afd7cbbb1b8c91a3faaee8bbf Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Tue, 18 Jan 2022 10:57:11 -0800 Subject: [PATCH 09/23] Respond to PR comments. --- text/0000-natvis.md | 45 ++++++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index c9a6558676f..95dc1334a3f 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -1,6 +1,6 @@ - Feature Name: `natvis` - Start Date: 2021-11-01 -- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- RFC PR: [rust-lang/rfcs#3191](https://github.com/rust-lang/rfcs/pull/3191) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) # Summary @@ -268,15 +268,18 @@ When viewed under WinDbg, the `fancy_rect` variable would be shown as follows: # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -In rustc, a new `-C natvis={comma-separated list of .natvis files}` flag -will be added which instructs the compiler to take the set of .natvis -files for a given crate and store them in the crate metadata. Rustc will not +In rustc, a new `-C natvis={path to single .natvis file}` flag will be added +which instructs the compiler to take a `.natvis` file for a given crate, serialize +the contents of this file and store it in the crate metadata. Rustc will not have any enforcement that the files specified via the new flag have the `.natvis` extension, but this is required in order for the Natvis to be loaded by both -the VS debugger and WinDbg. This will be gated by the existing `-Z unstable-options` -flag to ensure it is only used in unstable Rust. When running the linker, -using the `MSVC` toolchain, a `/NATVIS` linker option would be set for each -`.natvis` file. This includes `.natvis` files from all crate dependencies, +the VS debugger and WinDbg. This flag can be used multiple times to allow for +multiple `.natvis` files to be embedded for each crate. This new flag will be gated +by the existing `-Z unstable-options` flag to ensure it is only used in unstable +Rust. When running the linker, using the `MSVC` toolchain, the contents of the +`.natvis` file will be extracted from the crate metadata and store them in the target +directory under a new `natvis` directory. A `/NATVIS` linker option would be set +for each `.natvis` file which includes `.natvis` files from all crate dependencies, if any exist, as well as the current crate and embed them into the PDB. The MSVC linker supports embedding debugger visualizations defined in a Natvis file @@ -310,22 +313,22 @@ natvis = ["a.natvis", "b.natvis"] This would generate a call to rustc similar to the following, (for simplicity purposes, most of the rustc command line has been removed): -`rustc -C natvis=path/to/file/a.natvis,path/to/file/b.natvis -Z unstable-options` +`rustc -C natvis=path/to/file/a.natvis -C natvis=path/to/file/b.natvis -Z unstable-options` -`.natvis` files that contain spaces within the path will be quoted to ensure -the path remains valid when passing command line options to rustc. Since the `-Z` -option is a comma-separated list, a comma would not be valid within the path -for a Natvis file. +`.natvis` files that contain spaces or commas within the path will be quoted to ensure +the path remains valid when passing command line options to rustc. The `CrateRoot` type would also need to be updated to account for `.natvis` -files for crates within the dependency graph. To reflect this a new field, -`natvis_files: Lazy<[PathBuf]>,` would be added. This will store the list of -`.natvis` files that were passed to the invocation of rustc for the specific crate. +files for crates within the dependency graph. To reflect this a new type +`pub struct NatvisFile` will be created to ensure the contents of a `.natvis` +file can be serialized and stored in the crate metadata. The `CrateRoot` would +contian the field, `natvis_files: Lazy<[NatvisFile]>`. Another change that would need to be made here is to add a new field to the -`CrateInfo` type, `pub natvis_files: Option>`. This will allow the `MsvcLinker` -type to query the list of Natvis files that exist within the crate dependency graph -and add the `/NATVIS` linker arg for each `.natvis` file. +`CrateInfo` type, `pub used_crates_natvis: Option>`. This will +allow the `MsvcLinker` type to query the list of Natvis files that exist within +the crate dependency graph and add the `/NATVIS` linker argument for each +`.natvis` file. # Drawbacks [drawbacks]: #drawbacks @@ -341,7 +344,7 @@ is not possible and so a manual definition would be required to have a debugger [rationale-and-alternatives]: #rationale-and-alternatives This design provides a simple mechanism for cargo to collect the list of -.natvis files specified for a given crate and embed them in the resulting +`.natvis` files specified for a given crate and embed them in the resulting pdb. It does not need any manual intervention by a Rust developer who is consuming such a crate to get the debugging experience to work when it is viewed under a debugger that supports the Natvis Framework. @@ -477,7 +480,7 @@ of each type. ## Auto-discover Natvis XML files We may want to auto-discover Natvis files by searching specific directories -for .natvis files. For example, developers create a file with the +for `.natvis` files. For example, developers create a file with the `.natvis` file extension, and place it within the `dbgvis/natvis` subdirectory of their crate. The `dbgvis` directory is reserved for debugger visualizations, and the `natvis` subdirectory is reserved for Natvis visualizations. (The From 9d17e9b7ceab66bacd557e0452d16cf9fcc9a54e Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Tue, 18 Jan 2022 11:04:46 -0800 Subject: [PATCH 10/23] Update RFC to respond to PR comment. --- text/0000-natvis.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 95dc1334a3f..5b65a86c0eb 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -280,7 +280,9 @@ Rust. When running the linker, using the `MSVC` toolchain, the contents of the `.natvis` file will be extracted from the crate metadata and store them in the target directory under a new `natvis` directory. A `/NATVIS` linker option would be set for each `.natvis` file which includes `.natvis` files from all crate dependencies, -if any exist, as well as the current crate and embed them into the PDB. +if any exist, as well as the current crate and embed them into the PDB. Any crate type +that would generate a PDB would have any applicable `.natvis` files embedded in the +generated PDB. The MSVC linker supports embedding debugger visualizations defined in a Natvis file (`.natvis`) into a PDB generated by LINK through the use of the `/NATVIS` linker flag. From b30449e6d93c64ba82e4b412fd026d7788769d40 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Mon, 24 Jan 2022 11:03:00 -0800 Subject: [PATCH 11/23] Respond to PR comments. --- text/0000-natvis.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 5b65a86c0eb..c7bff1b738c 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -324,13 +324,14 @@ The `CrateRoot` type would also need to be updated to account for `.natvis` files for crates within the dependency graph. To reflect this a new type `pub struct NatvisFile` will be created to ensure the contents of a `.natvis` file can be serialized and stored in the crate metadata. The `CrateRoot` would -contian the field, `natvis_files: Lazy<[NatvisFile]>`. +contain the field, `natvis_files: Lazy<[NatvisFile]>`. Another change that would need to be made here is to add a new field to the -`CrateInfo` type, `pub used_crates_natvis: Option>`. This will +`CrateInfo` type, `pub natvis_files: Option>`. This will allow the `MsvcLinker` type to query the list of Natvis files that exist within the crate dependency graph and add the `/NATVIS` linker argument for each -`.natvis` file. +`.natvis` file. This will also be the stage at which the Natvis files for each +crate are copied to the target directory which they will be linked from. # Drawbacks [drawbacks]: #drawbacks From 2fafa9aaff1e8a9efd413f9013f438e8a53ab7ca Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Tue, 25 Jan 2022 12:34:54 -0800 Subject: [PATCH 12/23] Update the RFC and move the Cargo feature to auto-discover Natvis files to the first iteration of this feature. --- text/0000-natvis.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index c7bff1b738c..20e4d58d7d3 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -289,12 +289,26 @@ The MSVC linker supports embedding debugger visualizations defined in a Natvis f Since the `MSVC` linker is the only one that supports embedding Natvis files into a PDB, this feature would be specific to the `MSVC` toolchain only. -Cargo would add new syntax in `Cargo.toml` for specifying the list of Natvis -files to be added to the crate. The new manifest key, `natvis` would be added -to a new `[debugger-visualizations]` section. Cargo would emit a warning if -any of the specified Natvis files do not have the `.natvis` extension. This -would be in control of setting the `-C natvis` flag in rustc as well as -`-Z unstable-options` to enable it. +Cargo would add a new feature to auto-discover Natvis files by searching a specific +directory, `dbgvis/natvis/` for `.natvis` files. For example, developers create a +file with the `.natvis` file extension, and place it within the `dbgvis/natvis/` +subdirectory of their crate. The `dbgvis` directory is reserved for debugger +visualizations, and the `natvis` subdirectory is reserved for Natvis visualizations. +(The name `dbgvis` was chosen to avoid conflicts with `Debug` directories created +by build systems or IDEs; often, `.gitignore` files ignore `Debug` directories.) + +Cargo automatically scans for `dbgvis/natvis/*.natvis` files. This behavior +can be overridden by specifying a new manifest key. + +Cargo would also add new syntax in `Cargo.toml` for specifying the list of Natvis +files to be added to the crate directly. The new manifest key, `natvis` would be +added to a new `[debugger-visualizations]` section. Cargo would emit a warning if +any of the specified Natvis files do not have the `.natvis` extension. + +As with the auto-discover feature, this would be in control of setting the +`-C natvis` flag in rustc as well as `-Z unstable-options` to enable it. Using +the new toml syntax to specify `.natvis` files would override Cargo from +auto-discovering Natvis files in the `dbgvis/natvis/` directory. For example: @@ -480,19 +494,6 @@ to, along with the XML fragment. This would give developers the freedom to place visualizations anywhere in their crate, rather than at the definition of each type. -## Auto-discover Natvis XML files - -We may want to auto-discover Natvis files by searching specific directories -for `.natvis` files. For example, developers create a file with the -`.natvis` file extension, and place it within the `dbgvis/natvis` subdirectory -of their crate. The `dbgvis` directory is reserved for debugger visualizations, -and the `natvis` subdirectory is reserved for Natvis visualizations. (The -name `dbgvis` was chosen to avoid conflicts with `Debug` directories created -by build systems or IDEs; often, `.gitignore` files ignore `Debug` directories.) - -Cargo automatically scans for `dbgvis/natvis/*.natvis` files. This behavior -can be overridden by specifying manifest keys. - # References * Natvis From ac3e06406de84b237197e3c53afac62daeaffe44 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Wed, 26 Jan 2022 09:24:12 -0800 Subject: [PATCH 13/23] Update RFC with a bit more detail in the Reference-level explanation. --- text/0000-natvis.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 20e4d58d7d3..9d3cafbad92 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -341,11 +341,15 @@ file can be serialized and stored in the crate metadata. The `CrateRoot` would contain the field, `natvis_files: Lazy<[NatvisFile]>`. Another change that would need to be made here is to add a new field to the -`CrateInfo` type, `pub natvis_files: Option>`. This will -allow the `MsvcLinker` type to query the list of Natvis files that exist within -the crate dependency graph and add the `/NATVIS` linker argument for each +`CrateInfo` type, `pub natvis_files: FxHashMap>`. +This will allow the `MsvcLinker` type to query the list of Natvis files that exist +within the crate dependency graph and add the `/NATVIS` linker argument for each `.natvis` file. This will also be the stage at which the Natvis files for each -crate are copied to the target directory which they will be linked from. +crate are copied to the target directory which they will be linked from. For example, +in a debug build, the `.natvis` files will be copied to `target/debug/deps/natvis`. +Each `.natvis` file that is copied over to the new directory will have a new name to +ensure it is unique across `.natvis` files for all crates. The naming scheme will be +`---.natvis`. # Drawbacks [drawbacks]: #drawbacks From 051e319ce24b8b7c2c78bd141452cf7441166739 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Mon, 31 Jan 2022 21:51:24 -0800 Subject: [PATCH 14/23] Respond to PR comments. Fill out the alternatives section in greater detail and explain the need for embedding the contents of Natvis files in the metadata for a crate. --- text/0000-natvis.md | 170 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 134 insertions(+), 36 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 9d3cafbad92..452f3a86c3d 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -192,15 +192,21 @@ limited support for generic types. Rust developers can add one or more `.natvis` files to their crate. The `Cargo.toml` file specifies the path of these Natvis files via a new -manifest key. Cargo would then pass the set of `.natvis` files that were -specified in `Cargo.toml` and pass it to the rustc invocation for the crate -via a new `-C` flag and enabling unstable options in Rust via the -`-Z unstable-options`. See the below example for how rustc would embed these -Natvis files in the debuginfo for a binary as well as new compiler options -to be added to rustc. +manifest key or by automatically finding `.natvis` files in a predetermined +directory, `dbgvis/natvis/` and its subdirectories. Cargo would then take the +set of `.natvis` files collected and pass it to the invocation of rustc +for the crate. This would be done via a new `-C` flag and will also enable +unstable options in Rust via the existing `-Z unstable-options` flag. Rustc +would be responsible for either passing encoding the contents of the +`.natvis` files in the crate metadata if the target is an `rlib` or passing +the `/NATVIS` flag to MSVC linker to embed the Natvis visualizations into the PDB. + +See the below example for how rustc would embed these Natvis files in the +debuginfo for a binary as well as new compiler options to be added to rustc. To provide Natvis files, developers create a file with the `.natvis` file -extension and specify this Natvis file in the `Cargo.toml` under a new section +extension and either place it in the predetermined directory for Cargo to +find it automatically or specify it in the `Cargo.toml` under a new section that will be added as part of this RFC. As an example, consider a crate `foo` with this directory structure: @@ -289,16 +295,14 @@ The MSVC linker supports embedding debugger visualizations defined in a Natvis f Since the `MSVC` linker is the only one that supports embedding Natvis files into a PDB, this feature would be specific to the `MSVC` toolchain only. -Cargo would add a new feature to auto-discover Natvis files by searching a specific -directory, `dbgvis/natvis/` for `.natvis` files. For example, developers create a -file with the `.natvis` file extension, and place it within the `dbgvis/natvis/` -subdirectory of their crate. The `dbgvis` directory is reserved for debugger -visualizations, and the `natvis` subdirectory is reserved for Natvis visualizations. -(The name `dbgvis` was chosen to avoid conflicts with `Debug` directories created -by build systems or IDEs; often, `.gitignore` files ignore `Debug` directories.) - -Cargo automatically scans for `dbgvis/natvis/*.natvis` files. This behavior -can be overridden by specifying a new manifest key. +Cargo would add a new feature to auto-discover Natvis files by searching a +specific directory, `dbgvis/natvis/`, including subdirectories for `.natvis` +files. For example, developers create a file with the `.natvis` file extension, +and place it anywhere within the `dbgvis/natvis/` subdirectory of their crate. +The `dbgvis` directory is reserved for debugger visualizations, and the `natvis` +subdirectory is reserved for Natvis visualizations. (The name `dbgvis` was chosen +to avoid conflicts with `Debug` directories created by build systems or IDEs; +often, `.gitignore` files ignore `Debug` directories.) Cargo would also add new syntax in `Cargo.toml` for specifying the list of Natvis files to be added to the crate directly. The new manifest key, `natvis` would be @@ -331,25 +335,50 @@ This would generate a call to rustc similar to the following, `rustc -C natvis=path/to/file/a.natvis -C natvis=path/to/file/b.natvis -Z unstable-options` -`.natvis` files that contain spaces or commas within the path will be quoted to ensure -the path remains valid when passing command line options to rustc. +`.natvis` files that contain spaces or commas within the path will be +quoted to ensure the path remains valid when passing command line options +to rustc. The `CrateRoot` type would also need to be updated to account for `.natvis` -files for crates within the dependency graph. To reflect this a new type -`pub struct NatvisFile` will be created to ensure the contents of a `.natvis` -file can be serialized and stored in the crate metadata. The `CrateRoot` would -contain the field, `natvis_files: Lazy<[NatvisFile]>`. +files for crates within the dependency graph. The `CrateRoot` would contain +the field, `natvis_files: Lazy<[NatvisFile]>`. The new type `pub struct NatvisFile` +will be created to ensure the contents of a `.natvis` file can be serialized +and stored in the crate metadata. + +There are a couple of reasons why the contents of a `.natvis` file passed into +rustc will be serialized and encoded in the crate metadata. + +First, Cargo is not the only build system used with Rust. There are others +such as Bazel and Meson that support directly driving Rust. That might be +a minor issue to the wider community but for the people that are working +on those systems it is beneficial to pass this information through crate +metadata. That way, the information enters the dependency graph only at +the leaf nodes, and the code building the dependency graph doesn't need to +know how or why it flows through the dependency graph. + +Secondly, there's also been interest within the community of supporting +binary crate packages. That is, compiling crates to rlibs, and then passing +around rlibs directly and not rebuilding the entire library. Having to +ensure that `.natvis` files are always passed along with rlibs as well +could become very difficult especially when other debugger visualizations +also become supported such as GDB's debugger scripts and WinDbg's JavaScript +debugger scripts. Packaging these sorts of things in the `rmeta` for an `rlib` +is simple, reliable and seems like the "right" thing to do here. Another change that would need to be made here is to add a new field to the -`CrateInfo` type, `pub natvis_files: FxHashMap>`. -This will allow the `MsvcLinker` type to query the list of Natvis files that exist -within the crate dependency graph and add the `/NATVIS` linker argument for each -`.natvis` file. This will also be the stage at which the Natvis files for each -crate are copied to the target directory which they will be linked from. For example, -in a debug build, the `.natvis` files will be copied to `target/debug/deps/natvis`. -Each `.natvis` file that is copied over to the new directory will have a new name to -ensure it is unique across `.natvis` files for all crates. The naming scheme will be -`---.natvis`. +`CrateInfo` type, `pub natvis_files: FxHashSet`. This will +allow the `MsvcLinker` type to query the list of Natvis files that exist +within the crate dependency graph. This will be the stage at which the +`.natvis` files that were previously encoded and stored in the `CrateMetadata` +for a given crate dependency, will be decoded and have the contents written +to a new file in the `target` directory. The path of this new file will be +what is passed to the `/NATVIS` linker flag. For example, in a debug build, +the contents of the `.natvis` files that were encoded in the crate metadata +will be written to new files in the directory `target/debug/deps/natvis`. +Each `.natvis` file that is written will have a new name to ensure it is +unique across `.natvis` files for all crates. The naming scheme will be +`-.natvis`. The `` value will be the hash of the +contents of the Natvis file. # Drawbacks [drawbacks]: #drawbacks @@ -364,6 +393,8 @@ is not possible and so a manual definition would be required to have a debugger # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives +## Rationale + This design provides a simple mechanism for cargo to collect the list of `.natvis` files specified for a given crate and embed them in the resulting pdb. It does not need any manual intervention by a Rust developer who is @@ -376,10 +407,77 @@ developers which may lead to a period of learning the syntax. Since this feature would be optional, a consumer of a crate that has natvis definitions for types would not need to go through this learning curve. -Not doing this would keep the existing debugging experience for external Rust crates. -Most Rust types, outside of the standard library, do not have any debugger views -defined for them by default which makes them difficult to interpret when viewed -under a debugger. +## Alternatives + +### Alternative 1: existing -C link-arg flag + +Supporting this option would mean that changes to rustc are not necessary. +The changes would be limited to Cargo, which would be responsible for collecting +the set of Natvis files and passing `-Clink-arg=/NATVIS:{file-path}` for each +Natvis file. + +The drawbacks for this option is that it will only collect Natvis files for the +top-most manifest. This will not walk the dependency graph and find all relevant +Natvis files so this will only work for targets that produce a `DLL` or `EXE` and +not an `.rlib`. + +### Alternative 2: custom build script to set /NATVIS linker flag + +Supporting this option would mean that changes to cargo and rustc are not necessary. +Each individual crate would be able to create a custom build script that would set +the rustc `link-arg` flag `cargo:rustc-link-arg=/NATVIS:{file-path}` for each Natvis +file. + +The drawbacks for this option is that it would force all Rust developers to manually +create a build script and ensure it is kept up-to-date whenever the set of Natvis files +are updated. This also would set the linker argument regardless of the linker being used. +This could lead to potential build errors if the linker being used returns an error for +an invalid argument. + +### Alternative 3: inline Natvis XML fragments via attributes only + +Supporting this option would mean that changes to cargo are not necessary. +This option could be implemented via an attribute and/or proc-macro which +would live outside of the compiler and could be ingested in via an outside crate. +Rustc would need some changes in order to collect all of the attribute usage from the +source code and create temporary files that could be passed to the MSVC linker via +the `/NATVIS` linker arg. For crate dependencies, the Natvis fragments can be combined +and embedded in the crate metadata so the Natvis can still be embedded in the final +PDB generated. + +The drawbacks for this option is that it would add a lot of bloat to the Rust source +code directly if only the attribute syntax was supported. For types with many fields +or types that need extensive amounts of Natvis to appropriately visualize them in a +meaninngful way, this could distract from the contents of the code. Without being able +to pull some of the more intricate Natvis descriptions into a separate standalone +`.natvis` file, there may become an issue with the visibility of the source code. +Also, if/when other debugger visualization formats are supported, it could become +very obscure to read the source with large amounts of visualization scripts from +multiple schemas all being directly embedded in source code. + +### Alternative 4: miri executes the MIR of a Debug impl within a debugger + +Supporting this option would mean that changes to cargo and rustc are not necessary. +This would have the added benefit of taking full advantage of existing implementations +of the `Debug` trait. Many Rust developers already implement the `Debug` trait which is +used to format how types should be viewed, this would only ease the debugging quality of +Rust when viewed under any debugger. This option also has the added benefit of not +requiring any changes to a crate from a Rust developer by leveraging existing `Debug` impls. + +The drawbacks for this option is that this has not been fully investigated to +determine its viability. This could be a great potential feature to ease +debugging Rust but without concrete data to push this towards a potential RFC, +I would assume supporting debugging in the systems that are already heavily used +by the Rust community to be a higher priority. If/when this option becomes a bit +more viable, there would be nothing stopping it from becoming a true feature. + +## Impact + +By not implementing the feature described by this RFC, the debugging quality of Rust, +especially on Windows, will be continue to be a difficult experience. The only +visualizations that exist today are for parts of the standard library. External crates +being consumed will not have debugging visualizations available and would make it +difficult to understand what is being debugged. # Prior art [prior-art]: #prior-art From 9ec6cf5c3701ffcc037ca6b2fa534bf1daf99111 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Mon, 7 Feb 2022 16:44:25 -0800 Subject: [PATCH 15/23] Update alternatives section. --- text/0000-natvis.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/text/0000-natvis.md b/text/0000-natvis.md index 452f3a86c3d..a4638f3e61c 100644 --- a/text/0000-natvis.md +++ b/text/0000-natvis.md @@ -430,9 +430,12 @@ file. The drawbacks for this option is that it would force all Rust developers to manually create a build script and ensure it is kept up-to-date whenever the set of Natvis files -are updated. This also would set the linker argument regardless of the linker being used. -This could lead to potential build errors if the linker being used returns an error for -an invalid argument. +are updated. This option would also have the same drawback as above, using a build +script would be able to set the linker argument for adding Natvis but only for the top +level crate. Any dependencies or transitive dependencies would not be able to set that +linker argument in order to embed Natvis into the generated PDB. Also, for crates that +generate an `rlib`, this would also run into an issue since a PDB isn't generated for +an `rlib`. ### Alternative 3: inline Natvis XML fragments via attributes only From 6534e12e88ca1982928f765ade0298c06ee53387 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Mon, 28 Feb 2022 12:43:19 -0800 Subject: [PATCH 16/23] Refactor RFC after latest RFC comments. Rename RFC. --- ...-natvis.md => 0000-debugger-visualizer.md} | 270 ++++++++++-------- 1 file changed, 150 insertions(+), 120 deletions(-) rename text/{0000-natvis.md => 0000-debugger-visualizer.md} (70%) diff --git a/text/0000-natvis.md b/text/0000-debugger-visualizer.md similarity index 70% rename from text/0000-natvis.md rename to text/0000-debugger-visualizer.md index a4638f3e61c..7d199065fe1 100644 --- a/text/0000-natvis.md +++ b/text/0000-debugger-visualizer.md @@ -1,4 +1,4 @@ -- Feature Name: `natvis` +- Feature Name: `debugger-visualizer` - Start Date: 2021-11-01 - RFC PR: [rust-lang/rfcs#3191](https://github.com/rust-lang/rfcs/pull/3191) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) @@ -7,7 +7,7 @@ [summary]: #summary This RFC aims to improve the debugging experience for Rust developers, by -enabling Rust developers to package debugger visualization scripts with their +enabling Rust developers to package debugger visualizer scripts with their crates. # Motivation @@ -24,9 +24,6 @@ views". Debugger views are merely a convenience for some types, such as `Vec`, but are essential for types such as `HashMap`, where non-trivial logic is needed in order to correctly display the contents of a type. -Since Natvis is supported via WinDbg and the VS Debugger, this support is -specific to Windows and the MSVC toolchain. - For example, given the following instance of `HashMap`: ```rust fn main() { @@ -68,14 +65,15 @@ With Natvis applied, WinDbg results in the following: ``` Currently, Rust provides visualizations for a handful of types defined in its -standard library via `.natvis` files or python scripts. However, this support -is inflexible; updating it requires modifying the Rust toolchain itself, -and either using a local build of the toolchain or waiting for a new upstream -build of the toolchain. It is not feasible for developers of ordinary crates -to update the Rust toolchain, solely to add visualizations for their crates. +standard library via `.natvis` files or pretty printers via python scripts. +However, this support is inflexible; updating it requires modifying the Rust +toolchain itself, and either using a local build of the toolchain or waiting +for a new upstream build of the toolchain. It is not feasible for developers of +ordinary crates to update the Rust toolchain, solely to add visualizations for +their crates. The expected outcome of this RFC is to design a way for developers to seamlessly -integrate Natvis debugger visualizations with their crates. This would mean: +integrate debugger visualizations with their crates. This would mean: * Any developer can add debugger visualizations to their crate. * If a Rust developer uses a crate that has debugger visualizations in it, @@ -90,7 +88,7 @@ integrate Natvis debugger visualizations with their crates. This would mean: # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -This RFC explores making Natvis debugger visualizations extensible, in Rust. +This RFC explores making debugger visualizations extensible in Rust via Natvis and/or pretty printers. The scenario that we want to enable is: * Alice publishes a crate, say, `cool_stuff`. Alice wrote debugger @@ -107,6 +105,8 @@ The scenario that we want to enable is: visualizations worked or that Alice had written any debugger visualizations. From Bob's point of view, debugging `CoolType` "just worked". +The same should be applied to pretty printers defined and viewed under LLDB and GDB. + ## An example: The `regex` crate To make this less hypothetical, let's consider an important community crate, @@ -172,10 +172,17 @@ What we _want_ is something like this: > 3: "// some developer comment" ``` -This RFC will describe how to support adding Natvis visualizations which is supported by: +This RFC will describe how to support adding Natvis as well as GDB's pretty printers. + +Natvis is supported by: * The Windows Debugger (WinDbg) -* Visual Studio Debugger. +* Visual Studio Debugger + +Pretty printers are supported by: + +* GDB +* LLDB It should be easy for Rust developers to add debugger visualizations to their crates. @@ -190,24 +197,17 @@ The Natvis files provide patterns, which match type names, and for matching types, a description of how to display those types. This allows for some limited support for generic types. -Rust developers can add one or more `.natvis` files to their crate. The -`Cargo.toml` file specifies the path of these Natvis files via a new -manifest key or by automatically finding `.natvis` files in a predetermined -directory, `dbgvis/natvis/` and its subdirectories. Cargo would then take the -set of `.natvis` files collected and pass it to the invocation of rustc -for the crate. This would be done via a new `-C` flag and will also enable -unstable options in Rust via the existing `-Z unstable-options` flag. Rustc -would be responsible for either passing encoding the contents of the -`.natvis` files in the crate metadata if the target is an `rlib` or passing -the `/NATVIS` flag to MSVC linker to embed the Natvis visualizations into the PDB. - -See the below example for how rustc would embed these Natvis files in the -debuginfo for a binary as well as new compiler options to be added to rustc. +Rust developers can add one or more `.natvis` files to their crate. Through +the use of a new Rust attribute, `#[debugger-visualizer]`, the compiler will +encode the contents of the `.natvis` file in the crate metadata if the target +is an `rlib`. If the target is a `dll` or `exe`, the `/NATVIS` MSVC linker flag is +set for each `.natvis` file which will embed the Natvis visualizations into the PDB. To provide Natvis files, developers create a file with the `.natvis` file -extension and either place it in the predetermined directory for Cargo to -find it automatically or specify it in the `Cargo.toml` under a new section -that will be added as part of this RFC. +extension. + +See the below example for how rustc would embed these Natvis files in the +debuginfo for a binary. As an example, consider a crate `foo` with this directory structure: @@ -271,81 +271,86 @@ When viewed under WinDbg, the `fancy_rect` variable would be shown as follows: > LowerRight: (15, 10) ``` +## Supporting Pretty Printers + +This section describes how GDB's pretty printers are supported in Rust. + +To use a pretty printer, developers write python scripts that describe how a type +should be displayed when loaded up in GDB/LLDB. (See: https://sourceware.org/gdb/onlinedocs/gdb/Pretty-Printing.html#Pretty-Printing) +The pretty printers provide patterns, which match type names, and for matching +types, descibe how to display those types. (For writing a pretty printer, see: https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Pretty_002dPrinter.html#Writing-a-Pretty_002dPrinter). + +Rust developers can add one or more pretty printers to their crate. This is done +in the Rust compiler via `.py` python scripts. Through the use of a new Rust attribute, +`#[debugger-visualizer]`, the compiler will encode the contents of the `.py` file in +the crate metadata if the target is an `rlib`. If the target is an executable, the +`.debug_gdb_scripts` section will include a reference to the pretty printer specified. + +To provide pretty printers, developers create a file with the `.py` file +extension. + # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -In rustc, a new `-C natvis={path to single .natvis file}` flag will be added -which instructs the compiler to take a `.natvis` file for a given crate, serialize -the contents of this file and store it in the crate metadata. Rustc will not -have any enforcement that the files specified via the new flag have the `.natvis` -extension, but this is required in order for the Natvis to be loaded by both -the VS debugger and WinDbg. This flag can be used multiple times to allow for -multiple `.natvis` files to be embedded for each crate. This new flag will be gated -by the existing `-Z unstable-options` flag to ensure it is only used in unstable -Rust. When running the linker, using the `MSVC` toolchain, the contents of the -`.natvis` file will be extracted from the crate metadata and store them in the target -directory under a new `natvis` directory. A `/NATVIS` linker option would be set -for each `.natvis` file which includes `.natvis` files from all crate dependencies, -if any exist, as well as the current crate and embed them into the PDB. Any crate type -that would generate a PDB would have any applicable `.natvis` files embedded in the -generated PDB. - -The MSVC linker supports embedding debugger visualizations defined in a Natvis file -(`.natvis`) into a PDB generated by LINK through the use of the `/NATVIS` linker flag. -Since the `MSVC` linker is the only one that supports embedding Natvis files -into a PDB, this feature would be specific to the `MSVC` toolchain only. - -Cargo would add a new feature to auto-discover Natvis files by searching a -specific directory, `dbgvis/natvis/`, including subdirectories for `.natvis` -files. For example, developers create a file with the `.natvis` file extension, -and place it anywhere within the `dbgvis/natvis/` subdirectory of their crate. -The `dbgvis` directory is reserved for debugger visualizations, and the `natvis` -subdirectory is reserved for Natvis visualizations. (The name `dbgvis` was chosen -to avoid conflicts with `Debug` directories created by build systems or IDEs; -often, `.gitignore` files ignore `Debug` directories.) - -Cargo would also add new syntax in `Cargo.toml` for specifying the list of Natvis -files to be added to the crate directly. The new manifest key, `natvis` would be -added to a new `[debugger-visualizations]` section. Cargo would emit a warning if -any of the specified Natvis files do not have the `.natvis` extension. - -As with the auto-discover feature, this would be in control of setting the -`-C natvis` flag in rustc as well as `-Z unstable-options` to enable it. Using -the new toml syntax to specify `.natvis` files would override Cargo from -auto-discovering Natvis files in the `dbgvis/natvis/` directory. - -For example: - -`Cargo.toml`: - -```toml -cargo-features = ["natvis"] - -[package] -name = "natvis" -version = "0.1.0" -edition = "2018" - -[debugger-visualizations] -natvis = ["a.natvis", "b.natvis"] -``` +In rustc, a new built-in attribute `#[debugger-visualizer]` will be added which +instructs the compiler to take the specified file path for a debugger visualizer +and add it to the current binary being built. The file path specified must be +relative to the location of the attribute. -This would generate a call to rustc similar to the following, -(for simplicity purposes, most of the rustc command line has been removed): +The `#[debugger-visualizer]` attribute will reserve multiple keys to be able to +specify which type of visualizer is being applied. The following keys will be +reserved as part of this RFC: -`rustc -C natvis=path/to/file/a.natvis -C natvis=path/to/file/b.natvis -Z unstable-options` +* `natvis_file` +* `gdb_script_file` -`.natvis` files that contain spaces or commas within the path will be -quoted to ensure the path remains valid when passing command line options -to rustc. +As more visualizer schemes arise, more keys may be added in the future to ensure +a great debugging experience for any debugger that the Rust community sees fit. -The `CrateRoot` type would also need to be updated to account for `.natvis` +For example, to specify that a `.natvis` file should be included in the binary +being built, the following attribute should be added to the Rust source: + +```rust +#![debugger-visualizer(natvis_file = "../foo.natvis")] +``` + +The same can be done to specify a GDB python debugger script: + +```rust +#![debugger-visualizer(gdb_script_file = "../foo.py")] +``` + +Depending on the Rust target, the correct debugger visualizer will be selected and embedded +in the output. + +The Rust compiler will serialize the contents of the file specified via the +`#[debugger-visualizer]` attribute and store it in the crate metadata. This attribute +can be used multiple times to allow for multiple debugger visualizer files to be +embedded for each crate. When generating the final binary, the contents of the +visualizer file will be extracted from the crate metadata and written to a new file +in the target directory under a new `visualizer` directory. + +In the case of a Natvis file, `#![debugger-visualizer(natvis_file = "../foo.natvis")]` +the compiler will set the `/NATVIS:{.natvis file}` MSVC linker flag for each of the +Natvis files specified for the current crate as well as transitive dependencies if +using the MSVC toolchain. This linker flag ensures that the specified Natvis files +be embedded in the PDB generated for the binary being built. Any crate type that +would generate a PDB would have all applicable `.natvis` files embedded. + +In the case of GDB pretty printer, `#![debugger-visualizer(gdb_script_file = "../foo.py")]` +the compiler will ensure that the set of pretty printers specified will be added to the +`.debug_gdb_scripts` section of the `ELF` generated. The `.debug_gdb_scripts` section +takes a list of null-terminated entries which specify scripts to load within GDB. The +Rust compiler currently embeds a visualizer for some types in the standard library via +the `.debug_gdb_scripts` section. + +The `CrateRoot` type would need to be updated to account for debugger visualizer files for crates within the dependency graph. The `CrateRoot` would contain -the field, `natvis_files: Lazy<[NatvisFile]>`. The new type `pub struct NatvisFile` -will be created to ensure the contents of a `.natvis` file can be serialized -and stored in the crate metadata. +the field, `debugger_visualizers: Lazy<[DebuggerVisualizerFile]>`. The new type +`pub struct DebuggerVisualizerFile` will be created to ensure the contents of a +debugger visualizer file can be serialized and stored in the crate metadata. -There are a couple of reasons why the contents of a `.natvis` file passed into +There are a couple of reasons why the contents of a visualizer file passed into rustc will be serialized and encoded in the crate metadata. First, Cargo is not the only build system used with Rust. There are others @@ -366,19 +371,20 @@ debugger scripts. Packaging these sorts of things in the `rmeta` for an `rlib` is simple, reliable and seems like the "right" thing to do here. Another change that would need to be made here is to add a new field to the -`CrateInfo` type, `pub natvis_files: FxHashSet`. This will -allow the `MsvcLinker` type to query the list of Natvis files that exist -within the crate dependency graph. This will be the stage at which the -`.natvis` files that were previously encoded and stored in the `CrateMetadata` -for a given crate dependency, will be decoded and have the contents written -to a new file in the `target` directory. The path of this new file will be -what is passed to the `/NATVIS` linker flag. For example, in a debug build, -the contents of the `.natvis` files that were encoded in the crate metadata -will be written to new files in the directory `target/debug/deps/natvis`. -Each `.natvis` file that is written will have a new name to ensure it is -unique across `.natvis` files for all crates. The naming scheme will be -`-.natvis`. The `` value will be the hash of the -contents of the Natvis file. +`CrateInfo` type, `pub debugger_visualizers: FxHashSet`. +This will allow the Rust compiler to query the list of visualizer files being +added for a specific crate. The compiler will then be responsible for collecting +the set of visualizer files that were previously encoded and stored in the +`CrateMetadata` for a given crate dependency, will be decoded and have the +contents written to a new file in the `target` directory. In the case of Natvis, +the path of this new file will be what is passed to the `/NATVIS` linker flag. +For example, in a debug build, the contents of the `.natvis` files that were encoded +in the crate metadata will be written to new files in the directory `target/debug/deps/visualizers`. +Each visualizer file that is written will have a new name to ensure it is unique +across visualizer files for all crates with a naming scheme of `-.`. +The `` value will be `.natvis` in the case of a Natvis file +and `.py` in the case of a pretty printer. The `` value will be the hash of the +contents of the visualizer file. # Drawbacks [drawbacks]: #drawbacks @@ -395,17 +401,17 @@ is not possible and so a manual definition would be required to have a debugger ## Rationale -This design provides a simple mechanism for cargo to collect the list of -`.natvis` files specified for a given crate and embed them in the resulting -pdb. It does not need any manual intervention by a Rust developer who is +This design provides a simple mechanism to specify a debugger visualizer file +for a given crate and embed them in the resulting PDB or ELF depending on the +target. It does not need any manual intervention by a Rust developer who is consuming such a crate to get the debugging experience to work when it is -viewed under a debugger that supports the Natvis Framework. +viewed under a debugger that supports the visualizer specified. -This design does not break any existing usage of cargo or rustc. This new feature -would be strictly opt-in. The Natvis syntax may not be familiar to many Rust -developers which may lead to a period of learning the syntax. Since this feature -would be optional, a consumer of a crate that has natvis definitions for types -would not need to go through this learning curve. +This design does not break any existing usage of rustc. This new feature would +be strictly opt-in. The Natvis or GDB pretty printer syntax may not be familiar +to many Rust developers which may lead to a period of learning the syntax. Since +this feature would be optional, a consumer of a crate that has debugger visualizer +for types would not need to go through this learning curve. ## Alternatives @@ -474,6 +480,25 @@ I would assume supporting debugging in the systems that are already heavily used by the Rust community to be a higher priority. If/when this option becomes a bit more viable, there would be nothing stopping it from becoming a true feature. +### Alternative 5: #[link] attribute to implement this feature + +```rust +#[cfg_attr(target_platform="msvc",link(file="foo.natvis", arg="/NATVIS"))] +struct Foo; +``` + +Supporting this option would mean that no new attributes would be needed for rustc. +This attribute currently exists today and implementing this feature on top of this +attribute would create an easy way to drop support for this feature in the future if +need be. + +The drawbacks for this option is that it seems a sub-optimal in terms of user +experience. It requires the author to operate at a lower level of abstraction by +having to use a more general attribute and annotating it to tackle a specific use +case. Having a more targeted attribute, i.e. `#[debugger-visualizer]` allows for the +author to simply specify which debugger visualizer file should be included and allow +the compiler to select the right one under the covers. + ## Impact By not implementing the feature described by this RFC, the debugging quality of Rust, @@ -531,7 +556,8 @@ printers. Pretty printers work in the similar manner as Natvis in which they tel the debugger to serve up a specific visualization when viewing a type in the debugger. Pretty printers are written as python scripts and then have to be imported in to the debugger. When a type is viewed under the debugger that has a pretty printer, that view -is automatically shown. +is automatically shown. The Rust compiler currently defines a pretty printer for a +limited set of types from within the standard library. # Unresolved questions [unresolved-questions]: #unresolved-questions @@ -604,4 +630,8 @@ of each type. * Natvis + [Create custom views of C++ objects in the debugger using the Natvis framework](https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects) + [Visual Studio native debug visualization (natvis) for C++/WinRT](https://docs.microsoft.com/en-us/windows/uwp/cpp-and-winrt-apis/natvis) - + https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/native-debugger-objects-in-natvis \ No newline at end of file + + https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/native-debugger-objects-in-natvis + +* Pretty Printers + + [Writing a Pretty Printer](https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Pretty_002dPrinter.html#Writing-a-Pretty_002dPrinter) + + [The .debug_gdb_scripts section](https://sourceware.org/gdb/onlinedocs/gdb/dotdebug_005fgdb_005fscripts-section.html) \ No newline at end of file From 275a333d2beb0ecce05fafbbdf2dde910b6aa80a Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Mon, 28 Feb 2022 13:13:13 -0800 Subject: [PATCH 17/23] Minor updates. --- text/0000-debugger-visualizer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-debugger-visualizer.md b/text/0000-debugger-visualizer.md index 7d199065fe1..6454ec8a46e 100644 --- a/text/0000-debugger-visualizer.md +++ b/text/0000-debugger-visualizer.md @@ -569,11 +569,11 @@ None. ## Inline Natvis XML fragments via an attribute -Natvis support for Rust could be improved upon by adding support for natvis in source via an attribute. Example: +Debugger visualizer support for Rust could be improved upon by adding support for in-source visualizer definitions via an attribute. Example: ```rust /// A rectangle in first quadrant -#[dbgvis( +#[debugger-visualizer( natvis(r#" ({x},{y}) + ({dx}, {dy}) ({x}, {y}) From 0a67d40bf0a6312d89f4ff88347e7f4b2359fd58 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Mon, 28 Feb 2022 15:35:32 -0800 Subject: [PATCH 18/23] Update attribute name. Remove extra whitespace. --- text/0000-debugger-visualizer.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/text/0000-debugger-visualizer.md b/text/0000-debugger-visualizer.md index 6454ec8a46e..3d7cdbf1d29 100644 --- a/text/0000-debugger-visualizer.md +++ b/text/0000-debugger-visualizer.md @@ -1,4 +1,4 @@ -- Feature Name: `debugger-visualizer` +- Feature Name: `debugger_visualizer` - Start Date: 2021-11-01 - RFC PR: [rust-lang/rfcs#3191](https://github.com/rust-lang/rfcs/pull/3191) - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) @@ -16,7 +16,7 @@ crates. Most, if not all, Rust developers will at some point have to debug an issue in their crate. Trying to view types as they are laid out in memory is not always the most telling. Furthermore when viewing types from external crates, -the information is even harder to interpret. +the information is even harder to interpret. Many languages and debuggers enable developers to control how a type is displayed in a debugger. These are called "debugger visualizations" or "debugger @@ -198,7 +198,7 @@ types, a description of how to display those types. This allows for some limited support for generic types. Rust developers can add one or more `.natvis` files to their crate. Through -the use of a new Rust attribute, `#[debugger-visualizer]`, the compiler will +the use of a new Rust attribute, `#[debugger_visualizer]`, the compiler will encode the contents of the `.natvis` file in the crate metadata if the target is an `rlib`. If the target is a `dll` or `exe`, the `/NATVIS` MSVC linker flag is set for each `.natvis` file which will embed the Natvis visualizations into the PDB. @@ -282,7 +282,7 @@ types, descibe how to display those types. (For writing a pretty printer, see: h Rust developers can add one or more pretty printers to their crate. This is done in the Rust compiler via `.py` python scripts. Through the use of a new Rust attribute, -`#[debugger-visualizer]`, the compiler will encode the contents of the `.py` file in +`#[debugger_visualizer]`, the compiler will encode the contents of the `.py` file in the crate metadata if the target is an `rlib`. If the target is an executable, the `.debug_gdb_scripts` section will include a reference to the pretty printer specified. @@ -292,12 +292,12 @@ extension. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -In rustc, a new built-in attribute `#[debugger-visualizer]` will be added which +In rustc, a new built-in attribute `#[debugger_visualizer]` will be added which instructs the compiler to take the specified file path for a debugger visualizer and add it to the current binary being built. The file path specified must be relative to the location of the attribute. -The `#[debugger-visualizer]` attribute will reserve multiple keys to be able to +The `#[debugger_visualizer]` attribute will reserve multiple keys to be able to specify which type of visualizer is being applied. The following keys will be reserved as part of this RFC: @@ -311,33 +311,33 @@ For example, to specify that a `.natvis` file should be included in the binary being built, the following attribute should be added to the Rust source: ```rust -#![debugger-visualizer(natvis_file = "../foo.natvis")] +#![debugger_visualizer(natvis_file = "../foo.natvis")] ``` The same can be done to specify a GDB python debugger script: ```rust -#![debugger-visualizer(gdb_script_file = "../foo.py")] +#![debugger_visualizer(gdb_script_file = "../foo.py")] ``` Depending on the Rust target, the correct debugger visualizer will be selected and embedded in the output. The Rust compiler will serialize the contents of the file specified via the -`#[debugger-visualizer]` attribute and store it in the crate metadata. This attribute +`#[debugger_visualizer]` attribute and store it in the crate metadata. This attribute can be used multiple times to allow for multiple debugger visualizer files to be embedded for each crate. When generating the final binary, the contents of the visualizer file will be extracted from the crate metadata and written to a new file in the target directory under a new `visualizer` directory. -In the case of a Natvis file, `#![debugger-visualizer(natvis_file = "../foo.natvis")]` +In the case of a Natvis file, `#![debugger_visualizer(natvis_file = "../foo.natvis")]` the compiler will set the `/NATVIS:{.natvis file}` MSVC linker flag for each of the Natvis files specified for the current crate as well as transitive dependencies if using the MSVC toolchain. This linker flag ensures that the specified Natvis files be embedded in the PDB generated for the binary being built. Any crate type that would generate a PDB would have all applicable `.natvis` files embedded. -In the case of GDB pretty printer, `#![debugger-visualizer(gdb_script_file = "../foo.py")]` +In the case of GDB pretty printer, `#![debugger_visualizer(gdb_script_file = "../foo.py")]` the compiler will ensure that the set of pretty printers specified will be added to the `.debug_gdb_scripts` section of the `ELF` generated. The `.debug_gdb_scripts` section takes a list of null-terminated entries which specify scripts to load within GDB. The @@ -495,7 +495,7 @@ need be. The drawbacks for this option is that it seems a sub-optimal in terms of user experience. It requires the author to operate at a lower level of abstraction by having to use a more general attribute and annotating it to tackle a specific use -case. Having a more targeted attribute, i.e. `#[debugger-visualizer]` allows for the +case. Having a more targeted attribute, i.e. `#[debugger_visualizer]` allows for the author to simply specify which debugger visualizer file should be included and allow the compiler to select the right one under the covers. @@ -573,7 +573,7 @@ Debugger visualizer support for Rust could be improved upon by adding support fo ```rust /// A rectangle in first quadrant -#[debugger-visualizer( +#[debugger_visualizer( natvis(r#" ({x},{y}) + ({dx}, {dy}) ({x}, {y}) From 27c1664e85b1304cc3dc89448b73574b55035adb Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Thu, 3 Mar 2022 10:00:42 -0800 Subject: [PATCH 19/23] Respond to PR feedback. --- text/0000-debugger-visualizer.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/text/0000-debugger-visualizer.md b/text/0000-debugger-visualizer.md index 3d7cdbf1d29..e955dcd062b 100644 --- a/text/0000-debugger-visualizer.md +++ b/text/0000-debugger-visualizer.md @@ -221,6 +221,8 @@ As an example, consider a crate `foo` with this directory structure: Where `main.rs` contains: ```rust +#[debugger_visualizer(natvis_file = "../Foo.natvis")] + /// A rectangle in first quadrant struct FancyRect { pub x: f32, @@ -287,7 +289,11 @@ the crate metadata if the target is an `rlib`. If the target is an executable, t `.debug_gdb_scripts` section will include a reference to the pretty printer specified. To provide pretty printers, developers create a file with the `.py` file -extension. +extension and reference it via the `#[debugger_visualizer]` attribute as follows: + +```rust +#[debugger_visualizer(gdb_script_file = "../foo.py")] +``` # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -311,13 +317,13 @@ For example, to specify that a `.natvis` file should be included in the binary being built, the following attribute should be added to the Rust source: ```rust -#![debugger_visualizer(natvis_file = "../foo.natvis")] +#[debugger_visualizer(natvis_file = "../foo.natvis")] ``` The same can be done to specify a GDB python debugger script: ```rust -#![debugger_visualizer(gdb_script_file = "../foo.py")] +#[debugger_visualizer(gdb_script_file = "../foo.py")] ``` Depending on the Rust target, the correct debugger visualizer will be selected and embedded @@ -330,14 +336,14 @@ embedded for each crate. When generating the final binary, the contents of the visualizer file will be extracted from the crate metadata and written to a new file in the target directory under a new `visualizer` directory. -In the case of a Natvis file, `#![debugger_visualizer(natvis_file = "../foo.natvis")]` +In the case of a Natvis file, `#[debugger_visualizer(natvis_file = "../foo.natvis")]` the compiler will set the `/NATVIS:{.natvis file}` MSVC linker flag for each of the Natvis files specified for the current crate as well as transitive dependencies if using the MSVC toolchain. This linker flag ensures that the specified Natvis files be embedded in the PDB generated for the binary being built. Any crate type that would generate a PDB would have all applicable `.natvis` files embedded. -In the case of GDB pretty printer, `#![debugger_visualizer(gdb_script_file = "../foo.py")]` +In the case of GDB pretty printer, `#[debugger_visualizer(gdb_script_file = "../foo.py")]` the compiler will ensure that the set of pretty printers specified will be added to the `.debug_gdb_scripts` section of the `ELF` generated. The `.debug_gdb_scripts` section takes a list of null-terminated entries which specify scripts to load within GDB. The From c9b1b841d118c26963701e7ca55aefc637d80489 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Thu, 10 Mar 2022 10:21:13 -0800 Subject: [PATCH 20/23] Respond to PR comments. Minor updates to how GDB scripts would be embedded to the `.debug_gdb_scripts` section. Updates to specify the `#[debugger_visualizer]` targets modules and can be used with or without the leading `!` for the attribute. --- text/0000-debugger-visualizer.md | 70 ++++++++++++++++++++++++-------- 1 file changed, 52 insertions(+), 18 deletions(-) diff --git a/text/0000-debugger-visualizer.md b/text/0000-debugger-visualizer.md index e955dcd062b..a23a075b990 100644 --- a/text/0000-debugger-visualizer.md +++ b/text/0000-debugger-visualizer.md @@ -221,7 +221,7 @@ As an example, consider a crate `foo` with this directory structure: Where `main.rs` contains: ```rust -#[debugger_visualizer(natvis_file = "../Foo.natvis")] +#![debugger_visualizer(natvis_file = "../Foo.natvis")] /// A rectangle in first quadrant struct FancyRect { @@ -286,13 +286,14 @@ Rust developers can add one or more pretty printers to their crate. This is done in the Rust compiler via `.py` python scripts. Through the use of a new Rust attribute, `#[debugger_visualizer]`, the compiler will encode the contents of the `.py` file in the crate metadata if the target is an `rlib`. If the target is an executable, the -`.debug_gdb_scripts` section will include a reference to the pretty printer specified. +`.debug_gdb_scripts` section will include a reference to the pretty printer specified +by embedding the contents of the pretty printer directly into this section. To provide pretty printers, developers create a file with the `.py` file extension and reference it via the `#[debugger_visualizer]` attribute as follows: ```rust -#[debugger_visualizer(gdb_script_file = "../foo.py")] +#![debugger_visualizer(gdb_script_file = "../foo.py")] ``` # Reference-level explanation @@ -301,7 +302,33 @@ extension and reference it via the `#[debugger_visualizer]` attribute as follows In rustc, a new built-in attribute `#[debugger_visualizer]` will be added which instructs the compiler to take the specified file path for a debugger visualizer and add it to the current binary being built. The file path specified must be -relative to the location of the attribute. +relative to the location of the attribute and is resolved in a manner that is +identical to how paths are resolved in the `include_str!` macro. This attribute +will directly target modules which means the syntax `#![debugger_visualizer]` is +also valid when placed at the module level. + +For example, the following uses of the attribute are valid: + +Where `main.rs` contains: + +```rust +#![debugger_visualizer(natvis_file = "../main.natvis")] + +#[debugger_visualizer(natvis_file = "../foo.natvis")] +mod foo; +``` + +and `bar.rs` contains: + +```rust +#![debugger_visualizer(natvis_file = "../bar.natvis")] +``` + +In the first case, the attribute is applied to the crate as a top-level attribute +using the inner attribute syntax and also added to the module foo using the outer +attribute syntax. In the second case, the attribute is applied to the module bar +via a top-level attribute as well which also is valid since it is still targeting +a module. The `#[debugger_visualizer]` attribute will reserve multiple keys to be able to specify which type of visualizer is being applied. The following keys will be @@ -317,17 +344,17 @@ For example, to specify that a `.natvis` file should be included in the binary being built, the following attribute should be added to the Rust source: ```rust -#[debugger_visualizer(natvis_file = "../foo.natvis")] +#![debugger_visualizer(natvis_file = "../foo.natvis")] ``` The same can be done to specify a GDB python debugger script: ```rust -#[debugger_visualizer(gdb_script_file = "../foo.py")] +#![debugger_visualizer(gdb_script_file = "../foo.py")] ``` -Depending on the Rust target, the correct debugger visualizer will be selected and embedded -in the output. +Depending on the Rust target, the correct debugger visualizer will be selected +and embedded in the output. The Rust compiler will serialize the contents of the file specified via the `#[debugger_visualizer]` attribute and store it in the crate metadata. This attribute @@ -346,9 +373,13 @@ would generate a PDB would have all applicable `.natvis` files embedded. In the case of GDB pretty printer, `#[debugger_visualizer(gdb_script_file = "../foo.py")]` the compiler will ensure that the set of pretty printers specified will be added to the `.debug_gdb_scripts` section of the `ELF` generated. The `.debug_gdb_scripts` section -takes a list of null-terminated entries which specify scripts to load within GDB. The -Rust compiler currently embeds a visualizer for some types in the standard library via -the `.debug_gdb_scripts` section. +takes a list of null-terminated entries which specify scripts to load within GDB. This +section supports listing files to load directly or embedding the contents of a script +that will be executed. The Rust compiler currently embeds a visualizer for some types +in the standard library via the `.debug_gdb_scripts` section using the former method. +This attribute will embed the contents of the debugger script so that it will not +need to reference a file in the search path. This has proven to be a more reliable +route than depending on file paths which can be unstable at times. The `CrateRoot` type would need to be updated to account for debugger visualizer files for crates within the dependency graph. The `CrateRoot` would contain @@ -385,12 +416,9 @@ the set of visualizer files that were previously encoded and stored in the contents written to a new file in the `target` directory. In the case of Natvis, the path of this new file will be what is passed to the `/NATVIS` linker flag. For example, in a debug build, the contents of the `.natvis` files that were encoded -in the crate metadata will be written to new files in the directory `target/debug/deps/visualizers`. -Each visualizer file that is written will have a new name to ensure it is unique -across visualizer files for all crates with a naming scheme of `-.`. -The `` value will be `.natvis` in the case of a Natvis file -and `.py` in the case of a pretty printer. The `` value will be the hash of the -contents of the visualizer file. +in the crate metadata will be written to new files in a temp directory where they will +be included from. Each visualizer file that is written will have a new name to +ensure it is unique across visualizer files for all crates. # Drawbacks [drawbacks]: #drawbacks @@ -575,7 +603,9 @@ None. ## Inline Natvis XML fragments via an attribute -Debugger visualizer support for Rust could be improved upon by adding support for in-source visualizer definitions via an attribute. Example: +Debugger visualizer support for Rust could be improved upon by adding support +for in-source visualizer definitions via the `#[debugger_visualizer]` attribute +or a new attribute. Example: ```rust /// A rectangle in first quadrant @@ -595,6 +625,10 @@ struct FancyRect { } ``` +Currently the `#[debugger_visualizer]` attribute is only allowed to target modules +but can be updated to allow targeting types as well if the same attribute was to be +re-used to support this. + ## Inline Natvis XML fragments via a macro We may want to allow developers to provide Natvis descriptions using a From e4220fbfaeae7717f6d2add3b8d4d0c24fa2a4c3 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Fri, 25 Mar 2022 13:10:49 -0700 Subject: [PATCH 21/23] Respond to latest round of feedback. Specify which kinds of entities this attribute is applicable to and what happens if used elsewhere. --- text/0000-debugger-visualizer.md | 127 +++++++++++++------------------ 1 file changed, 51 insertions(+), 76 deletions(-) diff --git a/text/0000-debugger-visualizer.md b/text/0000-debugger-visualizer.md index a23a075b990..9a5ce9047ef 100644 --- a/text/0000-debugger-visualizer.md +++ b/text/0000-debugger-visualizer.md @@ -65,7 +65,7 @@ With Natvis applied, WinDbg results in the following: ``` Currently, Rust provides visualizations for a handful of types defined in its -standard library via `.natvis` files or pretty printers via python scripts. +standard library via Natvis files or pretty printers via python scripts. However, this support is inflexible; updating it requires modifying the Rust toolchain itself, and either using a local build of the toolchain or waiting for a new upstream build of the toolchain. It is not feasible for developers of @@ -197,23 +197,20 @@ The Natvis files provide patterns, which match type names, and for matching types, a description of how to display those types. This allows for some limited support for generic types. -Rust developers can add one or more `.natvis` files to their crate. Through -the use of a new Rust attribute, `#[debugger_visualizer]`, the compiler will -encode the contents of the `.natvis` file in the crate metadata if the target +Rust developers can add one or more Natvis files to their crate. Through +the use of a new Rust attribute, `#![debugger_visualizer]`, the compiler will +encode the contents of the Natvis file in the crate metadata if the target is an `rlib`. If the target is a `dll` or `exe`, the `/NATVIS` MSVC linker flag is -set for each `.natvis` file which will embed the Natvis visualizations into the PDB. +set for each Natvis file which will embed the Natvis visualizations into the PDB. -To provide Natvis files, developers create a file with the `.natvis` file -extension. +To provide Natvis files, developers create a file using the Natvis XML syntax +and reference it via the new `#![debugger_visualizer]` attribute that this RFC proposes. -See the below example for how rustc would embed these Natvis files in the -debuginfo for a binary. - -As an example, consider a crate `foo` with this directory structure: +As an example for how to use this attribute, consider a crate `foo` with this directory structure: ```text /Cargo.toml -/Foo.natvis (Note: the .natvis file does not have to match the name of the crate.) +/Foo.natvis (Note: the Natvis file does not have to match the name of the crate.) +-- src +-- main.rs ``` @@ -283,14 +280,12 @@ The pretty printers provide patterns, which match type names, and for matching types, descibe how to display those types. (For writing a pretty printer, see: https://sourceware.org/gdb/onlinedocs/gdb/Writing-a-Pretty_002dPrinter.html#Writing-a-Pretty_002dPrinter). Rust developers can add one or more pretty printers to their crate. This is done -in the Rust compiler via `.py` python scripts. Through the use of a new Rust attribute, -`#[debugger_visualizer]`, the compiler will encode the contents of the `.py` file in -the crate metadata if the target is an `rlib`. If the target is an executable, the -`.debug_gdb_scripts` section will include a reference to the pretty printer specified -by embedding the contents of the pretty printer directly into this section. +in the Rust compiler via python scripts. Through the use of the new Rust attribute +this RFC proposes, `#![debugger_visualizer]`, the compiler will encode the contents +of the pretty printer in the `.debug_gdb_scripts` section of the `ELF` generated. -To provide pretty printers, developers create a file with the `.py` file -extension and reference it via the `#[debugger_visualizer]` attribute as follows: +To provide pretty printers, developers create a pretty printer using the syntax provided +above and reference it via the `#![debugger_visualizer]` attribute as follows: ```rust #![debugger_visualizer(gdb_script_file = "../foo.py")] @@ -299,38 +294,28 @@ extension and reference it via the `#[debugger_visualizer]` attribute as follows # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -In rustc, a new built-in attribute `#[debugger_visualizer]` will be added which -instructs the compiler to take the specified file path for a debugger visualizer -and add it to the current binary being built. The file path specified must be -relative to the location of the attribute and is resolved in a manner that is -identical to how paths are resolved in the `include_str!` macro. This attribute -will directly target modules which means the syntax `#![debugger_visualizer]` is -also valid when placed at the module level. +In rustc, a new built-in crate-level attribute `#![debugger_visualizer]` will be +added which instructs the compiler to take the debugger visualizer specified by the +attribute and add it to the current binary being built. The file path specified must +be relative to the location of the attribute and is resolved in a manner that is +identical to how paths are resolved in the `include_str!` macro. + +Since the `#![debugger_visualizer]` is a crate-level attribute, any usage of this +attribute in a different context (i.e. targeting types, modules, etc) will cause a +compiler error. -For example, the following uses of the attribute are valid: +For example, the `#![debugger_visualizer]` attribute may be used in the following: Where `main.rs` contains: ```rust #![debugger_visualizer(natvis_file = "../main.natvis")] - -#[debugger_visualizer(natvis_file = "../foo.natvis")] -mod foo; -``` - -and `bar.rs` contains: - -```rust -#![debugger_visualizer(natvis_file = "../bar.natvis")] ``` -In the first case, the attribute is applied to the crate as a top-level attribute -using the inner attribute syntax and also added to the module foo using the outer -attribute syntax. In the second case, the attribute is applied to the module bar -via a top-level attribute as well which also is valid since it is still targeting -a module. +The attribute must be placed at the top of `main.rs` where other crate-level +attributes are placed. -The `#[debugger_visualizer]` attribute will reserve multiple keys to be able to +The `#![debugger_visualizer]` attribute will reserve multiple keys to be able to specify which type of visualizer is being applied. The following keys will be reserved as part of this RFC: @@ -340,7 +325,7 @@ reserved as part of this RFC: As more visualizer schemes arise, more keys may be added in the future to ensure a great debugging experience for any debugger that the Rust community sees fit. -For example, to specify that a `.natvis` file should be included in the binary +For example, to specify that a Natvis file should be included in the binary being built, the following attribute should be added to the Rust source: ```rust @@ -357,20 +342,20 @@ Depending on the Rust target, the correct debugger visualizer will be selected and embedded in the output. The Rust compiler will serialize the contents of the file specified via the -`#[debugger_visualizer]` attribute and store it in the crate metadata. This attribute +`#![debugger_visualizer]` attribute and store it in the crate metadata. This attribute can be used multiple times to allow for multiple debugger visualizer files to be embedded for each crate. When generating the final binary, the contents of the -visualizer file will be extracted from the crate metadata and written to a new file -in the target directory under a new `visualizer` directory. +visualizer file will be extracted from the crate metadata and written to a temp +directory. -In the case of a Natvis file, `#[debugger_visualizer(natvis_file = "../foo.natvis")]` +In the case of a Natvis file, `#![debugger_visualizer(natvis_file = "../foo.natvis")]` the compiler will set the `/NATVIS:{.natvis file}` MSVC linker flag for each of the Natvis files specified for the current crate as well as transitive dependencies if using the MSVC toolchain. This linker flag ensures that the specified Natvis files be embedded in the PDB generated for the binary being built. Any crate type that -would generate a PDB would have all applicable `.natvis` files embedded. +would generate a PDB would have all applicable Natvis files embedded. -In the case of GDB pretty printer, `#[debugger_visualizer(gdb_script_file = "../foo.py")]` +In the case of GDB pretty printer, `#![debugger_visualizer(gdb_script_file = "../foo.py")]` the compiler will ensure that the set of pretty printers specified will be added to the `.debug_gdb_scripts` section of the `ELF` generated. The `.debug_gdb_scripts` section takes a list of null-terminated entries which specify scripts to load within GDB. This @@ -381,12 +366,6 @@ This attribute will embed the contents of the debugger script so that it will no need to reference a file in the search path. This has proven to be a more reliable route than depending on file paths which can be unstable at times. -The `CrateRoot` type would need to be updated to account for debugger visualizer -files for crates within the dependency graph. The `CrateRoot` would contain -the field, `debugger_visualizers: Lazy<[DebuggerVisualizerFile]>`. The new type -`pub struct DebuggerVisualizerFile` will be created to ensure the contents of a -debugger visualizer file can be serialized and stored in the crate metadata. - There are a couple of reasons why the contents of a visualizer file passed into rustc will be serialized and encoded in the crate metadata. @@ -401,24 +380,20 @@ know how or why it flows through the dependency graph. Secondly, there's also been interest within the community of supporting binary crate packages. That is, compiling crates to rlibs, and then passing around rlibs directly and not rebuilding the entire library. Having to -ensure that `.natvis` files are always passed along with rlibs as well +ensure that Natvis files are always passed along with rlibs as well could become very difficult especially when other debugger visualizations also become supported such as GDB's debugger scripts and WinDbg's JavaScript debugger scripts. Packaging these sorts of things in the `rmeta` for an `rlib` is simple, reliable and seems like the "right" thing to do here. -Another change that would need to be made here is to add a new field to the -`CrateInfo` type, `pub debugger_visualizers: FxHashSet`. -This will allow the Rust compiler to query the list of visualizer files being -added for a specific crate. The compiler will then be responsible for collecting -the set of visualizer files that were previously encoded and stored in the -`CrateMetadata` for a given crate dependency, will be decoded and have the -contents written to a new file in the `target` directory. In the case of Natvis, -the path of this new file will be what is passed to the `/NATVIS` linker flag. -For example, in a debug build, the contents of the `.natvis` files that were encoded -in the crate metadata will be written to new files in a temp directory where they will -be included from. Each visualizer file that is written will have a new name to -ensure it is unique across visualizer files for all crates. +The Rust compiler will be responsible for collecting the entire set of visualizer +files that were specified via the `#![debugger_visualizer]` attribute across all +transitive crate dependencies and embedding them in the `.debug_gdb_scripts` +section for a pretty printer or passing them to the `/NATVIS` MSVC linker flag. +For example, in the case of a Natvis file, the contents of the Natvis files that +were specified will be written to new files in a temp directory where they will +be included from. The path of these files in the temp directory is what will be +passed to the `/NATVIS` MSVC linker flag. # Drawbacks [drawbacks]: #drawbacks @@ -493,7 +468,7 @@ code directly if only the attribute syntax was supported. For types with many fi or types that need extensive amounts of Natvis to appropriately visualize them in a meaninngful way, this could distract from the contents of the code. Without being able to pull some of the more intricate Natvis descriptions into a separate standalone -`.natvis` file, there may become an issue with the visibility of the source code. +Natvis file, there may become an issue with the visibility of the source code. Also, if/when other debugger visualization formats are supported, it could become very obscure to read the source with large amounts of visualization scripts from multiple schemas all being directly embedded in source code. @@ -529,7 +504,7 @@ need be. The drawbacks for this option is that it seems a sub-optimal in terms of user experience. It requires the author to operate at a lower level of abstraction by having to use a more general attribute and annotating it to tackle a specific use -case. Having a more targeted attribute, i.e. `#[debugger_visualizer]` allows for the +case. Having a more targeted attribute, i.e. `#![debugger_visualizer]` allows for the author to simply specify which debugger visualizer file should be included and allow the compiler to select the right one under the covers. @@ -557,7 +532,7 @@ Briefly, we cover some of the known systems for debugger views: Natvis is a framework that customizes how native types appear when viewed under a debugger. The Visual Studio Natvis framework is supported out of the box on -WinDbg. The debugger has the ability to load `.natvis` files via the `.nvload` +WinDbg. The debugger has the ability to load Natvis files via the `.nvload` command and directly apply them to types within loaded modules. WinDbg is also able to load `.natvis` files that have been embedded in the PDB for a binary and serve up the resulting views after applying those visualizations as well. This @@ -604,12 +579,12 @@ None. ## Inline Natvis XML fragments via an attribute Debugger visualizer support for Rust could be improved upon by adding support -for in-source visualizer definitions via the `#[debugger_visualizer]` attribute +for in-source visualizer definitions via the `#![debugger_visualizer]` attribute or a new attribute. Example: ```rust /// A rectangle in first quadrant -#[debugger_visualizer( +#![debugger_visualizer( natvis(r#" ({x},{y}) + ({dx}, {dy}) ({x}, {y}) @@ -625,9 +600,9 @@ struct FancyRect { } ``` -Currently the `#[debugger_visualizer]` attribute is only allowed to target modules -but can be updated to allow targeting types as well if the same attribute was to be -re-used to support this. +Currently the `#![debugger_visualizer]` attribute is only allowed to be used as a +crate-level attribute. This can be updated to allow targeting types as well if the +same attribute was to be re-used to support this. ## Inline Natvis XML fragments via a macro From 1889d4887a26dfe4d1e58418fcd1d300ca1c4fd4 Mon Sep 17 00:00:00 2001 From: ridwanabdillahi <91507758+ridwanabdillahi@users.noreply.github.com> Date: Fri, 25 Mar 2022 22:08:49 -0700 Subject: [PATCH 22/23] Add support for the #[debugger_visualizer] attribute to target modules. --- text/0000-debugger-visualizer.md | 49 +++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/text/0000-debugger-visualizer.md b/text/0000-debugger-visualizer.md index 9a5ce9047ef..61b11f95bb5 100644 --- a/text/0000-debugger-visualizer.md +++ b/text/0000-debugger-visualizer.md @@ -294,28 +294,44 @@ above and reference it via the `#![debugger_visualizer]` attribute as follows: # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -In rustc, a new built-in crate-level attribute `#![debugger_visualizer]` will be -added which instructs the compiler to take the debugger visualizer specified by the -attribute and add it to the current binary being built. The file path specified must -be relative to the location of the attribute and is resolved in a manner that is -identical to how paths are resolved in the `include_str!` macro. - -Since the `#![debugger_visualizer]` is a crate-level attribute, any usage of this -attribute in a different context (i.e. targeting types, modules, etc) will cause a -compiler error. - -For example, the `#![debugger_visualizer]` attribute may be used in the following: +In rustc, a new built-in attribute `#[debugger_visualizer]` will be added which +instructs the compiler to take the specified file path for a debugger visualizer +and add it to the current binary being built. The file path specified must be +relative to the location of the attribute and is resolved in a manner that is +identical to how paths are resolved in the `include_str!` macro. This attribute +will directly target modules which means the syntax `#![debugger_visualizer]` is +also valid when placed at the module level. This would allow for this attribute to +be used as a crate-level attribute as well which is different than a typical module item +when placed at the top-level crate file, `lib.rs` or `main.rs`. + +For example, the following uses of the attribute are valid: Where `main.rs` contains: ```rust #![debugger_visualizer(natvis_file = "../main.natvis")] + +#[debugger_visualizer(natvis_file = "../foo.natvis")] +mod foo; +``` + +and `bar.rs` contains: + +```rust +#![debugger_visualizer(natvis_file = "../bar.natvis")] ``` -The attribute must be placed at the top of `main.rs` where other crate-level -attributes are placed. +In the first case, the attribute is applied to the crate as a crate-level attribute +using the inner attribute syntax on the top-level crate source file. It is also +added to the module foo using the outer attribute syntax. In the second case, +the attribute is applied to the module bar using the inner attribute syntax which +also is valid since it is still targeting a module. + +The only valid targets for this attribute are modules or as a crate-level attribute. +Using this attribute on any other target, for instance a type or a function, will +cause rustc to raise a compiler error that will need to be resolved. -The `#![debugger_visualizer]` attribute will reserve multiple keys to be able to +The `#[debugger_visualizer]` attribute will reserve multiple keys to be able to specify which type of visualizer is being applied. The following keys will be reserved as part of this RFC: @@ -600,8 +616,9 @@ struct FancyRect { } ``` -Currently the `#![debugger_visualizer]` attribute is only allowed to be used as a -crate-level attribute. This can be updated to allow targeting types as well if the +Currently the `#[debugger_visualizer]` attribute is only allowed to target modules +which includes being used as crate-level attribute when targeting the top-level +`*.rs` source file. This can be updated to allow targeting types as well if the same attribute was to be re-used to support this. ## Inline Natvis XML fragments via a macro From 777a74c41788feaaee9d7ac1cb82701040449977 Mon Sep 17 00:00:00 2001 From: Wesley Wiser Date: Mon, 11 Apr 2022 10:20:00 -0400 Subject: [PATCH 23/23] Update 3191 to merge --- ...{0000-debugger-visualizer.md => 3191-debugger-visualizer.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename text/{0000-debugger-visualizer.md => 3191-debugger-visualizer.md} (99%) diff --git a/text/0000-debugger-visualizer.md b/text/3191-debugger-visualizer.md similarity index 99% rename from text/0000-debugger-visualizer.md rename to text/3191-debugger-visualizer.md index 61b11f95bb5..69321222a47 100644 --- a/text/0000-debugger-visualizer.md +++ b/text/3191-debugger-visualizer.md @@ -1,7 +1,7 @@ - Feature Name: `debugger_visualizer` - Start Date: 2021-11-01 - RFC PR: [rust-lang/rfcs#3191](https://github.com/rust-lang/rfcs/pull/3191) -- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) +- Rust Issue: [rust-lang/rust#95939](https://github.com/rust-lang/rust/issues/95939) # Summary [summary]: #summary