-
Notifications
You must be signed in to change notification settings - Fork 15.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[C#] Fix trim warnings #9182
[C#] Fix trim warnings #9182
Conversation
I took a look and I think I found a few more problematic areas: #9183 |
How did you find those? I tested build warnings with RC1. Were more warnings added in RC2 and final? |
I tested by trimming from the LinkedTestClient in the |
@MichalStrehovsky @agocke @jkotas All warnings fixed. The one thing I'm unsure about is this code: protobuf/csharp/src/Google.Protobuf/JsonFormatter.cs Lines 884 to 898 in 7ccf4d8
Enum types are specified in code-generated descriptor code here -
Type , so the dynamic members attribute isn't allowed. Have you considered allowing that attribute on Type[] ?
|
Unfortunately the problem with arrays isn't really that we couldn't allow the attribute, it's that the attribute represents the ability to track the variable, and tracking the individual elements of an array would probably be too difficult/cost prohibitive. That said, an improvement to investigate for the future. |
@@ -77,6 +80,7 @@ internal CustomOptions(IDictionary<int, IExtensionValue> values) | |||
/// <param name="field">The field to fetch the value for.</param> | |||
/// <param name="value">The output variable to populate.</param> | |||
/// <returns><c>true</c> if a suitable value for the field was found; <c>false</c> otherwise.</returns> | |||
[RequiresUnreferencedCode(UnreferencedCodeMessage)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CustomOptions
is obsolete so I just marked its methods as incompatible with trimming instead of finding a way to fix it.
clrType == typeof(string) ? "" | ||
: clrType == typeof(ByteString) ? ByteString.Empty | ||
: Activator.CreateInstance(clrType); | ||
object defaultValue = GetDefaultValue(descriptor); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the one notable runtime change. A default value is returned based on the field type instead of using reflection.
#endregion | ||
|
||
#if !NET5_0_OR_GREATER | ||
// Copied with permission from https://github.com/dotnet/runtime/tree/8fbf206d0e518b45ca855832e8bfb391afa85972/src/libraries/System.Private.CoreLib/src/System/Diagnostics/CodeAnalysis |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add copies of these code analysis attributes for framework versions that don't have them.
Avoids the need to add #ifdef
everywhere a code analysis attribute is used.
You'll need to update protobuf/csharp/install_dotnet_sdk.ps1 Line 19 in 13d559b
and
Ideally, we'd have a separate PR that upgrades the pre-installed SDKs and switches to netcoreapp3.1 targets (they won't run unless you have 3.1 runtime installed), after which we could look into the actual changes in this PR. |
Let's rebase once #9205 is merged (I'll merge as soon as the test finish). |
1cbf5b9
to
2e9da70
Compare
Rebased. |
Seems like tests are failing. |
2e9da70
to
73eb0d7
Compare
@jtattermusch Fixed. |
nit: Makefile.am needs to be updated
https://source.cloud.google.com/results/invocations/503d13e5-df5f-416e-8974-3b2a19a427ec/log |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few things I'd like to clarify:
-
I understand this was tested with a test app with trimming enabled, but it's unclear to me what was the coverage of the test app. IIRC, it might well be true that there are some codepath that could still generate trimming warnings, but the test app hasn't exercised them? So is this PR meant as an incremental improvement that removes many (but possibly not all) trimming issues? Is it even possible to identify all the potential trimming problems at compile time or are the trimming warnings basically heuristics?
-
how do we make sure new trimming warnings don't get reintroduced in the future? Manually testing with a test app seems quite fragile, and there is currently nothing in this PR that would prevent regressions. If at some point in the future we introduce a new API that has to do with reflection, it seems quite easy to just forget that the API needs to be annotated with the trimming-related attributes.
-
what is the failure mode if we don't get the trimming related annotations right and someone enables trimming for project that uses Google.Protobuf? Will the resulting application crash because required APIs may have been trimmed? It sounds like the trimming warnings cannot be 100% reliable in the sense "if it builds, it will work".
public GeneratedClrTypeInfo( | ||
// Preserve all public members on message types when trimming is enabled. | ||
// This ensures that members used by reflection, e.g. JSON serialization, are preserved. | ||
[DynamicallyAccessedMembers(MessageAccessibility)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. Is the trick there that we know that each generated message type will call the
GeneratedClrTypeInfo constructor, and so by piggybacking on GeneratedClrTypeInfo we can make sure that trimming will be avoided for all members of generated proto messages?
-
if so, I think the comment could do a better job at explaining this
-
is this really the best place for placing the DynamicallyAccessedMembers attribute?
-
I can imagine there being use cases where one uses large .proto files (quite common e.g. at google) and only uses a subset of the fields in the message. So one use case to think about is whether trimming can be made to strip all the unused protobuf fields (assuming the user doesn't use reflection on json serialization). I think this is probably quite difficult to achieve, but is something to think about in the future. Also this should be listed as a "limitation" of the current approach?.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this really the best place for placing the DynamicallyAccessedMembers attribute?
The idea of dynamically accessed members attribute is to annotate those places so Type
arguments and generic arguments that are passed to them can be used to include members on those types, without having them explicitly referenced by code. Having GeneratedClrTypeInfo
included in code generation seems like a good way to include members on those types that could be used dynamically.
I can imagine there being use cases where one uses large .proto files (quite common e.g. at google) and only uses a subset of the fields in the message. So one use case to think about is whether trimming can be made to strip all the unused protobuf fields (assuming the user doesn't use reflection on json serialization). I think this is probably quite difficult to achieve, but is something to think about in the future. Also this should be listed as a "limitation" of the current approach?.
Including members on types like this could definitely pull in more than you need. There is a tradeoff between everything just working but the app size is larger, and some functionality breaking but smaller apps.
Right now I'm not exactly sure how granular and smart trimming is. For example, imagine a library assembly that has generated types from two large proto files. If your app doesn't use any types from one of the library's proto files, will its static ctor never be evaluated, and therefore GeneratedClrTypeInfo
calls are trimmed and so none of its types will be included?
FTR since I don't know what the "test app" you tested against was, I ran a small experiment myself. That experiment gave me these warnings:
|
@@ -109,14 +113,48 @@ internal SingleFieldAccessor(PropertyInfo property, FieldDescriptor descriptor) | |||
// While presence isn't supported, clearing still is; it's just setting to a default value. | |||
var clrType = property.PropertyType; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like the clrType
is now unused?
I tested using this app: https://github.com/grpc/grpc-dotnet/tree/dce1ef72d8b88a380136a699a1fb26ca3793a597/testassets/LinkerTestsClient (although I modified it to reference Google.Protobuf source code while testing) There is a grpc-dotnet unit test that publishes the app with trimming and then executes it: https://github.com/grpc/grpc-dotnet/blob/82dec0ff513059786de5254e62d71c5f55e0ed5b/test/FunctionalTests/Linker/LinkerTests.cs Actually testing trimming is really difficult because you can't have multiple tests in an individual app because each test will potentially include types and members that otherwise wouldn't be included if tests were run individually. However, the trimming warnings shouldn't be impacted by this. It analysis all of an assembly, not just the parts that an app uses.
The warnings you've listed are from the test project. |
Fixed |
4e63393
to
22462b0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM from the perspective that this enables the use of .NET6 trimming with Google.Protobuf (and removes the warning we currently know about).
Currently there are some limitations of the solution proposed in this PR:
- we don't know whether the Google.Protobuf trimming will work realiably with a real life application (we only tried for a small test project)
- we don't know how effective the trimming is when used with protobuf messages
- we don't have any continuous test that prevents reintroduction of trimming warnings in the future
So for now I'd refrain from claims like "Google.Protobuf supports trimming now" when mentioning this change. I think the current state of things is now more like "initial work to support trimming has been done and Google.Protobuf may work well - but users should test carefully to make sure that there aren't any regressions."
@JamesNK if you are happy with this summary, I can merge once the tests are green.
There is a small test project for unit testing, but in real-world apps there are many who are using trimming + gRPC + Google.Protobuf in Blazor. Whether the JSON functionality works with trimming is still an open question.
That's fine. The goal here is aimed at removing warnings. |
Sorry, I missed the original comments here -- I wanted to note that the static analysis for trimming is meant to be sound, so if no trim warnings are produced, the application should not have any different behavior vs. an untrimmed app at runtime. Any situation in which the behavior of a trimmed app changes without a warning should be considered a bug on the https://github.com/dotnet/linker tool. |
Background:
.NET 6 introduces trim analysis and warnings. When Google.Protobuf is included in an app, and trimming is enabled, the app will get a list of trim-related warnings from Google.Protobuf. These are typically places in code that reflection is used but the type could have been trimmed at build time. Use those areas of code could fail after trimming at runtime.
For more information, see https://docs.microsoft.com/en-us/dotnet/core/deploying/trimming/prepare-libraries-for-trimming
This PR fixes the trim warnings for Google.Protobuf by adding trimming annotations. They either preserve type information so the trimmer can make better decisions, or warn the caller of Google.Protobuf that the API is not compatible with trimming.
I tested all trim warnings are fixed by using Google.Protobuf with a test app in the grpc-dotnet repo.
Also, I updated the test projects to use netcoreapp3.1. The previous target, netcoreapp2.1, is no longer supported and build was generating warnings.
cc @jtattermusch @captainsafia