cmd/link: option to omit DWARF vendor attributes #25297

gwik · 2018-05-08T20:56:11Z

Hi,

I'm trying to make the crashlytics symbol extractor to accept to parse go libraries generated with gomobile.
However it crashes because go use "vendor" or "user" DWARF attributes (e.g: DW_AT_go_kind = 0x2900).
Usage is perfectly standard, yet it crashes every time it encounters such attribute. I guess that, while standard, this is uncommon for "native" libraries. I suppose they are written in C or C++ most of the time and don't include vendor attributes.

My first approach was to try to strip the attributes from the binary. However, this is fairly involved and probably error prone, the DWARF binary encoding doesn't allow much flexibility. It would require to entirely rewrite many DWARF sections.

I figured that making Go not include those attributes in the first place was straightforward and not intrusive since there was already a single place that filters out the attributes before they are written out:

go/src/cmd/internal/dwarf/dwarf.go

Line 941 in cd1976d

// only the ones actually listed in the Abbrev will be written out.

Thanks to the amazing work of @eliasnaur lately, this is the last piece that prevent go to fully support the mobile debugging and tooling ecosystem. I hope you will consider adding the option to go tool link.
CL follows.

The text was updated successfully, but these errors were encountered:

gopherbot · 2018-05-08T20:57:13Z

Change https://golang.org/cl/112215 mentions this issue: cmd/link: add option to omit go DWARF user attributes

eliasnaur · 2018-05-08T21:07:54Z

A new flag is a heavy handed solution. Have you reported the issue to Crashlytics? Is the problem for both iOS and Android or only one? What are the vendor DWARF attributes for? Perhaps they can be filtered unconditionally for the problematic platforms, at least until Crashlytics catches up?

gwik · 2018-05-08T22:54:03Z

Yes, I reported the issue to Crashlytics, I'm waiting for a reply. The issue is on Android only, the crashlytics gradle plugin embeds a ELF/DWARF parser and generate "cSym" file which has the same purpose as the dSym on darwin.

The vendor attributes aims at adding higher level go concepts to the debug information. They can be used by debuggers that are go aware, I suppose. Although it doesn't feel like they are required, DWARF already have a lot of different concepts and was designed for representing any language.

Filtering on the target platform instead of a flag is a good idea, it propably would be a much less binding solution for go.

steeve · 2018-05-09T07:15:07Z

Although we've only seen the issue with Crashlytics on Android (and they have been informed), one could make the argument that some DWARF parsers could choke on Go metadata, in which case this flag could remedy the situation. Although that might be far fetched.

It is true that this could be enabled by default when GOOS=android, where Crashlytics is likely to be used.

eliasnaur · 2018-05-09T08:00:15Z

I'm still not convinced of a new flag; crashlytics is a client side program and as soon as they release a fix, the problem is gone for all their users. On the other hand, we'll have to support a new flag indefinitely. Even if a flag were the way forward, I'm not sure you'll get it in Go 1.11 anyway, because of the freeze.

Other than waiting for a response for Crashlytics, I think the best bet is @gwik's stripping of custom attributes. How far did that get? Can you "zero out" the offending attributes, or make them more standard instead of stripping them so you avoid more invasive changes to the DWARF structure?

I'm assuming the Crashlytics parser isn't open source. If it is, one could put in a hack to that instead.

steeve · 2018-05-09T08:17:03Z

I think there are two issues there:

wether link should dump Go attributes or not
wether that behaviour should be exposed externally in the command line

From what I understand, you are mainly concerned with 2, right? Because 1 could be restricted to GOOS=android.

Ultimately it is true that the burden lies with Crashlytics (or their parser). But since Crashlytics is Google now, I'm not sure a timely fix will be happening.

gwik · 2018-05-09T08:56:02Z

Can you "zero out" the offending attributes, or make them more standard instead of stripping them so you avoid more invasive changes to the DWARF structure?

That's what I hoped, but I don't think this is possible. Zero attribute marks the end of the attributes and the end of the entry. Although the spec tells that null entries are valid, they mark the end of siblings in the tree and crashlytics parser use a stack to build up its tree representation and unconditionally pops when it sees a null entry.

I also tried to replace with some unused attributes (for crashlytics) that would match the value class but vendor attributes end up being two bytes and "standard" attributes are one byte after variable size encoding (LEB128).

Rewriting the .debug_abbrev and .debug_info section seems inevitable, but then you have to rewrite .debug_aranges, .debug_pubnames, .debug_loc because they reference offsets in the .debug_info table. You end up rewrite everything.

gwik · 2018-05-09T09:11:07Z

I'm assuming the Crashlytics parser isn't open source. If it is, one could put in a hack to that instead.

Unfortunately it's not. Looking at decompiled code is straightforward but you can't modify it.

eliasnaur · 2018-05-09T14:55:38Z

Pinging @aclements (the owner of debug/dwarf) for good ideas.

ianlancetaylor · 2018-05-09T15:30:16Z

I don't know if it will help, but I think we should give Crashlytics a few days to respond. I've also pinged them internally.

thanm · 2018-05-09T15:54:52Z

I agree with Ian, seems like it would be better to fix the DWARF consumer.
Crashing on all vendor-specific attributes seems fairly unfriendly; they are not really all that uncommon.

gwik · 2018-05-09T16:38:25Z

I agree with Ian, seems like it would be better to fix the DWARF consumer.

Off course. Patching go was the easiest way for me to move forward and testing that it would work without the attributes. I perfectly understand that adding the flag is heavy-handed.

Thank you @ianlancetaylor for pinging them, here the answer I got from support (maybe the people you ping didn't communicate with the person who answered me):

Thanks for reaching out on this Antonin, unfortunately, we don't support Go, so the team won't be able to dig into this at the moment. If that changes, I'll let you know!

I will insist and explain them this is not related to Go, but it doesn't start well.

TKBurner · 2018-05-10T06:01:24Z

Thanks. I'm from the Crashlytics team and it's really interesting to hear what you are up to with Go! Right now it's not on our roadmap to add support for the build process offered by Gomobile.

I really am interested how these crash reports symbolicate if you follow manual symbolication: https://stackoverflow.com/questions/3832900/how-to-manually-symbolicate-ios-crash-to-view-crash-logs

This is generally how we begin to look into these sorts of problems when we consider if we are able to take on new work.

Thanks all and always feel free to reach out 👍

gwik · 2018-05-10T19:12:08Z

@TKBurner thanks for reaching out here. The issue we are talking about is not related to go nor gomobile. We are talking about how the crashlytics gradle android plugin fails when any native library (NDK) contains DWARF standard user attributes like go ones does.
I’ve run a debugger and looked at the decompiled java code where it raises an exception and the fix is probably a few “if” statements so it can just ignore the user attributes and complete the symbolication process.
Again, this is not a go issue, it is just about fixing the support for DWARF symbol extraction that you already support. All you need to do is to skip the user attribute range from 0x2000 to 0x3fff.

TKBurner · 2018-05-10T19:16:50Z

Thanks. Please send me an email at [email protected], mentioning me. I'll pass this along to our ndk team.

…

On Thu, May 10, 2018, 12:13 PM Antonin Amand ***@***.***> wrote: @TKBurner <https://github.com/TKBurner> thanks for reaching out here. The issue we are talking about is not related to go nor gomobile. We are talking about how the crashlytics gradle andoirs plugin fails when any native library (NDK) contains DWARF standard user attributes like go ones does. I’ve run a debugger and looked at the decompiled java code where it raises an exception and the fix is probably a few “if” statements so it can just ignore the user attributes and complete the symbolication process. Again, this is not a go issue, it is just about fixing the support for DWARF symbol extraction that you already support. All you need to do is to skip the user attribute range from 0x2000 to 0x3fff. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25297 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB5SJGBUVDBx-6T1qq0hP5iyrRZ1ALUfks5txJFbgaJpZM4T3U4E> .

steeve · 2018-05-10T19:19:55Z

Thanks Todd, we already have a thread going on with Michael. We'll let him know.

eliasnaur · 2018-11-29T11:44:18Z

What's the status of this?

steeve · 2018-11-29T17:07:47Z

We sent a patch to the Crashlytics team to add support for the Go DWARF types, which we did thanks to a Java decompiler (fernflower, used by Android Studio).

They have acknowledged the patch, but I'm not sure it was released/mainlined (don't see anything related to that in the Changelog at https://docs.fabric.io/android/changelog.html#september-27-2018).
@TKBurner should know more hopefully.

In the mean time, we recompiled the Crashlytics gradle plugin with the patch. It works well.

This patch basically adds the Go DWARF attributes to the hashmap responsible for holding them.

--- DWAttribute.java.orig	2018-08-23 20:51:48.000000000 +0200
+++ DWAttribute.java	2018-08-23 20:51:48.000000000 +0200
@@ -251,7 +251,12 @@
    APPLE_MAJOR_RUNTIME_VERS(16357, "APPLE_major_runtime_vers"),
    APPLE_RUNTIME_CLASS(16358, "APPLE_runtime_class"),
    APPLE_OMIT_FRAME_PTR(16359, "APPLE_omit_frame_ptr"),
-   HI_USER(16383, "hi_user");
+   HI_USER(16383, "hi_user"),
+   GO_KIND(0x2900, "go_kind"),
+   GO_KEY(0x2901, "go_key"),
+   GO_ELEM(0x2902, "go_elem"),
+   GO_EMBEDDED_FIELD(0x2903, "go_embedded_field"),
+   GO_RUNTIME_TYPE(0x2904, "go_runtime_type");
 
    private static final String PREFIX = "DW_AT_";
    private static final Map LOOKUP;

I can attach the final jar is people are interested, but I'd rather have it mainlined.

Finally, I think we can close that issue, since we've changed course.

aclements · 2018-11-29T17:18:31Z

The correct fix on Crashlytics' side would be to ignore all unrecognised vendor attributes. Adding to the list of known vendor attributes just defers the problem. Though perhaps that's harder to do by rewriting the byte code. :)

steeve · 2018-11-29T17:21:43Z

@aclements you nailed it. Calling javac on this single file was easier.

Here is our Makefile (minus our internal stuff), if people are tempted:

ARTIFACT_GROUP_ID := io.fabric.tools
ARTIFACT_NAME := gradle
ARTIFACT_VERSION := 1.25.4
MAVEN_REPOSITORY := https://maven.fabric.io/public
ARTIFACT_URL_PREFIX := $(MAVEN_REPOSITORY)/$(subst .,/,$(ARTIFACT_GROUP_ID))/$(ARTIFACT_NAME)/$(ARTIFACT_VERSION)
JAR_FILE := $(ARTIFACT_NAME)-$(ARTIFACT_VERSION).jar
POM_FILE := $(ARTIFACT_NAME)-$(ARTIFACT_VERSION).pom
PATCHED_ARTIFACT_VERSION := $(ARTIFACT_VERSION).zenly
PATCHED_JAR_FILE := $(ARTIFACT_NAME)-$(PATCHED_ARTIFACT_VERSION).jar
PATCHED_POM_FILE := $(ARTIFACT_NAME)-$(PATCHED_ARTIFACT_VERSION).pom
FERNFLOWER_VERSION := 2.5.0.Final
FERNFLOWER := fernflower.jar

all: deploy clean

$(FERNFLOWER):
	curl -L -o $(@) http://central.maven.org/maven2/org/jboss/windup/decompiler/fernflower/fernflower/$(FERNFLOWER_VERSION)/fernflower-$(FERNFLOWER_VERSION).jar

$(JAR_FILE):
	curl -L -o $(JAR_FILE) $(ARTIFACT_URL_PREFIX)/$(JAR_FILE)

$(POM_FILE):
	curl -L -o $(POM_FILE) $(ARTIFACT_URL_PREFIX)/$(POM_FILE)

$(PATCHED_JAR_FILE): $(JAR_FILE) $(FERNFLOWER)
	unzip -o $(JAR_FILE) 'com/crashlytics/tools/utils/dwarf/DWAttribute.class'
	java -cp $(FERNFLOWER) org.jetbrains.java.decompiler.main.decompiler.ConsoleDecompiler \
		com/crashlytics/tools/utils/dwarf/DWAttribute.class .
	patch -p0 < DWAttribute_golang_dwarf.patch
	javac -d . DWAttribute.java
	rm -f DWAttribute.java
	cp $(JAR_FILE) $(PATCHED_JAR_FILE)
	jar -uf $(PATCHED_JAR_FILE) com/

$(PATCHED_POM_FILE): $(POM_FILE)
	sed 's/>$(ARTIFACT_VERSION)</>$(PATCHED_ARTIFACT_VERSION)</g' < $(POM_FILE) > $(PATCHED_POM_FILE)

gopherbot · 2018-12-29T11:53:22Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

tmm1 · 2019-04-13T18:28:55Z

Was the patch merged into any of the new fabric gradle releases this year?

firesteps · 2019-04-15T11:00:07Z

As a workaround you can use go prior to 1.11
https://golang.org/doc/go1.11

DWARF sections are now compressed by default because of the expanded and more accurate debug information produced by the compiler. This is transparent to most ELF tools (such as debuggers on Linux and *BSD) and is supported by the Delve debugger on all platforms, but has limited support in the native tools on macOS and Windows. To disable DWARF compression, pass -ldflags=-compressdwarf=false to the go tool when building a binary.

aclements · 2019-04-15T18:17:43Z

@firesteps, DWARF compression is unrelated to the use of vendor attributes. I believe Go has been emitting DWARF vendor attributes for a very long time.

steeve · 2019-04-29T11:15:56Z

@tmm1 it wasn't. No response from the Fabric team at Google.
I'm thinking we should distribute our fork on Maven for others to use.

@firesteps @aclements is right, this is because of Go vendor attributes not supported by the Fabric Gradle plugin.

steeve · 2019-04-29T11:19:42Z

Here is our makefile

ARTIFACT_GROUP_ID := io.fabric.tools
ARTIFACT_NAME := gradle
ARTIFACT_VERSION := 1.28.1
MAVEN_REPOSITORY := https://maven.fabric.io/public
ARTIFACT_URL_PREFIX := $(MAVEN_REPOSITORY)/$(subst .,/,$(ARTIFACT_GROUP_ID))/$(ARTIFACT_NAME)/$(ARTIFACT_VERSION)
JAR_FILE := $(ARTIFACT_NAME)-$(ARTIFACT_VERSION).jar
POM_FILE := $(ARTIFACT_NAME)-$(ARTIFACT_VERSION).pom
PATCHED_ARTIFACT_VERSION := $(ARTIFACT_VERSION).zenly
PATCHED_JAR_FILE := $(ARTIFACT_NAME)-$(PATCHED_ARTIFACT_VERSION).jar
PATCHED_POM_FILE := $(ARTIFACT_NAME)-$(PATCHED_ARTIFACT_VERSION).pom
FERNFLOWER_VERSION := 2.5.0.Final
FERNFLOWER := fernflower.jar

all: $(PATCHED_JAR_FILE)

$(FERNFLOWER):
	curl -L -o $(@) http://central.maven.org/maven2/org/jboss/windup/decompiler/fernflower/fernflower/$(FERNFLOWER_VERSION)/fernflower-$(FERNFLOWER_VERSION).jar

$(JAR_FILE):
	curl -L -o $(JAR_FILE) $(ARTIFACT_URL_PREFIX)/$(JAR_FILE)

$(POM_FILE):
	curl -L -o $(POM_FILE) $(ARTIFACT_URL_PREFIX)/$(POM_FILE)

$(PATCHED_JAR_FILE): $(JAR_FILE) $(FERNFLOWER)
	unzip -o $(JAR_FILE) 'com/crashlytics/tools/utils/dwarf/DWAttribute.class'
	java -cp $(FERNFLOWER) org.jetbrains.java.decompiler.main.decompiler.ConsoleDecompiler \
		com/crashlytics/tools/utils/dwarf/DWAttribute.class .
	patch -p0 < DWAttribute_golang_dwarf.patch
	javac -d . DWAttribute.java
	rm -f DWAttribute.java
	cp $(JAR_FILE) $(PATCHED_JAR_FILE)
	jar -uf $(PATCHED_JAR_FILE) com/

$(PATCHED_POM_FILE): $(POM_FILE)
	sed 's/>$(ARTIFACT_VERSION)</>$(PATCHED_ARTIFACT_VERSION)</g' < $(POM_FILE) > $(PATCHED_POM_FILE)

clean:
	rm -rf \
		$(FERNFLOWER) \
		$(JAR_FILE) \
		$(POM_FILE) \
		$(PATCHED_JAR_FILE) \
		$(PATCHED_POM_FILE) \
		com/

And the patch:

--- DWAttribute.java.orig	2018-08-23 20:51:48.000000000 +0200
+++ DWAttribute.java	2018-08-23 20:51:48.000000000 +0200
@@ -251,7 +251,12 @@
    APPLE_MAJOR_RUNTIME_VERS(16357, "APPLE_major_runtime_vers"),
    APPLE_RUNTIME_CLASS(16358, "APPLE_runtime_class"),
    APPLE_OMIT_FRAME_PTR(16359, "APPLE_omit_frame_ptr"),
-   HI_USER(16383, "hi_user");
+   HI_USER(16383, "hi_user"),
+   GO_KIND(0x2900, "go_kind"),
+   GO_KEY(0x2901, "go_key"),
+   GO_ELEM(0x2902, "go_elem"),
+   GO_EMBEDDED_FIELD(0x2903, "go_embedded_field"),
+   GO_RUNTIME_TYPE(0x2904, "go_runtime_type");
 
    private static final String PREFIX = "DW_AT_";
    private static final Map LOOKUP;

ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 8, 2018

ianlancetaylor added this to the Unplanned milestone May 8, 2018

sashabaranov mentioned this issue Nov 29, 2018

x/mobile: unable to include symbols for App Store submission #28997

Closed

eliasnaur added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 29, 2018

gopherbot closed this as completed Dec 29, 2018

sfroment mentioned this issue Mar 6, 2019

[IOS] App freeze then crash berty/berty#1004

Closed

golang locked and limited conversation to collaborators Apr 28, 2020

gopherbot added the FrozenDueToAge label Apr 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/link: option to omit DWARF vendor attributes #25297

cmd/link: option to omit DWARF vendor attributes #25297

gwik commented May 8, 2018

gopherbot commented May 8, 2018

eliasnaur commented May 8, 2018

gwik commented May 8, 2018

steeve commented May 9, 2018 •

edited

Loading

eliasnaur commented May 9, 2018

steeve commented May 9, 2018 •

edited

Loading

gwik commented May 9, 2018

gwik commented May 9, 2018

eliasnaur commented May 9, 2018

ianlancetaylor commented May 9, 2018

thanm commented May 9, 2018

gwik commented May 9, 2018

TKBurner commented May 10, 2018

gwik commented May 10, 2018 •

edited

Loading

TKBurner commented May 10, 2018 via email

steeve commented May 10, 2018

eliasnaur commented Nov 29, 2018

steeve commented Nov 29, 2018 •

edited

Loading

aclements commented Nov 29, 2018 via email

steeve commented Nov 29, 2018 •

edited

Loading

gopherbot commented Dec 29, 2018

tmm1 commented Apr 13, 2019

firesteps commented Apr 15, 2019

aclements commented Apr 15, 2019

steeve commented Apr 29, 2019 •

edited

Loading

steeve commented Apr 29, 2019

cmd/link: option to omit DWARF vendor attributes #25297

cmd/link: option to omit DWARF vendor attributes #25297

Comments

gwik commented May 8, 2018

gopherbot commented May 8, 2018

eliasnaur commented May 8, 2018

gwik commented May 8, 2018

steeve commented May 9, 2018 • edited Loading

eliasnaur commented May 9, 2018

steeve commented May 9, 2018 • edited Loading

gwik commented May 9, 2018

gwik commented May 9, 2018

eliasnaur commented May 9, 2018

ianlancetaylor commented May 9, 2018

thanm commented May 9, 2018

gwik commented May 9, 2018

TKBurner commented May 10, 2018

gwik commented May 10, 2018 • edited Loading

TKBurner commented May 10, 2018 via email

steeve commented May 10, 2018

eliasnaur commented Nov 29, 2018

steeve commented Nov 29, 2018 • edited Loading

aclements commented Nov 29, 2018 via email

steeve commented Nov 29, 2018 • edited Loading

gopherbot commented Dec 29, 2018

tmm1 commented Apr 13, 2019

firesteps commented Apr 15, 2019

aclements commented Apr 15, 2019

steeve commented Apr 29, 2019 • edited Loading

steeve commented Apr 29, 2019

steeve commented May 9, 2018 •

edited

Loading

steeve commented May 9, 2018 •

edited

Loading

gwik commented May 10, 2018 •

edited

Loading

steeve commented Nov 29, 2018 •

edited

Loading

steeve commented Nov 29, 2018 •

edited

Loading

steeve commented Apr 29, 2019 •

edited

Loading