[MONO] Add JIT_CODE_DEBUG_INFO record functionality for Jitdump #85107

saitama951 · 2023-04-20T14:21:00Z

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

vargaz · 2023-04-20T20:15:12Z

src/mono/mono/mini/mini-runtime.c

@@ -2128,7 +2145,75 @@ mono_emit_jit_dump (MonoJitInfo *jinfo, gpointer code)

 		record.code_index = ++code_index;

-		// TODO: write debugInfo and unwindInfo immediately before the JitCodeLoadRecord (while lock is held).
+		DebugEntry ent;


This should follow the mono coding conventions, see the rest of the file for examples.

done. let me know if there is anything else

saitama951

changed the style to adhere to the mono coding conventions

saitama951 · 2023-04-21T04:51:11Z

src/mono/mono/mini/mini-runtime.c

@@ -2128,7 +2145,75 @@ mono_emit_jit_dump (MonoJitInfo *jinfo, gpointer code)

 		record.code_index = ++code_index;

-		// TODO: write debugInfo and unwindInfo immediately before the JitCodeLoadRecord (while lock is held).
+		DebugEntry ent;


done. let me know if there is anything else

fanyang-mono

Please go through your change and fix the coding style. I've pointed out a few examples for you.

fanyang-mono · 2023-04-21T19:46:48Z

src/mono/mono/mini/mini-runtime.c

+        guint32 line;
+        guint32 discrim;
+        char name[];
+}DebugEntry;


Suggested change

}DebugEntry;

} DebugEntry;

fanyang-mono · 2023-04-21T19:46:59Z

src/mono/mono/mini/mini-runtime.c

+        guint64 code_addr;
+        guint64 nr_entry;
+        DebugEntry debug_entry[];
+}JitCodeDebug;


Suggested change

}JitCodeDebug;

} JitCodeDebug;

fanyang-mono · 2023-04-21T19:47:24Z

src/mono/mono/mini/mini-runtime.c


+static void add_basic_JitCodeDebug_info(JitCodeDebug *record);


Suggested change

static void add_basic_JitCodeDebug_info(JitCodeDebug *record);

static void add_basic_JitCodeDebug_info (JitCodeDebug *record);

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

fanyang-mono · 2023-04-21T19:50:19Z

src/mono/mono/mini/mini-runtime.c

+                rec.code_addr = (guint64)dmji->code_start;
+                rec.header.total_size = sizeof(rec) + sizeof(ent) + 1;
+                rec.nr_entry=1;
+                for(i=0;i < dmji->num_line_numbers;++i){


Fix code style

fanyang-mono · 2023-04-21T19:51:00Z

src/mono/mono/mini/mini-runtime.c

+
+			loc = mono_debug_lookup_source_location_by_il(jinfo->d.method,dmji->line_numbers[i].il_offset,NULL);
+
+                        if(!(loc)


Suggested change

if(!(loc)

if(!loc)

saitama951 · 2023-04-24T04:17:11Z

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

saitama951 · 2023-05-04T09:39:28Z

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

directhex · 2023-05-17T20:34:21Z

@vargaz @lambdageek can someone see if this is ready to merge now?

shivanirmishra · 2023-05-25T05:58:48Z

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

saitama951 · 2023-05-26T09:01:15Z

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

shivanirmishra · 2023-05-29T07:15:31Z

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

saitama951 · 2023-05-29T07:35:08Z

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

shivanirmishra · 2023-05-31T10:50:25Z

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

saitama951 · 2023-05-31T10:57:41Z

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

shivanirmishra · 2023-06-12T14:14:11Z

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

fanyang-mono · 2023-06-12T20:30:41Z

/backport to release/7.0-staging

github-actions · 2023-06-12T20:30:53Z

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

shivanirmishra · 2023-06-13T06:47:20Z

@saitama951 @fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

saitama951 · 2023-06-13T06:56:19Z

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

shivanirmishra · 2023-06-13T15:28:47Z

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

fanyang-mono · 2023-06-13T19:58:17Z

@saitama951 @fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

fanyang-mono · 2023-06-14T19:40:37Z

@shivanirmishra We will get this fix in the next possible servicing release.

shivanirmishra · 2023-06-20T10:58:59Z

@saitama951 @fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

saitama951 · 2023-07-04T09:27:33Z

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

shivanirmishra · 2023-07-04T09:32:41Z

Hi @saitama951 @fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

saitama951 · 2023-07-04T09:36:58Z

@shivanirmishra have you applied the diff changes ?

shivanirmishra · 2023-07-06T07:14:19Z

@saitama951 Yes, It's working.
Thank you.

saitama951 and others added 2 commits April 20, 2023 15:48

added support for JITDEBUG_INFO

f015afa

added comments

9c1fdd7

saitama951 requested review from vargaz, lambdageek and SamMonoRT as code owners April 20, 2023 14:21

dotnet-issue-labeler bot added the area-Codegen-JIT-mono label Apr 20, 2023

ghost added the community-contribution Indicates that the PR has been added by a community member label Apr 20, 2023

Update mini-runtime.c

bf53413

build-analysis bot mentioned this pull request Apr 20, 2023

[wasm] interpreter timeouts when WebSocket closes unexpectedly #84101

Closed

vargaz reviewed Apr 20, 2023

View reviewed changes

made changes to coding conventions

b8c7554

saitama951 commented Apr 21, 2023

View reviewed changes

saitama951 requested a review from vargaz April 21, 2023 04:52

fanyang-mono reviewed Apr 21, 2023

View reviewed changes

saitama951 added 2 commits April 24, 2023 09:43

Update mini-runtime.c

2abb0f4

Update mini-runtime.c

c68edd6

saitama951 requested a review from fanyang-mono April 24, 2023 04:17

saitama951 added 2 commits April 24, 2023 09:50

Update mini-runtime.c

202c5de

Update mini-runtime.c

4b359cf

janani66 mentioned this pull request May 17, 2023

Perf is not working for JITed code in DotNet dotnet/performance#2988

Closed

fanyang-mono approved these changes May 17, 2023

View reviewed changes

fanyang-mono merged commit a5a485c into dotnet:main May 17, 2023

github-actions bot mentioned this pull request Jun 12, 2023

[release/7.0-staging] [MONO] Add JIT_CODE_DEBUG_INFO record functionality for Jitdump #87428

Merged

fanyang-mono mentioned this pull request Jun 12, 2023

[Mono JIT] Extend Linux perf jitdump functionality #36774

Closed

2 tasks

saitama951 mentioned this pull request Jul 4, 2023

[MONO] Incorrect timestamp for perf events #88373

Closed

saitama951 mentioned this pull request Jul 5, 2023

[MONO] Incorrect timestamps for perf events #88428

Merged

ghost locked as resolved and limited conversation to collaborators Aug 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MONO] Add JIT_CODE_DEBUG_INFO record functionality for Jitdump #85107

[MONO] Add JIT_CODE_DEBUG_INFO record functionality for Jitdump #85107

saitama951 commented Apr 20, 2023 •

edited

Loading

vargaz Apr 20, 2023

saitama951 Apr 21, 2023

saitama951 left a comment •

edited

Loading

saitama951 Apr 21, 2023

fanyang-mono left a comment

fanyang-mono Apr 21, 2023

fanyang-mono Apr 21, 2023

fanyang-mono Apr 21, 2023

fanyang-mono Apr 21, 2023

fanyang-mono Apr 21, 2023

fanyang-mono Apr 21, 2023

saitama951 commented Apr 24, 2023

saitama951 commented May 4, 2023 •

edited

Loading

directhex commented May 17, 2023

shivanirmishra commented May 25, 2023 •

edited

Loading

saitama951 commented May 26, 2023 •

edited

Loading

shivanirmishra commented May 29, 2023

saitama951 commented May 29, 2023

shivanirmishra commented May 31, 2023

saitama951 commented May 31, 2023 •

edited

Loading

shivanirmishra commented Jun 12, 2023

fanyang-mono commented Jun 12, 2023

github-actions bot commented Jun 12, 2023

shivanirmishra commented Jun 13, 2023

saitama951 commented Jun 13, 2023

shivanirmishra commented Jun 13, 2023

fanyang-mono commented Jun 13, 2023

fanyang-mono commented Jun 14, 2023

shivanirmishra commented Jun 20, 2023

saitama951 commented Jul 4, 2023

shivanirmishra commented Jul 4, 2023

saitama951 commented Jul 4, 2023 •

edited

Loading

shivanirmishra commented Jul 6, 2023


		static void add_basic_JitCodeDebug_info(JitCodeDebug *record);

	static void add_basic_JitCodeDebug_info(JitCodeDebug *record);
	static void add_basic_JitCodeDebug_info (JitCodeDebug *record);


		loc = mono_debug_lookup_source_location_by_il(jinfo->d.method,dmji->line_numbers[i].il_offset,NULL);

		if(!(loc)

[MONO] Add JIT_CODE_DEBUG_INFO record functionality for Jitdump #85107

[MONO] Add JIT_CODE_DEBUG_INFO record functionality for Jitdump #85107

Conversation

saitama951 commented Apr 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saitama951 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanyang-mono left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saitama951 commented Apr 24, 2023

saitama951 commented May 4, 2023 • edited Loading

directhex commented May 17, 2023

shivanirmishra commented May 25, 2023 • edited Loading

saitama951 commented May 26, 2023 • edited Loading

shivanirmishra commented May 29, 2023

saitama951 commented May 29, 2023

shivanirmishra commented May 31, 2023

saitama951 commented May 31, 2023 • edited Loading

shivanirmishra commented Jun 12, 2023

fanyang-mono commented Jun 12, 2023

github-actions bot commented Jun 12, 2023

shivanirmishra commented Jun 13, 2023

saitama951 commented Jun 13, 2023

shivanirmishra commented Jun 13, 2023

fanyang-mono commented Jun 13, 2023

fanyang-mono commented Jun 14, 2023

shivanirmishra commented Jun 20, 2023

saitama951 commented Jul 4, 2023

shivanirmishra commented Jul 4, 2023

saitama951 commented Jul 4, 2023 • edited Loading

shivanirmishra commented Jul 6, 2023

saitama951 commented Apr 20, 2023 •

edited

Loading

saitama951 left a comment •

edited

Loading

saitama951 commented May 4, 2023 •

edited

Loading

shivanirmishra commented May 25, 2023 •

edited

Loading

saitama951 commented May 26, 2023 •

edited

Loading

saitama951 commented May 31, 2023 •

edited

Loading

saitama951 commented Jul 4, 2023 •

edited

Loading