profiling streaming TraceLog #2385

vaind · 2023-05-22T15:06:46Z

Addresses multiple issues with our current TraceLog integration:

originally, we processed profiles by writing them to a file and then loading that, processing events and removing - this PR ensures all events are processed as they happen
originally, we started a new DiagnositcsClient profiler session on each transaction - this is very slow (100+ms) so this PR changes the logic to have a single profiler session running all the time and attach/detach our processing logic for each transaction

Additionally, this PR fixes some bugs in the profiler & adds it as a submodule, with a ProjectReference dependency on it. Currently, the submodule points to microsoft/perfview#1867

#skip-changelog because this isn't public-facing

mattjohnsonpint · 2023-05-22T19:16:34Z

Looking forward to getting this in. Thanks so much for your diligence! 🎉

bruno-garcia · 2023-08-20T18:59:12Z

branched from this into this: #2555

commit 6a1df09 Author: Ivan Dlugos <[email protected]> Date: Mon Aug 21 14:19:53 2023 +0200 timeout update commit 833a319 Merge: eaa320b c1e4dad Author: Ivan Dlugos <[email protected]> Date: Mon Aug 21 13:13:51 2023 +0200 Merge branch 'main' into feat/profiling-low-overhead commit eaa320b Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 20:30:09 2023 -0400 debugging on sample commit 1a593bc Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 20:10:34 2023 -0400 fuck this slnf mess commit b5cccee Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 17:53:03 2023 -0400 add profile category commit aa01035 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 16:35:52 2023 -0400 pack profiling commit 20286f4 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 16:35:18 2023 -0400 new full sln file commit 7c7404d Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 15:11:59 2023 -0400 verify commit f1ea1ec Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 14:34:44 2023 -0400 fix tfm in script commit 3a05994 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 14:16:24 2023 -0400 fix benchmarks. Package it commit bec4641 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 13:53:03 2023 -0400 ignore obj commit 791cdc6 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 13:52:01 2023 -0400 fixed sln file with all but mobile commit 915a0f2 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 13:29:28 2023 -0400 sample commit 67ef890 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 12:22:18 2023 -0400 log on collection failure commit 53715f4 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 11:46:30 2023 -0400 set correct platform to profile commit 43f78a0 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 11:20:07 2023 -0400 gitignore on module commit 40b6add Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 11:19:01 2023 -0400 in SentryCore sln commit 1c98746 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 11:14:02 2023 -0400 builds! commit 19e24a2 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 11:01:21 2023 -0400 resolve errors commit 6cb6cd1 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 10:20:55 2023 -0400 fix structure commit b346567 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 09:55:40 2023 -0400 ignore build files from submodule perfview commit 790d774 Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 09:50:11 2023 -0400 internal perfview commit 6e2ae7c Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 09:41:09 2023 -0400 remove profiling from sln/slnf commit 3376bea Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 09:38:10 2023 -0400 perfview under profiling commit aee692b Author: Bruno Garcia <[email protected]> Date: Sun Aug 20 08:54:41 2023 -0400 perfview sampling commit 0bbb0bd Merge: 5ff8f1a b4f9485 Author: Bruno Garcia <[email protected]> Date: Sat Aug 19 18:56:53 2023 -0400 Merge branch 'chore/integrate-traceevent' into feat/profiling-low-overhead commit 5ff8f1a Author: Bruno Garcia <[email protected]> Date: Sat Aug 19 18:54:43 2023 -0400 profiling in core sln commit b4f9485 Author: Ivan Dlugos <[email protected]> Date: Wed Aug 16 20:26:55 2023 +0200 script & project updates commit abc81db Merge: 3a7a10f f457e3a Author: Ivan Dlugos <[email protected]> Date: Wed Aug 16 19:23:29 2023 +0200 Merge branch 'feat/streaming-tracelog' into chore/integrate-traceevent commit 3a7a10f Merge: 4ffb45a d915f98 Author: Stefan Jandl <[email protected]> Date: Wed Aug 16 16:28:53 2023 +0200 Merge branch 'main' into chore/integrate-traceevent commit 4ffb45a Author: Bruno Garcia <[email protected]> Date: Mon Aug 14 15:08:31 2023 -0400 submodule perfview sentry fork commit 162ea2a Author: Ivan Dlugos <[email protected]> Date: Sun Apr 23 23:04:20 2023 +0200 prepare traceevent integration

github-actions · 2023-08-22T17:25:21Z

	Messages
📖	Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against 614ae76

bruno-garcia · 2023-08-22T17:27:01Z

benchmarks/Sentry.Benchmarks/ProfilingBenchmarks.cs

@@ -125,9 +139,9 @@ public void DiagnosticsSessionStartCopyStop(bool rundown, string provider)
    {
        EventPipeProvider[] providers = provider switch
        {
-            "runtime" => new[] { new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Informational, (long)ClrTraceEventParser.Keywords.Default) },
+            "runtime" => new[] { new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Informational, (long)BenchmarkDotNetTransientTraceEvent::Microsoft.Diagnostics.Tracing.Parsers.ClrTraceEventParser.Keywords.Default) },


We'll need to fix this in a follow up PR. Since our customers will experience the same thing. We need to change namespaces, or make stuff internal

related to #2385 (comment)

bruno-garcia · 2023-08-22T17:30:33Z

src/Sentry.Profiling/SampleProfilerSession.cs

+        //                | ThreadTransfer | GCHeapAndTypeNames | Codesymbols | Compilation,
+        new EventPipeProvider(ClrTraceEventParser.ProviderName, EventLevel.Informational, (long) ClrTraceEventParser.Keywords.Default),
+        new EventPipeProvider(SampleProfilerTraceEventParser.ProviderName, EventLevel.Informational),
+        // new EventPipeProvider(TplEtwProviderTraceEventParser.ProviderName, EventLevel.Informational, (long) TplEtwProviderTraceEventParser.Keywords.Default)


This isnt' needed now because there's no file involved?

Suggested change

// new EventPipeProvider(TplEtwProviderTraceEventParser.ProviderName, EventLevel.Informational, (long) TplEtwProviderTraceEventParser.Keywords.Default)

There were some experiments with this but it didn't give that much value in the past. However, the idea is that it would become necessary for profiling individual activities. As is, it cannot be used because the "activity computer" in perfview (that was used to handle these events) isn't yet updated to support streaming event pipe. It would have made the perfview PR too large to hope for a merge so I didn't do it yet. Hoping to get back to it once all this shakes down.

bruno-garcia · 2023-08-22T17:31:51Z

src/Sentry.Profiling/Sentry.Profiling.csproj

    <PackageReference Include="Microsoft.Diagnostics.NETCore.Client" Version="0.2.421201" />
-    <PackageReference Include="Microsoft.Diagnostics.Tracing.TraceEvent" Version="3.1.2" />
+    <ProjectReference Include="../../modules/perfview/src/TraceEvent/TraceEvent.csproj" />


I imagine we'll need to figure out distributing this, also OK on a follow up PR since profilign isn't packaged yet

yes, that was my thinking too

src/Sentry/Internal/FastSerialization/GrowableArray.cs

src/Sentry/Internal/SparseArray.cs

bruno-garcia · 2023-08-22T18:15:01Z

Added 11f2b34 to deal with: microsoft/perfview#1900

If we need more packages from that feed, we need to opt-in by adding the specific name to the packageSource node

bruno-garcia

I'd say this is good to merge, and we have follow-ups. It's still not getting packed for NuGet so we are free to iterate on main

vaind added 15 commits May 4, 2023 22:21

wip: in-memory tracelog

68bafaa

in-memory tracelog support

f6657f5

profiler - remove temp dir

9f8727f

tracelog streaming

3833b5f

fix sample collection

ad5e21f

migrate downsampling & fix tests

1510734

fix: SparseArray

886a2e9

update benchmark & profile builder

2b70a79

minor changes rundown issue investigation

ba3b4e1

Merge branch 'main' into feat/streaming-tracelog

079a72c

separate rundown session during tracelog init

25592ba

fixes

aa3ff4d

Merge branch 'main' into feat/streaming-tracelog

3f7b063

update profiling benchmark results

9a780eb

Merge branch 'main' into feat/streaming-tracelog

61d2718

vaind added 6 commits May 23, 2023 19:31

update tests

cc6119f

Merge branch 'main' into feat/streaming-tracelog

9005697

update tests

f88a20d

fixes

f864e7c

update tests to support streaming tracelog

04d38a5

Merge branch 'main' into feat/streaming-tracelog

8b3691d

vaind mentioned this pull request May 31, 2023

TraceLog streaming/in-memory EventPipe support microsoft/perfview#1867

Merged

vaind added 5 commits June 4, 2023 12:32

Merge branch 'main' into feat/streaming-tracelog

9b51611

Merge branch 'main' into feat/streaming-tracelog

d171989

update to latest perfview

c7f6929

fix test code

5b159e1

Merge branch 'main' into feat/streaming-tracelog

f457e3a

bruno-garcia closed this Aug 20, 2023

Merge branch 'main' into feat/streaming-tracelog

b4693e8

vaind reopened this Aug 21, 2023

vaind added 2 commits August 21, 2023 21:47

perfview submodule setup - unalatered branch

670b6fe

vaind mentioned this pull request Aug 21, 2023

feat: profiling low overhead #2555

Closed

vaind and others added 3 commits August 21, 2023 22:31

chore: update changelog

e89d574

fixup: squash

b7ca7b4

build

33e59c6

bruno-garcia marked this pull request as ready for review August 22, 2023 17:23

bruno-garcia requested review from bitsandfoxes and jamescrosswell as code owners August 22, 2023 17:23

bruno-garcia changed the title ~~WIP: streaming TraceLog~~ profiling streaming TraceLog Aug 22, 2023

Merge branch 'main' into feat/streaming-tracelog

4e4b63e

bruno-garcia reviewed Aug 22, 2023

View reviewed changes

add perfview feed

11f2b34

bruno-garcia and others added 5 commits August 22, 2023 14:22

ms feed pkg

223913a

ms feed pkg

1617ef2

ms feed pkg

0f5fc2f

ms feed pkg

d970fbf

test: profiler snapshot update

12e53e5

bruno-garcia approved these changes Aug 22, 2023

View reviewed changes

fix CI

7a00db9

vaind force-pushed the feat/streaming-tracelog branch from 708d0bd to 7a00db9 Compare August 23, 2023 06:42

vaind added 2 commits August 23, 2023 12:19

update perfview

ba89304

fix CI

614ae76

vaind merged commit 047acad into main Aug 23, 2023

vaind deleted the feat/streaming-tracelog branch August 23, 2023 11:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

profiling streaming TraceLog #2385

profiling streaming TraceLog #2385

vaind commented May 22, 2023 •

edited

Loading

mattjohnsonpint commented May 22, 2023

bruno-garcia commented Aug 20, 2023

github-actions bot commented Aug 22, 2023 •

edited

Loading

bruno-garcia Aug 22, 2023

vaind Aug 22, 2023

bruno-garcia Aug 22, 2023

vaind Aug 22, 2023

bruno-garcia Aug 22, 2023

vaind Aug 22, 2023

bruno-garcia commented Aug 22, 2023

bruno-garcia left a comment

profiling streaming TraceLog #2385

profiling streaming TraceLog #2385

Conversation

vaind commented May 22, 2023 • edited Loading

mattjohnsonpint commented May 22, 2023

bruno-garcia commented Aug 20, 2023

github-actions bot commented Aug 22, 2023 • edited Loading

bruno-garcia Aug 22, 2023

Choose a reason for hiding this comment

vaind Aug 22, 2023

Choose a reason for hiding this comment

bruno-garcia Aug 22, 2023

Choose a reason for hiding this comment

vaind Aug 22, 2023

Choose a reason for hiding this comment

bruno-garcia Aug 22, 2023

Choose a reason for hiding this comment

vaind Aug 22, 2023

Choose a reason for hiding this comment

bruno-garcia commented Aug 22, 2023

bruno-garcia left a comment

Choose a reason for hiding this comment

vaind commented May 22, 2023 •

edited

Loading

github-actions bot commented Aug 22, 2023 •

edited

Loading