Performance: Token Type Utility Analyzer: Avoid allocations

[martin-strecker-sonarsource]

The current implementation of the TokenTypeAnalyzerBase has the following problems:

• It allocates a new TokenClassifierBase per Token
• It does an "OrderBy" after all classification is collected
• It does a span-based text lookup in the tree for each classification

This PR changes how the TokenClassifierBase operates.

Instead of one new TokenClassifierBase per Token, the TokenClassifierBase is now created once per document. The token is passed to ClassifyToken. ClassifyToken was changed

• from adding TokenInfo into the collecting list
• to returning the TokenInfo to the caller

The trivia handling was extracted into TriviaClassifierBase to allow TokenClassifierBase.ClassifyToken() to return a single TokenInfo instead of a list. The trivia is now processed before (LeadingTrivia) and after (TrailingTrivia) the token and so the ordering is guaranteed.

The calculation of the TokenInfo.TextRange was simplified, and the GetSubText call in string.IsNullOrWhiteSpace(token.SyntaxTree.GetText().GetSubText(span).ToString()) check could be removed.

This PR is the groundwork needed to tackle ClassifyIdentifier next, which can be improved by avoiding querying the semantic model in many cases (e.g., an identifier in a parameter syntax is known to be an IParameterSymbol and there is no need to look up the symbol).

[Tim-Pohlmann]

Do we even need the inner class here at all? I think getting rid of it, makes the whole thing much easier to understand and extend.

[martin-strecker-sonarsource]

@pavel-mikula-sonarsource can you please have a second look and merge if appropriate? The reasoning for the change is given in the PR description.

[pavel-mikula-sonarsource]

@martin-strecker-sonarsource can you run the full analysis for Akka.Net and Lucene.Net projects to measure the impact?

[martin-strecker-sonarsource]

This PR is more of a refactoring and a precondition to improving ClassifyIdentifier which queries the semantic model twice for each identifier token (We can easily avoid this per syntactic check in a lot of cases). I only expect this PR to save some allocations because we created a classifier class per token before. The results are within the margin of error.

The results are:
Allocations

	Master	PR	Diff
Akka	8,191.27818 MB	7,672.96046 MB	-6.33%
Lucene	11,316.58781 MB	9,753.20441 MB	-13.81%

Runtime

	Master	PR	Diff
Akka	49.745s	53.301s	+7.12%
Lucene	57.709s	54.049s	-6.5%

A second run of Akka for the PR gave a runtime of 47s (slightly better than the master) with 7,688.16615 MB allocations.

dotnet AnalyzerRunner.dll C:\Projects\sonar-dotnet\analyzers\packaging\binaries C:\Projects\sonar-dotnet\analyzers\its\sources\akka.net\src\Akka.sln /a TokenTypeAnalyzer

dotnet AnalyzerRunner.dll C:\Projects\sonar-dotnet\analyzers\packaging\binaries C:\Projects\DeepSast\lucenenet\Lucene.Net.sln /a TokenTypeAnalyzer

Master
Akka.Net
Found 0 diagnostics in 82145ms (8589177704 bytes allocated)
Statistics for TokenTypeAnalyzer:
Concurrent:                     False
Execution time (ms):            49745,2454

Lucene.Net
Found 0 diagnostics in 97372ms (11866302384 bytes allocated)
Statistics for TokenTypeAnalyzer:
Concurrent:                     False
Execution time (ms):            57709,4617

https://github.com/SonarSource/sonar-dotnet/pull/6785

Akka.Net
Found 0 diagnostics in 91173ms (8045682184 bytes allocated)
Statistics for TokenTypeAnalyzer:
Concurrent:                     False
Execution time (ms):            53301,8113

Lucene.Net
Found 0 diagnostics in 95608ms (10226976072 bytes allocated)
Statistics for TokenTypeAnalyzer:
Concurrent:                     False
Execution time (ms):            54049,4831

[pavel-mikula-sonarsource]

LGTM, that's a very nice change!

[pavel-mikula-sonarsource]

Please run it without AnalyzerRunner to have more realistic view?
What I usually do:
run S4NET begin step
change SonarAnalysisConfig.xml to point to local "release" DLLs.
Run the build 3x to gather the times

[martin-strecker-sonarsource]

Please run it without AnalyzerRunner to have a more realistic view?

There shouldn't be any difference in runtime as the work done is the same as before. We avoid allocations in the hot path, which just reduces the pressure on the garbage collector. The analyzer runner also measures allocations, and we see some reduction there (the allocation measure doesn't fluctuate as much as the runtime does, so the ~10% reduction there is real).
I can still do the S4NET approach, but I don't think we will see significant changes until we get rid of the GetDeclaredSymbol and GetSymbolInfo calls in ClassifyIdentifier. For instance:

MyClass(List<String> list) // constructor

Causes two semantic model calls for each identifier token:

• MyClass
• List
• String
• list

We can easily avoid these in this case by checking the parent:

• MyClass -> token == (token.Parent as ConstructorDeclarationSyntax).Identifier -> Classify as type
• List -> token == (token.Parent as GenericNameSyntax).Identifier -> Must be a type
• String -> token.Parent is IdentifierName { Parent: TypeArgumentListSyntax } -> Must be a type
• list -> token == (token.Parent as ParameterSyntax).Identifier -> Its a parameter name. No classification

That alone saves 2x8 semantic model calls.

[pavel-mikula-sonarsource]

While the memory footprint is good to know too, the purpose is to have realistic data in terms of user experience.

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

96.4% Coverage
11.0% Duplication

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[martin-strecker-sonarsource]

This is just a single run for Lucene only, but it shows no difference in performance for the TokenAnalyzer. As long as the old SE engine is active, we will not see any impact here at all as it is responsible for 75% of the overall performance.

[martin-strecker-sonarsource]

I looked into the code colorization on peach for some projects. I could not find anything unusual.

[pavel-mikula-sonarsource]

Peach deployment failed and this was not deployed on Peach yet