Merge main to tokenizer branch #10781

333fred · 2024-08-22T19:09:55Z

Minor touchup of usings and baselines necessary, no functional changes.

Fixes something. Possibly breaks other things :)

…lerts.

…elease/dev17.8-to-release/dev17.10 * upstream/release/dev17.8: Manually reference Microsoft.IO.Redist 6.0.1 where needed to fix CG alerts. Upgrade testing analyzers to address CG alert.

This is an automatically generated pull request from release/dev17.8 into release/dev17.10. Once all conflicts are resolved and all the tests pass, you are free to merge the pull request. 🐯 ## Troubleshooting conflicts ### Identify authors of changes which introduced merge conflicts Scroll to the bottom, then for each file containing conflicts copy its path into the following searches: - https://github.com/dotnet/razor/find/release/dev17.8 - https://github.com/dotnet/razor/find/release/dev17.10 Usually the most recent change to a file between the two branches is considered to have introduced the conflicts, but sometimes it will be necessary to look for the conflicting lines and check the blame in each branch. Generally the author whose change introduced the conflicts should pull down this PR, fix the conflicts locally, then push up a commit resolving the conflicts. ### Resolve merge conflicts using your local repo Sometimes merge conflicts may be present on GitHub but merging locally will work without conflicts. This is due to differences between the merge algorithm used in local git versus the one used by GitHub. ``` bash git fetch --all git checkout -t upstream/merges/release/dev17.8-to-release/dev17.10 git reset --hard upstream/release/dev17.10 git merge upstream/release/dev17.8 # Fix merge conflicts git commit git push upstream merges/release/dev17.8-to-release/dev17.10 --force ```

Adds a single telemetry point so that we if we receive feedback that something unusual behavior occurs, we can determine if Fuse was engaged or not engaged.

…ocument lookups

…d remove duplicate

…t decides to skip a few things

…nlyList

…x tree for a C# document

This was a subtle future bug, that we would have hit when we removed the C# syntax tree bits in future. By checking for Razor GTD first we weren't deferring to Roslyn for plain attributes (ie, the case without `@bind-` in the tests) which means at best we do some extra unnecessary work, and at worst we could lose features for things Roslyn supports but we don't. I did not intend to find this bug when I started. Sorry for missing it in the PR Dustin. The key is that `ignoreAttributes` should have been `true` when porting over the code, and changing that would have made the test fail in your branch.

See comment in the file, but essentially this is the only way for the code that returns the generated documents syntax tree to be hit in cohosting.

In light of "should we get rid of document context" convo this morning, figured this made sense to do (and I needed _something_ on IDocumentSnapshot in order to actually make this whole idea work)

* Added new tests for the specific regressed scenario. * Allow `@@` sequences. Fixes dotnet/sdk#42730 for now. When we move to the new lexer, these will once again be disallowed; attempting to use runtime compilation will necessitate using the native lexer. * Fixup legacy test baselines

…x tree (dotnet#10765) This PR is a classic self-nerd-snipe, and resolves this comment: dotnet#10750 (comment) Also inadvertently found a slight "bug" with the existing go to def code in cohosting. Bug is in quotes because the actual user behaviour probably was identical in 99% of cases.

* Moving RazorFormattingService to common layer * Cleanup * Switch Razor LSP server code to use RazorFormattingService from common layer * Removing HTML formatting from common layer (for now) It requires services not easily available and will require extra work. For now leaving it in LSP server layer. Current need for the formatting service in cohosting is for OnAutoInsert which doesn't need HTML formatting. * Move AdhocServices, AdhoWorkspaceServices to common layer * Cleanup moved resources * Basic implementations for remote formatting classes * Removing hardcoded option value * Cleanup per CR suggestions * Cleanup doc-comments

# Conflicts: # eng/targets/Services.props # src/Razor/src/Microsoft.AspNetCore.Razor.LanguageServer/RazorLanguageServer.cs # src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/Remote/RazorServices.cs

Fixes dotnet#10689 Roslyn side: dotnet/roslyn#74730 Looks good, despite the Roslyn side having VS deps for images: ![image](https://github.com/user-attachments/assets/44d2ac8d-f7c1-46ec-9572-15bfd91bed28)

This is an automatically generated pull request from release/dev17.11 into release/dev17.12. Once all conflicts are resolved and all the tests pass, you are free to merge the pull request. 🐯 ## Troubleshooting conflicts ### Identify authors of changes which introduced merge conflicts Scroll to the bottom, then for each file containing conflicts copy its path into the following searches: - https://github.com/dotnet/razor/find/release/dev17.11 - https://github.com/dotnet/razor/find/release/dev17.12 Usually the most recent change to a file between the two branches is considered to have introduced the conflicts, but sometimes it will be necessary to look for the conflicting lines and check the blame in each branch. Generally the author whose change introduced the conflicts should pull down this PR, fix the conflicts locally, then push up a commit resolving the conflicts. ### Resolve merge conflicts using your local repo Sometimes merge conflicts may be present on GitHub but merging locally will work without conflicts. This is due to differences between the merge algorithm used in local git versus the one used by GitHub. ``` bash git fetch --all git checkout -t upstream/merges/release/dev17.11-to-release/dev17.12 git reset --hard upstream/release/dev17.12 git merge upstream/release/dev17.11 # Fix merge conflicts git commit git push upstream merges/release/dev17.11-to-release/dev17.12 --force ```

This is an automatically generated pull request from release/dev17.12 into main. Once all conflicts are resolved and all the tests pass, you are free to merge the pull request. 🐯 ## Troubleshooting conflicts ### Identify authors of changes which introduced merge conflicts Scroll to the bottom, then for each file containing conflicts copy its path into the following searches: - https://github.com/dotnet/razor/find/release/dev17.12 - https://github.com/dotnet/razor/find/main Usually the most recent change to a file between the two branches is considered to have introduced the conflicts, but sometimes it will be necessary to look for the conflicting lines and check the blame in each branch. Generally the author whose change introduced the conflicts should pull down this PR, fix the conflicts locally, then push up a commit resolving the conflicts. ### Resolve merge conflicts using your local repo Sometimes merge conflicts may be present on GitHub but merging locally will work without conflicts. This is due to differences between the merge algorithm used in local git versus the one used by GitHub. ``` bash git fetch --all git checkout -t upstream/merges/release/dev17.12-to-main git reset --hard upstream/main git merge upstream/release/dev17.12 # Fix merge conflicts git commit git push upstream merges/release/dev17.12-to-main --force ```

* upstream/main: (71 commits) Fix after merge PR feedback Bump Roslyn version Moving formatting service to common layer (dotnet#10761) Move GetSyntaxTree to document snapshot Inject file path service into the document snapshot Remove code document parameter and just use document snapshot Update NOTICE.txt (dotnet#10768) Allow @@ as a fallback (dotnet#10752) Rework how we get generated documents Directly test the component definition service in cohosting Add missing test case Defer to C# for component attribute GTD in cohosting Allow LSP and cohosting to provide specialized methods to get a syntax tree for a C# document Dev Container (dotnet#10751) Use a proper Try pattern Add tests for co-hosted GTD Rework IDocumentPositionInfoStrategy and use correctly in co-hosted GTD Add DocumentMappingSerice to RazorDocumentServiceBase Move IDocumentPositionInfoStrategy and friends to Workspaces layer ...

333fred · 2024-08-22T19:10:31Z

...re.Razor.Language/test/TestFiles/ParserTests/CSharpBlockTest/EscapedIdentifiers_13.stree.txt

+                    CSharpCodeBlock - [3..36)::33
+                        CSharpStatementLiteral - [3..34)::31 - [ var validationMessage = new { ] - Gen<Stmt>
+                            Whitespace;[ ];
+                            Keyword;[var];


Identifier->Keyword

333fred · 2024-08-22T19:11:33Z

...re.Razor.Language/test/TestFiles/ParserTests/CSharpBlockTest/EscapedIdentifiers_11.stree.txt

+                    CSharpCodeBlock - [3..117)::114
+                        CSharpStatementLiteral - [3..77)::74 - [ var validationMessage = @Html.ValidationMessage(Model.Binding, "", new { ] - Gen<Stmt>
+                            Whitespace;[ ];
+                            Keyword;[var];


Identifier->Keyword

333fred · 2024-08-22T19:11:59Z

src/Compiler/Microsoft.CodeAnalysis.Razor.Compiler/src/Language/Legacy/ParserContext.cs

@@ -4,7 +4,6 @@
 using System;
 using System.Collections.Generic;
 using System.Diagnostics;
-using System.IO;


There was conflict here that needed to be fixed up by removing System.IO.

davidwengier and others added 30 commits August 1, 2024 08:33

Allow anyone to opt in to FUSE

376b86c

Allow anyone to opt in to FUSE (dotnet#10690)

7af4670

Fixes something. Possibly breaks other things :)

Upgrade testing analyzers to address CG alert.

085aa32

Manually reference Microsoft.IO.Redist 6.0.1 where needed to fix CG a…

6353caa

…lerts.

Merge remote-tracking branch 'upstream/release/dev17.8' into merges/r…

47438b5

…elease/dev17.8-to-release/dev17.10 * upstream/release/dev17.8: Manually reference Microsoft.IO.Redist 6.0.1 where needed to fix CG alerts. Upgrade testing analyzers to address CG alert.

Add simple telemetry to initialize endpoint

f65c6b6

Update System.Formats.Asn1 to address CG alert.

f8c1f45

Upgrade System.Security.Cryptography.Xml to avoid CG alerts.

b5c4983

PR feedback

e817d39

Add simple telemetry to initialize endpoint (dotnet#10715)

f9c15e4

Adds a single telemetry point so that we if we receive feedback that something unusual behavior occurs, we can determine if Fuse was engaged or not engaged.

Extract rename functionality to a separate service

17671b6

Cleanups and tweaks

f388156

Create a remote rename service, and dependencies, to perform the rename

47665e0

Sync the ReturnCodeActionAndRenamePathsWithPrefixedSlash flag to OOP

c1e7181

Add rename endpoint

7d514b3

Add component rename test to validate Razor behaviour

fb87026

Move EditMappingService, and create Remote and Lsp versions that do d…

6e0bb70

…ocument lookups

Add C# rename test

3308f4f

Rename TryGetTextDocumentChanges to something better, and cleanup, an…

ee44c75

…d remove duplicate

Fix up tests because sometimes you ask Visual Studio to compile and i…

ef49d71

…t decides to skip a few things

Extract some common code

1297fea

Fix after rebase

4bbedc6

Add CopyTo(Span<T>) extension methods to Utilities.Shared

aedbb64

Add XML doc comments to ImmutableArray Order extensions

027dd04

Add ImmutableArray<T>.Unsafe() extension with in-place order methods

45e64fb

Un-nest ComparerHelper<T> and DescendingComparer<T>

4c0c279

Add IReadOnlyList<T> OrderAsArray extension methods

a3c1375

Add IEnumerable<T> OrderAsArray extension methods

72be234

Add tests to call OrderByArray methods through IEnumerable and IReadO…

4eb9cb7

…nlyList

davidwengier and others added 20 commits August 20, 2024 17:06

Allow LSP and cohosting to provide specialized methods to get a synta…

efceb90

…x tree for a C# document

Add missing test case

ad16b16

Directly test the component definition service in cohosting

a8b4259

See comment in the file, but essentially this is the only way for the code that returns the generated documents syntax tree to be hit in cohosting.

Rework how we get generated documents

7470352

In light of "should we get rid of document context" convo this morning, figured this made sense to do (and I needed _something_ on IDocumentSnapshot in order to actually make this whole idea work)

Update NOTICE.txt (dotnet#10768)

1d3c82c

Remove code document parameter and just use document snapshot

7d5b206

Inject file path service into the document snapshot

53fec86

Move GetSyntaxTree to document snapshot

a3edec0

Bump Roslyn version

0bb4e6e

PR feedback

ca93e0e

Merge remote-tracking branch 'upstream/main' into CohostDocumentSymbol

55c3058

# Conflicts: # eng/targets/Services.props # src/Razor/src/Microsoft.AspNetCore.Razor.LanguageServer/RazorLanguageServer.cs # src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/Remote/RazorServices.cs

Fix after merge

687a473

Cohost Document Symbols (dotnet#10728)

2723187

Fixes dotnet#10689 Roslyn side: dotnet/roslyn#74730 Looks good, despite the Roslyn side having VS deps for images: ![image](https://github.com/user-attachments/assets/44d2ac8d-f7c1-46ec-9572-15bfd91bed28)

333fred requested review from a team as code owners August 22, 2024 19:09

chsienki approved these changes Aug 22, 2024

View reviewed changes

333fred commented Aug 22, 2024

View reviewed changes

333fred enabled auto-merge August 22, 2024 19:12

333fred merged commit 647126b into dotnet:features/roslyn-tokenizer Aug 22, 2024
12 checks passed

333fred deleted the merge-main branch September 19, 2024 21:33

This was referenced Nov 4, 2024

[Automated] PRs inserted in VS build main-35503.21 #11145

Closed

[Automated] PRs inserted in VS build feature.debugger.main-35507.203 #11186

Closed

dotnet-bot mentioned this pull request Nov 19, 2024

[Automated] PRs inserted in VS build feature.debugger.shadowDebug-35518.109 #11227

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge main to tokenizer branch #10781

Merge main to tokenizer branch #10781

333fred commented Aug 22, 2024

333fred Aug 22, 2024

333fred Aug 22, 2024

333fred Aug 22, 2024

Merge main to tokenizer branch #10781

Merge main to tokenizer branch #10781

Conversation

333fred commented Aug 22, 2024

333fred Aug 22, 2024

Choose a reason for hiding this comment

333fred Aug 22, 2024

Choose a reason for hiding this comment

333fred Aug 22, 2024

Choose a reason for hiding this comment