Add the ability to quote token streams
#5284
Labels
enhancement
New feature or request
Comments
|
Since token streams are essentially a list of tokens I'm thinking it'd be useful to expose them directly as such. Instead of a |
5 tasks
github-merge-queue bot
pushed a commit
that referenced
this issue
Jun 24, 2024
# Description ## Problem\* Resolves #5284 ## Summary\* Adds a new `Quoted` type which is the new default output of a `quote { ... }` expression. In addition, a `quote` expression now quotes an unparsed, raw vector of tokens terminated by the matching closing brace. This has a number of advantages: - Quoting code fragments: Since these token streams are not parsed, there is almost no restriction on their grammar. Code fragments such as `quote { ) ( }` are valid as long as they're grammatically valid by the time the macro is unquoted at its call site. The only restriction on the grammar of these is that they cannot have an unmatched `}` since that would close the `quote` expression itself. Matched `{` and `}` paris however, are fine: `quote { look, a block: { foo } }`. - Since we don't need to parse to a specific grammar rule, we don't need a new quoted type for each grammar rule we quote (see the existing `Expr`, `Type`, and `TopLevelItem` types). - The implementation details are simpler (e.g. this PR net removes lines). The current plan is to keep the other macro types (`Expr`, `Type`, `TopLevelItem`, etc) as optional outputs of `quote` only if they're specified after the keyword, e.g. `quote Type { Foo<A, B> }`. Since the new approach of using token streams by default is much more flexible, there's less of a need for these however. ## Additional Context The formatting of quote expressions has also been fixed. This was a necessary change since these expressions hold token streams now instead of ASTs and thus can't be formatted the old way. Edit: This does not fix the existing issue with formatting comptime blocks, only `quote` blocks which were similarly affected. ## Documentation\* Check one: - [ ] No documentation needed. - [ ] Documentation included in this PR. - [x] **[For Experimental Features]** Documentation to be submitted in a separate PR. # PR Checklist\* - [x] I have tested the changes locally. - [x] I have formatted the changes with [Prettier](https://prettier.io/) and/or `cargo fmt` on default settings. --------- Co-authored-by: Tom French <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
Quoting is currently impossible for code fragments which are only part of a larger expression.
Consider
quote { foo: 1 + 1 }which would currently fail sincefoo: 1 + 1is only valid as part of a constructor initializer expression's inner field expressions. The current approach to getting this working would require we introduce a new quoted type for each parser rule we want to support this way, which is infeasible.Happy Case
Add the ability to quote token streams which aren't parsed until they're inserted at a macro's call site. This way users can quote any snippet of code, even ones that are temporarily invalid until they are fixed later or combined with other snippets.
See #4594 (comment) for more details
Project Impact
None
Impact Context
No response
Workaround
None
Workaround Description
No response
Additional Context
No response
Would you like to submit a PR for this Issue?
None
Support Needs
No response
The text was updated successfully, but these errors were encountered: