parser: use references instead of smart pointers for Tokens #691
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR does a few things that are inherently tightly coupled unfortunately. The main motivation is to leverage kevinmehall/rust-peg#269 and use
&'input Token<'a>
as theElement
type ofParseElem
instead ofRc<Token<'a>>
.This change allows us to
rust-peg==0.8.0
instead of an unpublished commitRc
while parsing. This turns out to be a significant cost, benchmarks show parsing to be ~33% faster, inflate to be ~65% faster, and overallparse_module
to be ~25% faster.To implement this, a couple of changes were necessary:
Deflated...
, which contains tokenrefs but not whitespace nodes, and as such has two lifetime parametersInflate
trait to be the bridge betweenDeflatedFoo
andFoo
nodes#[cst_node]
proc macro to facilitate all of this without lots of code duplicationclone()
in the grammar implementationTest Plan
libcst_derive