Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parser: use references instead of smart pointers for Tokens #691

Merged
merged 4 commits into from
Jun 7, 2022

Conversation

zsol
Copy link
Member

@zsol zsol commented May 28, 2022

Summary

This PR does a few things that are inherently tightly coupled unfortunately. The main motivation is to leverage kevinmehall/rust-peg#269 and use &'input Token<'a> as the Element type of ParseElem instead of Rc<Token<'a>>.
This change allows us to

  • upgrade to rust-peg==0.8.0 instead of an unpublished commit
  • more importantly, it removes the overhead of managing the refcount in Rc while parsing. This turns out to be a significant cost, benchmarks show parsing to be ~33% faster, inflate to be ~65% faster, and overall parse_module to be ~25% faster.

To implement this, a couple of changes were necessary:

  • create a parallel node hierarchy called Deflated..., which contains tokenrefs but not whitespace nodes, and as such has two lifetime parameters
  • change the Inflate trait to be the bridge between DeflatedFoo and Foo nodes
  • implement a #[cst_node] proc macro to facilitate all of this without lots of code duplication
  • remove all calls to clone() in the grammar implementation

Test Plan

  • added some trybuild tests to have basic coverage of various macros in libcst_derive
  • rely on existing tests to prevent regressions

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 28, 2022
@codecov-commenter
Copy link

codecov-commenter commented May 28, 2022

Codecov Report

Merging #691 (81a47ad) into main (f3811a0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #691   +/-   ##
=======================================
  Coverage   94.82%   94.82%           
=======================================
  Files         246      246           
  Lines       25653    25653           
=======================================
  Hits        24325    24325           
  Misses       1328     1328           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3811a0...81a47ad. Read the comment docs.

@zsol zsol force-pushed the native/two-lifetimes branch from d93246a to 81a47ad Compare May 29, 2022 09:31
@zsol zsol merged commit 380f045 into Instagram:main Jun 7, 2022
@zsol zsol deleted the native/two-lifetimes branch June 7, 2022 10:08
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Sep 16, 2022
0.4.7 - 2022-07-12

Fixed
* Fix get_qualified_names_for matching on prefixes of the given name by @lpetre in Instagram/LibCST#719

Added
* Implement lazy loading mechanism for expensive metadata providers by @Chenguang-Zhu in Instagram/LibCST#720


0.4.6 - 2022-07-04

New Contributors
- @superbobry made their first contribution in Instagram/LibCST#702

Fixed
- convert_type_comments now preserves comments following type comments by @superbobry in Instagram/LibCST#702
- QualifiedNameProvider optimizations
  - Cache the scope name prefix to prevent scope traversal in a tight loop by @lpetre in Instagram/LibCST#708
  - Faster qualified name formatting by @lpetre in Instagram/LibCST#710
  - Prevent unnecessary work in Scope.get_qualified_names_for_ by @lpetre in Instagram/LibCST#709
- Fix parsing of parenthesized empty tuples by @zsol in Instagram/LibCST#712
- Support whitespace after ParamSlash by @zsol in Instagram/LibCST#713
- [parser] bail on deeply nested expressions by @zsol in Instagram/LibCST#718


0.4.5 - 2022-06-17

New Contributors

-   @zzl0 made their first contribution in Instagram/LibCST#704

Fixed

-   Only skip supported escaped characters in f-strings by @zsol in Instagram/LibCST#700
-   Escaping quote characters in raw string literals causes a tokenizer error by @zsol in Instagram/LibCST#668
-   Corrected a code example in the documentation by @zzl0 in Instagram/LibCST#703
-   Handle multiline strings that start with quotes by @zzl0 in Instagram/LibCST#704
-   Fixed a performance regression in libcst.metadata.ScopeProvider by @lpetre in Instagram/LibCST#698


0.4.4 - 2022-06-13

New Contributors

-   @adamchainz made their first contribution in Instagram/LibCST#688

Added

-   Add package links to PyPI by @adamchainz in Instagram/LibCST#688
-   native: add overall benchmark by @zsol in Instagram/LibCST#692
-   Add support for PEP-646 by @zsol in Instagram/LibCST#696

Updated

-   parser: use references instead of smart pointers for Tokens by @zsol in Instagram/LibCST#691
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants