Implement lexer

[victor-pogor]

Background and Motivation

Implementing a lexer for PDF is essential for efficiently parsing and analyzing PDF documents. Unlike traditional programming languages, PDF documents have a unique structure and encoding, requiring a specialized lexer to interpret the document's syntax and content accurately.

By creating a dedicated lexer, we can ensure more precise parsing, improve performance, and enhance the maintainability of the PDF codebase, ultimately leading to a better user experience in handling and displaying PDF files.

Acceptance criteria

• The lexer should handle all PDF types and tokens, according to the ISO 32000-1:2008 specification
• The lexer should include the trivia, used later for Lossless syntax trees
• The lexer should have error handling
• Minimal performance improvements are required as part of this story
• Unit Tests
• Mutation Testing

Open questions

• Do we need a quick scanner like Roslyn has?
• How to lex the trivia? Should they be included in tokens or be separated in the syntax tree?
• How do different languages/libs handle the lexing phase?
  • Roslyn returns a full SyntaxToken object that includes text, value, errors, and syntax trivia
  • Swift lexer works in a similar way as Roslyn
  • Rust does not attach the whitespace characters as trivia to tokens, but there was a discussion on that. Rust Analyzer however is implemented like Roslyn or Swift
  • pdf.js has a scannerless parser

Resources

• Adding Matt operator to Roslyn - Syntax, Lexer and Parser
• Swift Syntax and Structured Editing Library
• Swift​Syntax — Written by Mattt
• An overview of SwiftSyntax
• Youtube — try! Swift NYC 2017 - Improving Swift Tools with libSyntax
• Mastering SwiftSyntax