Make Yi syntax-aware

[ethercrow]

Original author: [email protected] (June 21, 2007 15:39:47)

We want yi to be "syntax-aware"

The (partial) parse tree should be available for

• syntax highlighting
• auto-indent
• checking syntax
• maybe syntax-directed edition

Here is what the TODO file used to say about ideas on how to do that:

    - run a parser in background thread (of Parser class), 
      producing a syn tree

    - abstract syn representation
        - produces syntax colouring ppr (not general ppr)
        - hints for layout (on \n, for example)

    - so we're mostly direct editing, and updating the syn tree
      where possible.

    - so how to take a syn tree and flatten it to produce a 
      drawable buffer?
        - can we deparse it directly?   
        - or can we consult the abs syn tree, as we draw 
          each buffer line?
            - walk the parse tree, drawing chars from the buffer in
              the colour specified by the tree.

    - need correct position information in syn tree, to colour
      tokens. comments aren't stored in the syn tree, so how do we
      colour them?
        * Centaur records comments in the lexer, with their locations.
          Then using location information in the syntree, hook
          comments back in.
      - what about comments?
          - record the comments, with their positions, in the lexer.
          - re-splice comments back into the parse tree.

    - set of token classes?

    - need multiple entry points to the parser, to reparse
      subexpressions?
        - ==> Centaur (for ML) has a good system.

ftp://ftp.inria.fr/INRIA/publication/publi-pdf/RR/part3/RR-3139.pdf

        - alredy available in Happy parser for GHC, via the %name
          directive, for:
            - parseModule module
            - parseStmt   maybe_stmt
            - parseIdentifier  identifier
            - parseIface iface
            - parseType ctype

    - can keep track of points to start reparsing from. see
      vim:syntax.c- syntax state stack cache

    - need to map tokens to attributes.

- good overview to structure editors
         http://www.cs.uu.nl/research/projects/proxima/

See also the references provides by Thomas Schilling

http://www.emacswiki.org/cgi-bin/wiki/ParEdit
http://wiki.opendylan.org/wiki/view.dsp?title=AppleDylanScreenshots

The Deuce source code is available online as part of the functional
developer source code:

http://www.opendylan.org/cgi-bin/viewcvs.cgi/trunk/fundev/sources/deuce/

http://harmonia.cs.berkeley.edu/papers/twagner-parsing.pdf

Original issue: http://code.google.com/p/yi-editor/issues/detail?id=25

[ethercrow]

From [email protected] on June 29, 2007 12:36:16
Other references:

• Generalized LR Parsing in Haskell
  wiki.di.uminho.pt/twiki/pub/Research/PURe/PurePublications/PUReTR041101.pdf
• There and back again
  portal.acm.org/citation.cfm?id=1088348.1088357
• Parsek
  http://www.cs.chalmers.se/Cs/Grundutb/Kurser/afp/code/week3/Parsek.hs

[ethercrow]

From [email protected] on July 25, 2007 18:48:36
I just pushed an incremental parser prototype.

(sketches/IncrementalParser.hs)

Please review/comment :)

[ethercrow]

From [email protected] on July 26, 2007 19:25:41
Looks very interesting. Applicative parsers are probably easier to debug and have
the advantage that they don't require an external language, however I haven't yet
read the paper. I think the most important questions are:

• Can we sufficiently approximate Haskell's grammar?

  One tricky part are operator definitions, but I think we can handle them with a
  pre-pass and when an operator definition changes we just have to invalidate the whole
  file and dependent modules, so that they get re-parsed when required.

• How much overhead is needed to store intermediate results?

  Haskell programs consist primarily of top-level definitions and we store the parse
  tree anyways in memory, we can simply restart parsing at the current top-level
  definition or where/let-definition. That should be pretty low overhead.

Good work! I like the light-weight/K.I.S.S. approach.

[ethercrow]

From [email protected] on April 11, 2008 10:23:39

• syntax-aware highlighting is now supported (see cleverHaskellMode)
• syntax-dependent functions can be added to the Mode and called with "withSyntax"