Pretty printer preserves comments #894

nhuurre · 2021-05-03T17:01:47Z

Another shot at #93 . This is less hacky than #603 and keeps all comments (the comments with "unexpect" placement get pushed a few lines forward).

The strategy here is not to attach the comments to AST nodes but to dump them all into a single list with location metadata for each. Since AST nodes already have location metadata (for reporting semantic errors) the pretty printer can use that to interweave the comments with the code it outputs.

This is draft because unfortunately something goes wrong with #include and I can't figure out why.

Release notes

Correct canonicalizing/auto-formatting of commented Stan programs; comments no longer discarded.

Copyright and Licensing

Copyright holder: Niko Huurre
By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the BSD 3-clause license (https://opensource.org/licenses/BSD-3-Clause)

(still doesn't work)

nhuurre · 2021-06-12T20:00:45Z

Ok, I figured it out. The location metadata coming out the parser is a bit off, the begin_loc of a statement is actually at the end of the previous statement and that means some hacks are required to correctly format the whitespace in between.

Anyway, this is ready for review.
I don't understand what's causing the Jenkins failure but it seems unrelated.

rok-cesnovar · 2021-06-28T07:28:41Z

Sorry Niko, just saw that this has been waiting for 2 weeks+.

@rybern would you be able to have a look at this, given that you already discussed solutions/reviewed in #603? Else I can take a look.

rybern · 2021-06-28T15:18:49Z

Sorry, probably not going to be able to help with this in a timely manner @rok-cesnovar

rok-cesnovar

Looks good to me, I do have a question about all the new newlines (see pretty.expected changes). Are they intentional and could we avoid adding them?

test/integration/good/vec-expr/pretty.expected

WardBrian · 2021-08-26T13:47:04Z

Is any help needed on this issue? I'd love to see it merged so I can start really using the auto formatting ability on my own programs without worrying about comments

…stanc3 into feature/pretty-printer-comments

nhuurre · 2021-08-27T11:32:05Z

I think I resolved the problem with extra lines around local blocks.

Is any help needed on this issue?

Just waiting for a review.

WardBrian · 2021-08-27T13:57:36Z

Do you still want to review this @rok-cesnovar? Otherwise I can take a look

rok-cesnovar · 2021-08-27T14:02:16Z

Go ahead Brian, thanks! I forgot about this one :/ sorry

WardBrian

Looks good to me -- just a few minor comments and a question. Also, looks like you need to make format to make Jenkins happy

src/frontend/lexer.mll

WardBrian · 2021-08-27T14:21:21Z

src/frontend/Pretty_printing.ml

+  let trim_tail col_num lines =
+    match lines with [] -> [] | hd :: tl -> hd :: trim (col_num - 2) tl
+  in
+  Fmt.pf ppf "@[<v>/*%a*/@]" Fmt.(list string) (trim_tail col_num lines)


How difficult would it be to preserve single-line comments as // style? I think this would be nice to have, but if it is too difficult I am OK with this behavior.

Actually, Bob pointed out to me that we have to preserve single line comments.

Consider the valid stan program

// this is a comment */

Converting this from a line comment to a block comment is invalid, and running this branch on it leads to:

Warning: Empty file 'test.stan' detected; this is a valid stan model but likely unintended! Uncaught exception: "Option.value_exn None" Raised at file "src/error.ml" (inlined), line 9, characters 14-30 Called from file "src/option.ml", line 69, characters 4-21 Called from file "src/frontend/Pretty_printing.ml", line 598, characters 33-74 Called from file "src/frontend/Pretty_printing.ml", line 606, characters 2-28 Called from file "src/stanc/stanc.ml", line 190, characters 18-60 Called from file "src/stanc/stanc.ml", line 268, characters 9-16

This needs to be addressed, and we should add said file to the test suite

src/frontend/Pretty_printing.ml

WardBrian · 2021-08-27T15:42:50Z

src/frontend/Pretty_printing.ml

+  let trim_tail col_num lines =
+    match lines with [] -> [] | hd :: tl -> hd :: trim (col_num - 2) tl
+  in
+  Fmt.pf ppf "@[<v>/*%a*/@]" Fmt.(list string) (trim_tail col_num lines)


Actually, Bob pointed out to me that we have to preserve single line comments.

Consider the valid stan program

// this is a comment */

Converting this from a line comment to a block comment is invalid, and running this branch on it leads to:

Warning: Empty file 'test.stan' detected; this is a valid stan model but likely unintended! Uncaught exception: "Option.value_exn None" Raised at file "src/error.ml" (inlined), line 9, characters 14-30 Called from file "src/option.ml", line 69, characters 4-21 Called from file "src/frontend/Pretty_printing.ml", line 598, characters 33-74 Called from file "src/frontend/Pretty_printing.ml", line 606, characters 2-28 Called from file "src/stanc/stanc.ml", line 190, characters 18-60 Called from file "src/stanc/stanc.ml", line 268, characters 9-16

This needs to be addressed, and we should add said file to the test suite

WardBrian · 2021-09-03T13:54:40Z

@nhuurre - would you mind if I take a crack at adapting what you've done to line comments?

nhuurre · 2021-09-03T14:15:48Z

@WardBrian
Sorry, I should have gotten back to this earlier. Feel free try if you want. Pushed a commit with partial progress right now.
I think you need to use Format.pp_force_newline to guarantee a line break after the comment without messing up the formatting but when I tried I got an empty line after every line comment.

WardBrian

Ah nevermind then, that is more or less exactly how I would have implemented it!

It seems like there is a merge issue for the AST with the addition of complex numbers, but that should be a simple fix.

Additionally, can you add the following as test/integration/good/comments.stan, and update the expected test outputs? We should make sure they preserve line comments and that the following program is preserved (except for #)

// this is a line comment */ should stay // /**/

# another one using # - should become //

// how ugly # can */ we make /* this one // # 

/* one line block */

/* a weird 
      ly formated
          block 
                */
                
/**
 * Doc comment
 * @return nothing
 */

 data {
   //nothing here either 
 }

WardBrian · 2021-09-03T14:11:09Z

src/frontend/Pretty_printing.ml

-  Fmt.pf ppf "@[<v>/*%a*/@]" Fmt.(list string) (trim_tail col_num lines)
+  if is_block then
+    Fmt.pf ppf "@[<v>/*%a*/@]" Fmt.(list string) (trim_tail col_num lines)
+  else Fmt.pf ppf "@[<v>/*%a */@]" Fmt.(list string) lines


I believe this line should be

else Fmt.pf ppf "//%a" Fmt.string (List.hd_exn lines)

or similar

Yes, this just produces a block comment at the moment.
A // comment needs a new line at the end.

It seems like newlines are preserved in most cases, it is only a few edge cases where weird things happen, like in test/integration/good/ode_good.stan we have this section:

functions { real[] harm_osc_ode(real t, real[] y, // state real[] theta, // parameters real[] x, // data int[] x_int) { // integer data

which fails in the naive solution I suggested above.

Exactly. That's where you need Format.pp_force_newline. But I'm afraid that knowing if you're in that context requires a large refactoring of the printer code.

I've updated the tests which work for now, leaving out the exceptions so the tests will still fail in CI. This will require some thought, but I think we could try a special case if a comma and a line comment are on the same line?

Edit: Hmm, I suppose you will have a problem if the comment appears inside any set of parenthesis, even if there isn't a comma, like

real f(int x //some comment ) { ...

Maybe the only state you need to track is 'are we inside parenthesis?', but that's not exactly simple either. Another alternative way is to have the lexer preserve newlines/eols in the string it uses to create a LineComment, and then try our best to remove unnecessary newlines later?

The difference is pp_list_of (inside parens) vs pp_list_of_statements (at toplevel). Those don't share as much code as I thought so maybe it's not so bad.

Hopefully it is simpler then. Feel free to ping me if you want help/someone else to look at the problem

…ty-printer-comments

WardBrian

LGTM! This is a really great feature, thanks for all your work!

nhuurre added 8 commits April 29, 2021 22:23

make auto-format preserve some empty lines

6071191

add comments and block locations to AST

acdcbd1

add single-line comments

a71c37f

multiline comments

0edf327

comments in expression lists

86297d9

make sure the custom compare is actually used

58b3863

(still doesn't work)

hack to get correct start locations, dune promote

26582ae

Merge branch 'master' into feature/pretty-printer-comments

6520133

nhuurre marked this pull request as ready for review June 12, 2021 20:00

Merge branch 'master' into feature/pretty-printer-comments

f17af54

rok-cesnovar reviewed Jun 29, 2021

View reviewed changes

test/integration/good/vec-expr/pretty.expected Show resolved Hide resolved

nhuurre added 3 commits August 27, 2021 12:21

Merge branch 'master' into feature/pretty-printer-comments

db36498

$sloc has better start position than $loc

99315eb

Merge branch 'feature/pretty-printer-comments' of github.com:nhuurre/…

283cb89

…stanc3 into feature/pretty-printer-comments

WardBrian reviewed Aug 27, 2021

View reviewed changes

WardBrian requested changes Aug 27, 2021

View reviewed changes

keep line and block comments distinct

7e2442a

WardBrian reviewed Sep 3, 2021

View reviewed changes

WardBrian added 3 commits September 3, 2021 10:32

Update working tests

a397bb8

Merge branch 'master' of github.com:stan-dev/stanc3 into feature/pret…

336c6ce

…ty-printer-comments

Update to latest AST

5186968

WardBrian and others added 2 commits September 3, 2021 13:01

Fix formatting

dbfd9a5

all line comments working, update tests

b758a75

nhuurre requested a review from WardBrian September 7, 2021 06:57

WardBrian linked an issue Sep 7, 2021 that may be closed by this pull request

Store comments on statements in AST for pretty printer #93

Closed

WardBrian approved these changes Sep 7, 2021

View reviewed changes

WardBrian merged commit a1740db into stan-dev:master Sep 7, 2021

WardBrian mentioned this pull request Sep 7, 2021

Store comments in AST #603

Closed

nhuurre deleted the feature/pretty-printer-comments branch September 7, 2021 15:42

WardBrian mentioned this pull request Sep 20, 2021

Run canonicalizer on all models stan-dev/example-models#210

Merged

WardBrian mentioned this pull request Sep 30, 2021

stanc3 online demo for pedantic mode, canonicalizer, auto format, etc #577

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pretty printer preserves comments #894

Pretty printer preserves comments #894

nhuurre commented May 3, 2021

nhuurre commented Jun 12, 2021

rok-cesnovar commented Jun 28, 2021

rybern commented Jun 28, 2021

rok-cesnovar left a comment

WardBrian commented Aug 26, 2021

nhuurre commented Aug 27, 2021

WardBrian commented Aug 27, 2021

rok-cesnovar commented Aug 27, 2021

WardBrian left a comment

WardBrian Aug 27, 2021

WardBrian Aug 27, 2021

WardBrian Aug 27, 2021

WardBrian commented Sep 3, 2021

nhuurre commented Sep 3, 2021

WardBrian left a comment

WardBrian Sep 3, 2021

nhuurre Sep 3, 2021 •

edited

Loading

WardBrian Sep 3, 2021

nhuurre Sep 3, 2021

WardBrian Sep 3, 2021 •

edited

Loading

nhuurre Sep 3, 2021

WardBrian Sep 3, 2021

WardBrian left a comment

Pretty printer preserves comments #894

Pretty printer preserves comments #894

Conversation

nhuurre commented May 3, 2021

Release notes

Copyright and Licensing

nhuurre commented Jun 12, 2021

rok-cesnovar commented Jun 28, 2021

rybern commented Jun 28, 2021

rok-cesnovar left a comment

Choose a reason for hiding this comment

WardBrian commented Aug 26, 2021

nhuurre commented Aug 27, 2021

WardBrian commented Aug 27, 2021

rok-cesnovar commented Aug 27, 2021

WardBrian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WardBrian commented Sep 3, 2021

nhuurre commented Sep 3, 2021

WardBrian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nhuurre Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WardBrian Sep 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WardBrian left a comment

Choose a reason for hiding this comment

nhuurre Sep 3, 2021 •

edited

Loading

WardBrian Sep 3, 2021 •

edited

Loading