Parse tree issue with Python3 grammar

[rlakhwani23]

We are using python3 grammar and the generated parse tree is not correct. The block node is not able to take proper indentation.
For example, statements which are part of one block. It should be present as child nodes of the same block. But in the parse tree, code statements are generated as a child node of the parent block node.

[kaby76]

Please provide which grammar you are using, an example input that demos the problem, and indicate which target you are using. Note, the targets all have different implementations, so there cannot be a generalization from one target to another.

I tried the python/python3 grammar with the CSharp target, and I cannot reproduce the problem as you say.

$ cat hw.py

def hi():
    if True:
        print("hi")
        print("there")

hi()

$ trparse hw.py | trtree
CSharp 0 hw.py success 0.0773933

( file_input
  ( NEWLINE
    (  text:'\r\n' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
  ) ) 
  ( stmt
    ( compound_stmt
      ( funcdef
        ( DEF
          (  text:'def' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
        ) ) 
        ( name
          ( intertoken text:' ' tt:-1
          ) 
          ( NAME
            (  text:'hi' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
        ) ) ) 
        ( parameters
          ( OPEN_PAREN
            (  text:'(' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
          ) ) 
          ( CLOSE_PAREN
            (  text:')' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
        ) ) ) 
        ( COLON
          (  text:':' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
        ) ) 
        ( block
          ( intertoken text:'\r\n  ' tt:-1
          ) 
          ( NEWLINE
            (  text:'  ' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
          ) ) 
          ( INDENT
            (  text:'    ' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
          ) ) 
          ( stmt
            ( compound_stmt
              ( if_stmt
                ( IF
                  (  text:'if' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                ) ) 
                ( test
                  ( or_test
                    ( and_test
                      ( not_test
                        ( comparison
                          ( expr
                            ( atom_expr
                              ( atom
                                ( intertoken text:' ' tt:-1
                                ) 
                                ( TRUE
                                  (  text:'True' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                ) ) ) ) ) ) ) ) ) ) 
                ( COLON
                  (  text:':' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                ) ) 
                ( block
                  ( intertoken text:'\r\n      ' tt:-1
                  ) 
                  ( NEWLINE
                    (  text:'  ' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                  ) ) 
                  ( INDENT
                    (  text:'        ' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                  ) ) 
                  ( stmt
                    ( simple_stmts
                      ( simple_stmt
                        ( expr_stmt
                          ( testlist_star_expr
                            ( test
                              ( or_test
                                ( and_test
                                  ( not_test
                                    ( comparison
                                      ( expr
                                        ( atom_expr
                                          ( atom
                                            ( name
                                              ( NAME
                                                (  text:'print' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                                          ) ) ) ) 
                                          ( trailer
                                            ( OPEN_PAREN
                                              (  text:'(' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                                            ) ) 
                                            ( arglist
                                              ( argument
                                                ( test
                                                  ( or_test
                                                    ( and_test
                                                      ( not_test
                                                        ( comparison
                                                          ( expr
                                                            ( atom_expr
                                                              ( atom
                                                                ( STRING
                                                                  (  text:'\"hi\"' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                                            ) ) ) ) ) ) ) ) ) ) ) ) 
                                            ( CLOSE_PAREN
                                              (  text:')' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                      ) ) ) ) ) ) ) ) ) ) ) ) ) 
                      ( intertoken text:'\r\n      ' tt:-1
                      ) 
                      ( NEWLINE
                        (  text:'  ' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                  ) ) ) ) 
                  ( stmt
                    ( simple_stmts
                      ( simple_stmt
                        ( expr_stmt
                          ( testlist_star_expr
                            ( test
                              ( or_test
                                ( and_test
                                  ( not_test
                                    ( comparison
                                      ( expr
                                        ( atom_expr
                                          ( atom
                                            ( name
                                              ( NAME
                                                (  text:'print' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                                          ) ) ) ) 
                                          ( trailer
                                            ( OPEN_PAREN
                                              (  text:'(' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                                            ) ) 
                                            ( arglist
                                              ( argument
                                                ( test
                                                  ( or_test
                                                    ( and_test
                                                      ( not_test
                                                        ( comparison
                                                          ( expr
                                                            ( atom_expr
                                                              ( atom
                                                                ( STRING
                                                                  (  text:'\"there\"' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                                            ) ) ) ) ) ) ) ) ) ) ) ) 
                                            ( CLOSE_PAREN
                                              (  text:')' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                      ) ) ) ) ) ) ) ) ) ) ) ) ) 
                      ( intertoken text:'\r\n' tt:-1
                      ) 
                      ( NEWLINE
                        (  text:'\r\n' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                  ) ) ) ) 
                  ( DEDENT
                    (  text:'\n' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
          ) ) ) ) ) ) 
          ( DEDENT
            (  text:'\n' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
  ) ) ) ) ) ) 
  ( stmt
    ( simple_stmts
      ( simple_stmt
        ( expr_stmt
          ( testlist_star_expr
            ( test
              ( or_test
                ( and_test
                  ( not_test
                    ( comparison
                      ( expr
                        ( atom_expr
                          ( atom
                            ( name
                              ( NAME
                                (  text:'hi' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                          ) ) ) ) 
                          ( trailer
                            ( OPEN_PAREN
                              (  text:'(' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
                            ) ) 
                            ( CLOSE_PAREN
                              (  text:')' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
      ) ) ) ) ) ) ) ) ) ) ) ) ) 
      ( NEWLINE
        (  text:'\r\n' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
  ) ) ) ) 
  ( EOF
    (  text:'' tt:0 chnl:DEFAULT_TOKEN_CHANNEL
) ) )

$ trparse hw.py | trxgrep ' //if_stmt/block/stmt' | trtext -c
CSharp 0 hw.py success 0.0739169
2
 

The two print statements (as stmt) are nested properly under one block, and the block is under one if_stmt.

I have been updating the python/python3 grammar quite a bit. #3540