Reerror (#2480)

* split reason_toolchain in a few sub-modules

* Remove menhir error messages

* removed more menhir-related error management

* refactor errors in semantic actions

* REMOVE ME: some notes

* move more error management code to Reason_errors

* recovery annotations

* add recovery tool

* build recovery

* WIP lexer -> declarative_lexer, remove state

* rewrite lexer to isolate state

* factor the deterministic parser out of reason_toolchain

* implement support for POSTFIX

* introduce reason-multi-parser

* Switch on recovery

* add TODO plan

* oops, better to recover more than one token

* fix invalid docstrings merging

* fix DOCSTRING parsing

* fishy printing

* recovery is now optional

* update TODO

* attach errors for intf and impl

* Implement slightly better error messages

* Update reason.opam

* handle EOF

* Use Migrate_parsetree.Ast_404 instead of Ast_404

Modules are being namespaced, direct access is now deprecated and will
be removed (at some unknown point in the future).

* Reenable previous error messages (through Reason_parser_explain)

* detect unclosed parentheses

* TEMPORARY: tweak unclosed parenthesis error message

Still broken, the analysis can be improved

* TEMPORARY: disable JSX error messages on recovery

Recovery generates empty closing tag

* wip: distinguish recovery errors

* insert extension nodes only in recovery mode

* build with 4.02

* instrument reason-error message printer

* raise fatal errors if not using Reason_config.recoverable

* no need for lexer_report_error

* fix reporting of errors

* refactor version-dependent definitions

* move error manipulation to reason_syntax_util

* menhir-recover: hide warnings about recover.cost attribute on production

* fix error reporting (pre 4.08)

* update more tests

* Abstract version-dependent compiler-libs definitions to ocaml_util.ml

Fix build with all versions of OCaml

* update tests

* add fix dependency to esy

* fix tests

PLAN

@@ -0,0 +1,27 @@
+Fix error recovery & error reporting
+
+Step 1: remove existing implementation
+
+OK  * Put reason parser in its own file
+OK  * Remove menhir error management:
+OK    - remove "error" token 
+OK    - remove error messages infrastructure
+OK    Now message defaults to "Syntax error"
+    * Fix reported error location (when recovering or not)
+
+Step 2: reintroduce recovery
+    * Preprocess grammar:
+      - check exhaustivity of recovery
+      - produce a mapping of automaton states to automaton-items suitable for
+        recovery
+    * Instrument parser:
+      - first, always complete the AST and drop user input (":'(")
+      - second, introduce an heuristic for recovering based on location
+
+Step 3: reintroduce error messages
+    * Ask the crowd: 
+        What should messages look like? 
+        Which situations are tricky or counter-intuitive?
+    * Make a testsuite representative of common syntax errors 
+    * ... Design an analysis sufficient to produce the messages automatically
+          :P

esy.json

@@ -5,6 +5,7 @@
   "version": "3.5.1",
   "dependencies": {
     "ocaml": " >= 4.2.0 < 4.10.0",
+    "@opam/fix": "*",
     "@opam/ocamlfind": "*",
     "@opam/menhir": " >= 20170418.0.0",
     "@opam/utop": " >= 1.17.0 < 2.3.0",

formatTest/errorTests/expected_output/reservedField.re

@@ -1,2 +1,2 @@
 File "reservedField.re", line 1, characters 11-15:
-Error: 1054: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead

formatTest/errorTests/expected_output/reservedField.re.4.08.0

@@ -1,5 +1,5 @@
 File "reservedField.re", line 1, characters 11-15:
 1 | let x = {< type >};
                ^^^^
-Error: 1054: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedField.re.4.09.0

@@ -1,5 +1,5 @@
 File "reservedField.re", line 1, characters 11-15:
 1 | let x = {< type >};
                ^^^^
-Error: 1054: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecord.re

@@ -1,2 +1,2 @@
 File "reservedRecord.re", line 1, characters 24-28:
-Error: 1863: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead

formatTest/errorTests/expected_output/reservedRecord.re.4.08.0

@@ -1,5 +1,5 @@
 File "reservedRecord.re", line 1, characters 24-28:
 1 | let foo = { foo: "bar", type: "qux" };
                             ^^^^
-Error: 1863: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecord.re.4.09.0

@@ -1,5 +1,5 @@
 File "reservedRecord.re", line 1, characters 24-28:
 1 | let foo = { foo: "bar", type: "qux" };
                             ^^^^
-Error: 1863: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecordPunned.re

@@ -1,2 +1,2 @@
 File "reservedRecordPunned.re", line 1, characters 22-26:
-Error: 3411: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead

formatTest/errorTests/expected_output/reservedRecordPunned.re.4.08.0

@@ -1,5 +1,5 @@
 File "reservedRecordPunned.re", line 1, characters 22-26:
 1 | let foo = { ...other, type };
                           ^^^^
-Error: 3411: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecordPunned.re.4.09.0

@@ -1,5 +1,5 @@
 File "reservedRecordPunned.re", line 1, characters 22-26:
 1 | let foo = { ...other, type };
                           ^^^^
-Error: 3411: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecordTy

@@ -1,2 +1,2 @@
 File "reservedRecordTypePunned.re", line 1, characters 11-15:
-Error: 770: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead

formatTest/errorTests/expected_output/reservedRecordType.re

@@ -1,2 +1,2 @@
 File "reservedRecordType.re", line 1, characters 11-15:
-Error: 770: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead

formatTest/errorTests/expected_output/reservedRecordType.re.4.08.0

@@ -1,5 +1,5 @@
 File "reservedRecordType.re", line 1, characters 11-15:
 1 | type x = { type: string };
                ^^^^
-Error: 770: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecordType.re.4.09.0

@@ -1,5 +1,5 @@
 File "reservedRecordType.re", line 1, characters 11-15:
 1 | type x = { type: string };
                ^^^^
-Error: 770: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecordTypePunned.re

@@ -1,2 +1,2 @@
 File "reservedRecordTypePunned.re", line 1, characters 11-15:
-Error: 770: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead

formatTest/errorTests/expected_output/reservedRecordTypePunned.re.4.08.0

@@ -1,5 +1,5 @@
 File "reservedRecordTypePunned.re", line 1, characters 11-15:
 1 | type x = { type };
                ^^^^
-Error: 770: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

formatTest/errorTests/expected_output/reservedRecordTypePunned.re.4.09.0

@@ -1,5 +1,5 @@
 File "reservedRecordTypePunned.re", line 1, characters 11-15:
 1 | type x = { type };
                ^^^^
-Error: 770: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
+Error: type is a reserved keyword, it cannot be used as an identifier. Try `type_` or `_type` instead
 

reason.json

@@ -9,6 +9,7 @@
   },
   "dependencies": {
     "ocaml": " >= 4.2.0 < 4.10.0",
+    "@opam/fix": "*",
     "@opam/ocamlfind": "*",
     "@opam/menhir": " >= 20170418.0.0",
     "@opam/merlin-extend": " >= 0.3",

reason.opam

@@ -17,6 +17,7 @@ depends: [
   "ocamlfind"     {build}
   "menhir"        {>= "20170418"}
   "merlin-extend" {>= "0.4"}
+  "fix"
   "result"
   "ocaml-migrate-parsetree"
 ]

src/menhir-recover/Makefile

@@ -0,0 +1,15 @@
+.PHONY: all clean
+
+all:
+	dune build main.exe
+
+clean:
+	dune clean
+
+test-dijkstra:
+	dune build main.exe demo/calc.exe
+	_build/default/main.exe _build/default/demo/parser.cmly
+
+test-custom:
+	dune build main.exe demo/calc.exe
+	_build/default/main.exe -custom _build/default/demo/parser.cmly