-
Notifications
You must be signed in to change notification settings - Fork 235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix [locate] to look for non-pp source files. #1219
Conversation
This patch resolves #894. I've attached a small dune project that reproduces the problem. Inside bin/main.ml, if you try jumping to definition on |
Note: If you modify |
Disclaimer: I haven't looked at your reproduction case yet. |
@trefis have you had a chance to take a closer look at the repro case? Is a Also, on the dune discord channel @rgrinberg said he expects preprocessing to preserve locations, and from a bit of experimenting with throwing some definitions mixed w/ ppx's into the files I seem to be jumping to the right place. But if there's a better way to approach this I'm happy to revise the patch accordingly. |
This behavior is intended. When dune preprocesses a module, it saves the result into |
@rgrinberg and how does that work w.r.t. error messages locations? Do you introduce line directives in the |
Well behaving preprocessors will insert such directives.
We do if the preprocessor is a ppx. The ppx driver will emit the serialized AST. Sorry for the late reply. Slipped by my inbox. |
Even if it's not, I think a majority of the reasonable preprocessors are able to emit a serialized AST. I finally looked at @ddickstein's example (sorry for the delay!) and there is indeed a I just pushed a (probably incorrectly) minimized version of your reproduction case to master, as a test (which you can build with
I say "probably incorrectly minimized" because it's lacking one level of indirection (:exploding_head: :gun:) so we're probably going to be missing on crucial digest along the way. |
I finally had a look. Sorry for the delay.
Probably because your test doesn't copy the source files to the build directory like dune does. Regarding the PR, it seems like an improvement but it should probably be resilient to the source being absent from the build dir and fail more gracefully. |
This commit addresses the request in ocaml#1219 (comment) It also adds a test analogous to dot-pp-dot-ml.t that builds with dune and demonstrates that the fix works. I'm still unsure what's wrong with the original test, but since it's hand-constructed I'm not sure that it's simulating a real case. Finally, it updates both tests to use different values for x in dep.ml to avoid triggering an unrelated Merlin bug - ocaml#1403 - which should be resolved separately.
8c700d5
to
05e7a56
Compare
This commit addresses the request in ocaml#1219 (comment) It also adds a test analogous to dot-pp-dot-ml.t that builds with dune and demonstrates that the fix works. I'm still unsure what's wrong with the original test, but since it's hand-constructed I'm not sure that it's simulating a real case. Finally, it updates both tests to use different values for x in dep.ml to avoid triggering an unrelated Merlin bug - ocaml#1403 - which should be resolved separately.
05e7a56
to
04eff2b
Compare
The fix doesn't seem to work on MacOS -- not yet sure why. Any advice on how to get the new test to play nicely on Windows? Also, if "A" and "B" in the test are replaced with (), you'll see #1403 |
Update: fixed the issue on MacOS -- now only failing on Windows. |
Not sure what is the issue here, my guess would be that there is an problem with the path starting by
However we don't have a strict policy for Windows testing yet. If patching the test is too complicated you can simply disable it (see #1298). |
Could you add a changelog entry ? |
And you can add your github handle to it, we usually credit explicitly external contributors :-) |
This commit addresses the request in ocaml#1219 (comment) It also adds a test analogous to dot-pp-dot-ml.t that builds with dune and demonstrates that the fix works. I'm still unsure what's wrong with the original test, but since it's hand-constructed I'm not sure that it's simulating a real case. Finally, it updates both tests to use different values for x in dep.ml to avoid triggering an unrelated Merlin bug - ocaml#1403 - which should be resolved separately.
I'm not sure why debian-11-4.13_x86_32 is now failing. Is that architecture test newly added or is the test flaky? |
Looking again at the test I think we should avoid escaping the sandbox. However I tried to rework it but it doesn't work quite as expected. I won't have the time to look more closely for the next week, here is a gist with my attempt if you want to play with it: |
When ppx is used, the cmt sourcefile that is recorded is the *.pp.mli? file, and we compare its digest to the digests of the files in the source directory. Often the .pp.mli? file isn't there, and even when it is we probably do not want to prefer it over the normal .mli? file for locate. This patch checks if the file is a .pp.mli? file, and if so uses the digest of the regular .mli? file instead.
This commit addresses the request in ocaml#1219 (comment) It also adds a test analogous to dot-pp-dot-ml.t that builds with dune and demonstrates that the fix works. I'm still unsure what's wrong with the original test, but since it's hand-constructed I'm not sure that it's simulating a real case. Finally, it updates both tests to use different values for x in dep.ml to avoid triggering an unrelated Merlin bug - ocaml#1403 - which should be resolved separately.
Clean up the test output for MacOS. The $TMPDIR rewrite wasn't working properly because $TMPDIR was /var/folders/... but /var is a symlink to /private/var so the output was appearing as /private$TMPDIR/... The fix in this feature does not seem to work on MacOS - I'm not yet sure why.
At least one problem with the Windows build is that '.' is not recognized as an internal or external command, operable program or batch file. Hopefully by invoking prep.exe with an absolute path instead of as ./prep.exe that problem will be resolved.
When sourcefile is a path it isn't a valid basename for Misc.exact_file_exists. This feature fixes it, which seems to resolve the test failure on MacOS.
The Windows test is failing due to some issue with the usage of $TMPDIR, but I don't have an easy way to test the failure myself. I'm disabling the test for now and someone who can run the Windows workflow locally can investigate.
0e4432d
to
5357fec
Compare
@ddickstein I just force-pushed to your branch to be able to benefit from recent changes to merlin-wrapper. I rewrote the test to stay in the sandbox (that is not using the /tmp directory) and use the dune configuration reader. However the last test is still not giving the expected result. Would you mind reading again the test case to check that I did not add any mistake ? I am wondering if that PR actually solves the issue or not...
|
When this variable is set, $TESTCASE_ROOT appears in the build directory, which causes the search for the source file to fail because the path does not contain "$TESTCASE_ROOT" literally.
The merlin-wrapper changes were needed for MacOS when the test escaped the sandbox. I suspect they are no longer needed.
The debian-11-4.13_x86_32 test generates the following extra output to stderr: ``` $ BUILD_PATH_PREFIX_MAP= dune build + ocamlopt liba/liba.cmxs + /usr/bin/ld: liba/liba.a(liba__Dep.o): warning: relocation in read-only section `.text' + /usr/bin/ld: warning: creating DT_TEXTREL in a shared object + ocamlopt libb/libb.cmxs + /usr/bin/ld: libb/libb.a(libb__Dep.o): warning: relocation in read-only section `.text' + /usr/bin/ld: warning: creating DT_TEXTREL in a shared object ``` The output of `dune build` isn't interesting for the purposes of this test, so we suppress it to accommodate this test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, thank you !
I can merge by squashing everything or if you want you can rebase first and squash some of the commits. Like the iterations on the tests, the back and forth in merlin-wrapper etc.
Squashing everything seems fine to me. Thank you!
…On Tue, Jan 25, 2022 at 10:03 AM Ulysse ***@***.***> wrote:
***@***.**** approved this pull request.
Looks good to me.
I can merge by squashing everything or if you want you can rebase first
and squash some of the commits. Like the iterations on the tests, the back
and forth in merlin-wrapper etc.
—
Reply to this email directly, view it on GitHub
<#1219 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAGD5T43W4TTO6DI7TYVDWDUX23VRANCNFSM4UPNU7KQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
* Fix [locate] to look for non-pp source files. When ppx is used, the cmt sourcefile that is recorded is the *.pp.mli? file, and we compare its digest to the digests of the files in the source directory. Often the .pp.mli? file isn't there, and even when it is we probably do not want to prefer it over the normal .mli? file for locate. This patch checks if the file is a .pp.mli? file, and if so uses the digest of the regular .mli? file instead. When sourcefile is a path it isn't a valid basename for Misc.exact_file_exists. This feature fixes it, which resolves test failures on MacOS. * Adds a test analogous to dot-pp-dot-ml.t that builds with dune and demonstrates that the fix works. In order for the test to work we had to unset BUILD_PATH_PREFIX_MAP= When this variable is set, $TESTCASE_ROOT appears in the build directory, which causes the search for the source file to fail because the path does not contain "$TESTCASE_ROOT" literally. Co-authored-by: Ulysse Gérard <[email protected]>
* Fix [locate] to look for non-pp source files. When ppx is used, the cmt sourcefile that is recorded is the *.pp.mli? file, and we compare its digest to the digests of the files in the source directory. Often the .pp.mli? file isn't there, and even when it is we probably do not want to prefer it over the normal .mli? file for locate. This patch checks if the file is a .pp.mli? file, and if so uses the digest of the regular .mli? file instead. When sourcefile is a path it isn't a valid basename for Misc.exact_file_exists. This feature fixes it, which resolves test failures on MacOS. * Adds a test analogous to dot-pp-dot-ml.t that builds with dune and demonstrates that the fix works. In order for the test to work we had to unset BUILD_PATH_PREFIX_MAP= When this variable is set, $TESTCASE_ROOT appears in the build directory, which causes the search for the source file to fail because the path does not contain "$TESTCASE_ROOT" literally. Co-authored-by: Ulysse Gérard <[email protected]>
CHANGES: Tue Apr 5 20:59:42 CEST 2020 + merlin binary - don't reset the environment when running merlin in single mode so that the parent environement is forwarded the the child processes (ocaml/merlin#1425) - filter dups in source paths (ocaml/merlin#1218) - improve load path performance (ocaml/merlin#1323) - fix handlink of ppx's under Windows (ocaml/merlin#1413) - locate: look for original source files before looking for preprocessed files (ocaml/merlin#1219 by @ddickstein, fixes ocaml/merlin#894) - handle `=` syntax in compiler flags (ocaml/merlin#1409) - expose all destruct exceptions in the api (ocaml/merlin#1437) - fix superfluous break in error reporting (ocaml/merlin#1432) - recognise binding operators in locate and occurrences (ocaml/merlin#1398, @mattiase) - remove dependency on Result (ocaml/merlin#1441, @kit-ty-kate) + editor modes - fix an issue in Neovim where the current line jumps to the top of the window on repeated calls to `MerlinTypeOf` (ocaml/merlin#1433 by @ddickstein, fixes ocaml/merlin#1221) - add module, module type, and class imenu items for emacs (ocaml/merlin#1244, @ivg) - add prefix argument to force or prevent opening in a new buffer in locate command (ocaml/merlin#1426, @panglesd) - add type-on-hover functionality for vim (ocaml/merlin#1439, @nilsbecker) - add a dedicated buffer `*merlin-errors*` containing the last viewed error (ocaml/merlin#1414, @panglesd) + test suite - make `merlin-wrapper` create a default `.merlin` file only when there is no `dune-project` to let tests use `dune ocaml-merlin` reader. (ocaml/merlin#1425) - cover locate calls on module aliases with and without dune - Add a test expliciting the interaction between locate and Dune's generated source files (ocaml/merlin#1444)
CHANGES: Tue Apr 5 21:12:42 CEST 2022 + merlin binary - don't reset the environment when running merlin in single mode so that the parent environement is forwarded the the child processes (ocaml/merlin#1425) - locate: look for original source files before looking for preprocessed files (ocaml/merlin#1219 by @ddickstein, fixes ocaml/merlin#894) - fix handlink of ppx's under Windows (ocaml/merlin#1413) - handle `=` syntax in compiler flags (ocaml/merlin#1409) - fix superfluous break in error reporting (ocaml/merlin#1432) - recognise binding operators in locate and occurrences (ocaml/merlin#1398, @mattiase) - improve load path performance (ocaml/merlin#1323) - remove dependency on Result (ocaml/merlin#1441, @kit-ty-kate) + editor modes - fix an issue in Neovim where the current line jumps to the top of the window on repeated calls to `MerlinTypeOf` (ocaml/merlin#1433 by @ddickstein, fixes ocaml/merlin#1221) - add module, module type, and class imenu items for emacs (ocaml/merlin#1244, @ivg) - add prefix argument to force or prevent opening in a new buffer in locate command (ocaml/merlin#1426, @panglesd) - add type-on-hover functionality for vim (ocaml/merlin#1439, @nilsbecker) - add a dedicated buffer `*merlin-errors*` containing the last viewed error (ocaml/merlin#1414, @panglesd) + test suite - make `merlin-wrapper` create a default `.merlin` file only when there is no `dune-project` to let tests use `dune ocaml-merlin` reader. (ocaml/merlin#1425)
CHANGES: Tue Apr 5 21:17:21 PM CET 2022 + merlin binary - don't reset the environment when running merlin in single mode so that the parent environement is forwarded the the child processes (ocaml/merlin#1425) - locate: look for original source files before looking for preprocessed files (ocaml/merlin#1219 by @ddickstein, fixes ocaml/merlin#894) - fix handling of ppx's under Windows (ocaml/merlin#1413) - handle `=` syntax in compiler flags (ocaml/merlin#1409) - fix superfluous break in error reporting (ocaml/merlin#1432) - recognise binding operators in locate and occurrences (ocaml/merlin#1398, @mattiase) - remove dependency on Result (ocaml/merlin#1441, @kit-ty-kate) + editor modes - update quick setup instructions for emacs (ocaml/merlin#1380, @ScriptDevil) - fix an issue in Neovim where the current line jumps to the top of the window on repeated calls to `MerlinTypeOf` (ocaml/merlin#1433 by @ddickstein, fixes ocaml/merlin#1221) - add module, module type, and class imenu items for emacs (ocaml/merlin#1244, @ivg) - add prefix argument to force or prevent opening in a new buffer in locate command (ocaml/merlin#1426, @panglesd) - add type-on-hover functionality for vim (ocaml/merlin#1439, @nilsbecker) - add a dedicated buffer `*merlin-errors*` containing the last viewed error (ocaml/merlin#1414, @panglesd) + test suite - make `merlin-wrapper` create a default `.merlin` file only when there is no `dune-project` to let tests use `dune ocaml-merlin` reader. (ocaml/merlin#1425)
CHANGES for 414: Tue Apr 5 20:51:42 CEST 2022 + merlin binary - don't reset the environment when running merlin in single mode so that the parent environement is forwarded the the child processes (ocaml/merlin#1425) - filter dups in source paths (ocaml/merlin#1218) - improve load path performance (ocaml/merlin#1323) - fix handlink of ppx's under Windows (ocaml/merlin#1413) - locate: look for original source files before looking for preprocessed files (ocaml/merlin#1219 by @ddickstein, fixes ocaml/merlin#894) - handle `=` syntax in compiler flags (ocaml/merlin#1409) - expose all destruct exceptions in the api (ocaml/merlin#1437) - fix superfluous break in error reporting (ocaml/merlin#1432) - recognise binding operators in locate and occurrences (ocaml/merlin#1398, @mattiase) - remove dependency on Result (ocaml/merlin#1441, @kit-ty-kate) - use the new "shapes" generated by the compiler to perform precise jump-to-definition (ocaml/merlin#1431) + editor modes - fix an issue in Neovim where the current line jumps to the top of the window on repeated calls to `MerlinTypeOf` (ocaml/merlin#1433 by @ddickstein, fixes ocaml/merlin#1221) - add module, module type, and class imenu items for emacs (ocaml/merlin#1244, @ivg) - add prefix argument to force or prevent opening in a new buffer in locate command (ocaml/merlin#1426, @panglesd) - add type-on-hover functionality for vim (ocaml/merlin#1439, @nilsbecker) - add a dedicated buffer `*merlin-errors*` containing the last viewed error (ocaml/merlin#1414, @panglesd) + test suite - make `merlin-wrapper` create a default `.merlin` file only when there is no `dune-project` to let tests use `dune ocaml-merlin` reader. (ocaml/merlin#1425) - cover locate calls on module aliases with and without dune - Add a test expliciting the interaction between locate and Dune's generated source files (ocaml/merlin#1444) CHANGES for 413: Tue Apr 5 20:59:42 CEST 2022 + merlin binary - don't reset the environment when running merlin in single mode so that the parent environement is forwarded the the child processes (ocaml/merlin#1425) - filter dups in source paths (ocaml/merlin#1218) - improve load path performance (ocaml/merlin#1323) - fix handlink of ppx's under Windows (ocaml/merlin#1413) - locate: look for original source files before looking for preprocessed files (ocaml/merlin#1219 by @ddickstein, fixes ocaml/merlin#894) - handle `=` syntax in compiler flags (ocaml/merlin#1409) - expose all destruct exceptions in the api (ocaml/merlin#1437) - fix superfluous break in error reporting (ocaml/merlin#1432) - recognise binding operators in locate and occurrences (ocaml/merlin#1398, @mattiase) - remove dependency on Result (ocaml/merlin#1441, @kit-ty-kate) + editor modes - fix an issue in Neovim where the current line jumps to the top of the window on repeated calls to `MerlinTypeOf` (ocaml/merlin#1433 by @ddickstein, fixes ocaml/merlin#1221) - add module, module type, and class imenu items for emacs (ocaml/merlin#1244, @ivg) - add prefix argument to force or prevent opening in a new buffer in locate command (ocaml/merlin#1426, @panglesd) - add type-on-hover functionality for vim (ocaml/merlin#1439, @nilsbecker) - add a dedicated buffer `*merlin-errors*` containing the last viewed error (ocaml/merlin#1414, @panglesd) + test suite - make `merlin-wrapper` create a default `.merlin` file only when there is no `dune-project` to let tests use `dune ocaml-merlin` reader. (ocaml/merlin#1425) - cover locate calls on module aliases with and without dune - Add a test expliciting the interaction between locate and Dune's generated source files (ocaml/merlin#1444) CHANGES for 412: Tue Apr 5 21:12:42 CEST 2022 + merlin binary - don't reset the environment when running merlin in single mode so that the parent environement is forwarded the the child processes (ocaml/merlin#1425) - locate: look for original source files before looking for preprocessed files (ocaml/merlin#1219 by @ddickstein, fixes ocaml/merlin#894) - fix handlink of ppx's under Windows (ocaml/merlin#1413) - handle `=` syntax in compiler flags (ocaml/merlin#1409) - fix superfluous break in error reporting (ocaml/merlin#1432) - recognise binding operators in locate and occurrences (ocaml/merlin#1398, @mattiase) - improve load path performance (ocaml/merlin#1323) - remove dependency on Result (ocaml/merlin#1441, @kit-ty-kate) + editor modes - fix an issue in Neovim where the current line jumps to the top of the window on repeated calls to `MerlinTypeOf` (ocaml/merlin#1433 by @ddickstein, fixes ocaml/merlin#1221) - add module, module type, and class imenu items for emacs (ocaml/merlin#1244, @ivg) - add prefix argument to force or prevent opening in a new buffer in locate command (ocaml/merlin#1426, @panglesd) - add type-on-hover functionality for vim (ocaml/merlin#1439, @nilsbecker) - add a dedicated buffer `*merlin-errors*` containing the last viewed error (ocaml/merlin#1414, @panglesd) + test suite - make `merlin-wrapper` create a default `.merlin` file only when there is no `dune-project` to let tests use `dune ocaml-merlin` reader. (ocaml/merlin#1425) CHANGES for 411: Tue Apr 5 21:17:21 PM CET 2022 + merlin binary - don't reset the environment when running merlin in single mode so that the parent environement is forwarded the the child processes (ocaml/merlin#1425) - locate: look for original source files before looking for preprocessed files (ocaml/merlin#1219 by @ddickstein, fixes ocaml/merlin#894) - fix handling of ppx's under Windows (ocaml/merlin#1413) - handle `=` syntax in compiler flags (ocaml/merlin#1409) - fix superfluous break in error reporting (ocaml/merlin#1432) - recognise binding operators in locate and occurrences (ocaml/merlin#1398, @mattiase) - remove dependency on Result (ocaml/merlin#1441, @kit-ty-kate) + editor modes - update quick setup instructions for emacs (ocaml/merlin#1380, @ScriptDevil) - fix an issue in Neovim where the current line jumps to the top of the window on repeated calls to `MerlinTypeOf` (ocaml/merlin#1433 by @ddickstein, fixes ocaml/merlin#1221) - add module, module type, and class imenu items for emacs (ocaml/merlin#1244, @ivg) - add prefix argument to force or prevent opening in a new buffer in locate command (ocaml/merlin#1426, @panglesd) - add type-on-hover functionality for vim (ocaml/merlin#1439, @nilsbecker) - add a dedicated buffer `*merlin-errors*` containing the last viewed error (ocaml/merlin#1414, @panglesd) + test suite - make `merlin-wrapper` create a default `.merlin` file only when there is no `dune-project` to let tests use `dune ocaml-merlin` reader. (ocaml/merlin#1425)
When ppx is used, the cmt sourcefile that is recorded is the *.pp.mli? file,
and we compare its digest to the digests of the files in the source directory.
Often the .pp.mli? file isn't there, and even when it is we probably do not
want to prefer it over the normal .mli? file for locate. This patch checks if
the file is a .pp.mli? file, and if so uses the digest of the regular .mli?
file instead.