merlin: communicate STDLIB directive

[nojb]

Make it so that dune communicates the standard library directory to merlin. This makes it possible to use merlin with an OCaml compiler other than the one it was compiled with.

Fixes #4188

[nojb]

There is a CI failure which I don't understand, any ideas?

./dune.exe runtest
File "test/blackbox-tests/test-cases/merlin/github4125.t/run.t", line 1, characters 0-0:
         git (internal) (exit 1)
(cd _build/.sandbox/259d57d1d05fbb64c642522c5909613e/default && /usr/bin/git --no-pager diff --no-index --color=always -u ../../../default/test/blackbox-tests/test-cases/merlin/github4125.t/run.t test/blackbox-tests/test-cases/merlin/github4125.t/run.t.corrected)
diff --git a/../../../default/test/blackbox-tests/test-cases/merlin/github4125.t/run.t b/test/blackbox-tests/test-cases/merlin/github4125.t/run.t.corrected
index 8df63f0..1639ec6 100644
--- a/../../../default/test/blackbox-tests/test-cases/merlin/github4125.t/run.t
+++ b/test/blackbox-tests/test-cases/merlin/github4125.t/run.t.corrected
@@ -8,7 +8,7 @@
   $ dune ocaml-merlin --dump-config="$(pwd)" |
   > sed 's#'$(opam config var --switch default prefix)'#OPAM_PREFIX#'
   Foo
-  ((STDLIB OPAM_PREFIX/lib/ocaml)
+  ((STDLIB /usr/lib/ocaml)
    (EXCLUDE_QUERY_DIR)
    (B
     $TESTCASE_ROOT/_build/cross/.foo.objs/byte)
Makefile:73: recipe for target 'test' failed
make: *** [test] Error 1
'make test' failed with code 2

[ghost]

It could be because in the CI ocaml is installed globally while on your machine ocaml was installed by opam. As a result the paths to the stdlib directory differs. You could try rewriting ocamlc -where rather than opam config var prefix.

BTW, instead of sed you can also set the environment variable BUILD_PATH_PREFIX_MAP from inside a .t file to rewrite paths:

  $ export BUILD_PATH_PREFIX_MAP="/OCAML=`ocamlc -where`:$BUILD_PATH_PREFIX_MAP"
  $ ocamlc -where
  /OCAML

[nojb]

It could be because in the CI ocaml is installed globally while on your machine ocaml was installed by opam. As a result the paths to the stdlib directory differs. You could try rewriting ocamlc -where rather than opam config var prefix.

Yes, but if I do that then the test will may/will fail when run in some developer's machine. Perhaps I should try to do both? I will investigate a bit more.

BTW, instead of sed you can also set the environment variable BUILD_PATH_PREFIX_MAP from inside a .t file to rewrite paths:
...

Thanks for the tip. I used sed because this was already used in some other merlin tests.

[nojb]

It could be because in the CI ocaml is installed globally while on your machine ocaml was installed by opam. As a result the paths to the stdlib directory differs. You could try rewriting ocamlc -where rather than opam config var prefix.

I pushed a fix for the test that appears to work always: opam exec --switch default -- ocamlc -where.

[ghost]

I would expect just ocamlc -where to be more reliable actually. Dune will use the ocamlc in the PATH (unless the test has a dune-workspace file that says otherwise). So for Dune the stdlib directory will be the output of ocamlc -where and that is what dune will communicate to merlin.

On the other hand, if the opam switch currently in use is not default (for instance because the developer is using a local opam switch), then opam exec --switch default -- ocamlc -where will return a different result.

[nojb]

(unless the test has a dune-workspace file that says otherwise)

That's actually the case in this particular test :) It uses the default opam switch. Since the rest of the testsuite is run under a non-default switch, the compiler in the path does not match the one of the default switch.

[ghost]

Ah, I see. That seems right then

[ghost]

LGTM. A changelog entry seems relevant for this BTW.

I'm not a merlin expert, but looking at the messages in the linked tickets, that does seem like the right thing to do to me.

[nojb]

Friendly ping @voodoos

[voodoos]

Looks good to me.

I looked at Context.stdlib_dir and it is always populated with the path from the ocamlc -config of the current compiler.

If this is the expected behavior then it's all good (I am asking confirmation because I am not an expert, but this looks like the correct thing to do).

Also it my be good to add a call to merlin in one of the tests to check that it understands the directive as expected, something like:
ocamlmerlin single dump-configuration -filename foo.ml < foo.ml | jq ".value.merlin.stdlib"

[nojb]

Looks good to me.

I looked at Context.stdlib_dir and it is always populated with the path from the ocamlc -config of the current compiler.

If this is the expected behavior then it's all good (I am asking confirmation because I am not an expert, but this looks like the correct thing to do).

Yes, that's the expected behaviour. The point is that the ocamlc in the path may not be the same compiler used to build merlin itself.

Also it my be good to add a call to merlin in one of the tests to check that it understands the directive as expected, something like:
ocamlmerlin single dump-configuration -filename foo.ml < foo.ml | jq ".value.merlin.stdlib"

OK, will do.

[nojb]

Also it my be good to add a call to merlin in one of the tests to check that it understands the directive as expected, something like:
ocamlmerlin single dump-configuration -filename foo.ml < foo.ml | jq ".value.merlin.stdlib"

OK, will do.

I tried doing this, but for some reason I always get the error: "No config found for file \"x.ml\". Try calling dune build.". You can see the output of the ocamlmerlin command in the merlin-stdlib.t test. Any ideas?

[rgrinberg]

Do we backport this to 2.8.3?

[voodoos]

Also it my be good to add a call to merlin in one of the tests to check that it understands the directive as expected, something like:
ocamlmerlin single dump-configuration -filename foo.ml < foo.ml | jq ".value.merlin.stdlib"

OK, will do.

I tried doing this, but for some reason I always get the error: "No config found for file \"x.ml\". Try calling dune build.". You can see the output of the ocamlmerlin command in the merlin-stdlib.t test. Any ideas?

Ok it works when adding a dune-workspace file at the root of the test.
I think the dune ocaml-merlin process started by Merlin ignore the INSIDE_DUNE env variable and try to locate its root which is then outside of the test sandbox.

The best is probably not to test it as it won't be reproducible if Dune is in a larger workspace.

(However it works as expected: "stdlib": "/Users/ulysse/.opam/next-dune/lib/ocaml",)

[nojb]

The best is probably not to test it as it won't be reproducible if Dune is in a larger workspace.

Thanks, got it. I will merge once the CI is green.

[nojb]

Do we backport this to 2.8.3?

I would say yes.