Fix opam install on a local directory not updating pinned packages' metadata #6209

kit-ty-kate · 2024-09-24T21:02:50Z

Fixes #5567
Fixes #4507
Queued on #6438

I have no idea why for_view was introduced in 9b2a67f but i've tested opam show and i'm not seeing any breaking nor does the testsuite.

The bypass here was causing local packages to be put in the already_pinned set. However this set is only used to update the <switch>/.opam-switch/sources/<pkg> directory, not the <...>/overlay directory as the assumption with this set is that the package description hasn't change.
Removing that bypass makes the function much more consistent.

The reinstall bit is a bit easier to understand: when using --deps-only or --show-action, the codepath goes through simulate_local_pinning which didn't take into account that some packages being pinned are already pinned. In that case the reinstall field must be updated to ensure the rest of the code knows that package should be reinstalled.

rjbou

I reviewed commit per commit, it's better to look at comments per commit.
I'm not sure about the third commit (reinstall packages), it raises a good question. we want to update the opam file internally, but do we want to upgrade the package itself? Usually, when users use --deps-only they don't want to launch an install of given package (no need, or don't want to double compile it). When the package is already installed, there is the "we want to have a consistent state" that leads us to want to upgrade the package. From CLI perspective, it is more consistent to not upgrade the package.
Maybe a note or warning to highlight the situation package updated internally, if you really want to reinstall it, launch opam install pkg is better than a full reinstall triggered by --deps-only.

tests/reftests/pin.test

src/client/opamAuxCommands.ml

tests/reftests/lock.test

src/client/opamAuxCommands.ml

tests/reftests/pin.test

src/client/opamAuxCommands.ml

master_changes.md

src/client/opamAuxCommands.ml

tests/reftests/pin.test

rjbou · 2024-10-24T17:11:49Z

tests/reftests/pin.test

+### ::: behaviour when the package is not pinned
+### opam install ./pin-change
+pin-change is now pinned to git+file://${BASEDIR}/pin-change#master (version dev)
+[NOTE] Ignoring uncommitted changes in ${BASEDIR}/pin-change (`--working-dir' not specified or specified with no argument).


If we have a more fine grained handling of pin update (opam file vs source update), we can even remove that note if only the uncommitted opam file is updated, as now we have the information.

same as with https://github.com/ocaml/opam/pull/6209/files#r1815454356 i believe. I think those two improvements can be their own tickets to be fixed later.

src/client/opamAuxCommands.ml

kit-ty-kate · 2025-03-14T22:00:12Z

rebased on master

rjbou · 2025-03-27T15:38:06Z

I'm not sure about the third commit (reinstall packages), it raises a good question. we want to update the opam file internally, but do we want to upgrade the package itself? Usually, when users use --deps-only they don't want to launch an install of given package (no need, or don't want to double compile it). When the package is already installed, there is the "we want to have a consistent state" that leads us to want to upgrade the package. From CLI perspective, it is more consistent to not upgrade the package.

We need to upgrade the package to keep in sync the switch.

rjbou · 2025-03-27T15:52:54Z

src/client/opamAuxCommands.ml

             OpamSwitchState.opam_opt st pinned_pkg,
             OpamFile.OPAM.read_opt nf.pin.pin_file
           with
           | Some opam0, Some opam ->
+             (* Keep synchronised with [OpamFile.OPAM.pp_raw_fields] added


i'm wondering if it would be better to have a function in OpamFile.OPAM that does all these additions in order to compare (or just returns the new opam file?), like that it's easier to keep synchronised as they would be in the same module / near in the code.

I've added two fixup commits (to cleanup of course) exploring that idea.
The first one rewrites the comparison to be more efficient (2 read instead of 4), and more correct.
In particular the available field was bugging me. The pp_minimal_opam_version function would rewrite the available field but i can't find any good reason for this.

This function was originally added in 1d32477 + ca84c3a but i can't find any reasoning for this change aside from "consistency". I personally think this makes the whole thing less consistent by introducing differences between original file and rewritten files, which is nearly impossible to detangle.

For this reason i chose to remove this part of the code which makes for a needed simplification, in my opinion.
What do you think?

I extracted the OpamFile.OPAM function extraction idea to #6446 as i think we need more time to think about this

rjbou · 2025-04-08T14:06:54Z

tests/reftests/pin.test

@@ -1068,3 +1068,73 @@ opam-version: "2.0"

 No solution found, exiting
 # Return code 20 #
+### :C:e: Make sure opam install works when the opam file changes and the package is pinned


Depending on the order of merge, these header may need a change in the last rebase (:C:e: is also used in another PR)

master_changes.md

src/client/opamAuxCommands.ml

rjbou · 2025-04-08T14:12:25Z

src/client/opamAuxCommands.ml

+          | Some opam0 ->
+            (match OpamFile.OPAM.read_opt nf.pin.pin_file with
+             | Some opam ->
+               let opam = OpamFile.OPAM.with_locked_opt nf.pin.pin_locked opam in


Commit 4494369 lost in the PR split

It was discarded on purpose as to me #6438 removes the actual change made to the opam definition that we have to think about. But i've readded it, i don't mind either way.

…l opam file changes while being pinned

…ected and ensure showing local opam files is working as expected

…ages' metadata The reinstall field from the switch state wasn't updated according to the implicit pin and thus opam didn't think there was anything to do.

kit-ty-kate added KIND: BUG AREA: ENGINE labels Sep 24, 2024

kit-ty-kate added this to the 2.4.0~alpha1 milestone Sep 24, 2024

kit-ty-kate mentioned this pull request Sep 24, 2024

opam install . --deps-only --working-dir incorrectly reports "Nothing to do" #5567

Closed

kit-ty-kate requested a review from rjbou September 24, 2024 21:08

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from ce39103 to e8fc621 Compare September 24, 2024 21:12

kit-ty-kate mentioned this pull request Oct 17, 2024

opam install --deps-only --locked has nothing to do or installs wrong version #6248

Closed

rjbou requested changes Oct 22, 2024

View reviewed changes

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch 4 times, most recently from 3a911c1 to a95f45f Compare October 22, 2024 21:18

kit-ty-kate requested a review from rjbou October 22, 2024 21:19

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from a95f45f to 5e531b5 Compare October 22, 2024 21:56

kit-ty-kate mentioned this pull request Oct 22, 2024

Ensure the right versions of packages are used when simulating pinning #6256

Merged

rjbou reviewed Oct 24, 2024

View reviewed changes

This was referenced Nov 14, 2024

with-test deps not installed after second call to opam install #4507

Closed

On install action, reinstall if test/doc dependencies are not present #4514

Draft

kit-ty-kate changed the title ~~Fix "opam install <local_dir>" not updating pinned packages' metadata~~ Fix opam install on a local directory not updating pinned packages' metadata Dec 16, 2024

kit-ty-kate added the IMPORTANT label Feb 9, 2025

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from fd9e61e to ffc0d4f Compare February 25, 2025 21:58

kit-ty-kate requested a review from rjbou February 25, 2025 21:59

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from ffc0d4f to f5b80ec Compare March 14, 2025 21:59

rjbou force-pushed the fix-opam-install-local-pinned branch from f5b80ec to 294713e Compare March 27, 2025 15:51

rjbou reviewed Mar 27, 2025

View reviewed changes

kit-ty-kate mentioned this pull request Mar 27, 2025

pin: only display a "is now pinned to" message if the package is not already pinned #6437

Draft

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from 294713e to e3b74f6 Compare March 28, 2025 03:59

kit-ty-kate mentioned this pull request Mar 28, 2025

Stop modifying the available field when handling the builtin x-* fields and enable lint E63 instead #6438

Merged

kit-ty-kate added PR: QUEUED Pending pull request, waiting for other work to be merged or closed and removed PR: QUEUED Pending pull request, waiting for other work to be merged or closed labels Mar 28, 2025

kit-ty-kate mentioned this pull request Mar 31, 2025

Move the equality check between a pinned package and its upstream to OpamFile.OPAM.effectively_equal_modulo_pin #6446

Draft

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from e3b74f6 to f50852d Compare March 31, 2025 11:44

kit-ty-kate requested a review from rjbou March 31, 2025 11:44

rjbou approved these changes Apr 8, 2025

View reviewed changes

rjbou added PR: NEEDS UPDATE STATE: READY TO MERGE labels Apr 8, 2025

kit-ty-kate and others added 4 commits April 8, 2025 18:38

reftest: add a test showing the behaviour of opam install when a loca…

16fc979

…l opam file changes while being pinned

reftest: add pinned overlay opam files test

ef6f3cc

reftest: add a test for opam show to highlight which opam file is sel…

0d6f546

…ected and ensure showing local opam files is working as expected

Fix "opam install <local_dir>" not updating pinned packages' metadata

6191b55

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from f50852d to e3f9de0 Compare April 8, 2025 17:50

Fix "opam install --deps/--show <local_dir>" not updating pinned pack…

3b8a10e

…ages' metadata The reinstall field from the switch state wasn't updated according to the implicit pin and thus opam didn't think there was anything to do.

kit-ty-kate force-pushed the fix-opam-install-local-pinned branch from e3f9de0 to 3b8a10e Compare April 8, 2025 17:59

kit-ty-kate removed the PR: NEEDS UPDATE label Apr 8, 2025

kit-ty-kate merged commit 1458aa7 into ocaml:master Apr 8, 2025
44 checks passed

kit-ty-kate deleted the fix-opam-install-local-pinned branch April 8, 2025 19:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix opam install on a local directory not updating pinned packages' metadata #6209

Fix opam install on a local directory not updating pinned packages' metadata #6209

kit-ty-kate commented Sep 24, 2024 •

edited

Loading

rjbou left a comment

rjbou Oct 24, 2024

kit-ty-kate Oct 24, 2024

rjbou Mar 27, 2025

kit-ty-kate commented Mar 14, 2025

rjbou commented Mar 27, 2025

rjbou Mar 27, 2025

kit-ty-kate Mar 28, 2025

kit-ty-kate Mar 31, 2025

rjbou Apr 8, 2025

rjbou Apr 8, 2025

kit-ty-kate Apr 8, 2025

Fix opam install on a local directory not updating pinned packages' metadata #6209

Fix opam install on a local directory not updating pinned packages' metadata #6209

Conversation

kit-ty-kate commented Sep 24, 2024 • edited Loading

rjbou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kit-ty-kate commented Mar 14, 2025

rjbou commented Mar 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kit-ty-kate commented Sep 24, 2024 •

edited

Loading