Package Archiving: Plan

Philosophy

The opam-repository is steadily growing, using a substantial amount of space and inodes. Yet a lot of packages have become stale or uninstallable. We could slim it significantly and reduce power and time waste in the CI and in local solves, by periodically pruning the uninstallable and unmaintained packages.

Plan

Phase 0

• Write and agree on the new policy for opam-repository

  • Reach out the infra team to make sure they are ok with the proposed document #ci
  • Publish it on the opam-repository wiki
  • Write a discuss post to announce the new policy (with a roadmap) (follow up from https://discuss.ocaml.org/t/discussions-on-the-future-of-the-opam-repository/13898): https://docs.google.com/document/d/1jjD-COUOkJPlvakbce0izc4ljNr8CO9cI1TJAgEEg-o/edit?tab=t.0 now at https://pad.data.coop/yxLY_y3IRKKhwM24WzPFrQ#
  • Publish the post on discuss https://discuss.ocaml.org/t/proposed-package-archiving-policy-for-the-opam-repository/15713/7
  • advertise the post on discord, mailing-list, ocaml-weekly-news, … https://ocaml.org/community
  • write the motivation for the change in the discuss post

• make sure everyone is using a version of opam >= to 2.1.6 or 2.2.0

Phase 1 (January 1st 2025)

Ready to implement

• Create the ocaml/opam-repository-archive github repository
  • Create a repo file
  • Create a README.md see https://github.com/ocaml/opam-repository-archive/pull/1
    • explaining the role of the repo
    • explaining the uses of the repo (opam repository add …)
    • explaining the additional x- fields in the files
    • explaining that files in here may be modified to improve installability and such
• Make a list of known unavailable or broken packages using opam-repo-ci, opam-health-check and opam admin check --installable (archive-opam --installable --pkg-all)
  • make a PR removing them from opam-repository https://github.com/ocaml/opam-repository/pull/27100
  • make a PR adding them to opam-repository-archive https://github.com/ocaml/opam-repository-archive/pull/3
• write a Discuss post listing all the packages that are going to be removed and wait a week for comments https://discuss.ocaml.org/t/opam-repository-archival-phase-1-unavailable-packages/15797/2
  • in the post, explain the problem with (g)patch (see Phase 0)
  • Push a new before-opam-repository-archive tag on opam-repository before any of the removal
  • Merge the PRs if the discussion above is ok

Preliminaries for Phase 2 from infra team #CI

• Create a CI for this new repository
  • it should simply lint the files
    • checks that every dependency constraint is of the form "dep" {... & <= "<latest version of dep available in opam-repository at the time of the PR or lower>"} except for "ocaml" (needed for phase 2)
    • checks if the x-reason-for-archival is set (needed for phase 3)
    • checks if the x-opam-repository-commit-hash-at-time-of-archival is set (needed for phase 3)
• Add the archive repo to the health-check but run it more infrequently (~1/month)

Phase 2 (OCaml < 4.08 -- the least aggressive pruning) - aim: 1 month from Phase 1 (February 1st 2025)

• Call opam admin check --installability archive-opam --installable --pkg-all, make sure the output is correct and move all the packages it lists to opam-repository-archive #opam
• Make a PR to move all ocaml packages < 4.08 (oldest version still in use in the wild according to https://repology.org/project/ocaml/versions) to the opam-repository-archive
• Add a check to the CI lint of the opam-repository: the supported OCaml version is >= 4.08 #ci
• Announce it in a Discuss post and wait a week for comments then merge

The above steps will be repeated each time we go on with this point with a more recent version of the compiler as bound.

Phase 3 (spring cleaning) - aim: March 1st 2025

• Write a Discuss post to advertise the extension fields for maintainers to declare intention of maintaining for their packages (advertised in https://discuss.ocaml.org/t/opam-repository-archival-phase-1-unavailable-packages/ and https://discuss.ocaml.org/t/proposed-package-archiving-policy-for-the-opam-repository/)
  • for usability, it is sufficient to add the field to the latest version of a package and then the tooling will resolve this
  • in case of multiple values for this field in multiple versions of a package, the latest version is taken into account
  • The idea is that only the packages with the x-maintenance-intent will be considered for removal when the times come, the others will stay
  • Syntax for the new x-maintenance-intent field: take a look into https://github.com/ocaml/opam-repository/blob/master/governance/policies/archiving.md#specification-of-the-x--fields-used-in-the-archiving-process for details
  • Phase 3 is planned to happen on March 1st 2025. Make sure that this proposal does allow to write any type of pattern that may apply to your packages and let us know if not, this is our chance to change it beforehand. (TODO: make this point boldface when doing the announcement)
  • TODO: do we need a mechanism to register interest in specific packages (a way to indicate "I depend on this package even though my work is not published in opam-repository")

To repeat each time

• Use a script that reads every opam-repository package, takes the latest version of each package that has the x-maintenance-intent field and lists all versions that are not maintained anymore #tools #ci
• use the installability check to ensure that we remove only packages that do not prevent installation of maintained packages #ci
• Ping maintainers of large sets of packages individually so they can add this field to the latest version of each of their packages
• Create the PR to remove the packages from the opam-repository and adds it to the opam-repository-archive
• Announce the removal on discuss
  • once everything is ok merge

Tooling (basically to do once)

• Support for intent field on dune release, opam publish, opam repo ci linter, ... to ensure it is present #ci #tools
• Create a script that reads every opam-repository package, takes the latest version of each package that has the x-maintenance-intent field and lists all versions that are not maintained anymore #tools #ci
• use the installability check to ensure that we remove only packages that do not prevent installation of maintained packages #ci

Phase 4 (coasting)

• Repeat Phase 3 every 3 months
• Repeat Phase 2 every year