[vcpkg] RFC: Versioning

[vicroms]

We are currently working on implementing package versioning, the model is based on Go's module management implementation with some modifications.

This feature allows users to specify package versions to be installed as well as allowing packages to specify the versions of their dependencies.

There are some rough edges that need to be smoothed out, we appreciate any feedback that helps improve the current design.

P.S.: is easier to read the spec directly from the branch.

Closes #1681 #10674

[BillyONeal]

Mostly nitpicks here, but the inconsistency where we say we want the latest version in some places but the earliest version in others needs to be resolved.

[JackBoosY]

Related: #10674.

[MVoz]

maybe should with the accepted \ established parameters

Major.Minor.Patch[-Suffix], where the components have the following meanings:

Major: Breaking changes
Minor: New features, but backwards compatible
Patch: Backwards compatible bug fixes only
-Suffix (optional): a hyphen followed by a string denoting a pre-release version (following the Semantic Versioning or SemVer 1.0 convention).

https://docs.microsoft.com/en-us/nuget/concepts/package-versioning

[Hoikas]

This seems like a much needed feature to me. I've only briefly skimmed the spec, but I notice that manifest files are specified as being JSON. Are these manifest files intended to be maintained by humans? If so, has there been any internal debate about utilizing YAML over JSON? This is merely a personal opinion, but it seems to me that YAML files are more human-maintainable than JSON. I realize there are more factors that go into such a decision aside from personal preference--I'm just curious if this has already been explored.

[strega-nil]

@Hoikas see #11203; we chose JSON for a reason.

[Hoikas]

@strega-nil I'm afraid I don't see the portion of that item where that rationale is given. Can you provide a more specific reference? Edit: My mistake, I see the section in the spec now. I only looked at the discussion initially. Thanks for pointing this out!

[strega-nil]

I also have much the same reservations as Billy; this feels like we should be talking in a full meeting, can you set something up?

[strega-nil]

Do we have version-scheme fields? Are we adding that?

[strega-nil]

Yeah, this seems unfortunate. This is definitely something I want to talk about in person; can we discuss this in the vcpkg team meeting next week?

[Simran-B]

Are the at symbol, colon and backtick reserved?

[JackBoosY]

Fixes #1681

[YehezkelShB]

Why can't it be kept added as a suffix (e.g. like deb-version, maybe with a different separating char (e.g. +)? Does it complicate the support of the various versioning schemes?
From user point of view it seems easier to look on only one field to understand why a package is upgraded, for example.

[ras0219]

Suggested change

	* Exact version requirement (`"version": "= 1.0.0"`)
	* Minimum version requirement (`"version": ">= 1.0.0"`)
	* Exact version requirement (`"version=": "1.0.0"`)
	* Minimum version requirement (`"version>=": "1.0.0"`)

This declaration is inconsistent with the rest of the doc.

[rosti-il]

How do you plan to maintain old versions of a port? I didn't find any explanation.

Today vcpkg uses one big Git repository to store all packages in the ports/ directory and tries to use only latest versions. This approach produces a linear versions chain of a port, for example:

1.0.1 --> 1.0.2 --> 1.0.2#1 --> 1.0.2#2 --> 1.1.1 --> 1.2.0 --> 1.2.4

With the current ports collection that tends to use latest versions of all ports this is alright but with versioning it isn't. How could you make a fix of a port of a specified old version? I mean something like 1.0.1#1 or 1.0.2#3? How could you make a port of a lower version when a higher one already exist? For example versions 1.0.3, 1.1.2 or 1.2.1. Also versions like 1.1.5 could be released in the upstream after 1.2.0 or 1.2.2 were released, i.e. your may not be able to keep your versions chain in the chronological order. Or you may have some version skipped because of a lack of interest at that time when a newer version was released. For example you may have no 1.5.x version between 1.4.x and 1.6.x but would like to add 1.5.1 after someone already made version 1.6.0 of that port.

With the linear versions chain this is just impossible to achieve. So you need to use Gt branches and if you want to use Git branches of port versions you must put each port into a separate Git repository, i.e. make them Git submodules of the main vcpkg Git repository.

I think the Git workflow of those submodules should be a little bit different to the usual. There should be a skeleton branch with only one commit of the port skeleton with upstream version 0 and port version 0. For each real upstream version you should make a new branch and then use those branches to commit future patches, if you need. For example:

[skeleton branch]---<skeleton commit>
                       |  |  |
[1.0.1 branch]         |  |  +----<1.0.1 create commit>----<1.0.1#1 fix commit>----<1.0.1#2 fix commit>
[1.0.2 branch]         |  +-------<1.0.2 create commit>----<1.0.2#1 fix commit>
[1.2.0 branch]         +----------<1.2.0 create commit>

Each x.y.z branch (or whatever version schema is used by the upstream) should be based on the skeleton branch only. Let's say you want to make 1.0.3 branch for the 1.0.3 upstream version of your port. How to do that? Very simple:

1. cd to the Git submodule directory of your port
2. run git checkout skeleton && git checkout -b 1.0.3
3. copy files of latest commit in some other branch, for example git archive 1.0.2 | tar x
4. modify, add and commit this port files.

With this approach you may use the same JSON files of ports, like your zlib.json example, but maintaining of the old port versions will be much easier and more flexible.

[klalumiere]

From what I understand, the problem you describe will be solved by registries take 2. This is suppose to be released during this semester along with versioning, as you can see on the roadmap.

[rosti-il]

@klalumiere That registries design seems over engineered to me. Also it still seems to use a single Git repository for many ports. If a registry is simply a set of packages (as stated in the "take 1" of the registries design document) then it could be just a set of Git repositories where each repository is just for one port/package only. Why to use anything else, for example a filesystem registry with sub-directories for versioning? How could it be maintained without any VCS? Why to invent a new database (the 😇 directory in "take 2") when you can define all submodules (i.e. the registry) in the .gitmodules file that Git understand? If some company or organization needs to use a different registry it just needs to use its own version of the .gitmodules file. Why to define baselines in that invented database instead of defining proper dependencies with their versions in the ports? The intention of the baseline is to have "a set of versions that are tested against each other". This is the exact purpose of defining proper dependencies with versions.

P.S. There is git push after git commit --amend in the "take 2" document. How it's supposed to work? You will need to do forced push i.e. git push -f but this rewrites history and can make many troubles. I was astonished to see it.

[rosti-il]

One thing I forgot to mention. A Git repository of a ports submodule (of a port) should use Git tags, so all versions of the port could easily be found by git tag command, without any other database. For example zlib-1.2.10, zlib-1.2.11-9, zlib-1.2.11-11, i.e. the format is <port name><dash><upstream version in any format><optional last dash><optional port version>.

[strega-nil]

git push after git commit --amend works if you don't push in between the initial commit and the final commit.

I'm not going to respond to the other questions; if you do want to ask these questions, please ask in the original thread (I will say that likely the things you're asking about do not scale, or require large scale changes across the ecosystem, which are both unacceptable. Existing package managers support on the order of tens of thousands of packages in an automated way, and vcpkg should be able to do the same.).

[rosti-il]

@strega-nil Ok, it's just an unusual practice to split commits, i.e. to use git commit --amend for something other than fixing mistakes.

Could you give me a link to that thread?

[strega-nil]

PR #13590

The reason we have to do this is to grab the tree object ID from git.

[wrobelda]

One thing I have to add here is that the current proposal misses an opportunity for allowing to delegate the maintenance of the portfile manifestos to their respective upstream projects. What @rosti-il suggests with .gitmodules would also let to switch from a vcpkg-handled portfile directly to a vanilla upstream git repos, should the upstream decide to support their portfiles directly (just like they support e.g. Debian packaging scripts nowadays). I am sure we all agree that this is something you will eventually have to consider, given the growing popularity of vcpkg and, by extension, the burden of its maintenance.

[cmpute]

I didn't go through all the conversations above, but I also want to put forward my points. I totally agree with the idea, as @rosti-il mentioned, of using multiple git repositories to maintain port files for each package, and use branches to maintain port files for each version. This scheme has been adopted by many famous package managers like conda, boost and ROS.

By this way, the maintaining burden is further reduced and it enabling modification of port files in previous versions. With current git id versioning, it's impossible to add feature to a previous version of a library (which I mentioned in my issue #2823). That really limited the functionality of vcpkg.

[cmpute]

Also utilizing git tricks makes vcpkg less user-friendly to developers that are not familiar with git.

[strega-nil]

Suggested change

	[Port versions database](#port-versions-database)
	[Git manipulation](#git-manipulation)
	[Versioned portfile provider](#versioned-portfile-provider)
	[Baseline provider](#baseline-provider)
	[Versioned dependency resolution](#versioned-dependency-resolution)
	[Lock files](#lock-files)
	* [Port versions database](#port-versions-database)
	* [Git manipulation](#git-manipulation)
	* [Versioned portfile provider](#versioned-portfile-provider)
	* [Baseline provider](#baseline-provider)
	* [Versioned dependency resolution](#versioned-dependency-resolution)
	* [Lock files](#lock-files)

[Be-ing]

What would be the recommended way to specify different versions for different OSes? We need to use different versions of Qt on macOS and Windows (for complex reasons).

[strega-nil]

@Be-ing it would probably be:

"dependencies": [
  {
    "name": "qt5",
    "version>=": "blah",
    "platform": "windows"
  },
  {
    "name": "qt5",
    "version>=": "blah",
    "platform": "osx"
  }
]

[Be-ing]

What is the status of implementation of this? As far as I understand, this PR is only planning the new feature. Has any of it been implemented yet?

[vicroms]

@Be-ing

Most of it is already in the codebase. The actual implementation deviated slightly from this specification but we are now redacting end-user documentation (see #15693).