[vcpkg] [RFC] Implement host-triplets and host-triplet dependencies

[ras0219]

This PR solves a long-standing issue with manifests being unable to correctly install boost while cross-compiling. This is done by enabling consumers to mark a dependency as coming from the host (via "host": true). The dependency can then be accessed in the portfile via ${CURRENT_HOST_INSTALLED_DIR}/subpath.

This PR does not automatically add any subpaths of the host installed dir to helpers such as vcpkg_configure_cmake(). For now, consumers are intended to use vcpkg_add_to_path() to pick individual required executable folders.

See docs/users/tools.md for more details.

[Neumann-A]

Future TODOs:

• check how vcpkg_configure_make needs to be changed (--host(VCPKG_TARGET_TRIPLET)/--build(VCPKG_HOST_TRIPLET))
• check how vcpkg_configure_meson needs to be adapted.
• change ARM uwp triplets to use VCPKG_HOST_TRIPLET ?
• change community triplets to use VCPKG_HOST_TRIPLET ?

[ras0219-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ras0219-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Neumann-A]

the capability would be added for exactly the one additional triplet (available as HOST_TRIPLET)

but you said I am able to manipulate that value on a per port basis in the triplet. So for different ports VCPKG_HOST_TRIPLET can be different and as such the dependency graph should contain multiple host triplets. I just want to have that list passed to cmake.

[ras0219]

I apologize if I've misstated something above, to clarify:

Manifest files gain the ability to mark a dependency edge as "host": true. This causes that edge to resolve to the singular host triplet determined by runtime settings (--host-triplet, %VCPKG_DEFAULT_HOST_TRIPLET%). For example, boost-system:x86-windows-static package might have that edge resolve as boost-build:x64-windows. There's no way for the dependency graph of a single package to result in more than exactly two triplets (either the original triplet or the host triplet) because there's no syntax possible for that. There is no part of the current design that would feed into a variable VCPKG_ALL_HOST_TRIPLETS more than just the already proposed HOST_TRIPLET. My comment above was intended only that if you wanted a variable available to your private ports that has arbitrary data (such as a list of triplets), you could always do that with existing methods.

However, that selected host triplet (e.g. x64-windows) as always has all the normal powers of a triplet. It could be your https://github.com/Neumann-A/my-vcpkg-triplets/blob/master/x64-windows-llvm.cmake which would cause certain packages still within that single host triplet, to be built with whatever settings that triplet applies. To make this happen, you'd pass --host-triplet=x64-windows-llvm and the graph will resolve all "host": true edges to x64-windows-llvm. This allows arbitrarily different toolchains to be applied to each host package build, but it does not allow any individual port to be built more than twice in a single graph: once for the "target" and once for the "host".

The above said, in classic mode, there is a not-by-design "bug" where you could get transitively different host triplets on accident via changing the host triplet settings while executing different install commands. However, this is unintended, discouraged behavior (that also doesn't work in manifest mode and so won't work with newer features that rely on manifest mode such as versioning).

[Neumann-A]

Im am a bit confused (after actually checking the changes i am less confussed but irritated). You are planning to add it to the vcpkg tool as a cmd line argument? Doesn't this leave the installed/<triplet> strange state if somebody does vcpkg install port1 --host=sometriplet and then vcpkg install port2 --host=someothertriplet and port1 and port2 have shared deps? What happens to the binary cache? does different host triplets resolve to different hashes? I mean there could very well be a compile regression in one of the host tools which could lead to strange and hard to reproduce errors/bugs. What happens if port2==port1?
Reminder: https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Specifying-Target-Triplets.html
I think the host triplet should be part of the triplet file. (--triplet probably needs renaming if it is not.)

For now lets step back a bit since I think we have two different objects here:

1. How should portfiles discover host tools/scripts/utilities (without the user needed to put everything on PATH)
2. How can vcpkg help provide these tools?

To the first point:
My main concern is in 1. As you said yourself vcpkg is not responsible for providing a complete toolchain but if that is the case there needs to be a general story on how to provide the complete toolchain. This is why I want to have an indirection instead of a direct dependency on CMAKE_HOST_CURRENT_INSTALLED_DIR. The user should have the power to directly supply search paths or other <triplets> to search for tools/scripts whatever.

To the second point:
I am totally ok with 1 port = 1 host triplet dep but I don't think it should be a cmd line argument to vcpkg.
There should be a flag to completly deactivate the build of host deps (also on a per port basis). (VCPKG_NO_HOST_TRIPLET_BUILD?)
Per port customization of the host triplet should also be allowed in the triplet. Never forget host dependencies are not hard dependencies. The user could choose to supply host tools/scripts themself.

[ras0219]

Doesn't this leave the installed/ strange state if somebody does vcpkg install port1 --host=sometriplet and then vcpkg install port2 --host=someothertriplet and port1 and port2 have shared deps?

The state can be "strange", but it doesn't violate any manifest constraints. For example, given the ports:

A -> B:$host
C -> A, D:$host
D -> B:$host

if the user ran:

vcpkg install --host-triplet host1 A:target

they would move to the installed state of

A:target {built against B:host1}
B:host1

If they then ran

vcpkg install --host-triplet host2 C:target

they would end up in the state

A:target {built against B:host1}
B:host1
B:host2
C:target {built against A:target and D:host2}
D:host2 {built against B:host2}

Notably, at build time of C:target, D is available in CURRENT_HOST_INSTALLED_DIR (host2) precisely as required in the dependencies list. However, A will not be rebuilt against B:host2. There is no syntax for expressing "I need my dependency A:target to be built specifically against the B:host2" and none of the options we've discussed above would allow for that.

However, there is a way for D to say that "I am a host tool and require this other package for the same host triplet": that's just a non-host dependency combined with a check in D's build that only allows it to be built when HOST_TRIPLET STREQUAL TARGET_TRIPLET.

To belabor the point further, the dependency expression above D -> B:$host specifically says that "I require B available for my host triplet" and if the user changes host triplets while installing different packages they'll get results that model that constraint. For example, you could say vcpkg install --host-triplet host1 D:host2 and then vcpkg install --host-triplet host2 C and get:

A:target {built against B:host2}
B:host1
B:host2
C:target {built against A:target and D:host2}
D:host2 {built against B:host1}

This may be "strange", but it follows all constraints expressed in the files. If D needs B to match its own triplet, it should be a normal, non-host dependency. When designing a host port that installs helpers, some care does need to be taken because HOST_TRIPLET is not constant in all contexts in classic mode. Mostly, this just means installed host port helpers with dependencies should avoid using HOST_TRIPLET and should instead reference files relatively from ${CMAKE_CURRENT_LIST_DIR}, which will always have the intended result.

Of course, all the above strange capabilities are due to the stateful, incremental nature of classic mode and are totally unavailable in manifest mode by design. There are already other "strange" situations that can occur in classic mode due to its statefulness, such as version skew (installing C in the above example also won't rebuild A if its version is out of date, for example). These are all reasons why we recommend users to use manifests for new projects and only use classic mode for trivial, lazy things or for developing packages.

What happens to the binary cache? does different host triplets resolve to different hashes?

All binary caching and dependency tracking would still be correct -- vcpkg remove B:host2 would correctly remove itself, C, and D. Every package has its own triplet added to its hash, so B:host1 is different than B:host2 and thus A:target built against B:host1 will also be a different package than A:target built against B:host2.

The user should have the power to directly supply search paths or other to search for tools/scripts whatever.

At this time, it isn't clear what a fully generic replacement system would look like (with full correct integration into binary caching etc). With this PR, I expect that in cases where we discover a need to perform tool replacements, we can introduce the appropriate indirection at that point. Once that indirection design has proven to be stable, useful, and effective across many different tools, we can promote it to a built-in helper. In the meantime, even helpers that "just put exes into /tools" can still be effectively-enough replaced using an overlay port that packages up a prebuilt exe. For example:

>find /overlays
/overlays
/overlays/make-tool
/overlays/make-tool/make.exe
/overlays/make-tool/vcpkg.json
/overlays/make-tool/portfile.cmake
>cat /overlays/make-tool/portfile.cmake
file(COPY ${CMAKE_CURRENT_LIST_DIR}/make.exe DESTINATION ${CURRENT_PACKAGES_DIR}/tools/make-tool)

This would work correctly with binary caching and would replace some build-dependent host port that is normally expected to put make.exe into /tools/make-tool/ with a custom prebuilt executable (perhaps from a system package manager or otherwise?). There is effort in copying the make.exe and any other files out of the origin and into the overlay, but some clunkiness is to be expected until we have sufficient experience to build improvements.

The user should have the power to directly supply search paths or other to search for tools/scripts whatever.

It is not clear yet precisely how that should work. For example, the user generally should not be replacing boost-build and should definitely not be replacing protoc; those both have medium to strong version-dependence with other ports in the tree. On the other hand, it might be reasonable for the user to replace yasm or make. It's unclear what general purpose mechanism beyond the existing overlays capability would be worth its complexity weight. As always, we would much prefer to standardize and tweak existing practices instead of trying to guess up front what will be required.

... but I don't think it should be a cmd line argument to vcpkg.

In the end, it will be a command line argument in every way except name; there's no realistic alternative. If it is configurable at all, and usable in classic mode, it can change between command lines. I do agree that it will be better for most users to use the environment variable VCPKG_DEFAULT_HOST_TRIPLET, but not offering a command line option seems unnecessarily "difficult".

For now lets step back a bit since I think we have two different objects here:

1. How should portfiles discover host tools/scripts/utilities (without the user needed to put everything on PATH)
2. How can vcpkg help provide these tools?

This PR is explicitly not attempting to universally solve (1) for all time and space; it is focused on incrementally improving the status quo with respect to tools that we must build (boost-build, grpc, protobuf). I do expect that once this feature is available, we will be able to make progress on (1), ideally designing through experience a pattern for tools capable of coming from the system (like vcpkg_find_acquire_program()). Once that pattern is proven to be effective and stable, we can elevate it to a convention with built-in tool support. This PR introduces low-level tool functionality that will eventually enable such a pattern to be developed.

For example, with this PR, I think it would be totally reasonable for a host port to have an expected consumption interface like:

include(${CURRENT_HOST_INSTALLED_DIR}/tools/<tool>/tool.cmake)

<tool>_find_acquire(OUT_VAR VAR)

There should be a flag to completly deactivate the build of host deps (also on a per port basis).

I don't understand this as written. The intention of "host": true is to say "I absolutely must have this available for the host at build time or I cannot build"; this would be the same as a flag that completely disables all dependency resolution. If the intent is to say "I don't want make-tool to do anything", we have that in the form of overlay ports -- just overlay make-tool with an empty package.

However, if I understand that your request to mean "how do I provide my own definition of how to get make.exe instead of vcpkg's curated make-tool build", see the many words above and below on various approaches to that.

Never forget host dependencies are not hard dependencies. The user could choose to supply host tools/scripts themself.

I think I've addressed this within the words above, but essentially the intent of all ports (not just host ports) is to express dependence on capabilities. Overlays are our current way that users can replace the way those capabilities are fulfilled. One port's capability may be "I can find_package(OpenSSL) inside my build", another might be "I can include($/share/&lt;tool&gt;/magic.cmake)" and another might be "I can vcpkg_add_to_path($/tools/<foo>) and then execute_process(COMMAND foo)".

I understand that you're very concerned about the particular chosen capability/contract design of any specific host port; we expect those capabilities to evolve over time the same way that every other port evolves. However, today, we do not believe we have enough experience with buildable host tools in vcpkg to design an higher level interface that is correct, stable, robust, complete, and worth the implementation complexity.

[ras0219-msft]

Posting the sqlpp11:arm-uwp error here for potential future investigation:

14>CUSTOMBUILD : CMake error : Cannot restore timestamp D:/buildtrees/sqlpp11/arm-uwp-rel/test_static_asserts/CMakeFiles/generate.stamp [D:\buildtrees\sqlpp11\arm-uwp-rel\test_static_asserts\sqlpp11_assert_having.vcxproj]

[vicroms]

Move this PR to vcpkg-tool repo.

[autoantwort]

If I see this right it was merged in microsoft/vcpkg-tool#5, but the documentation was lost. So is this available?

[vicroms]

You can try it out by cloning the vcpkg-tool repository and building vcpkg yourself. You won't get this feature from bootstrapping until we release a new version of the vcpkg tool (we are reviewing our release process, there's no next release scheduled yet).

Documentation will be added once the feature is released properly.