don't use triplet file hash. Hash variables instead #596

autoantwort · 2022-06-18T23:24:09Z

Fixes microsoft/vcpkg#19984

Neumann-A · 2022-06-19T15:07:34Z

src/vcpkg/cmakevars.cpp

+    print_variables("15fe3605-2429-46b9-b6b1-c5c008621f79")
    vcpkg_triplet_file(${VCPKG_TRIPLET_ID})
+    print_variables("9dbc61d9-e274-40b4-b38a-4d2d3fe91341")


I think you also need to probe the environment before and after the triplet.

Maybe, but for example you don't want to have the PATH in the abi. Another solution would be that you have to set a local variable to the value of the environment variable.

Do you have an example in which a environment variable should be included?

You need to include whatever the triplet sets. So the only way to do that is comparing before/after and filtering unrelevant variables out.

A problem would be that if a user sets a env variable outside of the triplet and uses this variable inside a port, this would be not caught (But this is probably impossible to detect. Except we include all env vars in the hash, but that would make the binarycache useless).

You need to include whatever the triplet sets. So the only way to do that is comparing before/after and filtering unrelevant variables out.

And if someone appends something to the PATH we should only hash the newly added stuff.

And if I look at
https://github.com/Neumann-A/my-vcpkg-triplets/blob/99d32a61fe890db22cc668fb4327891aa42f2699/armabi-v7a.cmake#L6
we should remove ${CMAKE_CURRENT_LIST_DIR} from the variable values before hashing.

https://github.com/microsoft/vcpkg/pull/15053/files#diff-e1605373cd9d4822d5c5db0dbeef7008e0657895640b0639f4a3ca5e72d40407 has code to extract changes in the environment from a batch script. You basically need something similar

Yeah I can use the code I already have. What I mean is that for example here:
https://github.com/Neumann-A/my-vcpkg-triplets/blob/99d32a61fe890db22cc668fb4327891aa42f2699/x64-windows-llvm.port.cmake#L22
you only want ${POSSIBLE_LLVM_BIN_DIR} gets hashed and not the whole ENV{PATH}

And POSSIBLE_LLVM_BIN_DIR gets hashed anyways. Hm, do we have an example where it would be really necessary to hash changes of the env?

Maybe we need something similar to VCPKG_ENV_PASSTHROUGH_UNTRACKED.

Something like VCPKG_TRIPLET_VARS_UNTRACKED and VCPKG_TRIPLET_ENV_VARS_(UN)TRACKED to explicitly enable/disable tracking of some (env) vars. Currently vcpkg just tracks everything from the triplet

really necessary to hash changes to the env?

Setting any variable which is related to building but not detected by the compiler detection: AR/ASM/FC

BillyONeal

Other than my ask for testing of a new parser, I think this from a code perspective is fine.

I would like to see @ras0219 / @ras0219-msft comment on what the product / ux implications may be before merging.

BillyONeal · 2022-06-28T18:11:51Z

src/vcpkg/cmakevars.cpp

@@ -318,6 +344,51 @@ endfunction()
                block_end = std::find(block_start, port_end, BLOCK_END_GUID);
            }

+            if (opt_triplet_hashes)
+            {
+                auto parse_variables = [port_start, port_end](StringView start_guid, auto callback) {


New parser should be extracted and tested. (And maybe use ParserBase et al. but since this is parsing our own content I don't think that's strictly necessary)

BillyONeal · 2022-06-28T18:14:57Z

src/vcpkg/cmakevars.cpp

@@ -318,6 +344,51 @@ endfunction()
                block_end = std::find(block_start, port_end, BLOCK_END_GUID);
            }

+            if (opt_triplet_hashes)


Suggested change

if (opt_triplet_hashes)

if (auto triplet_hashes = opt_triplet_hashes.get())

BillyONeal · 2022-06-28T18:15:14Z

src/vcpkg/cmakevars.cpp

+                    hasher->add_bytes(key.begin(), key.end());
+                    hasher->add_bytes(value.begin(), value.end());
+                });
+                opt_triplet_hashes.get()->push_back(hasher->get_hash());


Suggested change

opt_triplet_hashes.get()->push_back(hasher->get_hash());

triplet_hashes->push_back(hasher->get_hash());

ras0219-msft · 2022-06-28T18:39:11Z

I haven't dug 100% into the specifics, but I see one very large obstacle blocking this direction: paths in variables. Similar to my comment at microsoft/vcpkg#19984 (comment), I don't see how this approach can be implemented in a way that doesn't make the triplets incredibly machine- and project-location-specific.

I'm also concerned about making the binary caching system even more opaque for users.

Consider values like VCPKG_CXX_FLAGS or VCPKG_VISUAL_STUDIO_PATH, which are almost certain to contain paths. With the current system, it's very simple to understand whether or not those will affect the package abi -- is the triplet file the same or different? It's also therefore reasonably simple to understand how to choose whether the absolute path should affect the abi or not -- just use VCPKG_ENV_PASSTHROUGH_UNTRACKED to bring in an environment variable and expand that if you want it "hidden".

My counter-proposal to solve #19984 is to eventually introduce a more declarative (json) format for triplets with a narrower set of capabilities than CMake that can be precisely analyzed. It would be structurally clear which parts of the document apply to what ports and it would be possible to "see through" variable expansions to determine semantically significant substitions.

Unfortunately, triplet file invalidation is not very high on our (the vcpkg core team) backlog of problems to solve, compared to bug fixing, docs improvements, artifacts, and PR review.

Neumann-A · 2022-08-11T07:21:10Z

With the current system, it's very simple to understand whether or not those will affect the package abi -- is the triplet file the same or different?

No its not. It doesn't consider include() which I actively use to undermine the triplet hash to deal with per port customization in manifest mode without nuking everything which is unaffected. Of course this leads to inconsistencies in the binary hashes which is kind of bad but vcpkg doesn't provide a proper solution to deal with that.
VCPKG_VISUAL_STUDIO_PATH is also a bad example since the compiler hash will be affected by that any way so hashing that is also kind of unnecessary.

For me the problem seems to be string(APPEND z_vcpkg_output "${z_vcpkg_variable_name}=${${z_vcpkg_variable_name}}56732a63-363f-4326-8eac-738ed6390780") which returns the actual value of the variable instead of the "intended" or "set" value. If we however run cmake in trace mode (--trace --trace-format=json-v1) we can get output like:

{"args":["BLA","somevalue"],"cmd":"set","file":"E:/vcpkg_folders/bla.cmake","frame":1,"line":2,"time":1660201353.3553562}
{"args":["BLA_PATH","$ENV{PATH}"],"cmd":"set","file":"E:/vcpkg_folders/bla.cmake","frame":1,"line":3,"time":1660201353.3563538}

Tracing all set/unset commands a list of triplet variables can be obtained without absolute paths and the correct hashing even if other files are included. There are no absolute paths since those don't get expanded from the variables and if there is an absolute path vcpkg can at least warn about it and its consequences (since then it is explicitly set as an absolute path).

vcpkg could use --trace-redirect= to already output the trace into another file.

I think this way is way better than the current approach since e.g. simple formatting changes in an if statement within the triplet won't affect everything anymore.

autoantwort mentioned this pull request Jun 18, 2022

[vcpkg|binarycache] don't use triplet file hash. Hash variables instead. microsoft/vcpkg#19984

Open

autoantwort force-pushed the hash-triplet-vars branch 3 times, most recently from 428cebc to 7dc8801 Compare June 19, 2022 10:46

don't use triplet file hash. Hash variables instead

64f065f

Fixes microsoft/vcpkg#19984

autoantwort force-pushed the hash-triplet-vars branch from 7dc8801 to 64f065f Compare June 19, 2022 10:58

Neumann-A reviewed Jun 19, 2022

View reviewed changes

BillyONeal reviewed Jun 28, 2022

View reviewed changes

autoantwort marked this pull request as draft October 1, 2022 11:13

Neumann-A mentioned this pull request Nov 9, 2022

Use CMake variables set in triplet for abi instead of hashing the triplet file #800

Draft

BillyONeal added the category:binary-caching-improvements label Nov 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

don't use triplet file hash. Hash variables instead #596

don't use triplet file hash. Hash variables instead #596

autoantwort commented Jun 18, 2022

Neumann-A Jun 19, 2022

autoantwort Jun 19, 2022 •

edited

Loading

Neumann-A Jun 19, 2022

autoantwort Jun 19, 2022

Neumann-A Jun 19, 2022

autoantwort Jun 19, 2022

autoantwort Jun 19, 2022 •

edited

Loading

Neumann-A Jun 20, 2022

BillyONeal left a comment

BillyONeal Jun 28, 2022

BillyONeal Jun 28, 2022

BillyONeal Jun 28, 2022

ras0219-msft commented Jun 28, 2022

Neumann-A commented Aug 11, 2022

	if (opt_triplet_hashes)
	if (auto triplet_hashes = opt_triplet_hashes.get())

	opt_triplet_hashes.get()->push_back(hasher->get_hash());
	triplet_hashes->push_back(hasher->get_hash());

don't use triplet file hash. Hash variables instead #596

Are you sure you want to change the base?

don't use triplet file hash. Hash variables instead #596

Conversation

autoantwort commented Jun 18, 2022

Choose a reason for hiding this comment

autoantwort Jun 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

autoantwort Jun 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BillyONeal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ras0219-msft commented Jun 28, 2022

Neumann-A commented Aug 11, 2022

autoantwort Jun 19, 2022 •

edited

Loading

autoantwort Jun 19, 2022 •

edited

Loading