[WIP] Git repository rule cache

[ittaiz]

fixes #5116 still WIP
Main open question which we'd love @aehlig's thoughts about is how to decide where to clone this?
Ideally it should be controlled by --repository_cache's value and just add a new key type (git?) which this code clones under it. WDYT?
If this sounds like a good idea we'd really appreciate pointers on how to access the command line flag's value from the repository rule.

thank you @aoded and @eyalmalron for your work on this

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

[ittaiz]

@aoded is a Wix employee and Wix has signed the corporate CLA.

…

On Fri, 17 Aug 2018 at 20:55 googlebot ***@***.***> wrote: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here <https://cla.developers.google.com/> to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5928 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFxEJBg12fCjDb8qCfMm8ER7Wmd0pks5uRwN3gaJpZM4WB9f7> .

[Globegitter]

Another interesting flag to support would be --experimental_repository_cache_hardlinks but I guess that could not be achieved with git and would really require a proper skylark api.

[edbaunton]

If you clone from the cache (on the same filesystem) I believe git will automatically create hardlinks for you https://git-scm.com/docs/git-clone

[ittaiz]

@aehlig can we get your feedback about this? Also, how do you recommend we handle the path?
I have a bit of hope of getting this before 0.17.0 RC1 is cut (probably next few days)

[Globegitter]

Ah nice @edbaunton the docs do talk about hard linking but only when the --local flag is provided. So should this flag maybe be passed in also?

[ittaiz]

@aehlig ping

[aehlig]

@aehlig ping

I didn't realize you already wanted a review for a pull request marked "Work in progress". In any case, the concern I already mentioned in direct communication is still valid: sharding the git cache by name(!) of the repository contradicts the general idea of our repository caches being conent-addressable. This also is unnecessary, as git already stores everything in a content-adressable way. Also, hard-coding cache paths is not a ready-to-merge solution. Using an appropriate environement variable (and falling back to not caching, if unset) is a solution for the time being, till #6016 is done.

…

-- Klaus Aehlig Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

[VolodymyrPolishchuk]

@aehlig ping
I didn't realize you already wanted a review for a pull request marked "Work in progress". In any case, the concern I already mentioned in direct communication is still valid: sharding the git cache by name(!) of the repository contradicts the general idea of our repository caches being conent-addressable. This also is unnecessary, as git already stores everything in a content-adressable way. Also, hard-coding cache paths is not a ready-to-merge solution. Using an appropriate environement variable (and falling back to not caching, if unset) is a solution for the time being, till #6016 is done.
…
-- Klaus Aehlig Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

Hi Klaus!
we are working on improvement of this PR to align with bazel's vision of local cache.
Afaiu, if we clone multiple repositories into temp directory, we may encounter naming collisions, since repositories end up being folders of the same name as repository, am i correct?

[ittaiz]

The idea is not to clone many repositories into the folder but rather have that folder be a git repository with many remotes (for the URLs needed from the various git_repository rules in the *various* bazel workspaces of the machine) and then have each git_repository use that as its remote. Is that clearer?

…

On Tue, 18 Sep 2018 at 18:33 Volodymyr Polishchuk ***@***.***> wrote: @aehlig <https://github.com/aehlig> ping I didn't realize you already wanted a review for a pull request marked "Work in progress". In any case, the concern I already mentioned in direct communication is still valid: sharding the git cache by name(!) of the repository contradicts the general idea of our repository caches being conent-addressable. This also is unnecessary, as git already stores everything in a content-adressable way. Also, hard-coding cache paths is not a ready-to-merge solution. Using an appropriate environement variable (and falling back to not caching, if unset) is a solution for the time being, till #6016 <#6016> is done. … <#m_-7824494803801880849_> -- Klaus Aehlig Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen <https://maps.google.com/?q=Erika-Mann-Str.+33,+80636+Muenchen&entry=gmail&source=g> Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado Hi Klaus! we are working on improvement of this PR to align with bazel's vision of local cache. Afaiu, if we clone multiple repositories into temp directory, we may encounter naming collisions, since repositories end up being folders of the same name as repository, am i correct? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5928 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF6o_PBagP_NQ9S78JkaJA9JAejZUks5ucRIwgaJpZM4WB9f7> .

[siedentop]

Thank you @aehlig for the nice collaboration today at the hackathon. My branch can be found here.

I've written what is definitely still missing inside the commit.

[meisterT]

What's the status of this PR? Can we close it in favor of @siedentop 's changes?