add CommitOperationCopy

[lhoestq]

close #1083

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Wauplin]

Hey @lhoestq thanks for opening a PR on this topic! I've made a few comments on your work. The 2 main things I would change in the logic:

• fail early in case of missing file on the repo or non-LFS file
• handle list_files_info differently (returned value is a bit misleading but basically you can't assume the output to be aligned 1-1 with the input) 😕

Also listed a few things to complete before being ready to merge. Could you:

1. Add in the existing test a second copy from same source but different destination (basically make a commit to copy twice the same file to 2 different location) => we should ensure it works (not the case currently)
2. Update operations argument in create_commit docstring
3. Add small section + example in the upload guide
4. Add CommitOperationCopy to package reference

Thanks a lot in advance! 🤗

Note: I made some changes to update _multi_commits.py (no need to care about that one) accordingly + make CommitOperationCopy a first-class citizen in huggingface_hub.__init__.py. You should pull from the branch before making changes :)

[Wauplin]

list_files_info can be a bit misleading but basically you cannot assume that the returned files will be in the same order as the input paths + you cannot even assume that the list will be the same size. In particular:

• if a path is duplicated (i.e. user is copying a file multiple times), the server will return only one RepoFile instance
• if a path is missing on the repo, the server will silently ignore it

So what I would advice to do is:

• take as input a List[CommitOperationCopy] instead of Iterable
• do a first pass with for src_revision, operations in groupby(copies, key=lambda op: op.src_revision):
  • list_files_info
  • if a returned RepoFile has .lfs = None => raise an issue
  • add to files_to_copy using RepoFile.rfilename, src_revision and RepoFile.lfs["sha256"]
• do a second pass on copies
  • if op.src_path_in_repo/op.src_revision pair is not found in the sha256 dictionary => raise the EntryNotFound error

(disclaimer: I haven't tested the above logic)

[lhoestq]

Sounds good ! why switch to List instead of Iterable ?

[Wauplin]

You can iterate on a list twice. On an iterable it's not guaranteed

[lhoestq]

Now I only iterate on it once using groupby so we're good

[Wauplin]

Perfect :)

[lhoestq]

Took all your comments into account :)

[Wauplin]

Looks good to me! Thanks for making the changes :)
I'll quickly update _multi_commits.py and we'll be ready to merge.

[lhoestq]

feel free to merge once it's good for you ;)