Push to hub save

[sgugger]

What does this PR do?

This PR changes the documentation of the push_to_hub training argument to be in sync with the actual behavior, and also makes sure that when push_to_hub=True, a push is done every time the model is saved.

Fixes #15313

[HuggingFaceDocBuilder]

The documentation is not available anymore as the PR was closed or merged.

[LysandreJik]

This works for me! Something to keep in mind here (might be a bit complex), is that git is aware of what commits have been pushed to the remote only when it has done pushing that commit.

It means the following:

repo.git_commit(...)
repo.git_push(blocking=False)
...
repo.git_commit(...)
repo.git_push(blocking=False) <--- if the previous push was not finished, it will push the two commits instead of the last one, resulting in a 2x bigger (and slower) push

This means that if the pushes are very frequent, or with very big models, then it wouldn't be surprising to have each git_push push the previous commit(s) alongside it, resulting in each following invocation pushing bigger and bigger pushes.

One way to prevent that would be to ensure that all previous pushes have been finished before starting a new one. I'm not sure there is a perfect solution to this.

[sgugger]

I think you're missing the fact that, apart from a push asked explicitly with trainer.push_to_hub() (or now trainer.save_model()) all other pushes are only done if the previous one is completely finished (see here). So, there are no frequent pushes (except if the user asks for it by putting trainer.save_model() in a loop but I don't think that will happen).

[LysandreJik]

Then that's perfect :)

[osanseviero]

Nit

Suggested change

	content will be pushed each time a save is triggered (depneding on your `save_strategy`). Calling
	content will be pushed each time a save is triggered (depending on your `save_strategy`). Calling

[osanseviero]

Nit

Suggested change

	[`~Trainer.save_model`] will also trigger a push
	[`~Trainer.save_model`] will also trigger a push.

[osanseviero]

I don't fully understand this. Don't we also want to push to the Hub automatically during training (which is an internal call)?

[sgugger]

push_to_hub calls save_model which calls push_to_hub which calls save_model which calls push_to_hub which calls save_model which calls push_to_hub which calls save_model which calls push_to_hub which calls save_model which calls push_to_hub which calls save_model which calls push_to_hub which calls save_model which calls push_to_hub which calls save_model which calls push_to_hub which calls save_model ...

This internal argument is here to avoid that ;-)

[osanseviero]

Thanks for explaining and for the PR!

Unfortunately I'm still getting the behaviour in which the model is getting saved during training but it's not getting pushed. https://colab.research.google.com/drive/1GAXf3egH2GDbk7M0btdKWbLerBqLoJPi?usp=sharing. I have a commit saving at step 500 but no push

[sgugger]

Hi Omar! I just checked your colab, and ran it. You have to wait for a solid 15 minutes after the fact to see the weights on your repo as Colab is uploading at an excruciatingly slow speed (those pushes during training are asynchronous to avoid slowing down training).

trainer.push_in_progress give you the job that is pushing, you can check its stdout attribute to see the progress it makes, its is_done attribute to see if it's finished or not and its stderr attribute to check if there was an error or not.

You can see on this repo that I eventually got my weights pushed with your code :-)

[osanseviero]

Aha! Thanks @sgugger for the investigation. I was not aware about trainer.push_in_progress, this is something I'll definitively use next time. Thanks once again.

[osanseviero]

Thanks @sgugger for the PR and clarifications on the pushing behaviour 🚀 🥳

[sgugger]

Ugh, sorry @osanseviero I was convinced I had accepted your suggestions before merging (must have skipped one click), will make a new commit with you as Co-author...