Uploading/Sharing large models to HuggingFace

[MXueguang]

Hi,

I am trying to upload a t5-3b based model to HuggingFace. The folder to upload has 11G.
When I am uploading, it will gives 'Connection aborted.', BrokenPipeError(32, 'Broken pipe').
Does it because the model is too large and there is a limitation? How could I deal with that?
Thank you for your help!

[julien-c]

There is no limit to the file sizes on the model hub, however, for uploads that large and if your connection is even slightly unstable, it can indeed fail.

If you have another host (S3 bucket or whatever) you can upload the file to, I can handle cping it to your namespace on huggingface.co

[MXueguang]

Actually It will abort at the very beginning of uploading process for the large file every time. All my other smaller models could be uploaded smoothly. so I feel it might not be my network issue.
My pytorch_model.bin is about 11G. I tried to use truncate to truncate the file size and noticed that I will keep aborting until I truncate the file to 5G

[MXueguang]

Btw, we have a public google cloud storage host. Does it work for you if i am still not able to upload the model?

[julien-c]

I can indeed reproduce. For now, can you upload to a GCS or S3 bucket, post the url here, and I'll cp the file?

Will take a note to investigate/fix this in the future.

[MXueguang]

gs://ron-random/castorini/monot5-3b-med-msmarco/
gs://ron-random/castorini/monot5-3b-msmarco/

Could you help us cp these two models to our organization castorini

Thank you very much for your help!

[julien-c]

Here you go: https://huggingface.co/castorini

[julien-c]

Will close this for now but we are tracking the "large file upload" issue internally