ConnectionError: ('Connection aborted.', OSError("(32, 'EPIPE')"))

[danyaljj]

Getting this error uploading a T5-3b model (~5 GB) to the model-hub.
I don't think it's my connection; I didn't have any issues with other smaller models, except this one.
Any thoughts on what could be the issue?

$ transformers-cli upload unifiedqa-t5-3b --organization allenai 
About to upload file /Users/danielk/ideaProjects/t2t-qa/experiments/upload_models/3b/unifiedqa-t5-3b/tokenizer_config.json to S3 under filename unifiedqa-t5-3b/tokenizer_config.json and namespace allenai
About to upload file /Users/danielk/ideaProjects/t2t-qa/experiments/upload_models/3b/unifiedqa-t5-3b/special_tokens_map.json to S3 under filename unifiedqa-t5-3b/special_tokens_map.json and namespace allenai
About to upload file /Users/danielk/ideaProjects/t2t-qa/experiments/upload_models/3b/unifiedqa-t5-3b/config.json to S3 under filename unifiedqa-t5-3b/config.json and namespace allenai
About to upload file /Users/danielk/ideaProjects/t2t-qa/experiments/upload_models/3b/unifiedqa-t5-3b/spiece.model to S3 under filename unifiedqa-t5-3b/spiece.model and namespace allenai
About to upload file /Users/danielk/ideaProjects/t2t-qa/experiments/upload_models/3b/unifiedqa-t5-3b/pytorch_model.bin to S3 under filename unifiedqa-t5-3b/pytorch_model.bin and namespace allenai
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Your file now lives at:                                                                                                                                                                  
https://s3.amazonaws.com/models.huggingface.co/bert/allenai/unifiedqa-t5-3b/tokenizer_config.json
Your file now lives at:                                                                                                                                                                  
https://s3.amazonaws.com/models.huggingface.co/bert/allenai/unifiedqa-t5-3b/special_tokens_map.json
Your file now lives at:                                                                                                                                                                  
https://s3.amazonaws.com/models.huggingface.co/bert/allenai/unifiedqa-t5-3b/config.json
Your file now lives at:                                                                                                                                                                  
https://s3.amazonaws.com/models.huggingface.co/bert/allenai/unifiedqa-t5-3b/spiece.model
  0%|                                                                                                                               | 9756672/11406640119 [00:02<1:41:40, 1868312.30it/s]Traceback (most recent call last):
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 331, in _send_until_done
    return self.connection.send(data)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1737, in send
    self._raise_ssl_error(self._ssl, result)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 987, in send
    self.sock.sendall(data)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 342, in sendall
    sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 337, in _send_until_done
    raise SocketError(str(e))
OSError: (32, 'EPIPE')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/http/client.py", line 987, in send
    self.sock.sendall(data)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 342, in sendall
    sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 337, in _send_until_done
    raise SocketError(str(e))
urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError("(32, 'EPIPE')"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/danielk/opt/anaconda3/bin/transformers-cli", line 10, in <module>
    sys.exit(main())
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/transformers/commands/transformers_cli.py", line 33, in main
    service.run()
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/transformers/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/transformers/hf_api.py", line 168, in presign_and_upload
    r = requests.put(urls.write, data=data, headers={"content-type": urls.type})
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/requests/api.py", line 131, in put
    return request('put', url, data=data, **kwargs)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/danielk/opt/anaconda3/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(32, 'EPIPE')"))

[LysandreJik]

cc @julien-c

[julien-c]

We've seen similar issues in the past @danyaljj.

We are going to release a new upload system for models in the coming week, can this wait till then? If it can't, and if you can upload your model to another bucket, we can copy it over manually. Let us know.

[danyaljj]

Hey, @julien-c 👋 Sounds fair, I think I can wait until your new system is rolled out.