Internal server error on upload

[haesemeyer]

OS: Ubuntu
Python: 3.10.4
dandi: 0.39.1

Hi,

I'm trying to upload data to a recently created dandiset. All steps in preparing the dataset run without error. When calling dandi upload initial steps complete fine as well. However the upload itself seems to fail with HTTP 500 errors. I attached a screenshot of the command line output as well as the complete log-file. I also pasted below what I think is the relevant section of the log-file illustrating the error. I have installed the correct API key using keyring.
I wonder if this is related to the dataset being under embargo and if I need to implement a similar workaround as for the initial download (How to upload to embargoed dandiset?).

Thanks,
Martin
20220418134033Z-120559.log

LINE 15561 in full log
2022-04-18T09:41:46-0400 [DEBUG   ] dandi 120559:140705536141056 sub-RSpinal-Random-Wave-04/sub-RSpinal-Random-Wave-04_ses-20211008T101450_ophys.nwb: Beginning upload
2022-04-18T09:41:46-0400 [DEBUG   ] dandi 120559:140705536141056 POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:46-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:47-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:47-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:47-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:48-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:49-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:49-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:49-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:49-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:51-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:51-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:51-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:51-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:53-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:54-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:54-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:54-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:57-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:57-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:41:57-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:41:57-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:01-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:01-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:01-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:01-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:06-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:06-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:06-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:06-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:12-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:12-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:12-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:12-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:19-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:19-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:20-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:20-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:29-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:29-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:29-0400 [WARNING ] dandi 120559:140705536141056 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:29-0400 [DEBUG   ] urllib3.connectionpool 120559:140705536141056 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:41-0400 [WARNING ] dandi 120559:140705544533760 Retrying POST https://api.dandiarchive.org/api/uploads/initialize/
2022-04-18T09:42:41-0400 [DEBUG   ] urllib3.connectionpool 120559:140705544533760 https://api.dandiarchive.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 500 145
2022-04-18T09:42:41-0400 [ERROR   ] dandi 120559:140705544533760 HTTP connection failed

[dchiquito]

This is an internal error with our API server having problems authenticating with S3. I'll update this issue as soon as we have resolved the problem.

[dchiquito]

@haesemeyer I put in a temporary fix for the issue, and it seems to be fixed on my end. Can you try uploading again?

[haesemeyer]

@dchiquito thanks! That seems to have fixed it (tried with one dandiset and it uploaded swimmingly).