File Upload API succeeds despite 408 Request Timeout

[iamarjun]

Reproducible in:

# test.py
import sys
# Enable debug logging
import logging

from slack_sdk import WebClient

def uploadToSlackChannel():
    client = WebClient(
        token="you-slack-token", timeout=300)
    try:
        filepath = "file-to-upload"
        response = client.files_upload(
            channels="your-slack-channel-name", file=filepath)
        assert response["file"]  # the uploaded file
    except Exception as e:
        print(e)

uploadToSlackChannel()        

The Slack SDK version

slack-sdk==3.13.0

Python runtime version

Python 3.9.10

OS info

ProductName: Mac OS X
ProductVersion: 10.15.7
BuildVersion: 19H1615
Darwin Kernel Version 19.6.0: Sun Nov 14 19:58:51 PST 2021; root:xnu-6153.141.50~1/RELEASE_X86_64

Steps to reproduce:

(Share the commands to run, source code, and project settings (e.g., setup.py))

1. copy the above snippet to any file (e.g. upload_to_slack.py)
2. run python3 upload_to_slack.py

Expected result:

Should succeed without any error

Actual result:

The file gets uploaded but also throws this error

Received a response in a non-JSON format: stream timeout
The server responded with: {'status': 408, 'headers': {'x-edge-backend': 'envoy-www', 'content-length': '14', 'content-type': 'text/plain', 'x-slack-edge-shared-secret-outcome': 'no-match', 'date': 'Wed, 26 Jan 2022 07:12:45 GMT', 'server': 'envoy', 'via': 'envoy-edge-bom-jbcm', 'connection': 'close'}, 'body': 'stream timeout'}

Requirements

For general questions/issues about Slack API platform or its server-side, could you submit questions at https://my.slack.com/help/requests/new instead. 🙇

Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

[seratch]

Hi @iamarjun, thanks for writing in!

We recently received a similar feedback and that was due to some server-side issue: #1162 (comment) In the case where HTTP status 408 is returned from the server-side, the timeout setting in WebClient does not affect at all.

I'm sorry for bothering you again but would you mind contacting our customer support agents from either https://my.slack.com/help/requests/new or /feedback in your Slack workspace? It'd be appreciated if you could understand this.

[iamarjun]

Hi @seratch, when I use curl to upload the file I get stream timeout within 9-10s, even for 5MB files I get this error

 sh "curl -s https://slack.com/api/files.upload -F initial_comment=\"" + initial_message + "\" -F token=\"" + access_token + "\" -F channels=\"" + channel_name + "\" -F title=\"" + file_name + "\" -F filename=\"" + file_name + "\" -F file=@" + file_path

Hence shifted to SDK, although the file gets uploaded I still get stream timeout, not sure what the problem is.

[seratch]

That may be due to some connectivity issue between your host and Slack but I'm unable to help you out for it.

Trying the same script in a different host may help for figuring out whether it's caused by Slack's workspace specific issue or network issues in your host. I hope this helps.

[iamarjun]

@seratch initially I thought it was the CI Server that is causing the issue, but I could reproduce it locally as well.

[iamarjun]

Hi @seratch here's the full log using curl

curl -v https://slack.com/api/files.upload -F token=access-token -F channels= "#ios-ajay-test1" -F file=@file_path

*   Trying 13.126.138.201...
* TCP_NODELAY set
* Connected to slack.com (13.126.138.201) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Slack Technologies, Inc.; CN=slack.com
*  start date: Apr 13 00:00:00 2021 GMT
*  expire date: Apr 18 23:59:59 2022 GMT
*  subjectAltName: host "slack.com" matched cert's "slack.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fcc26010800)
> POST /api/files.upload HTTP/2
> Host: slack.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 195338476
> Content-Type: multipart/form-data; boundary=------------------------ce8e5e2beebe823d
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
* We are completely uploaded and fine
< HTTP/2 408 
< x-edge-backend: envoy-www
< content-length: 14
< content-type: text/plain
< x-slack-edge-shared-secret-outcome: no-match
< date: Wed, 26 Jan 2022 08:48:33 GMT
< server: envoy
< via: envoy-edge-bom-3mz6
< 
* Connection #0 to host slack.com left intact
stream timeout* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection 0

[seratch]

@iamarjun Thanks for sharing more details!

I've checked internal reports and found that other people are also facing this server-side issue with relatively large size files (I know that your file is not so large but it seems that the situation can arise even with mega-byte-level size).

The server-side engineering teams are working hard to eliminate (at least mitigate) the issue now but I cannot tell when the situation can be improved.

The only workaround that I can suggest is that, if you receive the 408 error, your app can check files.list API results with the same token to know whether the upload was actually successful. If there is no new item in the response, your app needs to retry the same files.upload API call.

We're sorry for the disruption.

[abhaynpai]

Even we are facing the same issue @seratch. We are not able to upload build artefacts to slack 😢

Keeping you in the loop here @jaypoojara. Let's follow this thread.

[sreejith-hme]

Facing the same issue, trying to upload 60MB file using curl request, failing with stream timeout error.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 61.0M    0     0  100 61.0M      0  50.2M  0:00:01  0:00:01 --:--:-- 50.2M
100 61.0M    0     0  100 61.0M      0  27.5M  0:00:02  0:00:02 --:--:-- 27.5M
100 61.0M    0     0  100 61.0M      0  18.9M  0:00:03  0:00:03 --:--:-- 18.9M
100 61.0M    0     0  100 61.0M      0  14.4M  0:00:04  0:00:04 --:--:-- 14.4M
100 61.0M    0     0  100 61.0M      0  11.6M  0:00:05  0:00:05 --:--:-- 11.6M
100 61.0M    0     0  100 61.0M      0   9.8M  0:00:06  0:00:06 --:--:--     0
100 61.0M    0     0  100 61.0M      0  8655k  0:00:07  0:00:07 --:--:--     0
100 61.0M    0     0  100 61.0M      0  7602k  0:00:08  0:00:08 --:--:--     0
100 61.0M    0     0  100 61.0M      0  6777k  0:00:09  0:00:09 --:--:--     0
100 61.0M  100    14  100 61.0M      1  6117k  0:00:14  0:00:10  0:00:04     2
100 61.0M  100    14  100 61.0M      1  6116k  0:00:14  0:00:10  0:00:04     3
stream timeout

[locomike]

Any other workaround? I'm trying the retry mechanism @seratch suggested but it is failing every time for me (408 and no actual message being posted).

[seratch]

@locomike I'm sorry to say this but I don't have anything further that I can suggest at this moment. We will continue communicating with the server-side teams.

[navjotbedi]

I am facing the same issue, let me know once it's resolved.

[AndreyMlashkin]

I am facing the same issue here. Subscribing for the updates

[picachuloves]

I am facing the same issue here. Subscribing for the updates

[navjotbedi]

Any other workaround? I'm trying the retry mechanism @seratch suggested but it is failing every time for me (408 and no actual message being posted).

Same here, failing all the time even after multiple retry.

[LunkRat]

I see these intermittently. Re-trying even a minute later and the same files upload successfully without issue.

The error was common two weeks ago, disappeared for a few weeks, and now it is back again today.

see also: https://stackoverflow.com/questions/70752546/receiving-an-error-from-server-when-trying-to-upload-a-file

[mattpr]

Is there any correlation with file type?
This started happening to us with ~33MB zip file.
Larger text (csv) files never had this issue...but our CSVs hit a ridiculous size and we started zipping them when they were over some threshold and then we started getting this issue.
Makes me suspicious that this might relate to file-type rather than just raw size.

Can anyone else commend on the file types you are experiencing the problem with (binary vs text and also mime-type)? Maybe there is a correlation that will make this easier for the devs to repro.

[LunkRat]

@mattpr we are seeing this issue with files that are 2kb-3kb in size, plain text, and have .log file extension. The issue appears in ~5% of uploads. A file that triggers the error will upload without issue on a re-try seconds later.

Thus, file size/type does not seem to be a factor here.

[mattpr]

Interesting. We get this error consistently on every try (100%) with the ~33MB zip that is generated each day.
We have retry logic on failure (3x before hard fail) but the upload succeeds each time. So the upload happens successfully 3 times and the 408 error happens each of the three times.

[LunkRat]

Maybe a temporary workaround for those who are experiencing this issue intermittently. It retries three times each with a 3s pause.

for file_name in files_list:
    # Try up to three times because Slack file uploads are recently kinda flaky.
    # @see https://github.com/slackapi/python-slack-sdk/issues/1165
    tries = 3
    for i in range(tries):
        if i > 0:
            print(f"Trying upload again for {file_name} (attempt {i+1} of {tries}) ...")
        try:
            # Limit to 20 calls per minute so we don't get rate-limited by Slack.
            # @see https://api.slack.com/docs/rate-limits
            if len(files_list) > 20:
                time.sleep(3)
            upload_result = client.files_upload(file=file_name)
            logger.info(upload_result)
        except SlackApiError as e:
            logger.error(f"Error uploading file {file_name}. Attempt {i+1} of {tries}: {e}")
            if i < tries - 1:
                time.sleep(3)
                continue
            else:
                raise
        break

Edit: We have been using the above for a week now. Normally it does not need more than one try. But occasionally it goes up to three tries, with the third try succeeding.

[ZaxR]

I'm facing this same issue as well, on ~90% of tries, starting on 2022/02/22. In my case the files are all .xlsx files in the 30-60MB range.

[abhaynpai]

curl https://slack.com/api/files.upload --limit-rate 423K -F token="xoxb-PUT-YOUR-TOKEN-HERE" -F channels="#test-channel" -F title="TESTING! FileName.apk" -F filename="filename.txt" -F file="@/path/to/file/test_empty_file.txt"

[hp7c]

curl https://slack.com/api/files.upload --limit-rate 423K -F token="xoxb-PUT-YOUR-TOKEN-HERE" -F channels="#test-channel" -F title="TESTING! FileName.apk" -F filename="filename.txt" -F file="@/path/to/file/test_empty_file.txt"

Thankyou 🙏

[kent-amplitude]

This issue is still happening for us. We're using Bolt JS, which automatically retries 9 times, and we end up spamming the customer's channel with the same message + file 10 times.

We've tried the curl --limit-rate workaround, but unfortunately, it's not working for us. We've gone as low as 13 kBps and have continued to get the same error.

[chrisedington]

Also happening for me, not sure how to fix it without a library update.

[seratch]

Hi all, let me share some updates on this issue.

Firstly, sincere apologies for taking a long time to resolve this issue on the Slack platform side. We do understand that this issue has been critical for many people.

As a solution, we just released v3.19.0, which includes a new method named WebClient#files_upload_v2(). This new method is much stabler and is mostly compatible with the existing files_upload() method. Please migrate to the new method if your app is affected by the server-side performance issue described here. Here is an example code demonstrating how to use the v2 method:

response = client.files_upload_v2(
    file="./logo.png",
    title="New company logo",
    # Note that channels still works but going with channel="C12345" is recommended
    # channels=["C111", "C222"] is no longer supported. In this case, an exception can be thrown 
    channels=["C12345"],
    initial_comment="Here is the latest version of our new company logo :wave:",
)

The new method eliminates the timeouts. In addition, it enables 3rd party app developers to upload multiple files like humans do in Slack clients. This feature addition can be useful for many use cases. Here is a simple example code:

response = client.files_upload_v2(
    file_uploads=[
        {
            "file": "./logo.png",
            "title": "New company logo",
        },
        {
            "content": "Minutes ....",
            "filename": "team-meeting-minutes-2022-03-01.md",
            "title": "Team meeting minutes (2022-03-01)",
        },
    ],
    channel="C12345",
    initial_comment="Here is the latest version of our new company logo :wave:",
)
response.get("files")  # returns the full metadata of all the uploaded files

Please refer to the release notes for more details: https://github.com/slackapi/python-slack-sdk/releases/tag/v3.19.0

[mattpr]

Thanks for the update.
I tested this today, updating to 3.19.0 and using the v2 method.

Unfortunately the new method has the exact same issue (practically) for me in that the file uploads apparently fail...except they don't. They actually post to slack despite the API/slack_sdk error. Now it is every file, not just the big ones. Apparently a permission issue that is not actually a permission issue since the files upload.

[7992]: SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/files.info)
[7992]: The server responded with: {'ok': False, 'error': 'missing_scope', 'needed': 'files:read', 'provided': 'files:write,chat:write'}
[7992]: Posting report to slack failed 2 times. Sleeping 5 seconds before trying again.
[7992]: Posting to slack failed 3 times.  Giving up.

Thev2 method also no longer takes the filename as the default title for the uploaded file... so if you upload a file without specifying the optional title parameter you will get a bunch of "unknown" files. Of course this can be worked around by getting the basename of the specified file and setting the title to that.

I guess I could fix the permission error issue by granting more permissions to the API token I am using but seems like a bug since the API allows me to post anyway?

[seratch]

Hi @mattpr, thanks for writing in with details!

[7992]: The server responded with: {'ok': False, 'error': 'missing_scope', 'needed': 'files:read', 'provided': 'files:write,chat:write'}

As the error code "missing_scope" indicates, the situation that you've encountered with v2 method is a different problem. I should have mentioned this, but the v2 method requires both files:write and files:read scopes while the legacy one requires only files:write scope.

So, please add files:read scope to your app's OAuth scopes and re-install the app to update the tokens to use. If you have a certain reason not to add files:read, you can use the underlying methods directly. Only files.info API calls in the last step inside files_upload_v2 require the read scope.

The v2 method also no longer takes the filename as the default title for the uploaded file... so if you upload a file without specifying the optional title parameter you will get a bunch of "unknown" files. Of course this can be worked around by getting the basename of the specified file and setting the title to that.

Thanks for sharing this. I will look into this later and definitely we can improve the v2 behavior to be consistent with v1 as much as possible.

[mattpr]

Thanks for the fast response and fixing the default title.

[mattpr]

Sorry, one more thing...

While this is mostly working I have also noticed that a successful call to files_upload_v2 followed by a call to chat_postMessage sometimes results in the second message posting in the channel first.

It appears that we are waiting for confirmation from the API that the request succeeded. The files_upload_v2 returns a SlackResponse which appears to be validated as successful and not an error.

So maybe this is a server-side "issue" where the file is uploaded successfully but not actually completely posted to the channel yet at the time of return and the subsequent chat_postMessage gets through to the channel faster?

While in general I prefer leaving things async as much as possible, in some cases (like posting a series of things to a channel) I might care about the ordering. Is there a way to cause files_upload_v2 to simulate a synchronous remote API so we can wait until the file is actually posted to the channel before continuing?

[seratch]

@mattpr Yes, the delay is one of the key differences in the new way.

I might care about the ordering. Is there a way to cause files_upload_v2 to simulate a synchronous remote API so we can wait until the file is actually posted to the channel before continuing?

When the ordering can be critical for some of your use cases, the only workaround that I can suggest is to wait for the message with files by periodically polling conversations.history/replies after the file upload is done. We are not planning to add this functionality to the files_upload_v2() method, so please go ahead with your own code for it. It'd be appreciated if you could understand this.

Lastly, when you (or anyone else) have further feedback on the v2 method, please start a new issue dedicated to the topics.

[Plebolution]

Hi!
We switched to files_upload_v2, the problem is gone, but looks like something broken with slack markdown in initial_comment. Link tags not working anymore - formatting example from docs sends to chat as text <http://www.example.com|This message is a link> (bold, italic and others works well)

Tried on files_upload again and link tag works

[BenIVRE]

Is there are issue around size limits, or is the jenkins plugin not using the new files_upload_v2? We are uploading the artifacts from all our platform builds, and there are no errors in the log around any of the uploads. All our .apk and other file type upload, but our PC build which is .zip and a bit larger never reaches the slack channel.

[RossRH]

Running file uploads through GitHub actions via either https://github.com/marketplace/actions/slack-file-upload or curl results in the same issue.
This happened after we upgraded our internet. limit-rate via curl (tested 1M) seemed to fix it although now uploads are too slow.

Going to upload to github releases and share a link in slack for now... but this is odd. A day debugging to find out the change was an internet upgrade 🤣

Not really prepared to use the new JS API for a non essential part of our CI so it would be nice if curl commands worked without a rate limit.