File Upload API succeeds despite 408 Request Timeout

[iamarjun]

Reproducible in:

# test.py
import sys
# Enable debug logging
import logging

from slack_sdk import WebClient

def uploadToSlackChannel():
    client = WebClient(
        token="you-slack-token", timeout=300)
    try:
        filepath = "file-to-upload"
        response = client.files_upload(
            channels="your-slack-channel-name", file=filepath)
        assert response["file"]  # the uploaded file
    except Exception as e:
        print(e)

uploadToSlackChannel()        

The Slack SDK version

slack-sdk==3.13.0

Python runtime version

Python 3.9.10

OS info

ProductName: Mac OS X
ProductVersion: 10.15.7
BuildVersion: 19H1615
Darwin Kernel Version 19.6.0: Sun Nov 14 19:58:51 PST 2021; root:xnu-6153.141.50~1/RELEASE_X86_64

Steps to reproduce:

(Share the commands to run, source code, and project settings (e.g., setup.py))

1. copy the above snippet to any file (e.g. upload_to_slack.py)
2. run python3 upload_to_slack.py

Expected result:

Should succeed without any error

Actual result:

The file gets uploaded but also throws this error

Received a response in a non-JSON format: stream timeout
The server responded with: {'status': 408, 'headers': {'x-edge-backend': 'envoy-www', 'content-length': '14', 'content-type': 'text/plain', 'x-slack-edge-shared-secret-outcome': 'no-match', 'date': 'Wed, 26 Jan 2022 07:12:45 GMT', 'server': 'envoy', 'via': 'envoy-edge-bom-jbcm', 'connection': 'close'}, 'body': 'stream timeout'}

Requirements

For general questions/issues about Slack API platform or its server-side, could you submit questions at https://my.slack.com/help/requests/new instead. 🙇

Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

[seratch]

Hi @iamarjun, thanks for writing in!

We recently received a similar feedback and that was due to some server-side issue: #1162 (comment) In the case where HTTP status 408 is returned from the server-side, the timeout setting in WebClient does not affect at all.

I'm sorry for bothering you again but would you mind contacting our customer support agents from either https://my.slack.com/help/requests/new or /feedback in your Slack workspace? It'd be appreciated if you could understand this.

[iamarjun]

Hi @seratch, when I use curl to upload the file I get stream timeout within 9-10s, even for 5MB files I get this error

 sh "curl -s https://slack.com/api/files.upload -F initial_comment=\"" + initial_message + "\" -F token=\"" + access_token + "\" -F channels=\"" + channel_name + "\" -F title=\"" + file_name + "\" -F filename=\"" + file_name + "\" -F file=@" + file_path

Hence shifted to SDK, although the file gets uploaded I still get stream timeout, not sure what the problem is.

[seratch]

That may be due to some connectivity issue between your host and Slack but I'm unable to help you out for it.

Trying the same script in a different host may help for figuring out whether it's caused by Slack's workspace specific issue or network issues in your host. I hope this helps.

[iamarjun]

@seratch initially I thought it was the CI Server that is causing the issue, but I could reproduce it locally as well.

[iamarjun]

Hi @seratch here's the full log using curl

curl -v https://slack.com/api/files.upload -F token=access-token -F channels= "#ios-ajay-test1" -F file=@file_path

*   Trying 13.126.138.201...
* TCP_NODELAY set
* Connected to slack.com (13.126.138.201) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Slack Technologies, Inc.; CN=slack.com
*  start date: Apr 13 00:00:00 2021 GMT
*  expire date: Apr 18 23:59:59 2022 GMT
*  subjectAltName: host "slack.com" matched cert's "slack.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fcc26010800)
> POST /api/files.upload HTTP/2
> Host: slack.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 195338476
> Content-Type: multipart/form-data; boundary=------------------------ce8e5e2beebe823d
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
* We are completely uploaded and fine
< HTTP/2 408 
< x-edge-backend: envoy-www
< content-length: 14
< content-type: text/plain
< x-slack-edge-shared-secret-outcome: no-match
< date: Wed, 26 Jan 2022 08:48:33 GMT
< server: envoy
< via: envoy-edge-bom-3mz6
< 
* Connection #0 to host slack.com left intact
stream timeout* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection 0

[seratch]

@iamarjun Thanks for sharing more details!

I've checked internal reports and found that other people are also facing this server-side issue with relatively large size files (I know that your file is not so large but it seems that the situation can arise even with mega-byte-level size).

The server-side engineering teams are working hard to eliminate (at least mitigate) the issue now but I cannot tell when the situation can be improved.

The only workaround that I can suggest is that, if you receive the 408 error, your app can check files.list API results with the same token to know whether the upload was actually successful. If there is no new item in the response, your app needs to retry the same files.upload API call.

We're sorry for the disruption.

[abhaynpai]

Even we are facing the same issue @seratch. We are not able to upload build artefacts to slack 😢

Keeping you in the loop here @jaypoojara. Let's follow this thread.

[sreejith-hme]

Facing the same issue, trying to upload 60MB file using curl request, failing with stream timeout error.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 61.0M    0     0  100 61.0M      0  50.2M  0:00:01  0:00:01 --:--:-- 50.2M
100 61.0M    0     0  100 61.0M      0  27.5M  0:00:02  0:00:02 --:--:-- 27.5M
100 61.0M    0     0  100 61.0M      0  18.9M  0:00:03  0:00:03 --:--:-- 18.9M
100 61.0M    0     0  100 61.0M      0  14.4M  0:00:04  0:00:04 --:--:-- 14.4M
100 61.0M    0     0  100 61.0M      0  11.6M  0:00:05  0:00:05 --:--:-- 11.6M
100 61.0M    0     0  100 61.0M      0   9.8M  0:00:06  0:00:06 --:--:--     0
100 61.0M    0     0  100 61.0M      0  8655k  0:00:07  0:00:07 --:--:--     0
100 61.0M    0     0  100 61.0M      0  7602k  0:00:08  0:00:08 --:--:--     0
100 61.0M    0     0  100 61.0M      0  6777k  0:00:09  0:00:09 --:--:--     0
100 61.0M  100    14  100 61.0M      1  6117k  0:00:14  0:00:10  0:00:04     2
100 61.0M  100    14  100 61.0M      1  6116k  0:00:14  0:00:10  0:00:04     3
stream timeout

[locomike]

Any other workaround? I'm trying the retry mechanism @seratch suggested but it is failing every time for me (408 and no actual message being posted).