curl fails to download after a certain time, but wget does not.

[manojkiraneda]

The container seems to use curl to download the .sh file and that is failing for some reason, but the wget on the same file seems to work. Can you guys help me to figure out what is the issue here ?

Even with wget, it seems like we are reconnection after certain time, do i need to enable some option in curl to get this working ?

----------------------------------------------------------------------------------------------------------------------
Attempting to download https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
#####                                                                      7.1%
curl: (18) transfer closed with 314315481 bytes remaining to read
Unable to download "https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh".


----------------------------------------------------------------------------------------------------------------------------
[Juliet@juliet ci-stage-area]$ wget https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
--2020-10-11 11:27:51--  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Resolving jenkins.openbmc.org (jenkins.openbmc.org)... 169.44.58.24
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 338280321 (323M) [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s   6%[=====>                                                                                                ]  20.84M   324KB/s    in 71s     

2020-10-11 11:29:05 (303 KB/s) - Connection closed at byte 21854168. Retrying.

--2020-10-11 11:29:06--  (try: 2)  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 338280321 (323M), 316426153 (302M) remaining [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s  12%[++++++=====>                                                                                          ]  39.44M   392KB/s    in 46s     

2020-10-11 11:29:54 (410 KB/s) - Connection closed at byte 41353000. Retrying.

--2020-10-11 11:29:56--  (try: 3)  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 338280321 (323M), 296927321 (283M) remaining [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s  19%[++++++++++++======> 

[rewitt1]

Hi @manojkiraneda,

Unfortunately seems to be caused by some interaction between the client and the server when download speeds are slow. I limited the download speeds on my router, and even when using a web browser, the download still fails.

Here is an example of a similar issue, although I wouldn't go so far as to say it is the exact same.

Outside of contacting openbmc, I'm not sure of a solution, because it seems to occur independent of the client.

[manojkiraneda]

Hi @rewitt1

Thanks for looking at this issue. I agree that it's probably because of the server, but wget seems to work as it is doing a re-try, can we do similar settings with curl as well, so that we don't break out with a partial file in case of bad servers? That would really help.

I guess something like this might help.

Here is a snip of wget doing a retry & resuming from a closed connection from server.

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s   6%[=====>                                                                                                ]  20.84M   324KB/s    in 71s     

2020-10-11 11:29:05 (303 KB/s) - Connection closed at byte 21854168. Retrying.

--2020-10-11 11:29:06--  (try: 2)  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 338280321 (323M), 316426153 (302M) remaining [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s  12%[++++++=====> 

[rewitt1]

The most obvious ways I can think of fixing this would be one of the following:

1. Add an extra argument to the entry point that tells curl to retry
2. Add a --curl-arguments parameter that allows for specifying arguments to curl in addition to the url
3. Replace --url with --curl-arguments

I'm not sure which of these I like best yet.

[manojkiraneda]

To solve only this problem, I would personally vote for 1 as retry only kicks-in when the connection is closed and has no effect in case of a better server.

But if you want to provide the user the facility to give his own parameters, then I would go for 2.

[manojkiraneda]

@rewitt1 sorry for asking, but when can i have this fix ? I was trying to use this container for our CI purpose and was blocked because of this.

[rewitt1]

Hi @manojkiraneda,

I would like to give you an ETA, but I'm unsure of when I'll be able to get to it.

If you need a solution sooner, you can build the container image locally, substituting in the curl arguments you need.

[manojkiraneda]

sure thanks @rewitt1 will do.