net/http: Client does not retry idempotent requests on transport failure

[gopherbot]

by patrick.allen.higgins:

rfc2616 section 8.1.4 says: "Client software SHOULD reopen the transport connection
and retransmit the aborted sequence of requests without user interaction so long as the
request sequence is idempotent"

http.Client.Get() does not do this and does not document that callers are responsible
for doing so.

[bradfitz]

Comment 1:

If only server authors consistently made their GET handlers be idempotent.
I don't think this is safe to do by default. It could be per-Client opt-in, or we could
just document it.

Status changed to Accepted.

[rsc]

Comment 2:

What does Chrome do?

[bradfitz]

Comment 3:

See also issue #3514 (a race between client re-using connections and the server shutting
down that connection).  Fixing this would kinda fix that, if it were safe to do this.
I don't know what Chrome does.

[bradfitz]

Comment 4:

[+willchan for Chrome question]

[evmar]

Comment 5:

Relevant Chrome code, I think:
"1264 // This method determines whether it is safe to resend the request after an
1265 // IO error.  It can only be called in response to request header or body
1266 // write errors or response header read errors.  It should not be used in
1267 // other cases, such as a Connect error."
http://git.chromium.org/gitweb/?p=chromium.git;a=blob;f=net/http/http_network_transaction.cc;h=caa8f11909ff40e5cee9dd6be3814233220d36f1;hb=HEAD#l1264

[bradfitz]

Comment 6:

And:
1342 bool HttpNetworkTransaction::ShouldResendRequest(int error) const {
1343   bool connection_is_proven = stream_->IsConnectionReused();
1344   bool has_received_headers = GetResponseHeaders() != NULL;
1345 
1346   // NOTE: we resend a request only if we reused a keep-alive connection.
1347   // This automatically prevents an infinite resend loop because we'll run
1348   // out of the cached keep-alive connections eventually.
1349   if (connection_is_proven && !has_received_headers)
1350     return true;
1351   return false;
1352 }

[gopherbot]

Comment 7 by willchan@chromium.org:

* If we get a RST at the TCP level when we try to reuse a persistent connection, we
retry (because middleboxes often time out connections and send RSTs if you still try to
use it). For desktop, you can be wasteful and use TCP keepalives to mitigate this too.
For mobile, it wakes up the radio and maybe costs money, so you shouldn't do that and
just deal with it. We close persistent connections after a fixed period to minimize the
times we get these errors. For mobile, we do deferred socket closes (wait for the radio
to wakeup due to other HTTP requests, and then close all timed out sockets).
* If you pipeline requests and get a transport error, we pray that HEADs and GETs are
actually idempotent and retry.

[rsc]

Comment 8:

Labels changed: added priority-later, removed priority-triage.

[rsc]

Comment 9:

Labels changed: added go1.1maybe, removed go1.1.

[robpike]

Comment 10:

Labels changed: added go1.2maybe, removed go1.1maybe.

[rsc]

Comment 12:

Labels changed: added feature.

[robpike]

Comment 13:

Not for 1.2.

Labels changed: removed go1.2maybe.

[rogpeppe]

Comment 14:

FWIW we just came across an issue that triggered this problem
in a way that happens reliably.
The problem happens when GOMAXPROCS=1 and there's something
doing lots of work (in our case it was reading a bunch of
data from disk and gzipping it) and then issues an http request.
The connection times out while the code is spinning, but
the http transport code doesn't get scheduled to see the EOF
until the code stops spinning, so we see the EOF almost exactly
at the same time as the request is issued, so the request
fails because it doesn't realise that the EOF is out of band.
The problem is exacerbated because the window for the race is larger than
it needs to be (the request is marked as in flight before the
request has been transferred to the writing goroutine, instead
of incrementing the count just before conn.Write is called).
Here's a program that demonstrates the issue reliably for me:
   http://play.golang.org/p/bVf9wsCJSx
The second GET request fails with "can't write HTTP request on broken connection" or
"EOF".
It succeeds when run with GOMAXPROCS>0.
For the time being we can work around this avoiding connection reuse
on all our http client connections, but this is less than ideal.

[rogpeppe]

Comment 15:

> It succeeds when run with GOMAXPROCS>0.
GOMAXPROCS>1, of course!

[rsc]

Comment 16:

Labels changed: added go1.3maybe.