libstore/filetransfer,http-binary-cache-store: disable on substituter 500s instead of throwing

Right now, if a substituter returns a 5xx other than 501, 505 or 511, nix will keep trying to use it even though it is broken/misconfigured at least four more times before trying to use the next one (given the necessary patch is applied, otherwise it throws), resulting in unnecessary log spam.
Also, if you are receiving 504s (such as if an nginx proxy is trying to connect to a substituter but timing out) this can lead to rather nasty waits - I have personally seen it take 20-30s, with it timing out at each individual request, before it timing out completely and giving up. Awful.

This doesn't really make sense because 500s wont go away unless the server is fixed? This change would make nix disable an http substituter on the first 500 it gets and immediately switch to the next substituter (intuitive default behaviour?)

This also preserves `settings.tryFallback` as it will still throw if false and all the substituters have been disabled because they have had (terminal) errors. (that logic is unrelated to the http binary cache store and is in substition-goal/store-api, see NixOS#13301 )

linked to NixOS#13301

![nix error log](https://github.com/user-attachments/assets/cc2a7911-4298-43fa-a3b6-b68eaf88132a)

Author: philipwilk

src/libstore/filetransfer.cc

@@ -496,11 +496,9 @@ struct curlFileTransfer : public FileTransfer
                     // Most 4xx errors are client errors and are probably not worth retrying:
                     //   * 408 means the server timed out waiting for us, so we try again
                     err = Misc;
-                } else if (httpStatus == 501 || httpStatus == 505 || httpStatus == 511) {
-                    // Let's treat most 5xx (server) errors as transient, except for a handful:
-                    //   * 501 not implemented
-                    //   * 505 http version not supported
-                    //   * 511 we're behind a captive portal
+                } else if (httpStatus >= 500 && httpStatus <= 599) {
+                    // a server is unlikely to 5xx unless there is a critical configuration/system error
+                    // we can't recover from an error that isn't our fault so give up
                     err = Misc;
                 } else {
 // Don't bother retrying on certain cURL errors either

src/libstore/http-binary-cache-store.cc

@@ -96,7 +96,7 @@ class HttpBinaryCacheStore : public virtual BinaryCacheStore
     void maybeDisable()
     {
         auto state(_state.lock());
-        if (state->enabled && settings.tryFallback) {
+        if (state->enabled) {
             int t = 60;
             printError("disabling binary cache '%s' for %s seconds", config->getHumanReadableURI(), t);
             state->enabled = false;
@@ -196,8 +196,14 @@ class HttpBinaryCacheStore : public virtual BinaryCacheStore
                     try {
                         (*callbackPtr)(std::move(result.get().data));
                     } catch (FileTransferError & e) {
-                        if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden)
+                        if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden) {
                             return (*callbackPtr)({});
+                        }
+                        // if the server is having errors then give up on it
+                        if (e.error == FileTransfer::Misc) {
+                            maybeDisable();
+                            return (*callbackPtr)({});
+                        }
                         maybeDisable();
                         callbackPtr->rethrow();
                     } catch (...) {
@@ -219,6 +225,10 @@ class HttpBinaryCacheStore : public virtual BinaryCacheStore
         } catch (FileTransferError & e) {
             if (e.error == FileTransfer::NotFound)
                 return std::nullopt;
+            if (e.error == FileTransfer::Misc) {
+                maybeDisable();
+                return std::nullopt;
+            }
             maybeDisable();
             throw;
         }