start job fails when trying to set status

[jdries]

After starting a job and setting the application id, we also set the status.
This is of lesser importance, as status will eventually be configured by job tracker.

However, it seems that the 'start_job' call can fail on this bit. This for instance makes integrationtests less stable, but can also diminish user experience.
In the case below, the job was still started and finished successfully, so there was no real reason to return this error. Also returning the error might cause user to try and start job again, which is not a good idea when it has already been started.

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 463, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 802, in urlopen
    retries = retries.increment(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/util/retry.py", line 552, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/opt/venv/lib64/python3.8/site-packages/urllib3/connectionpool.py", line 463, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib64/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 379, in _do_request
    response = do_request()
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 363, in <lambda>
    do_request = lambda: self._session.request(
  File "/opt/venv/lib64/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib64/python3.8/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/venv/lib64/python3.8/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/users/auth.py", line 95, in decorated
    return f(*args, **kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/views.py", line 1013, in start_job
    backend_implementation.batch_jobs.start_job(job_id=job_id, user=user)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py", line 1710, in start_job
    self._start_job(job_id, user, _get_vault_token,proxy_user=proxy_user)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/backend.py", line 2262, in _start_job
    dbl_registry.set_status(job_id, user_id, JOB_STATUS.QUEUED)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/job_registry.py", line 872, in set_status
    self.elastic_job_registry.set_status(job_id=job_id, status=status, started=started, finished=finished)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 547, in set_status
    return self._update(job_id=job_id, data=data)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 553, in _update
    return self._do_request("PATCH", f"/jobs/{job_id}", json=data)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/jobregistry.py", line 382, in _do_request
    raise EjrApiError(f"Failed to do EJR API request `{method} {url}`") from e
openeo_driver.jobregistry.EjrApiError: Failed to do EJR API request `PATCH https://jobregistry.vgt.vito.be/jobs/j-2502240709194ee69e95d3c4a6bdc4fb`

https://kibana-infra.vgt.vito.be/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(columns:!(message),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'907d8590-4f1b-11ed-8cc4-3747d5233c59',key:req_id,negate:!f,params:(query:r-2502240709194b6380a2a158a27cf4f3),type:phrase),query:(match:(req_id:(query:r-2502240709194b6380a2a158a27cf4f3,type:phrase))))),index:'907d8590-4f1b-11ed-8cc4-3747d5233c59',interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))