The Publish Beam SDK Snapshots job is flaky

[github-actions]

The Publish Beam SDK Snapshots is failing over 80% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_Publish_Beam_SDK_Snapshots.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=1&var-Workflow=Publish%20Beam%20SDK%20Snapshots

[damccorm]

Looking at https://github.com/apache/beam/actions/workflows/beam_Publish_Beam_SDK_Snapshots.yml this seems like it was a false positive, this has not been >80% flaky

[github-actions]

Reopening since the workflow is still flaky

[github-actions]

Reopening since the workflow is still flaky

[shunping]

I confirm that it started to turn flaky again since Jan 15, 2025.

[liferoad]

@Amar3tto let us prioritize this. Thanks.

[liferoad]

Traceback (most recent call last):
  File "/home/runner/work/beam/beam/sdks/java/container/license_scripts/pull_licenses_java.py", line 68, in pull_from_url
    url_read = urlopen(Request(url, headers=***
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open
    return self.do_open(http.client.HTTPConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 1347, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>
ERROR:root:Invalid url for jFormatString-3.0.0: http://www.gnu.org/licenses/lgpl.html. Retrying...

could be blocked by gpu.org.

[Amar3tto]

I agree to try to optimize pull_licenses_java.

But it also fails for python:container and it's not related to licenses.

[ahmedabu98]

We can track the python failures here #33744

[ahmedabu98]

@shunping mentioned we could optimize pull_licenses_java by caching them in the repo: https://github.com/apache/beam/tree/master/sdks/java/container/license_scripts/manual_licenses

I'm not sure how often licenses change, but it would be good to at least use the cache as a fall-back instead of failing after exhausting the retries

[shunping]

.take-issue

I am working on a PR to cache the licenses.