[Bug]: Beam's licence check is failing

[stankiewicz]

What happened?

it looks like moving from beautifulsoup4-4.12.3 to beautifulsoup4-4.13.0 there is missing typing_extensions dependency.

good run

Collecting beautifulsoup4<5.0,>=4.9.0
  Downloading beautifulsoup4-4.12.3-py3-none-any.whl (147 kB)
Collecting pyyaml<7,>=6.0.1
  Downloading PyYAML-6.0.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (746 kB)
Collecting tenacity<9.0,>=6.1.0
  Downloading tenacity-8.5.0-py3-none-any.whl (28 kB)
Collecting soupsieve>1.2
  Downloading soupsieve-2.6-py3-none-any.whl (36 kB)
Installing collected packages: soupsieve, beautifulsoup4, pyyaml, tenacity
Successfully installed beautifulsoup4-4.12.3 pyyaml-6.0.2 soupsieve-2.6 tenacity-8.5.0

bad run

Collecting beautifulsoup4<5.0,>=4.9.0
  Downloading beautifulsoup4-4.13.0-py3-none-any.whl (184 kB)
Collecting pyyaml<7,>=6.0.1
  Downloading PyYAML-6.0.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (746 kB)
Collecting tenacity<9.0,>=6.1.0
  Downloading tenacity-8.5.0-py3-none-any.whl (28 kB)
Collecting soupsieve>1.2
  Downloading soupsieve-2.6-py3-none-any.whl (36 kB)
Requirement already satisfied: typing-extensions in /home/runner/.local/lib/python3.8/site-packages/setuptools/_vendor (from beautifulsoup4<5.0,>=4.9.0->-r /runner/_work/beam/beam/sdks/java/container/license_scripts/requirement.txt (line 19)) (4.12.2)
Installing collected packages: soupsieve, beautifulsoup4, pyyaml, tenacity
Successfully installed beautifulsoup4-4.13.0 pyyaml-6.0.2 soupsieve-2.6 tenacity-8.5.0
Executing python /runner/_work/beam/beam/sdks/java/container/license_scripts/pull_licenses_java.py --license_index=/runner/_work/beam/beam/sdks/java/container/build/reports/dependency-license/index.json        --output_dir=/runner/_work/beam/beam/sdks/java/container/build/target/java_third_party_licenses        --dep_url_yaml=/runner/_work/beam/beam/sdks/java/container/license_scripts/dep_urls_java.yaml        --manual_license_path=/runner/_work/beam/beam/sdks/java/container/license_scripts/manual_licenses        --use_license_cache
Traceback (most recent call last):
  File "/runner/_work/beam/beam/sdks/java/container/license_scripts/pull_licenses_java.py", line 33, in <module>
    from bs4 import BeautifulSoup
  File "/home/runner/.local/lib/python3.8/site-packages/bs4/__init__.py", line 64, in <module>
    from .builder import (
  File "/home/runner/.local/lib/python3.8/site-packages/bs4/builder/__init__.py", line 24, in <module>
    from bs4.element import (
  File "/home/runner/.local/lib/python3.8/site-packages/bs4/element.py", line 9, in <module>
    from bs4.css import CSS
  File "/home/runner/.local/lib/python3.8/site-packages/bs4/css.py", line 27, in <module>
    from bs4._typing import _NamespaceMapping
  File "/home/runner/.local/lib/python3.8/site-packages/bs4/_typing.py", line 16, in <module>
    from typing_extensions import (
ModuleNotFoundError: No module named 'typing_extensions'

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[stankiewicz]

maybe related https://bugs.launchpad.net/beautifulsoup/+bug/2097262

update: same with beautifulsoup4-4.13.1

[liferoad]

We probably should not use Python 3.8 with :sdks:java:container:pullLicenses

[stankiewicz]

I think this is the default python that is running under eclipse-temurin java container.

[lostluck]

Not sure exactly what caused this, but we may want to bump the go-licenses version to the v2 tree instead. (google/go-licenses#270)

https://pkg.go.dev/github.com/google/go-licenses/v2?tab=versions

That would be my guess to fix the Go side of the problem. Litterally change it to go-licenses/v2@latest from go-licenses@latest here:

https://github.com/apache/beam/blob/master/release/go-licenses/Dockerfile#L23

[lostluck]

Nevermind, confirmed that it's the same beautiful soup issue, and not this warning from the Go side.

+ go-licenses save github.com/apache/beam/sdks/v2/java/container --save_path=/output/licenses
W0203 18:14:00.748774      28 library.go:101] "golang.org/x/sys/unix" contains non-Go code that can't be inspected for further dependencies:
/go/pkg/mod/golang.org/x/[email protected]/unix/asm_linux_amd64.s
+ go-licenses csv github.com/apache/beam/sdks/v2/java/container

[liferoad]

Opened #33844 to track upgrading Python.