Missing binary on Linux install via pypi in newer versions (2.39.0 - 2.40.0)

[bnubald]

What happened?

Installing newer versions of eccodes not finding/including binary.

When attempting to use eccodes (latest -> v2.40.0), I was getting an error, and using self check, showed me the included error. In trying to find out if it was a system issue, I tried installing versions since the binary was included, and found the earlier versions working, here is a list:

• 2.37.0 (working)
• 2.38.0 (working)
• 2.38.1 (working)
• 2.38.3 (working)
• 2.39.0 (not working)
• 2.39.1 (not working)
• 2.39.2 (not working)
• 2.40.0 (not working)

This is the output when working, and I'm able to use these versions to read data without issue:

❯ export ECCODES_PYTHON_TRACE_LIB_SEARCH=1
❯ pip install eccodes==2.37
❯ python -m eccodes selfcheck
eccodes lib search: trying to find binary wheel
eccodes lib search: looking in /data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes.libs
eccodes lib search: returning wheel library from /data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes.libs/libeccodes-aadb7c6a.so
eccodes lib search: versions: {'eccodes': '2.37.0'}
Found: ecCodes v2.37.0.
Library: /data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes.libs/libeccodes-aadb7c6a.so
Definitions: /MEMFS/definitions
Samples: /MEMFS/samples
Your system is ready.

What are the steps to reproduce the bug?

❯ pip install eccodes
❯ export ECCODES_PYTHON_TRACE_LIB_SEARCH=1
❯ python -m eccodes selfcheck

Version

v2.39.0, v2.39.1, v2.39.2, v2.40.0

Platform (OS and architecture)

CentOS Linux 7 (Core), and on Rocky Linux 9.5 (Blue Onyx)

Relevant log output

eccodes lib search: trying to find binary wheel
eccodes lib search: looking in /data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes.libs
eccodes lib search: looking in /data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes/.dylibs
eccodes lib search: looking in /data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes
eccodes lib search: did not find library from wheel; try to find as separate lib
Traceback (most recent call last):
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes/__init__.py", line 13, in <module>
    from .eccodes import *  # noqa
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes/eccodes.py", line 12, in <module>
    from gribapi import (
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/gribapi/gribapi.py", line 34, in <module>
    from gribapi.errors import GribInternalError
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/gribapi/errors.py", line 16, in <module>
    from .bindings import ENC, ffi, lib
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/gribapi/bindings.py", line 102, in <module>
    library_path = find_binary_libs("eccodes")
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/gribapi/bindings.py", line 95, in find_binary_libs
    import findlibs
  File "/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/findlibs/__init__.py", line 78, in <module>
    lib_name: str, pkg_name: str, preload_deps: bool | None = None
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

Relevant diff?: 2.38.3...2.39.0

Accompanying data

https://github.com/pydata/xarray-data/blob/d90be9863e8f8bbba980c1070b70628a18ab8d48/era5-2mt-2019-03-uk.grib

Organisation

British Antarctic Survey

[bnubald]

The following eccodes.libs directory that was created when installing 2.37-2.38.3 is missing with 2.39.0+:

/data/hpcdata/users/username/miniconda3/envs/ffmpeg_del/lib/python3.9/site-packages/eccodes.libs

[iainrussell]

Hi @bnubald, I suspect the reason you are seeing this change of behaviour is that from version 2.39.0 of ecCodes, we changed the version of the Docker image we use to build the wheels from manylinux_2_17_x86_64 to manylinux_2_28_x86_64. This was because we found that GihUb Actions could no longer run on the older version. This means that the binary wheel will not work on such old OS versions as Centos 7. I think that one (hopefully) easy solution for you, since you seem to be in a conda environment, is to 'conda install eccodes -c conda-forge' in order to get the binary installed.

[dschneider-wxs]

Hi - I'm seeing this same issue installing eccodes with pip in a docker image using python:3.11-slim and python:3.10-slim. I had 2.40.0 pinned locally in a venv (macos) which includes the binary, but I can't find a version that installs the binary in my container. I tried 2.37.0, 2.38.0, 2.38.3, and 2.40.0.

root@ffd3859b124b:~#  ECCODES_PYTHON_TRACE_LIB_SEARCH=1 python -m cfgrib selfcheck
eccodes lib search: trying to find binary wheel
eccodes lib search: looking in /app/.venv/lib/python3.11/site-packages/eccodes.libs
eccodes lib search: looking in /app/.venv/lib/python3.11/site-packages/eccodes/.dylibs
eccodes lib search: looking in /app/.venv/lib/python3.11/site-packages/eccodes
eccodes lib search: did not find library from wheel; try to find as separate lib
eccodes lib search: findlibs returned None
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/app/.venv/lib/python3.11/site-packages/cfgrib/__init__.py", line 20, in <module>
    from .cfmessage import COMPUTED_KEYS
  File "/app/.venv/lib/python3.11/site-packages/cfgrib/cfmessage.py", line 29, in <module>
    from . import abc, messages
  File "/app/.venv/lib/python3.11/site-packages/cfgrib/messages.py", line 28, in <module>
    import eccodes  # type: ignore
    ^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/eccodes/__init__.py", line 13, in <module>
    from .eccodes import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/eccodes/eccodes.py", line 12, in <module>
    from gribapi import (
  File "/app/.venv/lib/python3.11/site-packages/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/gribapi/gribapi.py", line 34, in <module>
    from gribapi.errors import GribInternalError
  File "/app/.venv/lib/python3.11/site-packages/gribapi/errors.py", line 16, in <module>
    from .bindings import ENC, ffi, lib
  File "/app/.venv/lib/python3.11/site-packages/gribapi/bindings.py", line 106, in <module>
    raise RuntimeError("Cannot find the ecCodes library")
RuntimeError: Cannot find the ecCodes library

any other solutions besides trying to install it manually?

[dschneider-wxs]

related #104

[bnubald]

Hi @bnubald, I suspect the reason you are seeing this change of behaviour is that from version 2.39.0 of ecCodes, we changed the version of the Docker image we use to build the wheels from manylinux_2_17_x86_64 to manylinux_2_28_x86_64. This was because we found that GihUb Actions could no longer run on the older version. This means that the binary wheel will not work on such old OS versions as Centos 7. I think that one (hopefully) easy solution for you, since you seem to be in a conda environment, is to 'conda install eccodes -c conda-forge' in order to get the binary installed.

Thanks @iainrussell, can confirm your suspicion, a direct install via pypi on Rocky Linux 9.5 (Blue Onyx) is working with the newer versions.

I was attempting to migrate away from needing conda and trying to strip such dependencies, I will probably set version specifier for now since we're in the process of migrating to Rocky 9.

Thanks again! Happy to close if @dschneider-wxs's issue is covered in #104