Breaking change in 7e5bae4 (importlib_metadata 5)

[inno]

entry_points previously returned a dict-like object and other projects have taken advantage of this functionality (stevedore, for example, iterates via .items()). As entry_points now returns a list-like object, it breaks these previously working dict-like methods.

Issue appears with this change: 7e5bae4

[bmrobin]

seconded. i don't have specific details like @inno but i am able to trace this back to upgrading to 5.0.0. the package that triggered the error was celery

for the time being i have downgraded to <5.0.0

[tsibley]

Breaks flake8 too, which uses .get().

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.14/x64/bin/flake8", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/cli.py", line 22, in main
    app.run(argv)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 363, in run
    self._run(argv)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 350, in _run
    self.initialize(argv)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 330, in initialize
    self.find_plugins(config_finder)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/main/application.py", line 153, in find_plugins
    self.check_plugins = plugin_manager.Checkers(local_plugins.extension)
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 357, in __init__
    self.namespace, local_plugins=local_plugins
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 238, in __init__
    self._load_entrypoint_plugins()
  File "/opt/hostedtoolcache/Python/3.7.14/x64/lib/python3.7/site-packages/flake8/plugins/manager.py", line 254, in _load_entrypoint_plugins
    eps = importlib_metadata.entry_points().get(self.namespace, ())
AttributeError: 'EntryPoints' object has no attribute 'get'

[tsibley]

…though it looks like that was intentional looking at 7e5bae4, which removed a previously-deprecated interface and corresponding warnings silencer specifically for flake8.

Ah. Right. This is the whole SelectableGroups debacle I'd managed to forget. :-(

[jaraco]

That's correct. Support for dict-based access was deprecated in importlib_metadata 3.6 (and Python 3.10) and intentionally removed in the 5.0 release (and planned for the Python 3.12 release).

Is it possible the usage in stevedore didn't trigger the deprecation warnings?

[jaraco]

Breaks flake8 too, which uses .get().

…though it looks like that was intentional looking at 7e5bae4, which removed a previously-deprecated interface and corresponding warnings silencer specifically for flake8.

The corresponding warnings silencer for flake8 was removed in #319. The only reason it shows up in that specific commit is because I made these removal commits against v4.4.0 to ensure that I wasn't removing anything that wasn't part of Python 3.10. The actual diff for this change can be seen in #405.

It's my understanding that later versions of flake8 provide a workaround for this issue but also that flake8 pins against and older version of importlib_metadata, so shouldn't get this behavior anyway. Any idea why an older version of flake8 is ending up with a brand new version of importlib_metadata?

[tsibley]

Any idea why an older version of flake8 is ending up with a brand new version of importlib_metadata?

Other things in the dep chain need a newer importlib_metadata, so pip walks back the flake8 versions until it finds flake8 3.9.2 which is the last release that includes importlib_metadata without a version pin. It doesn't work, of course, but pip can't know that.

If I manually pin to flake8 >=4.0.0, the first release which includes the importlib_metadata pin, then pip instead walks back the versions of other things to a point where the version of importlib_metadata they need is compatible with the older version flake8 needs. (I'm not sure yet if this is acceptable or not for our usage, but at least it's something.)

[jaraco]

Other things in the dep chain need a newer importlib_metadata, so pip walks back the flake8 versions until it finds flake8 3.9.2 which is the last release that includes importlib_metadata without a version pin.

Oh, interesting. That makes sense now that you explain it.

I'm not sure yet if this is acceptable or not for our usage

I'd expect that to be the case, as importlib_metadata is mostly compatible between 3.6 and 5.0.

I do believe that flake8 has removed the usage of the deprecated code paths in late releases, so they probably can remove the pin and help ease the pain.

If I manually pin to flake8 >=4.0.0, the first release which includes the importlib_metadata pin, then pip instead walks back the versions

That also seems like a reasonable workaround.

[jaraco]

By the way, if it would help to yank the release and delay the break for a day or a week or a month to stop the bleeding, that’s certainly possible.

[tony]

Workaround for poetry + flake8 + python 3.7+: This pin fixed the resolution for me

flake8 = [
  { version = "*", python = "^3.7" },
  { version = ">=5", python = "^3.8" },
]

I truly don't know why the resolution has these issues. One possibility I haven't ruled out: https://github.com/PyCQA/flake8/blob/e94ee2b5f1801354b940cfe830b9160852915aec/setup.cfg#L45

CC: Two earlier issues #406 #407

[inno]

Unfortunately importlib_metadata is used deep within many automated systems (build systems, containers, etc...), so the deprecation warnings are likely unseen by most and ignored by many others. It's super easy to see "this functionality is deprecated" and treat it as "I won't use it more in the future" instead of "I will stop using it now".

On that note, it might be helpful to have messaging regarding a date or version when breaking changes are coming. That way folks can plan for it, especially for a commonly used package.

[jaraco]

I really appreciate the feedback.

On that note, it might be helpful to have messaging regarding a date or version when breaking changes are coming. That way folks can plan for it, especially for a commonly used package.

That's useful feedback. I've been operating under a less rigorous versioning strategy and I've only seen a few examples of projects that have adopted a rigorous strategy for breaking changes (Python recently with the two-version deprecation period, and pip with the breaking changes annually). I've found that different breaking changes have different blast radii, so it's difficult to devise a specific strategy that accommodates the variety of impacts. I try to ascertain the level of impact and use that to devise a timeline for deprecation/removal. Often, I will put a specific date in the code "not to be removed before YYYY-MM-DD" to provide downstream consumers a minimum time. I have found that attempting to say specifically when or in what version a breaking change will occur is folly and often gets missed.

I didn't put a particular date on this change in part because the deprecation itself was so impactful, I'd expected the major consumers to have already addressed the issue early and the 16 month delay gave sufficient time for systems to upgrade.

I do think it would have helped to put a minimum date on the deprecation warning. I'll do that for the future and possibly remaining deprecation warnings.

It's super easy to see "this functionality is deprecated" and treat it as "I won't use it more in the future" instead of "I will stop using it now".

That's an interesting perspective. I've never taken that perspective. Whenever I've encountered a DeprecationWarning, that says to me that the functionality is slated for removal and could be removed at any time in any future version (unless stated otherwise). Interestingly, the Python documentation doesn't clarify what the meaning of a DeprecationWarning is.

If some maintainers hold the opinion that DeprecationWarning just means limit additional usage, they will necessarily encounter the breakage every time it happens (even if it's years in the future). That is, unless no project ever changes any expectation ever.

I could imagine some projects employing a PendingDeprecationWarning to signal what you're suggesting, that further use be curtailed, but that existing uses are still okay for now... but it still implies that a deprecation is imminent and when that happens, one should take action and adapt.

Per the Python deprecation policy, this behavior is expected to change in Python 3.12 as well, which is why I wanted to get the backport change out sooner, so users would have a chance to adapt where they have more control over the behavior (pinning dependencies, adding backports, etc).

Unfortunately importlib_metadata is used deep within many automated systems (build systems, containers, etc...), so the deprecation warnings are likely unseen by most and ignored by many others.

Good point. If there's a specific example where the directly-affected project (the project using the interface) did not have visibility to the deprecation warning, I'd like to investigate that, because that's an assumption I rely on to enact these changes. I know other projects downstream of the direct-consuming projects will be affected, and I rely on the direct consumers to take action when deprecations affect their projects and their users.

My current understanding is that the big projects affected are openstack (stevedore) and flake8, both projects with which I engaged directly at the time of the deprecation (or prior to it) and proposed multiple fixes to address the breakage early. If any project had reached out and requested that the removal be delayed, I certainly would have considered that and likely built it into the approach. Stability and predictability are important, but if a project chooses to ignore the warnings, they do so at their own risk (true of any project).