Build platform-specific wheels containing libmagic #294

ddelange · 2023-09-06T12:59:42Z

This PR builds self-contained wheels as discussed in #233. For Windows users, this renders python-magic-bin from @julian-r obsolete.

pip install these wheels

pip can install from GitHub Release assets from my fork:

# space separated list of URLs for --find-links
export PIP_FIND_LINKS='https://github.com/ddelange/python-magic/releases/expanded_assets/0.4.28.post9'
pip install --force-reinstall python-magic

- python-magic-0.4.27.tar.gz
+ python-magic-0.4.28.tar.gz
- python_magic-0.4.27-py2.py3-none-any.whl
+ python_magic-0.4.28-py2.py3-none-macosx_10_9_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-macosx_11_0_arm64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_s390x.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-win32.whl
+ python_magic-0.4.28-py2.py3-none-win_amd64.whl

The wheels:

Come bundled with libmagic (.dylib on mac, .so on nix, .ddl on win) - no additional user action needed
- latest libmagic (5.46)
  - building from source on linux and macos with maximum backwards compatibility
  - using MSYS2 built libmagic on windows (mingw32 and mingw64)
- skipping i686 linux wheels as they don't pass pytest ref ca4def3
Are forward compatible, they will install on cpython 2.7 and all current and future cpython 3.5+ distributions
Are built for the major cpu architectures (and also for alpine linux)
- ~~Linux architectures limited by availability: https://pkgs.org/search/?q=file-libs~~ now building from source on linux
- For windows I couldn't find compiled libmagic for ARM64, skipped it
Are uploaded as GitHub Release assets, as well as upload to PyPI, whenever a GitHub Release is (pre)released.
- 🚩 @ahupp for this, you still need to add wheels.yml as trusted publisher here

CI/CD

dists build with official cibuildwheel on GitHub Actions, and they build in parallel:

The PyPI publish step is failing here ^ because my fork is not a trusted publisher
For any system not covered by these wheels, pip will fall back to the source distribution, which will check for proper libmagic on the systen at install time.

fix #137, fix #288, fix #225, fix #276, fix #248, fix #87, fix #139, fix #233, fix #73, fix #60, fix #34, fix #293, fix #233, fix #278, fix #262, fix #248, fix #238, fix #145, fix #61, fix #12, fix #295, fix #311, fix #312, fix #313, fix #321, fix #332, fix #249, fix #333

apirogov · 2023-09-12T13:49:39Z

This is nice! Hope this will be merged soon!

Just ran into issues with my library being not usable by Mac and Windows users because I rely on python-magic. If there are wheels, I don't need to find a workaround or replace the library :)

python-magic-bin did not work for some of them, by the way.

stumpylog · 2023-09-12T14:40:45Z

.github/workflows/main.yml

+      with:
+        files: dist/*
+
+    - name: Upload to PyPI


A small improvement here might be to use the PyPa Action instead: https://github.com/pypa/gh-action-pypi-publish

The big advantage is trusted publishing, instead of storing a password or token as a secret

that's cool, thanks for sharing!

@ahupp shall I make that change and you set it up on PyPI side?

Sounds like trusted publishing is the way to go. I recently got this email:

Hi ddelange!
Earlier this year, we announced that PyPI would require all users to enable a form of two-factor authentication on their accounts by the end of 2023.

Keeping your PyPI account secure is important to all of us. We encourage you to enable two-factor authentication on your PyPI account as soon as possible.

What forms of 2FA can I use?
We currently offer two main forms of 2FA for your account:

Security device including modern browsers (preferred) (e.g. Yubikey, Google Titan)
Authentication app (e.g. Google Authenticator)
Once one of these secure forms is enabled on your account, you will also need to use either Trusted Publishers (preferred) or API tokens to upload to PyPI.

What do I do if I lose my 2FA device?
As part of 2FA enrollment, you will receive one-time use recovery codes. One of them must be used to confirm receipt before 2FA is fully active. Keep these recovery codes safe - they are equivalent to your 2FA device. Should you lose access > to your 2FA device, use a recovery code to log in and swap your 2FA to a new device.

Read more aboutrecovery codes.

Why is PyPI requiring 2FA?
Keeping all users of PyPI is a shared responsibility we take seriously. Strong passwords combined with 2FA is a recognized secure practice for over a decade.

We are requiring 2FA to protect your account and the packages you upload, and to protect PyPI itself from malicious actors. The most damaging attacks are account takeover and malicious package upload.

To see this and other security events for your account, visit your account security history.

Read more on this blog post.

If you run into problems, read the FAQ page. If the solutions there are unable to resolve the issue, contact us via [email protected].

Thanks,
The PyPI Admins

@ahupp so the last thing for you to do is adding this repo as trusted publisher to https://pypi.org/manage/project/python-magic/settings/publishing/

.github/workflows/main.yml

ahupp · 2023-09-28T17:18:10Z

This is huge, thank you! Apology for the delay I thought I'd commented earlier but guess not. I'll look this over soon; I didn't quite understand how bad the binary dep situation was expecially on windows.

.github/workflows/main.yml

jean-humann · 2023-12-03T17:02:05Z

@ahupp @stumpylog could we have this one merged (and released) by the end of the year please ?

Haiz14 · 2024-12-10T06:48:09Z

When will 0.4.28 be released? Like, this year?

edit: sorry if it came out as rude. It takes quite some complexity to merge complex PR's. And asking a developer about when there is a new release, is never really the right thing.

C0rn3j · 2024-12-28T17:40:52Z

@ahupp This PR has 6 approvals, and from the looks of it, 0 controversy, would it be possible to review & pull it sometime soon?
I just ran into one of the issues this fixes myself.

Thanks in advance!

nightlark · 2025-02-04T20:50:18Z

I'm trying to understand what the current situation is with using libmagic for file type identification in Python, and I've found at least 13 separate packages on PyPI that are in various states of not being maintained -- of them, this PR seems to be the version that includes the most complete set of wheels with pre-compiled copies of libmagic (aside from not being available on PyPI).

Would it make sense to create a fork of python-magic based on this PR by @ddelange, perhaps under an org with at least 2 maintainers that can review PRs and publish updated wheels to help minimize the burden for a single person and hopefully avoid adding to the unmaintained package count?

@ahupp has done an great job with python-magic and this is in no way a criticism of their work or minimizing the time they've spent working on and maintaining it -- of the libmagic packages I found on PyPI, python-magic seems to be in by far the best overall shape. Life happens and situations change so time isn't as plentifully as it once was, and I know how it is for things to get really busy and maintaining an open source project can be draining, leading to burnout. This happens literally all the time (as evidenced by all of the libmagic PyPI packages that haven't been updated since before 2019).

ddelange · 2025-02-04T22:20:15Z

@ahupp fwiw I'm happy to become a maintainer of this project 👍

There is an ongoing debate on how to embed libmagic library in the pypi package. For windows we have to rely on python-magic-bin for the time being. See ahupp/python-magic#294

stumpylog · 2025-02-14T16:11:35Z

I would also be happy to help maintain. We use this in paperless-ngx, so I'm happy to work on keeping it up to date, resolving issues, etc

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: update magic/compat.py simplify tests into something more delarative

ddelange · 2025-02-14T18:08:13Z

Hi @ahupp 👋

I fixed the broken tests I just pulled from master. Did you forget to track the test.py file by any chance? I commented out that test for now.

I also upgraded this PR to build wheels and run CI using the newly released file-5.46.

Any response from you on this PR would be greatly appreciated 🙏

.github/workflows/ci.yml

.github/workflows/wheels.yml

Co-authored-by: Christian Clauss <[email protected]>

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: add support for python 3.13 Unbreak various things

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: format with ruff

ddelange · 2025-03-18T21:32:55Z

so I pulled master again and now Windows wheels are failing with the following error. I don't know why it now started failing. I don't see changes to https://github.com/julian-r/file-windows, nor relevant changes in master. Anyway, better to get rid of that dependency so I pushed for a more official libmagic wheel build for Windows. Fix incoming, with latest libmagic 4.46 as a bonus! 🎉

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-l6641oui\cp312-win_amd64\venv-test\Lib\site-packages\magic\__init__.py", line 120, in __init__
      magic_load(self.cookie, magic_file)
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-l6641oui\cp312-win_amd64\venv-test\Lib\site-packages\magic\__init__.py", line 384, in magic_load
      return _magic_load(cookie, coerce_filename(filename))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-l6641oui\cp312-win_amd64\venv-test\Lib\site-packages\magic\__init__.py", line 275, in errorcheck_negative_one
      raise MagicException(err)
  magic.MagicException: b"File 5.44 supports only version 18 magic files. `C:\\Users\\runneradmin\\AppData\\Local\\Temp\\cibw-run-l6641oui\\cp312-win_amd64\\venv-test\\Lib\\site-packages\\magic\\magic.mgc' is version 20"

ddelange · 2025-03-18T21:59:44Z

for anyone interested in latest libmagic on windows: I created a new GitHub release on my fork, and updated the pip install instructions in this PR description.

ddelange changed the title ~~Build ABI3 wheels containing libmagic~~ Build platform-specific wheels containing libmagic Sep 7, 2023

ddelange force-pushed the abi3-wheels branch 2 times, most recently from a98f13b to dc9c393 Compare September 7, 2023 07:26

ddelange mentioned this pull request Sep 7, 2023

Build platform-specific wheels containing libmagic ddelange/python-magic#1

Closed

ddelange force-pushed the abi3-wheels branch 12 times, most recently from d672b91 to 14f7dbb Compare September 7, 2023 10:03

Build platform-specific wheels containing libmagic

ec952d7

ddelange force-pushed the abi3-wheels branch from 14f7dbb to ec952d7 Compare September 7, 2023 10:09

jspraul mentioned this pull request Sep 9, 2023

Update file to 5.44 hey-red/Mime#49

Closed

apirogov mentioned this pull request Sep 12, 2023

python-magic has no wheels for Windows/Mac Materials-Data-Science-and-Informatics/metador-core#85

Open

stumpylog reviewed Sep 12, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

Move persmissions into job scope, remove ABI3 reference

a437409

ahupp mentioned this pull request Oct 10, 2023

Binary distribution for libmagic on Windows #293

Open

Switch to PyPI trusted publishing

4a715e2

ddelange mentioned this pull request Nov 23, 2023

bugfix: add path to load libmagic bin on Windows #249

Open

ddelange commented Nov 23, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

ddelange mentioned this pull request Dec 10, 2024

magic.from_file(file) generate "File Not Found" exception when file is a linux (symbolic) link #336

Open

C0rn3j mentioned this pull request Dec 28, 2024

Tauon blindly trusts file extension, causing files to be read wrong Taiko2k/Tauon#1362

Open

ddelange added 4 commits February 14, 2025 19:28

Merge branch 'master' of https://github.com/ahupp/python-magic into a…

43c0c99

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: update magic/compat.py simplify tests into something more delarative

Fix tests

3e51048

Use older typing syntax

d7b1171

Use older typing syntax

620d78f

cclauss reviewed Feb 14, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

cclauss reviewed Feb 14, 2025

View reviewed changes

.github/workflows/wheels.yml Show resolved Hide resolved

cclauss reviewed Feb 14, 2025

View reviewed changes

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

ddelange commented Feb 14, 2025

View reviewed changes

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

ddelange and others added 2 commits February 14, 2025 21:27

Apply suggestions from code review

d3e886c

Co-authored-by: Christian Clauss <[email protected]>

Merge branch 'master' of https://github.com/ahupp/python-magic into a…

2bb9fa8

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: add support for python 3.13 Unbreak various things

ddelange mentioned this pull request Feb 20, 2025

Python 3.13 support #339

Closed

ahupp force-pushed the master branch 2 times, most recently from 7c1a018 to 5a89644 Compare March 2, 2025 01:12

ddelange added 4 commits March 12, 2025 16:42

Merge branch 'master' of https://github.com/ahupp/python-magic into a…

c8a599b

…bi3-wheels * 'master' of https://github.com/ahupp/python-magic: format with ruff

Run ruff format

e1b154c

Skip install_source on windows

d111ace

Typo

33c827e

Get latest libmagic from MSYS2 for Windows (#10)

688edf0

ddelange mentioned this pull request Mar 18, 2025

Get latest libmagic from MSYS2 for Windows ddelange/python-magic#10

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build platform-specific wheels containing libmagic #294

Build platform-specific wheels containing libmagic #294

ddelange commented Sep 6, 2023 •

edited

Loading

apirogov commented Sep 12, 2023

stumpylog Sep 12, 2023

ddelange Sep 12, 2023

ddelange Oct 26, 2023 •

edited

Loading

ddelange Nov 13, 2023

ddelange Nov 22, 2023

ahupp commented Sep 28, 2023

jean-humann commented Dec 3, 2023

Haiz14 commented Dec 10, 2024 •

edited

Loading

C0rn3j commented Dec 28, 2024 •

edited

Loading

nightlark commented Feb 4, 2025 •

edited

Loading

ddelange commented Feb 4, 2025

stumpylog commented Feb 14, 2025

ddelange commented Feb 14, 2025

ddelange commented Mar 18, 2025 •

edited

Loading

ddelange commented Mar 18, 2025

Build platform-specific wheels containing libmagic #294

Are you sure you want to change the base?

Build platform-specific wheels containing libmagic #294

Conversation

ddelange commented Sep 6, 2023 • edited Loading

pip install these wheels

CI/CD

apirogov commented Sep 12, 2023

stumpylog Sep 12, 2023

Choose a reason for hiding this comment

ddelange Sep 12, 2023

Choose a reason for hiding this comment

ddelange Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

ddelange Nov 13, 2023

Choose a reason for hiding this comment

ddelange Nov 22, 2023

Choose a reason for hiding this comment

ahupp commented Sep 28, 2023

jean-humann commented Dec 3, 2023

Haiz14 commented Dec 10, 2024 • edited Loading

C0rn3j commented Dec 28, 2024 • edited Loading

nightlark commented Feb 4, 2025 • edited Loading

ddelange commented Feb 4, 2025

stumpylog commented Feb 14, 2025

ddelange commented Feb 14, 2025

ddelange commented Mar 18, 2025 • edited Loading

ddelange commented Mar 18, 2025

ddelange commented Sep 6, 2023 •

edited

Loading

ddelange Oct 26, 2023 •

edited

Loading

Haiz14 commented Dec 10, 2024 •

edited

Loading

C0rn3j commented Dec 28, 2024 •

edited

Loading

nightlark commented Feb 4, 2025 •

edited

Loading

ddelange commented Mar 18, 2025 •

edited

Loading