Pip packaging

[reginaldl]

Hi,

First of all, thank you for tkrzw. I love it!!!

I would like to add proper python packaging for this awesome library.
So whenever someone needs tkrzw-python, they could simply do: pip install tkrzw

I am wondering how we could ship the original tkrzw library (estraier/tkrzw) as part of this python package?
(So we wouldn't have to compile+install tkrzw before installing the python wrapper)

I'm sending a first diff that updates setup.py a little bit in order to allow easy packaging.
I think there are probably 2 ways to include the original tkrzw:

• Compile the python wrapper with the static version of tkrzw. This only requires to change a few compile flags I assume.
• Add the tkrzw.(so|dylib) to the python package (although I'd have to look up how to correctly update ldconfig for python). This is doable by adding packages and packages_data to the setup function.

I'd appreciate some input, thank you!

[estraier]

Hi Reginald,
Thanks for the suggestion.
I think that supporting pip is a great idea to be more user-friendly.
I'm checking your pull request now.

Though I'm not sure how to make pip packages, separating the core C++ library and the Python pip package seems reasonable. If we include binary objects in the pip package, the maintainer have to prepare many files for each platforms x86, x64, ppc, arm etc. Now that Tkrzw's binary packages are going to be ready on Fedora and Debian, and hopefully other platforms including MacOS in the future, the installation of the core library becomes easier.

[reginaldl]

Oh Debian packages are coming up?! (and Ubuntu I assume?). That would be great!

As for maintaining per architecture package, I guess you’d have to do that anyway since it’s CPython module, right?
Wouldn’t be safer to bundle the lib and the python module together to avoid dealing with API dependencies?
Thank you!

[estraier]

Though I'm not sure how the approval process is, there's an on-going Debian package proposal.
https://salsa.debian.org/debian/tkrzw

I have no experience of making PIP packages for native libraries.
However, this project for a native DBM library doesn't include binary files but include some source files.
https://pypi.org/project/py-mdbm/#files
I think it's normal to package C or C++ source code in the pip package.

[reginaldl]

Hi Mikio,

ah I see what you mean now. You'd prefer to package sources and let the package installer to compile them.
In that case I'm not sure there's a benefit to having a pip package (besides not typing make/make install).
I just tried py-mdbm, it does attempt to compile its sources and it actually fails on my machine :) (macOS).
Alternatively, I believe pip supports both types of package for a given version, I'd need to double check but basically you can ship compiled version of your lib for some platforms (say the main ones for instance, linux x86_64, macos, windows) and ship a source package for all other platforms (which will automatically compile on the host machine).
As an example, I posted a wheel package here:
https://gitlab.com/reginaldl/tkrzw-python/-/blob/staticlink/dist/tkrzw-0.1-cp38-cp38-linux_x86_64.whl

This is a compiled version of tkrzw-python, statically linked with tkrzw (which wouldn't be necessary with tkrzw system packages). This is just a zip file you can list files with unzip -l tkrzw-0.1-cp38-cp38-linux_x86_64.whl if you're curious. Note the platform linux_x86_64. This is just an example in the context of this discussion.
Eventually, I believe this could be automated on Github Actions (they support the 3 major OSes)

[estraier]

Nice. If we can use an automation platform to server binary packages, it's reasonable to utilize it for user's convenience.

[estraier]

BTW, as we discussed in the pull request, the package name can be either small letters or capitalized.
Is the package name used as the name in the command line "pip install xxx"?
If so, using small letter seems better.
Moreover, the Python guideline recommends us to use small letter for module names and package names.
https://www.python.org/dev/peps/pep-0008/#package-and-module-names
Currently, the module name "tkrzw" and the package name is "Tkrzw".
As I'm not particular to this case issue, I just want to be conforming to the convention of the community.

[reginaldl]

Nice catch! I've updated the pull request accordingly.

[jruere]

I've not found the library in PyPI. Is there anything blocking this?

[rhizoome]

@estraier First I am a big fan of your database-software since many years

I read in your FAQ that you would prefer other people to provide binaries. I did a prototype of building a wheel for pypi.org. (I will fully automate this of course)

1. Either I could put this on a github-repository under my user and use my pypi-account to upload to pypi.org (you can get ownership of the package)
2. Or I could prepare everything in a pull-request on this repository. In that case you would have to create a pypi-account and enter the secrets in the github-environment settings on this project.

Definitely 2. is more correct, but if we do 1:

• If you have/create a pypi.org account I can give you ownership of the package, but still upload through my account.
• I can fix problems with the build myself
• There are no PRs, so things move faster
• There is less communication needed and it is less a burden for you

Let me know what you think.

EDIT: I started the automation: https://github.com/ganwell/tkrzw-wheels/actions/runs/3578797497

[estraier]

Thanks for the suggestion.

I'm not knowledgeable of pypi.org so I appreciate that you take initiative to manage the pypi package.
To me, either #1 or #2 is OK. If you prefer #2, let's go with it.

[rhizoome]

I'm not knowledgeable of pypi.org so I appreciate that you take initiative to manage the pypi package. To me, either #1 or #2 is OK. If you prefer #2, let's go with it.

Thanks for the interest. I started the automated build of wheels (python binary packages). I plan to support at least musllinux, "normal linux" and macOS. If there aren't any mayor problems I will also support windows. See also PR #5.

I do the variant 1. and give you full owner access to both the github- and the pypi-project. So you have full access in case something happens and I can't support the builds anymore.

I hope this will increase the user-base of your great database-software. Easy install is important for many users.

EDIT: here is how a build looks https://github.com/ganwell/tkrzw-wheels/actions/runs/3588311518. It's not important just to show you how it is automated.

[rhizoome]

The github workflows are now building and uploading musllinux_1_2_x86_64 and manylinux_2_27_x86_64 to test.pypi.org.

https://test.pypi.org/project/tkrzw/0.1.28.6/#files

https://github.com/ganwell/tkrzw-wheels/actions/runs/3606283119

It was quite some work to get all the details right. I hope I can add macOS next week.