GH-41105: [Python][Docs] Update PyArrow installation docs for conda package split

[amoeba]

Do not merge until some discussion is had about how to time this relative to conda-forge/arrow-cpp-feedstock#1376. Additionally, consider hot-patching this into arrow-site if appropriate.

What changes are included in this PR?

Updates to the Python installation docs to reflect the in-progress change splitting PyArrow on conda-forge into three separate packages. Specifically:

1. Add a note in the conda section highlighting that there are three packages and linking to a new section (2) in order to provide more information
2. Add a new section, linked from (1), providing a comparison of each package as a table

Are these changes tested?

These are just docs changes. I have built them locally and they look fine.

Are there any user-facing changes?

Just docs.

• GitHub Issue: [Python][Docs] Update documentation for installing split PyArrow packages from conda #41105

[amoeba]

Note to reviewers: I used "Core" here for lack of a better word. I wasn't aware of any common way of referring to the functionality libarrow provides to PyArrow.

[AlenkaF]

Maybe Arrow C++ minimal or Arrow minimal? if it isn't too long (mentioned here: https://arrow.apache.org/docs/dev/developers/cpp/building.html#optional-components)

[amoeba]

Thanks for pointing that out, it might work.

[amoeba]

The rows in the table above are based on @raulcd's breakdown on the mailing list but I think we want to be careful about what's listed here. I think the best thing would be for this table to fully align with the submodules exposed in PyArrow since that's what the user is most familiar with. And we might even consider renaming the "Component" column to "Module" and using the literal module names. i.e., parquet instead of Parquet so it's clear we're talking about being able to run import pyarrow.parquet or not.

This would mean the table isn't complete yet (json, csv, filesystems, orc, more?).

[AlenkaF]

Agree with aligning with PyArrow on the names 👍

[amoeba]

In looking closer, I think it makes sense to align on conda package names too so in cccc7ff I updated the table to include the names of the conda package for each component.

I need to look at where the modules I mentioned above (JSON, etc) live because I think that should be clear.

[jorisvandenbossche]

I need to look at where the modules I mentioned above (JSON, etc) live because I think that should be clear.

Right now all the things you mentioned (json, csv, filesystems, orc) are included in the main libarrow.so, and so can only be either enabled or disabled (cannot be installed separately).
For the filesystems, there is work under way to actually build them as separate libraries, so we will have to update this doc section once that is done. For ORC I am actually wondering if we should open an issue about splitting that in a separate library (although liborc is not that big of a dependency, so it might not be super important)

[amoeba]

Thanks both of you. The latest draft now tries to make it clear which Python modules each package provides. It'd be good to get reviews on the overall style and also the correctnes/completeness.

[raulcd]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 4fc75b7

Submitted crossbow builds: ursacomputing/crossbow @ actions-eab855c0d2

Task	Status
preview-docs

[jorisvandenbossche]

We might want to add an example about how to create an env with a custom selection by installing the core package plus just one additional module. I think that will be something like conda install pyarrow-core libparquet

[amoeba]

Oh, that's really neat. I added a subsection about custom selections but let me know if you think it needs higher billing.

Next steps on this PR are:

• I want to wrap up GH-41105: [Python][Docs] Update PyArrow installation docs for conda package split #41135 (comment) which is to understand where some of the Python components that aren't in their own conda package exist
• Actually test the custom selections which I assume we can do once Create subsets of pyarrow package with pyarrow-core < pyarrow < pyarrow-all and update to Arrow v16.0.0 conda-forge/arrow-cpp-feedstock#1376 is merged.

[amoeba]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: be37b52

Submitted crossbow builds: ursacomputing/crossbow @ actions-c33978f0b1

Task	Status
preview-docs

[ianmcook]

Thanks for doing this @amoeba. If possible, can we include some more information about which parts of pyarrow.compute are available / usable in the pyarrow-core package? See the discussion about that here: conda-forge/arrow-cpp-feedstock#1255 (comment)

[amoeba]

Thanks @ianmcook, I debated how to present this. The two options were to either list the Python modules or the shared libraries and I went with the latter in the current revision, though this may have been mostly out of convenience. Your suggestion would be to go with the former and that's probably better.

I'll take a look a that and see about updating the table.

[ianmcook]

Presenting the table showing how the three packages map onto the shared libraries seems worthwhile regardless, because it's a clean mapping.

I don't know whether the Python modules map as cleanly onto these three packages. It sounds like maybe they don't. The comment from @h-vetinari suggests that some pyarrow.compute functions require libarrow-acero while others don't.

So maybe a longer prose explanation or a bulleted list will be required to explain the latter.

[amoeba]

Good points, thanks.

[raulcd]

I thought pyarrow.compute required acero but that might not be the case. We will have to investigate if that's the case or in the other hand compute does not require acero at all. Probably someone with more knowledge on pyarrow like @jorisvandenbossche can help answer that one.

[jorisvandenbossche]

I just answered on the feedstock issue (conda-forge/arrow-cpp-feedstock#1255 (comment)) before seeing the discussion here, but indeed, the individual compute kernels (what is in pyarrow.compute) are fully usable without acero being present. You only cannot compose them / run then in parallel on batches / ... for which you need am Acero plan.

We have an issue about splitting the compute kernels into its own shared library (#25025), and that would allow it to be installed separately as well like a libarrow-compute, but until then it's either all included in the main libarrow.so or not (if you would build libarrow.so without compute support)

[jorisvandenbossche]

If we list that here, I think we should also say that those cloud filesystem are planned to moved out of pyarrow-core in the next release, and so you should install pyarrow if you want to rely on those being present

[jorisvandenbossche]

pyarrow.fs itself is always available, though (with at a minimum just the LocalFileSystem)

[amoeba]

Good point, thanks. Changed in 51f2fde.

[amoeba]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 51f2fde

Submitted crossbow builds: ursacomputing/crossbow @ actions-a5a94423db

Task	Status
preview-docs

[amoeba]

Thanks everyone for the reviews. I've addressed what comments have been left (thank you!) and I think this is very close to merge-able. The preview for the latest changeset is at http://crossbow.voltrondata.com/pr_docs/41135/python/install.html#using-conda.

[amoeba]

Thanks for the reviews all.

I think we'll want this to appear on the website sooner than the 17 release I've filed apache/arrow-site#520.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit f5ac05c.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

[jorisvandenbossche]

I think we'll want this to appear on the website sooner than the 17 release I've filed apache/arrow-site#520.

Thanks for dong that! I also labeled the issue here with backport-candidate label, so that in case we do a 16.1.1 release, it will get included