META: Desired test behavior

[stefanv]

The current spin release's test command does not work for the majority of packages in the ecosystem. This is related to support added for editable installs, and the subsequent decision to add --pyargs to each pytest invocation.

This led to related problems, including #208, #204, #203, and numpy/numpy#26525

This issue is an attempt to write up the requirements for spin test for various configurations, which is a cross-product of:

• Installation methodology: Editable install vs built into build-install.
• Source or classic layout: Are sources in src/ or packagename/?
• Tests bundled / unbundled: Are tests only in ./tests, ./packagename/**/tests, or included as part of the package?
• conftest.py: Does the package include a conftest.py (which may, perhaps, import the package)?

I was hoping to find a Single Invocation of pytest that would handle all of these cases, but I suspect that that may not be possible. Instead, I am now creating test packages of the various flavors to see to what degree we'd have to customize the pytest call.

In a future comment, I may document the implications of each of the various configurations.

The biggest obstacle I have right now is how to support filename test specifications, such as

spin test numpy/linalg/tests/test_linalg.py
spin test tests/some/test_module.py

for the case where tests are not shipped along with the package.