Shiny Ska3 for maintainers

This describes notes and actions related to development of the Shiny Ska3 distribution. The current definition of shiny is:

• Python 3.8.3
• Contains the latest releases of all the core dependencies. Currently numpy is at 1.18.5.
• Adds:
  • Jupyterlab
  • bootstrap-astropy for docs building
• Conda 4.8.3
• Includes distributions built for linux (CentOS-7), MacOS (Catalina), Windows (Windows 10).

To do / notes

• Console scripts don't work on Windows. Need to change packages with console scripts to be arch-specific.
• Make ska3-flight and ska3-core with the right versions
• Update combine_arch.py to work with Windows
• Remove Ska.Table
• Perl for linux and OSX
• Full tests on HEAD
• Deprecation warning in bleach that causes sparkles post_check fail
• Check xija_gui_fit
• Check aca_view [FAIL]
• Latest kadi with continuity fix
• Rsync cheru installation to get the latest kadi

Installing

This is in flux, but the following should work to install the latest. It will pull core packages from defaults, astropy, and sherpa channels.

Standalone

conda create -n ska3-shiny --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/shiny ska3-flight

On linux and MacOSX you can optionally install ska3-perl-latest as well. This is mostly for aspect-specific apps like starcheck.

Build process

Minimal build environment

conda create -n ska-builder setuptools_scm gitpython conda-build conda-verify jinja2 pyyaml python=3.8

On MacOSX Catalina:

cd ~/git/skare3
git checkout master
git pull
conda activate ska-builder

rm -rf builds

# Build everything at master.
python ska_builder.py --tag master

# Ska3-static (perl)
cd ../skare3-static
python build.py --build-root=../skare
cd ../skare3

# Make a temporary environment using external package repos to solve for
# package configuration and download all necessary packages.
# Note that conda-forge contains a quaternion package, so channel order matters.
conda create -n shiny-tmp -c file://$PWD/builds -c defaults -c astropy -c sherpa -c conda-forge \
 ska3-flight-latest ska3-core-latest ska3-perl-latest

# Get into the environment and upload all the packages to the upstream
# repository package provider.
conda activate shiny-tmp
rm -rf repo-shiny
python upload_packages.py --repo-dir=repo-shiny
conda index repo-shiny

# Now do a dry run to create environment using only the uploaded packages. If this
# succeeds to resolve packages then move on to next step.
# --override-channels is important to avoid accidentally getting defaults packages.
conda create -n test-test --dry-run --override-channels -c file://$PWD/repo-shiny \
 ska3-flight-latest ska3-core-latest ska3-perl-latest

# Put them up on the server.  Note that the output directory (shiny) along with the
# subdirs noarch, {win,osx,linux}-64 need to all exist already.
python upload_packages.py --user=aca --host=kady \
  --repo-dir=/proj/sot/ska/www/ASPECT_ICXC/ska3-conda/shiny [--dry-run]

#################
# As aca on kady
# Use any existing conda executable with a recent-enough version:
# conda >= 4.8 and conda-build >= 3.18
#################
/export/tom/miniconda37/bin/conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/shiny

# Final install, back as user on CentOS-7 or Windows
conda create -n ska3-shiny --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/shiny \
  ska3-flight-latest ska3-core-latest ska3-perl-latest

On linux (CentOS-7) and Windows-10

# For linux this needs to be done on HEAD (CentOS-7+) so the 
# Sybase-dependent perl packages can build.

cd ~/git/skare3
git checkout shiny
git pull

rm -rf builds  # linux
rm -r -fo builds  # Window

conda activate ska-builder

python ska_builder.py --arch-specific --tag master

# Perl packages [ linux ONLY ]
cd ../skare3-static
python build.py all --build-root=../skare
cd ../skare3

# Need to force using the conda-forge version of perl because the defaults linux
# build appears to be broken. Do this by putting this build into our builds/ area
# where it gets priority in subsequent `conda create` commands.
wget https://conda.anaconda.org/conda-forge/linux-64/perl-5.26.2-h516909a_1006.tar.bz2
mv perl-5.26.2-*.tar.bz2 builds/linux-64
conda index builds

# Note: cannot put conda-forge in front of shiny because then it picks up the other 
# quaternion package instead of our Quaternion.
# linux
conda create -n shiny-tmp -c file://$PWD/builds -c defaults -c astropy -c sherpa \
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/shiny -c conda-forge \
  ska3-flight-latest ska3-core-latest ska3-perl-latest
# Windows
conda create -n shiny-tmp -c file:///Users/aldcroft/git/skare3/builds -c defaults -c sherpa -c astropy `
  -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/shiny -c conda-forge `
  ska3-flight-latest ska3-core-latest

conda activate shiny-tmp
rm -rf repo-shiny  # linux
rm -r -fo repo-shiny  # Windows
python upload_packages.py --repo-dir repo-shiny
conda index repo-shiny

# linux
conda create -n test-test  --dry-run --override-channels \
  -c file://$PWD/repo-shiny \
  ska3-flight-latest ska3-core-latest ska3-perl-latest
# Windows
conda create -n test-test  --dry-run --override-channels `
  -c file:///Users/aldcroft/git/skare3/repo-shiny `
  ska3-flight-latest ska3-core-latest

# Upload packages to kady server
python upload_packages.py --user=aca --host=kady --repo-dir=/proj/sot/ska/www/ASPECT_ICXC/ska3-conda/shiny [--dry-run]

#################
# As aca on kady
# Use any existing conda executable with a recent-enough version:
# conda >= 4.8 and conda-build >= 3.18
#################
/export/tom/miniconda37/bin/conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/shiny

# Final install, back as user on CentOS-7 or Windows
# linux
conda create -n ska3-shiny --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/shiny \
  ska3-flight-latest ska3-core-latest ska3-perl-latest
# Windows
conda create -n ska3-shiny --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/shiny `
  ska3-flight-latest ska3-core-latest

Production HEAD

As aca user:

wget https://repo.anaconda.com/miniconda/Miniconda3-py38_4.8.3-Linux-x86_64.sha
bash Miniconda3-py38_4.8.3-Linux-x86_64.sh -b -p /proj/sot/ska3/shiny  

setenv PATH /proj/sot/ska3/shiny/bin:$PATH

conda install --override-channels -c https://icxc.cfa.harvard.edu/aspect/ska3-conda/shiny \
  ska3-flight ska3-perl

cd /proj/sot/ska3/shiny
# (final production environment will have these links, but shiny environment will have some links and some real directories in checkout phase).
#ln -s /proj/sot/ska/data ./
#ln -s /proj/sot/ska/www ./

Production GRETA

As SOT user:

rsync -avz [email protected]:/proj/sot/ska3/shiny/ /proj/sot/ska3/shiny/

Testing

Set up local data directories.

• Install arc from master and run.
• Copy a subset of cheta content directories (probably same as used for cheta_sync and/or the ska_testr regression) and run shadow update process via task_schedule.
• Copy kadi data and run shadow update process.

Known issues

Forward and fix

Some packages are not that easy to test on the side but can tolerate a downtime of a week or so.

• Mica?

Package details

Chandra.Time has a shiny branch that includes https://github.com/sot/Chandra.Time/pull/46 which changes the default behavior for times like 2001:001 (without hr/min/sec). Since this is likely to break some processing code, it is only being included in the shiny distribution.

Migration hints

See https://github.com/sot/xija/pull/87 for an example.

Fixing times for the Chandra.Time default change

Many instances of times like 2001:001, especially those in regression tests, need to be changed to a fully-qualified value like 2001:001:12:00:00. These commands will do the trick, but only do this within a git-configured directory so you can easily see the changes. See

find . -name '*.py' | xargs perl -pi -e  "s/'([12]\d\d\d:\d\d\d)'/'\1:12:00:00'/g" 
find . -name '*.py' | xargs perl -pi -e  's/"([12]\d\d\d:\d\d\d)"/"\1:12:00:00"/g'

# For thermal model specs with an epoch built in
find . -name '*.json' | xargs perl -pi -e  's/"([12]\d\d\d:\d\d\d)"/"\1:12:00:00"/g'

pytest.ini

Modern pytest shows all warnings in the test logging by default. There are certain warnings that show up which are benign and should be ignored with the following in pytest.ini.

[pytest]
filterwarnings =
    # See https://github.com/numpy/numpy/issues/11788 for why this is benign
    ignore:numpy.ufunc size changed:RuntimeWarning
    ignore:the imp module is deprecated in favour of importlib:DeprecationWarning
    ignore:parse functions are required to provide a named argument:PendingDeprecationWarning

Note that if there is a pytest.ini file then anything which was in setup.cfg is ignored. So check for that file and move any pytest-related config into pytest.ini.

Namespace packages

For namespace packages like Chandra.cmd_states, put something like below in at least one of the test modules.

# Check that this test file is in the same package as the imported Chandra.cmd_states.
# Due to subtleties with pytest test collection and native namespace pacakges,
# running `pytest Chandra/cmd_states` in the git repo will end up getting the installed
# Chandra.cmd_states not the local one.  Use `python setup.py test` instead.
assert Path(__file__).parent.parent == Path(Chandra.cmd_states.__file__).parent

Assembling Package Meta

There is a Github workflow that creates the meta.yaml for the ska3 meta-packages. To run it, first make sure you have skare3_tools installed and run:

from skare3_tools import github
repository = github.Repository('sot/skare3')
repository.dispatch_event('conda-meta-yaml', client_payload={'skare3_branch': '2020.13+shiny', 'version': '2020.13+shiny'})

Upon success, got to the corresponding workflow page in the actions tab, and you can download a zip file containing the generated meta.yaml files.