Where is wavelength in results?, no binned spectrum created.

[tjamqt]

First attempt at running SPIFF on WIndows 11. UltraNest install requires a C++ compiler, which I currently don't have, so used --scipy-only option. The job ran to completion, though lengthy at 3.5 hours - lots of download time and bandwidth apparently.

X:\Temp>spiff-lv2 --ra 35.21992062 --dec 24.962390 --reference-crd-epoch-yr 2016.0 --reference-pmra-masyr 0.0 --reference-pmdec-masyr 0.0 --target-name J022052 --outdir ./runs --save-figs --scipy-only

This status message was printed repeatedly while running:
[batch] Download attempt 1/15 succeeded in 51.38s (downloader=s3, timeout=10000s, curl_resume=False)
-> source: DataLink (access_url) resolved to FITS
-> saved to C:\Users\tjask\AppData\Local\Temp\spherex_dl_1dh48qqp.fits (20.70 MB)
-> warning: could not read header metadata: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

After the last file was processed the job ended with these status messages:
[batch] Wrote 266 rows to X:\Temp\runs\J022052_RA35.219921_DEC24.962390\results.csv
[batch] Wrote JSONL with covariances to X:\Temp\runs\J022052_RA35.219921_DEC24.962390\results.jsonl
[batch] warning: could not write binned SPHEREx spectrum: No valid SPIFF-like rows remain for SPHEREx binning after filtering.
[batch] Total FITS analyzed (non-skipped, including failures): 266
[LV2_ANALYZED_COUNT] 266

No binned spectrum file was created and the results.jsonl file was length 0 (empty). Results.csv was created with detailed information for each image cutout file, but I could find no data column indicating an estimated wavelength for the target pixel location in the images.

I tried using spiff-compile-results, which successfullly created compiled_results.csv (data content mostly the same as results.csv), but faulted in created the binned output:

X:\Temp>spiff-compile-results --outdir ./runs/J022052_RA35.219921_DEC24.962390 --write-binned-spectrum
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Program Files\Python310\Scripts\spiff-compile-results.exe_main.py", line 5, in
File "C:\Program Files\Python310\lib\site-packages\spiff\results.py", line 97, in main
out = compile_results_csvs(
File "C:\Program Files\Python310\lib\site-packages\spiff\results.py", line 69, in compile_results_csvs
write_binned_spherex_spectrum_csv(out, binned_output_csv)
File "C:\Program Files\Python310\lib\site-packages\spiff\spherex_binning.py", line 174, in write_binned_spherex_spectrum_csv
binned = build_binned_spherex_spectrum(frame)
File "C:\Program Files\Python310\lib\site-packages\spiff\spherex_binning.py", line 103, in build_binned_spherex_spectrum
raise ValueError("No valid SPIFF-like rows remain for SPHEREx binning after filtering.")
ValueError: No valid SPIFF-like rows remain for SPHEREx binning after filtering.

This software package does a hugely impressive amount of analysis and I'm looking forward to getting it work!

[tjamqt]

I found the wavelength columns in results.csv (duh!)

I tried running lv2 again with more options specified (maybe default weren't working?) and still no binned output:

X:\Temp>spiff-lv2 --ra 35.21992062 --dec 24.962390 --reference-crd-epoch-yr 2016.0 --reference-pmra-masyr 0.0 --reference-pmdec-masyr 0.0 --target-name J022052 --outdir ./runs --save-figs --scipy-only --use-sia2 --keep-fits --write-binned-spectrum --binned-spectrum-path ./runs

[tjamqt]

Re: binned output
In spherex_binning.py, line 61, in the case of --scipy_only (no psf_un.* columns), the test:

if {"wavelength_um", "flux_ujy", "flux_err_ujy"}.issubset(columns):

fails as the column names do not match those that appear in the frame that is written to results.csv. (lv2.py CSV_FIELDS?)