NIRC2 emulation

[astronomerdave]

I'm titling this issue as "NIRC2 emulation" because there is first a problem with the emulator to solve, in order to reproduce the problem locally.

NIRC2 is using a heavily modified version of camerad, which is experiencing reliability problems (it crashes routinely).

The branch currently in use for observing is the first commit (e06df1b) of the version.W branch.

I am not currently able to reproduce the problem locally with the emulator. But here are the steps to follow which mimic those used with the real instrument. Perhaps the failure to crash locally has something to do with the emulator?

These are the steps to follow locally.

1. branch version.W

2. checkout e06df1b

3. You will need to modify camerad/CMakeLists.txt for your local environment, specifically the location of OpenCV, cfitsio and CCFits, because the commit is using particular versions of those, which are hard-coded (that was one of the attempts to solve the problem). This should involve removing or commenting-out the following lines:

# set( ENV{OpenCV_DIR} "/usr/local/caltech/opencv-454" )
# include_directories( "/usr/local/caltech/heasarc/include" )
# set(CCFITS_LIB "/usr/local/caltech/heasarc/lib/libCCfits.so" CACHE FILEPATH "Path to CCfits library" FORCE)
# set(CFITS_LIB "/usr/local/caltech/heasarc/lib/libcfitsio.so" CACHE FILEPATH "Path to cfitsio library" FORCE)

and changing these lines:

find_library(CCFITS_LIB CCfits ...
find_library(CFITS_LIB cfitsio ...

to something like:

find_library(CCFITS_LIB CCfits NAMES libCCfits PATHS /usr/lib/x86_64-linux-gnu)
find_library(CFITS_LIB cfitsio NAMES libcfitsio  PATHS /usr/lib/x86_64-linux-gnu)

4. Download the attached alad_camerad.cfg.TXT file, modifying the directories for your environment, of course (e.g. IMDIR, EMULATOR_SYSTEM, DEFAULT_FIRMWARE). You will use this file to run camerad and emulator. NB. I had to add .TXT so that github would accept them.

5. Download the attached nirc2.system.TXT file and nirc2.acf.TXT files (specified in the config as EMULATOR_SYSTEM=/path/to/nirc2.system and DEFAULT_FIRMWARE=/path/to/nirc2.acf)

6. Build using:

cd build
cmake .. -DINSTR=nirc2
make

I suggest running the emulator in the foreground so you can see what it's doing (it otherwise does not log to a file):

(base) dhale@milliways:~/Projects/camera-interface/build$ ../bin/emulator ../demo/alad_camerad.cfg
(Emulator::main) 38 lines read from ../demo/alad_camerad.cfg
(Emulator::Server::configure_server) applied 5 configuration lines to server
2024-08-01T23:37:38.371 (Archon::Interface::configure_controller) complete

Running camerad in the foreground or background matters less since there is logging, but if you run it in the foreground there are a few progress indicators that only go to stdout:

(base) dhale@milliways:~/Projects/camera-interface/build$ ../bin/camerad ../demo/alad_camerad.cfg
2024-08-01T23:37:51.472  (Camera::main) config:TM_ZONE_LOG=local
2024-08-01T23:37:51.473  (Common::FitsKeys::addkey) [DEBUG] added key: TM_ZONE=UTC (STRING) // time zone
2024-08-01T23:37:51.473  (Camera::main) config:TM_ZONE=UTC
2024-08-01T23:37:51.473  (Camera::main) config:TZ_ENV=PST8PDT,M3.2.0/2,M11.1.0/2
2024-08-01T16:37:51.473  (Camera::main) this version built Aug  1 2024 16:35:27
2024-08-01T16:37:51.473  (Common::FitsKeys::addkey) [DEBUG] added key: CAMD_VER=Aug  1 2024 16:35:27 (STRING) // camerad build date
2024-08-01T16:37:51.473  (Camera::main) 38 lines read from ../demo/alad_camerad.cfg
2024-08-01T16:37:51.473  (Camera::Camera::longerror) true
2024-08-01T16:37:51.473  (Camera::Server::configure_server) applied 5 configuration lines to server
2024-08-01T16:37:51.473  (Camera::Camera::imdir) image directory: /tmp
2024-08-01T16:37:51.473  (Camera::Camera::basename) base name is n
2024-08-01T16:37:51.473  (Archon::Interface::configure_controller) [DEBUG] default_sampmode=3 1 1
2024-08-01T16:37:51.473  (Archon::Interface::configure_controller) [DEBUG] default_exptime=1000
2024-08-01T16:37:51.473  (Archon::Interface::configure_controller) [DEBUG] default_roi=1024 1024
2024-08-01T16:37:51.473  (Archon::Interface::configure_controller) applied 25 configuration lines to controller

Connect to camerad using a script or telnet or whatever; I will show only the commands (as if typed in a telnet session):

open
load
longexposure false
basename eraseme
sampmode 4 1 1
roi 128 152
exptime 0  # this will respond with 11 msec, the min exptime for this roi
expose

This should create a file e.g. /imdir/yyymmdd/eraseme_unp_yyymmddhhmmss.fits

expose

Another file, so far, so good

sampmode 2 1 1000
expose

This will take some time because it will take 1000 exposures. While this is happening, the fits files are created and being added to (because they are multi-extension, adding each as they come in) and while writing a ".writing" suffix is appended. Here is where the emulator departs from the real instrument. The real instrument will segfault because of a CCFits error (it's in CCFits and not something thrown that I can catch). But using the emulator there is a different problem -- I am getting a timeout waiting for readout. The emulator has sent the correct number of frames; you can check this by sending

printstatus

If you follwed this example exactly, then printstatus should show that the current frame counter is at 2002. That is 2 for the first two expose commands with sampmode 4 1 1, and 2000 frames for the sampmode 2 1 1000.

Explanation:
sampmode 4 is UTR (Up The Ramp) and 4 1 1 means UTR with 1 sample and 1 "ramp" with each ramp as an extension. So 4 1 1 is basically a single frame. This example did that twice, so 2 frames.

sampmode 2 is CDS (Correlated Double Sampling) where each "exposure" generates two frames so 2 1 1000 means CDS, 1000 times (the middle 1 never changes for this mode). 1000 CDS frames generates 1000 pairs, or 2000 frames, so if the frame counter is now at 2002 then the emulator has generated all the frames it was supposed to. camerad got lost along the way.

This is a problem specific to the emulator. It would be nice to fix that problem so that we could better reproduce the actual problem.