Sometimes number of sequences drops

[RemingtonRohel]

We've seen this at PGR and INV now. The number of sequences will sometimes drop precipitously, or bounce back and forth between normal and not. The rx_signal_processing log shows the problem to be a drastic increase in the time to transfer samples between the ringbuffer and the GPU, reasons unknown. Theo noticed that at INV the swap was sometimes full when this occurred, and decreasing the reliance on swap for the computer made the problem go away temporarily. The most recent occurrence (today, at PGR) did not have an increased usage of swap. Unsure what the core problem is here, but the ringbuffer being moved into swap is certainly still a possibility. We may be able to move the entire ringbuffer into pinned memory (https://stackoverflow.com/questions/5736968/why-is-cuda-pinned-memory-so-fast) to avoid swapping and increase reliability. I'm not sure why this has only recently become an issue, and only on some sites (namely, those that haven't been updated this year? coincidence?).

[RemingtonRohel]

So usrp_driver does call mlock to keep the ringbuffer in memory, but there is no check that the call returns success. In fact, on the lab computer, it fails with ENOMEM, which could stem from several different things (see https://www.man7.org/linux/man-pages/man2/mlock.2.html). I'll keep looking into this, but bottom line is that mlock probably has been failing at all sites for some time, and we simply didn't have the code set up to log it.

[RemingtonRohel]

We can add the following line to /etc/security/limits.conf:

radar - memlock unlimited

This will remove any limitation on the maximum size of page-locked memory (and amount of shared memory) that a process started by the radar user can allocate. Doing this on the lab computer made both mlock calls (for ringbuffer and pulse buffer) successful. Updating the installation docs and adding error checks with logging to the usrp_driver.cpp script are both pertinent changes in light of this.

[RemingtonRohel]

I'm hopeful that #563 will fix this issue, but I'll hold off closing this issue until it has been implemented on INV and PGR and sufficient time has passed for this solution to seem solid.

[RemingtonRohel]

PGR is having issues again, despite the memlock change. There is nothing in the usrp_driver logs indicating that the mlock calls failed. Intermittently, the processing time for a sequence will shoot up to 500ms or more, but the next sequence will be back to normal. The swappiness value was set to 60 for PGR, so I've reduced it to 10 and rebooted the radar.

Swap used:

Process IDs: