[Request]: Update NWBIO Class to allow for chunck-by-chunk conversion

[FrancescoNegri]

Describe the bug
When Neo is used to load and export proprietary formats in .nwb files, the source electrophysiological recording is fully loaded into memory, converted and ultimately saved in the output format.
This procedure is particularly problematic when converting large files, where the process is interrupted due to saturation of the available memory.
Unlike Neo, the neuroconv package sequentially converts smaller chunks of the original signal, avoiding any memory-related issue.
Wouldn't it be appropriate to modify the NWBIO class so that it follows the same approach as neuroconv?
These considerations hold for any output format, as reading the whole electrophysiological recording in many cases might be unfeasible.

To Reproduce
The issue arises when reading and subsequently writing any large enough recording to .nwb. In my case, I tried to export a .pl2 Plexon file with a size of about 50 GB.

Expected behaviour
The desirable behaviour, as previously stated, would be a chunk-by-chunk conversion, without the need to load in-memory the whole recording, avoiding memory saturation and subsequent process interruption.

Environment:

• OS: Ubuntu (in a Windows-based Docker container)
• Python version: 3.12
• Neo version: 0.14.0

[zm711]

@apdavison manages nwb here. So he's best to comment here.

[h-mayorquin]

As a maintainer of neuroconv I am curious on whether neuroconv is missing any feature that would make you use neo instead of neuroconv.

Implementing buffered reading might be a major effort on neo side, effort that might be better spread to other cases that are not solved in the community.

[FrancescoNegri]

Sure @h-mayorquin, I will let @TommasoLambresa elaborate on your question!

[TommasoLambresa]

Hi @h-mayorquin,

We are currently working with Intan .rhs data, and we've encountered a limitation with neuroconv. Specifically, neuroconv can only convert the first data stream provided by the amplifier, as this behavior is hardcoded in the IntanRecordingInterface class. This is an issue for us because we need to save all the streams provided by the acquisition system (as the USB digital input/output channel).

On the other hand, neo allows access to all streams, but it has drawbacks: the organization of NWB metadata is messy when writing .nwb files, and there are memory-related issues as pointed by @FrancescoNegri.

Our current approach is to first convert the data using neuroconv, as it provides better metadata organization and buffered conversion, and then read the additional streams with neo, appending them to the existing NWB file. We attempted to parametrize the stream_id within the IntanRecordingInterface, but we ran into an issue due to the way neuroconv handles stream data by default. When running the conversion, neuroconv treats the stream as an ElectricalSeries, which requires associated electrodes. However, some Intan streams do not have electrodes, making a more general TimeSeries class a better fit.

[h-mayorquin]

Hi @TommasoLambresa. It should be really easy for us to add support to other streams in Intan and I have been meaning to do it but I don't have a user that could help us identify the nature of the data in those streams. If you are willing to help us with feedback I could add this feature this week. If you are interested please send me an email h . mayorquin @ gmail (without spaces) and we can get the ball rolling. Basically, I just need feedback from an experimentalist to assure ourselves that we are mapping the stream to the correct nwb data types.

Meanwhile, we should continue this conversation on neuroconv so we don't hijack this topic further:
catalystneuro/neuroconv#519