Plexon2RawIO's implementation of _get_analogsignal_chunks can be improved

[nikhilchandra]

Is your feature request related to a problem? Please describe.
The latest version of the PL2FileReader API includes a function capable of retrieving subsets of data from an analog channel. Switching _get_analogsignal_chunks to this function should (hopefully) remove the need to load entire channels into self._analogsignal_cache (this takes up a LOT of memory) and might also speed things up.

Describe the solution you'd like
I want to do the following:

• Add content to the currently empty pypl2lib function "pl2_get_analog_data_subset"
• Replace the underlying DLL to the latest version
• In plexon2rawio.py, add code to retrieve fragment (i.e., segment) information for each stream to the _parse_header function
• Rewrite _get_analogsignal_chunks to use the subset function in conjunction with fragment information in lieu of the present cached solution
• Run tests to check whether the new method is faster

Describe alternatives you've considered
There are none that I can think of.

Related Question
What is the proper way to handle requests that exceed the limits of a recording's segments? For example, suppose I have a multi-trial recording in which 2 recording segments run from [0, 10] and [50, 60] (units in seconds). Suppose the user requests data from [5, 15] seconds -- obviously we have data from [5, 10] from the first segment. But for the interval [10, 15] -- do we just return zeros?

[zm711]

Sounds cool to me. I'm sure @h-mayorquin is also interested.

[h-mayorquin]

That sounds great. Concerning the question of what to do when asking for out of bound data. That is not clear to me, is this something that you have made a made a decision about @zm711 ?

[samuelgarcia]

Hi. On spikeinterface side we decided to raise an error.
On neo side this have never been clear.
I think the most often case is to clip not to zero pad.

[zm711]

I'm looking right now and a bunch of the memmap ones just do slicing [start:stop, :] so that would just follow whatever the numpy memmap convention allows, no. Where is the clipping machinery?

[nikhilchandra]

@samuelgarcia Please, could you clarify what you mean by "clipping"?

[samuelgarcia]

Hi sorry just reading this.
I am a bit fuzzy.
cliping was for : if stop in [start:stop, :] si greater than the shape[0] then stop is is "clip" to shape[0].
In short very often neo behave like numpy for upper boudary.
but spikeinterface do not, it is more strick.

[nikhilchandra]

@h-mayorquin It looks like plexon2rawio.py downloads the relevant DLLs from this location:

https://raw.githubusercontent.com/Neuralensemble/pypl2/master/bin/{file_name}

where "file_name" can take on the value PL2FileReader.dll or PL2FileReader64.dll. How does one go about submitting updated DLLs to this location?

[nikhilchandra]

Another problem -- the functions _segment_t_start and _segment_t_stop take block_index and seg_index as arguments. However, for PL2 files the precise start/stop times for a given segment depend on the stream. For example, here are segment start times for 3 wideband and 3 field potential channels from the same recording.

WB001 segment timestamps: [ 67 3609805 7217229 10824175 14430721 18036311 21642619]
WB002 segment timestamps: [ 67 3609805 7217229 10824175 14430721 18036311 21642619]
WB003 segment timestamps: [ 67 3609805 7217229 10824175 14430721 18036311 21642619]

FP001 segment timestamps: [ 67 3609827 7217267 10824187 14430747 18036347 21642627]
FP002 segment timestamps: [ 67 3609827 7217267 10824187 14430747 18036347 21642627]
FP003 segment timestamps: [ 67 3609827 7217267 10824187 14430747 18036347 21642627]

Note that within streams the values are identical, but across streams they differ slightly. I am not sure what the correct approach is to incorporate these observations into the existing framework.

Note: all values are in units of ticks, not seconds.

[zm711]

This is because the data is organized in packets?

How could the time stamps of the Wideband and field potential be different if it is the same data just filtered differently? Unless the headstage or equipment applies a slight delay to do the onboard filtering and then sends that stream with a slight delay?

[nikhilchandra]

WB and FP sampling rates are 40 kHz and 1 kHz, respectively. Each recorded sample is assigned an integer value (tick) that corresponds to a clock running at 40 kHz. Therefore, successive samples recorded for WB have tick values that increment by 1, while successive samples recorded for FP have tick values that increment by 40. Each segment timestamp in my previous message equals the tick value of the first sample recorded for that segment. Here are some predictions based on the preceding statements:

(1) For a given segment, the first recorded samples for WB vs FP channels should have tick values that differ by a maximum of 40
(2) For FP, tick values for all recorded samples must differ by a multiple of 40.

The attached calculations from an Excel spreadsheet show that both predictions work out.

In the row "WB-FP", we see that the difference in tick values for FP and WB segments never exceeds 40.

In the row "FP(n) - FP(0) % 40", we see that the difference in tick values for each FP segment compared with the first (67) is always a multiple of 40 .

I suspect that the correct course of action depends on exactly how _segment_t_start and _segment_t_end are used, will keep digging and make recommendations based on what I find.

[nikhilchandra]

Could someone please provide insight as to how to incorporate new 32- and 64-bit DLLs into the project (see my previous message from last week)? Thank you! @zm711 @h-mayorquin

[zm711]

Either @h-mayorquin or @JuliaSprenger set this up originally. So to be honest I don't know. Hopefully Heberto can comment on how he set it up. I only helped with the shell commands for it :)

[h-mayorquin]

HI, the DLLs home is here:

https://github.com/NeuralEnsemble/pypl2

I guess the route would be to make a PR to this but I don't know the ownership situation of that repo

@samuelgarcia @apdavison?

Maybe I or @nikhilchandra could take ownership over it now that Julia is sadly not as present as she used to be.

[zm711]

Just checked and I definitely don't have any writing permission there. We likely need to get Julia to share with a couple people so we don't lose access to it !