FeatureExtraction Example

[ahabdallah27]

I am trying to create a FeatureExtraction based on this description, but I don't understand how to call the electrodes. There's a "get_electrode_group", but not one to get electrodes in the nwb file so I don't know how to reference the electrodes that I created. Additionally, once the FeatureExtraction is created does it automatically get pulled into the nwb file? I saw in some cases you have to reference the nwb file, but when creating a feature extraction there's no reference to the nwb file so how does it get put in the nwb file?

This is what my code for the FeatureExtraction looks like

featureExtraction = []
for jdx in range(0,1):        
    featureExtraction.append(FeatureExtraction(
                    electrodes=[jdx],
                    description=['neuralfeature','samplewidth'],
                    times=times,
                    features=[sampleWidths, neuralFeatures]
                    ))

I'm appending to a list called featureExtraction becasue I don't understand how to create a new feature as a row in the nwb file. Once I see how to attach the FeatureExtraction to the nwb file this code will be adjusted. Ultimately an example/tutorial with the FeatureExtraction would be helpful. I did not see one anywhere. I looked in the documentation book as well as the website.

This is what my nwb file looks like

[CodyCBakerPhD]

Going to ping @oruebel @bendichter @rly on this one

I'm not personally aware of the answer or intention of that neurodata type

[oruebel]

I'm not personally aware of the answer or intention of that neurodata type

FeatureExtraction is (to the best of my knowledge) intended for storing features extracted from electrical recordings stored in SpikeEventSeries or ElectricalSeries, where features may be from dimension reduction, such as PCA. @rly @bendichter may be more familiar with FeatureExtraction than I am, but I'll do my best. Hopefully the below will help answer at least some of the questions.

There's a "get_electrode_group", but not one to get electrodes in the nwb file s

I believe the electrodes argument of FeatureExtraction behaves the same way as in ElectrodeGroup (the tutorial here shows an example for how this it works with ElectricalSeries https://pynwb.readthedocs.io/en/stable/tutorials/domain/ecephys.html#extracellular-recordings). I think NWBFile.create_electrode_table_region is the function to help with creating DynamicTableRegion references to the electrodes table.

how to create a new feature as a row in the nwb file

Since you are using multiple FeatureExtraction objects, I believe the approach to add them to the NWBFile should be to create a ProcessingModule. Something along the line of:

feature_module = nwbfile.create_processing_module(name="feature_extractions", description="features extracted from ......")
feature_module.add(......my FeatureExtraction object....)

Ultimately an example/tutorial with the FeatureExtraction would be helpful.

Agreed

[ahabdallah27]

Thanks!

[ahabdallah27]

Is there an easy to to convert the TimeSeries into a DataFrame? I see documentation on converting unit and trials to a dataframe, but not TimeSeries. I have a bunch of TimeSeries saved in acquisition and some may be saved in a processing module. How do I convert all of the TimeSeries in acquisition in a single DataFrame and all of the ones in the processing module in another single DataFrame?

[ahabdallah27]

This was my attempt, but it is taking a little less than a second per iteration (through 7 keys) so this method isn't scalable and therefore practical. Furthremore, I noticed I had to read the datafile and was unable to use the below method from a created nwbfile object because of the compression that I applied. Is there a way to convert a compressed TimeSeries into a DataFrame?

nwbTimeSeriesDict = dict()
for key in read_nwbfile.acquisition.keys():
    nwbTimeSeriesDict[key] = []
    dataLen = len(read_nwbfile.acquisition[key].data)

for i in range(0, dataLen):
    for key in read_nwbfile.acquisition.keys():
        nwbTimeSeriesDict[key].append(read_nwbfile.acquisition[key].data[i])

pd.DataFrame.from_dict(nwbTimeSeriesDict)

[CodyCBakerPhD]

First thing first; the units and trials neurodatatypes are DynamicTables and TimeIntervals (which descends from DynamicTable respectively) and so are more 'table-like' which is why they have an inherent .to_dataframe() -> pandas.DataFrame method

One thing to know about a pandas.DataFrame is that is that, like a numpy.array, it loads all data into memory

So for TimeSeries, while you can certainly form a tabular structure out a single 2D one by of several of them if they have consistent shapes, this is not usually recommended based on how heavy they can be - there are many NWB files on DANDI that if you attempted to load them into memory like this would overload and crash your system (I know from experience 😆)

So with that addendum, let's look at performance...

You should generally avoid iterating single vales of any array-like entity in Python if the equivalent operation can be done with slicing

So instead of doing

for i in range(0, dataLen):
    for key in read_nwbfile.acquisition.keys():
        nwbTimeSeriesDict[key].append(read_nwbfile.acquisition[key].data[i])

you should just be able to do

for key in read_nwbfile.acquisition.keys():
    nwbTimeSeriesDict[key].append(read_nwbfile.acquisition[key].data[:])

Of course, pandas has compatibility with h5py.Dataset objects anyway so you shouldn't even need to slice them at all. My best attempt to replicate this, without seeing your data, is

import pandas
from pynwb.testing.mock.base import mock_TimeSeries

list_of_timeseries = [mock_TimeSeries(), mock_TimeSeries(), mock_TimeSeries()]
pandas.DataFrame.from_dict({time_series.name: time_series.data for time_seriesin list_of_timeseries})

[ahabdallah27]

Thanks, it worked! I needed to also convert my np array into a list before turning it inserting them into a dictionary.