Multiple errors locating data linux version

[iezqrom]

Describe the issue

I get multiple errors when trying to locate data and I can't get pass the screen. I get errors at this screen both when i tick 'Would you like to locate these sessions programmatically' and when I don't. The errors are different though. Here I focus on the errors when I tick to locate the sessions programmatically

First, not all files are listed, only one per subject.

Second, I click next and see the errors logged in the console below.

Steps to Reproduce

1. Create a new conversation pipeline
2. Check both 'Will this pipeline be run on multiple sessions' and 'Would you like to locate these sessions programmatically'
3. Video - VideoInterface format
4. Choose base directory and string path
5. Click Next, it tries to locate and errors are logged in the console.

Operating System

Ubuntu 20.04.6 LTS

GUIDE Version

linux-fix

Code of Conduct

Yes

Did you confirm this issue was not already reported?

Yes

[iezqrom]

As the error suggests, in the file https://github.com/NeurodataWithoutBorders/nwb-guide/blob/linux-fix/src/renderer/src/stories/pages/guided-mode/data/GuidedPathExpansion.js, keepExistingData is not defined before it's used for the conditional statement.

[iezqrom]

I managed to solve the issue of not moving pass that screen by adding the line

const keepExistingData = false;

to the file GuidedPathExpansion.js.

However, I can't still list all the available files (they are actually found by DeepDict, but this doesn't inject vertically to the GUI and API. Then, once completing the table. I'm getting another error:

[garrettmflynn]

Thank you for flagging these @iezqrom! I've submitted fixes for both of the issues.

[CodyCBakerPhD]

Thanks for the report @iezqrom!

As a linux user, you'll be a little more exposed to day-to-day bugs like this, so be sure to always be pulling the latest state of that branch from Git

@garrettmflynn has started a general fix to this on #667, it will take a little while to trickle down to the linux-fix branch

[iezqrom]

@garrettmflynn @CodyCBakerPhD thanks for the quick reply both!

We're looking into implementing this in our lab, so I'm exploring it. I'm keen on contributing when I get a handle of things.

When do you expect to fix these issues in the linux-fix branch?

On a related note, is there a way to programmatically fill in all the columns of the subject metadata?

[CodyCBakerPhD]

@iezqrom We actually just re-imagined the way we do linux installation to help increase its stability

Can you give the linux-latest branch a try? Things may look very different from current main but they will correspond identically to the stable state of the application at time of latest release (in our case, 2 weeks ago) - so it matches latest Windows and Mac release too

[CodyCBakerPhD]

On a related note, is there a way to programmatically fill in all the columns of the subject metadata?

If you programmatically generate a table file, you should be able to copy and paste from that table file into the interactive table on the GUIDE

Otherwise, purely programmatic access is the domain of NeuroConv, which the GUIDE is simply a graphical wrapper of (plus some additional 'nice' features like file visualization)

[iezqrom]

Thanks for the replies and for suggesting linux-latest. This branch works more smoothly. However, I'm still getting two errors which prevent me from completing the conversation pipeline. The first is that not all videos are listed in Locate data screen. I'd expect to see 4 videos listed, but I can only see the baselines ones, see image:

Then, I'm getting a video is not allowed warning in the source data screen, see image below. I think this is the step that's preventing me from moving on to the next Conversation preview step and hence completing the pipeline

[garrettmflynn]

To get around this quickly, can you specify your source data manually instead of programmatically?

You have uncovered a more complex issue related to the VideoInterface, which will only accept one file in the file_paths property at the moment.

[iezqrom]

Thanks. This worked, I'm waiting for the API key to complete the process.

I still have the issue that I can only associate one video per mouse in the source data information page, but maybe there's something I'm missing? See screenshot below

[CodyCBakerPhD]

I still have the issue that I can only associate one video per mouse in the source data information page, but maybe there's something I'm missing? See screenshot below

We hope to have an advanced tutorial to clarify these kinds of details of how to use this interface soon

Until then I'll describe here

A single VideoInterface is for attaching a lossy-compressed video stream from a single camera to an NWB file

The reason it takes file_paths: List[FilePath] is for the case where the user buffered the contiguous acquisition of this stream on disk by creating multiple files; for example, "Camera1/001.mp4" (with first 300 frames), "Camera1/002.mp4" (with next 300 frames, no other gap between 001 and 002), and so on

I think we only allow one path at a time for the moment due to lack of a component for easily specifying the order of those files in the contiguous schema (one would hope natsort would work in most if not all cases, but user might still need to be able to offer manual corrections); maybe @garrettmflynn remembers where we are on that?

At any rate, is that the case you're talking about needing this for? Otherwise, if this is for including data from multiple cameras, then all you need to do is add multiple VideoInterfaces (one per camera) to your data formats and select a different video as the source data for each one (you can technically use that as a workaround to the other use case for the time being too)

[garrettmflynn]

Yes, this is correct. We've tabled the ability for users to specify multiple video files for the reasons that Cody described above.

We haven't yet discussed natsort and the ability to handle this automatically.

[iezqrom]

Right, that makes sense, thanks so much for the thorough explanation. Indeed, I was trying to add videos that were from the same session and mice taken one after the other (with a small break in between). I might also need the case for including data from multiple cameras soon, so thanks so much for the explanation!

[CodyCBakerPhD]

Closing this for now, will ping you again when we add more support for this case to the VideoInterface (the gap in between videos is especially tricky and requires special treatment in a feature we have yet to support)

[iezqrom]

sure, thanks!