Z-Stack Data Storage and Image Quality

[jmdelahanty]

Hey everyone!

Do you have any advice on how many frames is adequate for a reasonable image quality in a z-stack for 2-photon images? Grabbing single images for a plane just looks like noise to me so I was planning on collecting 8 or 16 stacks individually and averaging them together.

Thanks for the tips and advice!

[bendichter]

@jmdelahanty It might be OK to average frames together as part of a preprocessing program, but I'd suggest storing all of the raw data to start with, even if individual frames look noisy to the eye.

[oruebel]

Do you have any rules of thumb for file size best practices based on your experience? As in when should I start to worry about size?

As a general rule, I would say, if you know the data is something that you want to keep (and share) then placing it in the file is fine. If the data is something you believe you may not want to delete, e.g., some preliminary analysis, then I would place it in a separate file so that you can easily remove it. In terms of file-size, it really depends on what you expect to do with the data. On AWS and Azure clouds the file size limit is around 5TB (I think), so I would not go larger than that. The main issues I've encountered with large files is that transferring the data becomes more challenging, in particular when moving from a high performance system (e.g,. cloud or supercomputer) to a less performant system (e.g., laptop or backup tape store) with limited network bandwidth or storage capacity. Rather than strictly worrying about size, I would ask the question "How do you typically use the data?" E.g,:

• If the data belongs together and you usually want to be able to use it all together, then placing it into one file is fine.
• If all the data is typically used in a write-once-read-many type of pattern (i.e., where you generate the data and then not modify it again) then placing it in the same file is fine.
• If some data is static but other parts are modified often (or need to be deleted) then I would place the parts that change in a separate file

[jmdelahanty]

Thank you so much for all the clarification Oliver! This all makes sense. Thankfully, we'll not be getting anywhere close to 5TB (at least I'm pretty darn sure we won't!). My plan is to have all the data in the file be static once we figure out how to optimally process our fluorescence microscopy data with Suite2P. From there people could read in the data on their local machines and do stats/etc. I'm hoping that the NWB file is something that users of the system basically don't have to build/interact with beyond loading it. Ideally, they won't write to it either since the NWB file should be a complete representation of the given recording session and writing to it might make it impossible for other users to load and analyze.

[oruebel]

writing to it might make it impossible for other users to load and analyze

when the HDF5 file opened normally (i.e., with 'w' or 'a') that is correct. HDF5 has support for "single writer multiple reader" access as well but that I would consider that to be an advanced use case (https://docs.h5py.org/en/latest/swmr.html#single-writer-multiple-reader-swmr)

[jmdelahanty]

Do you think that it's a good idea to restrict write access? In other words, is it best practice to basically have a pipeline that produces a complete, static NWB file that users then analyze or should I try to see how flexible the NWB file for a session should be? The only thing that I'm worried about is that since using NWB is totally new to the lab that when we're analyzing it we might accidentally ruin it when running things on it.

Edit: Maybe it's better to say that my goal for the pipeline is to automatically process data to an analyzable state without any end user input/handholding required so people can focus on their science and not their data formatting. I volunteered in a lab at Davis that basically had an automatic pipeline that would let you know when an animal's data for a session was ready for analysis and they had premade scripts that could be called on any file they produced and let them visualize it, analyze it, and interact with it. It's a dream to do that here!

[oruebel]

Do you think that it's a good idea to restrict write access? In other words, is it best practice to basically have a pipeline that produces a complete, static NWB file that users then analyze or should I try to see how flexible the NWB file for a session should be?

In general, I think that's probably a good rule of thumb, at least for all data that you want to be static (e.g., for raw recordings and data from standardized pipelines). For analyses (which tend to change), saving results to separate NWB is probably preferred, at least until you are confident that the analysis results are good and are what you want to share.