error: Can't open path/to/data/input_R1.fastq.gz_sample.fastq.gz

[morien]

I've got the above error (i modified the path and file name to simplify the question) with a set of data straight from the sequencing facility. if there are no reads associated with a particular barcode, could that cause this error?

The error occurs after IDEMP has finished parsing the fastq files.

Here's my input command:
/git/idemp/idemp -b data/metadata/demultiplexing_file.txt -I1 data/raw_data/input_I1.fastq.gz -R1 data/raw_data/input_R1.fastq.gz -R2 data/raw_data/input_R2.fastq.gz -m 1 -o data/raw_data/demultiplexed/

i am having trouble imagining why this is happening. i have used IDEMP successfully many times before with the same kind of experiment.

[yhwu]

I don't think so. Without data, it's hard to know what's going on.

[morien]

Okay, I'd like to figure out the issue. I'll try to make a small dataset that recreates the problem. The data are too large to share practically right now.

[catalicu]

Was this resolved? I am getting the same error and cannot figure out what is wrong.

[morien]

I wasn't able to re-create the issue with a small dataset. Since last year, some of my research group have noticed that this only happens when the dataset has a large number of samples. Somewhere north of 250. I'm not sure whether the developers will have any idea about that, but that's what I and others have observed "in the wild"

So, for datasets with huge numbers of samples, we usually just break up the samples into two chunks, or use a different demultiplexing tool.

[catalicu]

That sounds about right, I have 350 samples. I ll try what you suggested, thank you so much for your quick reply!

[yhwu]

Thanks for offering the workaround! This program opens multiple files to write. If you have 100 samples, it writes 100 files at the same time. You may have reached that limit. You can use ulimit -n to check. On my machine, it is only 256. If you are able to change that number, I bet it would work. Otherwise, @morien 's workaround is easy to implement, and it would give you the correct files.