Replace the sed snippets with Python ones

[abhi18av]

Hey @LennertVerboven and @TimHHH ,

I've accommodated both Python scripts within the two processes. Here's some context

1. SNPEFF

Continuing from #163 (comment), I tweaked that script for the python2 environment which is in magma-env-2 // magma-container-2

2. GATK_VARIANTS_TO_TABLE

Continuing from #163 (comment) , the generated results are a bit different from the previous file.

Question: Is params.median_coverage_cutoff no longer necessary as it has been substituted by params.site_representation_cutoff ?

==========

However, one problem that I perceived is that the exact expected file isn't being generated in both steps hence I created a DRAFT PR. Once the debugging is done we can merge this PR.

[LennertVerboven]

Question: Is params.median_coverage_cutoff no longer necessary as it has been substituted by params.site_representation_cutoff ?

They should both remain, the median coverage is used to filter samples but was mistakenly also used here.

I dont understand the issue with the files not being generated

[abhi18av]

They should both remain, the median coverage is used to filter samples but was mistakenly also used here.

I see - therefore I assume that we don't need the cohort stats file to reflect this cut-off right?

I dont understand the issue with the files not being generated

Yeah, strange for me too. But what I suggest is that you run the pipeline and then do debugging based on the work directory file.

.command.sh contains the final shell script

.command.run is the driver script, which stages files, activates conda/docker and then runs .command.sh

I suggest, you copy the python script there and then update the .command.sh to use ./python_script.py syntax and then just run .command.run for debugging.

[LennertVerboven]

Added a newline that I forgot here to fix the issue with the snpeff replacemnt of sed

[abhi18av]

Okay, I tested again with the change, but the result is same still. The downstream INDEX_FEATURE_FILE process fails

gatk IndexFeatureFile --java-options "-Xmx4G" \
       \
      -I joint.raw_variants.annotated.vcf.gz

Error log

  13:46:45.420 INFO  IndexFeatureFile - Shutting down engine
  [August 16, 2023 at 1:46:45 PM UTC] org.broadinstitute.hellbender.tools.IndexFeatureFile done. Elapsed time: 0.01 minutes.
  Runtime.totalMemory()=2147483648
  ***********************************************************************

  A USER ERROR has occurred: Error while trying to create index for joint.raw_variants.annotated.vcf.gz. Error was: htsjdk.tribble.TribbleException.InvalidHeader: Your input file has a malformed header: We never saw the required CHROM header line (starting with one #) for the input VCF file

  ***********************************************************************
  Set the system property GATK_STACKTRACE_ON_USER_EXCEPTION (--java-options '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true') to print the stack trace.

[abhi18av]

Okay, this looks functional now , had to tweak the variants to table process 8e3e1f3 and now the pipeline is finishing.

I'm opening this PR for review, please do another pipeline-level test on your side - to check the final analysis - and then we can merge.