Error: --read-freq variant ID appears multiple times in main dataset.

[h171e]

Pipeline completed without errors previously when using a single --pgs_id but trying to use a few --trait_efo id's results in this error even after creating a plink2 pfile set with the --rm-dup exclude-all argument.

The error reads Error: --read-freq variant ID '10:17813004:G:A' appears multiple times in main dataset. and was produced towards the end of step 21 APPLY_SCORE:PLINK2_SCORE

Did not try using the VCF file in the samplesheet for the pipeline due to limited storage memory and ram so tried with pfiles created with the following plink2 command:

plink2 \
   --threads 1 \
   --memory 2000 \
   --missing vcols=fmissdosage,fmiss \
   --new-id-max-allele-len 100 missing \
   --rm-dup exclude-all \
   --vcf /chr1.imputed.vcf.gz \
   --allow-extra-chr \
   --chr 1-22, X, Y, XY \
   --make-pgen vzs \
   --out /chr1

Also tried --rm-dup with the force-first and exclude-mismatch modes and got the same errors.

This was created from an imputed VCF file had split multi-allelic variants which could not be merged back with bcftools with the norm -m+any command since the process was interrupted with error messages for eleven of the twenty-two chromosomes, these messages looked like this:

Chr1: The REF prefixes differ: C vs A (1,1)
Failed to merge alleles at chr1:1715541

Used the following PGSC command:

nextflow run pgscatalog/pgsc_calc \
    -profile docker \
    --input /samplesheet.csv \
    --target_build GRCh38 \
    --trait_efo EFO_0004698,EFO_0003888 \
    --run_ancestry /pgsc_HGDP+1kGP_v1.tar.zst

Output error message was.

Command error:
  PLINK v2.00a3.3 SSE4.2 (3 Jun 2022)            www.cog-genomics.org/plink/2.0/
  (C) 2005-2022 Shaun Purcell, Christopher Chang   GNU General Public License v3
  Logging to Phased-Imputed-EAGLEIMP-Server3_10_additive_0.log.
  Options in effect:
    --allow-extra-chr
    --extract Imputed_10_additive_0.scorefile.gz
    --memory 11264
    --out Imputed_10_additive_0
    --pfile vzs GRCh38_Imputed_10
    --read-freq Imputed.afreq_ALL_relabelled.gz
    --score Imputed_10_additive_0.scorefile.gz zs header-read cols=+scoresums,+denom,-fid list-variants
    --score-col-nums 3-17
    --seed 31
    --threads 2

  Start time: Tue Nov  7 00:58:22 2023
  11908 MiB RAM detected; reserving 11264 MiB for main workspace.
  Using up to 2 compute threads.
  1 sample (0 females, 0 males, 1 ambiguous; 1 founder) loaded from
  GRCh38_Imputed_10.psam.
  3968807 variants loaded from
  GRCh38_Imputed_10.pvar.zst.
  Note: No phenotype data present.
  --extract: 432963 variants remaining.
  --read-freq: PLINK 2 --freq file detected.

  --read-freq: Frequencies for 10k variants loaded.
  --read-freq: Frequencies for 20k variants loaded.
  --read-freq: Frequencies for 30k variants loaded.
  --read-freq: Frequencies for 40k variants loaded.
  --read-freq: Frequencies for 50k variants loaded.
  --read-freq: Frequencies for 60k variants loaded.
  --read-freq: Frequencies for 70k variants loaded.
  Error: --read-freq variant ID '10:17813004:G:A' appears multiple times in main
  dataset.

[nebfield]

My first thought is that this might be a caching issue. Could you try changing the work directory with the -w parameter:

$ nextflow run pgscatalog/pgsc_calc \
     -w new_traits/ \
     -profile docker \
     --input /samplesheet.csv \
     --target_build GRCh38 \
     --trait_efo EFO_0004698,EFO_0003888 \
     --run_ancestry /pgsc_HGDP+1kGP_v1.tar.zst 

(alternatively delete your existing work directory, and try to run the larger traits)

[h171e]

The error points to the reference and not the sample? I believed it was my sample causing the error since before using plink2 on the sample data with the --rm-dup exclude-all argument the error pointed to a different duplicate position in chr10.

Installed ubuntu again and the same error occurs in the default work directory and a different one with the -w parameter.

Tried using the other reference called pgsc_1000G_v1.tar.zst and got the same error but with position 10:50240776:T:C

Command error:
  PLINK v2.00a3.3 SSE4.2 (3 Jun 2022)            www.cog-genomics.org/plink/2.0/
  (C) 2005-2022 Shaun Purcell, Christopher Chang   GNU General Public License v3
  Logging to Phased-Imputed-EAGLEIMP-Server4_10_additive_1.log.
  Options in effect:
    --allow-extra-chr
    --extract Imputed_10_additive_1.scorefile.gz
    --memory 11264
    --out Imputed_10_additive_1
    --pfile vzs GRCh38_Imputed_10
    --read-freq Imputed.afreq_ALL_relabelled.gz
    --score Imputed_10_additive_1.scorefile.gz zs header-read cols=+scoresums,+denom,-fid list-variants
    --score-col-nums 3-15
    --seed 31
    --threads 3

  Start time: Wed Nov  8 16:31:04 2023
  11908 MiB RAM detected; reserving 11264 MiB for main workspace.
  Using up to 3 compute threads.
  1 sample (0 females, 0 males, 1 ambiguous; 1 founder) loaded from
  GRCh38_Imputed_10.psam.
  3968807 variants loaded from
  GRCh38_Imputed_10.pvar.zst.
  Note: No phenotype data present.
  --extract: 29342 variants remaining.
  --read-freq: PLINK 2 --freq file detected.

  --read-freq: Frequencies for 10k variants loaded.Error: --read-freq variant ID '10:50240776:T:C' appears multiple times in main
  dataset.

  End time: Wed Nov  8 16:31:08 2023

The Options in effect: list above that contain lines with the word Imputed as well as below where it says GRCh38_Imputed_10.psam and GRCh38_Imputed_10.pvar.zst, refers to the imputed chr input sample indicated in the samplesheet.csv file, not the reference file.

This execution had the following steps stored even thou I did not use this reference before and I wonder how I could either clear the cache, delete the folders storing this info, using a parameter that disables the use of stored data, or re-installing the pipeline to delete these references:

MAKE_COMPATIBLE:PLINK2_RELABELPVAR
ANCESTRY_PROJECT:EXTRACT_DATABASE
ANCESTRY_PROJECT:INTERSECT_VARIANTS
ANCESTRY_PROJECT:INTERSECT_THINNED

Would renaming the samplesheet and sampleset names disable using stored reference data?

I still need to know how to completely uninstall the pipeline and remove all associated data or at least most since running the pipeline with different samples and references stores large amounts of data and deleting the work folder did not seem to reclaim all the used space for this folder because it looks like the data is restored afterwards.

Would like to point out that the names of the downloaded references posted on the pgsc website https://pgsc-calc.readthedocs.io/en/latest/how-to/ancestry.html differ from the name of the reference posted here by @smlmbrt just in case it is a different reference. Downloaded references: pgsc_1000G_v1.tar.zst / pgsc_HGDP+1kGP_v1.tar.zst

Used the zstdgrep 10:17813004:G:A and zstdcat| cut -f 3 | uniq -d commands with the second reference. The first command did not output anything in terminal and the second posted this:

user@pc:~$ 61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;2

user@pc:~$ 61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c61;6;7;22;23;24;28;32;42c

I have not retried the pipeline with either reference after this modification attempt that could have resulted in no modification to the reference file since I don’t know how to reset the stored or cached steps in the pipeline.

[h171e]

@nebfield
Is there an argument that could be added to the run command or a modification to the config files that can be used to ignore and bypass this error?

[nebfield]

If some processes are stored, then the cache hasn't been deleted from your computer. The cache is stored in the work directory by default. I suggest finding and deleting this directory to try and fix the error:

$ find $HOME -type d -name 'work' 2&>/dev/null
(a path to a directory should appear here)
$ rm -rf /path/to/work/directory

Be careful not to accidentally delete your personal data, this command will find any directories with the name work.

[h171e]

@nebfield
Can a different version of this reference be used with vcf.gz files for each chr and indicated in the run command with the --run_ancestry argument?

--run-ancestry /chr1.vcf.gz /chr2.vcf.gz ... /chr22.vcf.gz

Would each chr have to be concatenated in a single vcf.gz file?

Or should it be converted to a plink2 pfile set for each chr and then concatenate into a single set? If a plink2 set is required or recommended, could you help me try it by showing the recommended plink2 command for conversion and concatenation?

[nebfield]

Sorry, I'm not sure I understand. We only provide two reference databases and these have to be structured in a specific format. Our reference panels are combined in a single plink2 fileset for genome build GRCh37 and GRCh38. The workflow handles multiple target genome chromosomes internally.