--slurm-logdir not working when first keyword starts with /

[oanegros]

Software Versions
snakemake 8.28
snakemake-executor-plugin-slurm 0.15.1
slurm 24.05.4

Describe the bug
When the first wildcard starts with / the --logdir redirection gets overwritten and the concatenation of wildcards is interpreted as a rooted path.

Minimal example
This writes log files to /rootname/folder1/_folder_to_wildcard/%J.log, where %J is jobid, even if --logdir is set.

rule example:
    input:
        "/rootname/folder1/testFiles/folder_to_wildcard/testfile.processed.txt"

rule touch_specific_file:
    input:
        "{wc1}folder1/testFiles/{wc2}/testfile.txt"
    output:
        "{wc1}folder1/testFiles/{wc2}/testfile.processed.txt"
    run:
        with open(output[0], 'a') as f:
            pass 

similarly, this writes logs to /folder1/_folder_to_wildcard

rule example:
    input:
        "/rootname/folder1/testFiles/folder_to_wildcard/testfile.processed.txt"

rule touch_specific_file:
    input:
        "/rootname{wc1}folder1/testFiles/{wc2}/testfile.txt"
    output:
        "/rootname{wc1}folder1/testFiles/{wc2}/testfile.processed.txt"
    run:
        with open(output[0], 'a') as f:
            pass 

--slurm-logdir gets successfully set in this, although due to the concatenation of wildcards, you do get them at slurm_logdir/rulename/rootname/folder1/_folder/%J.log

rule example:
    input:
        "/rootname/folder1/testFiles/folder_to_wildcard/testfile.processed.txt"

rule touch_specific_file:
    input:
        "/{wc1}testFiles/{wc2}/testfile.txt"
    output:
        "/{wc1}testFiles/{wc2}/testfile.processed.txt"
    run:
        with open(output[0], 'a') as f:
            pass 

[cmeesters]

Thank you for brining this issue to our attention! Your minimal examples do not show the setting of the logdir. Perhaps you can show how and where (on the command line, in a configuration file) your logdir is set and what its value is?

[oanegros]

I set the slurm-logdir with --slurm-logdir: e.g. --slurm-logdir ~/slurm_logdir_name

This was functional on my cluster: (with not-generalized paths, so that it actually runs)
i've run with snakemake -s ./touchpath_specific.smk example --slurm-logdir ~/slurm_logdir_name --executor slurm --jobs 1

  input:
        "/g/koehler/scratch/testFiles/AiryScan2/testfile.processed.txt"

rule touchfspecific:
    input:
        "/{root}scratch/testFiles/{wc1}/testfile.txt"
    output:
        "/{root}scratch/testFiles/{wc1}/testfile.processed.txt"
    run:
        with open(output[0], 'a') as f:
            pass 

and logged to /home/hecht/slurm_logdir_name/rule_touchfspecific/g/koehler/_AiryScan2/21084436.log

while

  input:
        "/g/koehler/scratch/testFiles/AiryScan2/testfile.processed.txt"

rule touchfspecific:
    input:
        "{root}scratch/testFiles/{wc1}/testfile.txt"
    output:
        "{root}scratch/testFiles/{wc1}/testfile.processed.txt"
    run:
        with open(output[0], 'a') as f:
            pass 

run in the same way logged to /g/koehler/_AiryScan2/21085507.log

[oanegros]

I'm running it with a profile now and this is just making me very confused and i don't fully understand what's going on.
my profile:

use-conda: True
executor: slurm

rerun-incomplete: True
jobs: 100 # max simultaneous
latency-wait: 60

default-resources:
  mem_mb: 2968
  runtime: 20
  slurm_partition : htc-el8
  cpus_per_task : 2
  constraint: bergamo|milan
  slurm_account: sikoehle
  slurm_logdir: "'/home/hecht/logs'"

logs to /g/koehler/Lab_common/git/Image_Analysis/src/.snakemake/slurm_logs/rule_touchfspecific/g/koehler/_AiryScan2/21088201.log when the first wildcard does not start with a slash ("/{root}scratch/testFiles/{wc1}/testfile.txt")
I defined a path with the keyword slurm_logs earlier (i deleted the profile in the meantime) and don't understand why this name persists, as i'm no longer referring to it - the path to .snakemake is to the folder where i run my command.

Additionally, I feel this is partially due to paths in the wildcards also remaining as paths in the log file name. This also causes for me very indented log files even with functional slurm-logdir redirection. This would be helped with an option that parses the slashes as underscores/dots some variable string?

Thanks for quickly replying and looking into it! 😁

[cmeesters]

Ah, sorry. Now, I begin to see your point. The SLURM logfile internally starts with the slurm_logdir, or more precisely:

slurm_logfile = self.slurm_logdir / group_or_rule / wildcard_str / "%j.log"

This is path concatenation using Python's pathlib operators.

Consider:

>>> p = Path("/home/someone")
>>> l = p / 'b' / 'test'
>>> print(l)
/home/someone/b/test
>>> l = p / 'b' / '/test'
>>> print(l)
/test

So, indeed, when a wildcard will start with a slash, the SLURM log directory, will start with /.

One might want to consider this a bug in Python, as <prefix>//<suffix> works in bash (I assume you are working in bash, but this hardly matters). We rely on this kind of concatenation, as other forms have their own quirks, and this is the current way to go in Python.

Now, I could implement a check to prevent wildcards from starting with /. Or, if you give a compelling reason that wildcards sometimes need to start with /, implement a different concatenation version.

[oanegros]

Ah okay, that makes sense; For me the arguments for different/configurable logpath would be unrelated to this issue, i can open a new issue for this.

[cmeesters]

Yes, you can open new issues. Yes, it seems easy to have a solution by replacing it here. I do not know whether there will be repercussions somewhere else in Snakemake's code. It is, therefore, that I started a discussion in the main repo. So: Is there a reason to let a wildcard start with a special character?

[oanegros]

I think it would be unintuitive to have the restriction, but also don't think there's another very strong argument for it.