log/job/ and work/ directory restructure

[matthewrmshin]

Currently we create files in log/job/ and directories in work/ like this:

log/job/%(task)s.%(cycle)s.%(submit_num)d%(suffix)s
work/%(task)s.%(cycle)s/

The convention has various problems, e.g.:

• It is not easy to tar up a cycle of logs.
• It is not easy to housekeep the work directory for a cycle.
• For the job scripts, .1 to .8 are normally suffixes for Unix manual pages.

We have now agreed to migrate to the following convention:

log/job/%(cycle)s/%(task)s.%(submit_num)02d%(suffix)s
# or
log/job/%(cycle)s/%(task)s.%(submit_num)02d/%(file)s
# where "%(file)s" would be "job", "job.out", "job.err", etc
work/%(cycle)s/%(task)s/

N.B. While making this change, we should consider adding symbolic links to allow users to locate the logs from the latest submit.

[matthewrmshin]

I am now going to implement:

log/job/%(cycle)s/%(task)s/%(submit_num)02d/%(file)s
log/job/%(cycle)s/%(task)s/NN
work/%(cycle)s/%(task)s/

where:

• %(file)s would be job, job.out, job.err, etc.
• log/job/%(cycle)s/%(task)s/NN is a symbolic link that points to the latest log/job/%(cycle)s/%(task)s/%(submit_num)02d.

Please raise any concerns as soon as possible.

[hjoliver]

Good. Two questions:

1. is "NN" literal? Why not something more obvious like "latest"?
2. should work directories be work/%(cycle)s/%(task)s/%(submit_num)02d/ so that job output from failed runs - which might be needed for diagnostic purposes - is not clobbered?

[matthewrmshin]

NN is literal. N is a familiar symbol for representing the final number in a series, regardless of language. (Latest is an English word, on the other hand, and may be less meaningful for international users.) I want something that has the same number of columns, so NN should be good for something with two digits.

Your suggestion for work/can be useful. I'll implement it as such.

[hjoliver]

Well, users who don't know English would have a hard time with cylc for many reasons other than that, but fair enough (and there is #51)

[matthewrmshin]

Well, users who don't know English would have a hard time with cylc for many reasons other than that, but fair enough ...

I agree. However, it is still worth making an effort to avoid having to translate anything at all.

[matthewrmshin]

Your suggestion for work/%(cycle)s/%(task)s/%(submit_num)02d/ can be useful. I'll implement it as such.

@dpmatthews just pointed out that this will break incremental modes of various Rose utilities, e.g. file install in rose task-run, rose_arch, etc. These utilities all rely on state files (as sqlite databases) under $PWD of the tasks.

@hjoliver do you feel strongly about work/%(cycle)s/%(task)s/%(submit_num)02d/? If not, I'll go with my original proposal.

[hjoliver]

I presume we're planning a few changes that will require corresponding changes to Rose - can the state file not just remain one level up, i.e. task-specific but not submit-specific. Hmmm, but another problem might be the need to install files into the task work directory? (rose-configured namelists?)

[matthewrmshin]

@hjoliver rose task-run incremental file install rely on state files under the current working directory. We can make an assumption to change it to ../ but this not ideal.

[hjoliver]

Well, it struck me some time ago that we should really keep work directories as well as log files for multiple submits. But if you don't think it's easy to adapt rose to that in a sensible way, I guess we could punt it to a low-priority issue to consider in the future.

[dpmatthews]

I don't think this is just a Rose issue - there are likely to be other tasks which assume that the work directory does not change on retry. For instance, long running model tasks can currently write any files they need to restart (in the event of a failure) to the work directory.

[hjoliver]

@dpmatthews - OK, fair point, we'll go with Matt's original plan.