On perlmutter, when file not present on server inputdata, the auto-download command may claim success and simply write a 0-sized file

[ndkeen]

I've noticed a few 0-size partition files at NERSC and I can reproduce by simply asking for odd-number partition file.

I see this in submit output:

Model mpassi missing file graph127 = '/global/cfs/cdirs/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127'
Trying to download file: 'ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127' to path '/global/cfs/cdirs/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127' using WGET protocol.
SUCCESS

but then:

login38% lr /global/cfs/cdirs/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127
-rw-rw-r--+ 1 ndk ndk 0 Aug 28 13:27 /global/cfs/cdirs/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127

[ndkeen]

I think we've figured out issue -- or actually, how to avoid it -- still not sure why this happens.

While LCRC was down, we added NERSC as a data server. Which maybe isn't a good idea to leave there
in general (as maybe better to have one true golden set), but also seems to have this issue of writing 0-size files.
Which I'm sure we could figure out and correct if needed

Easier reproducer:

login22% rm o
login22% rm /global/cfs/cdirs/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127
rm: cannot remove '/global/cfs/cdirs/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127': No such file or directory
login22% wget https://portal.nersc.gov/project/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127 -nc --output-document o
--2023-08-28 13:51:55--  https://portal.nersc.gov/project/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127
Resolving portal.nersc.gov (portal.nersc.gov)... 128.55.206.109, 128.55.206.106, 128.55.206.111, ...
Connecting to portal.nersc.gov (portal.nersc.gov)|128.55.206.109|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-08-28 13:51:55 ERROR 404: Not Found.

login22% ls -l o
-rw-rw-r-- 1 ndk ndk 0 Aug 28 13:51 o

[rljacob]

I don't understand the reproducer. Did it also create a 0 size file called mpas-seaice.graph.info.230424.part.127 ?

[ndkeen]

For the reproducer, I was trying to simplify by asking it write output file named o:
wget https://portal.nersc.gov/project/e3sm/inputdata/ice/mpas-seaice/oEC60to30v3/partitions/mpas-seaice.graph.info.230424.part.127 -nc --output-document o

which it does, and not sure why.
So just ask for any file that is not there.

login22% rm o
rm: cannot remove 'o': No such file or directory
login22% wget https://portal.nersc.gov/project/e3sm/inputdata/foo -nc --output-document o
--2023-08-28 14:42:02--  https://portal.nersc.gov/project/e3sm/inputdata/foo
Resolving portal.nersc.gov (portal.nersc.gov)... 128.55.206.110, 128.55.206.111, 128.55.206.108, ...
Connecting to portal.nersc.gov (portal.nersc.gov)|128.55.206.110|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-08-28 14:42:02 ERROR 404: Not Found.

login22% ls -l o
-rw-rw-r-- 1 ndk e3sm 0 Aug 28 14:42 o

[rljacob]

related CIME issue ESMCI/cime#4480

[ndkeen]

Actually, I think the issue might be --output-document argument? Without that, it will not write 0-size file for me. I think that is actually expected behavior. As noted in the CIME issue, the python is then supposed to detect and cleanup that 0-size file.