[Feature Request] append to existing lcurve for restart training

[anyangml]

Summary

Currently, when restart a training, the lcurve.out file will be overwritten. Maybe it's more natural to keep adding to the existing file, so that we can keep the entire training history in a single file.

Detailed Description

When a training crashes, and I would like to restart it from the latest ckpt, I have to move the ckpt and input file into a separate folder to prevent log files from being overwritten. This creates more complexity for downstream workflows such as training history visualization. If there is no technical issue, I believe adding to the existing log file is a more desirable behavior than overwriting it.

Further Information, Files, and Links

No response

[njzjz]

Currently, when restart a training, the lcurve.out file will be overwritten.

I don't reproduce it using the water example. Do you have any specific configuration (e.g. multi-task)?

[anyangml]

Interesting, this happens to me for both single-task and multi-task. Maybe it's something associated with the cloud server, Ali-Pai, I will look into that.

here is the single task training input

command: dp --pt train input.json --restart model.ckpt-200200.pt --skip-neighbor-stat

[njzjz]

Are you sure the latest code is used? At least after #3985

[anyangml]

I see, I didn't notice this is already available.