Fix auto-resume training from checkpoint

[jncasey]

What does this PR do?

This fixes a few minor issues with training auto-resume, as discussed here

1. checkpoints = [path for path in content if _re_checkpoint.search(path) is not None and os.path.isdir(path)] was returning empty. I changed os.path.isdir(path) to os.path.isdir(os.path.join(folder, path)) and now it returns a list of the checkpoint folders as expected.
2. Similarly, the get_last_checkpoint function was returning the basename of the checkpoint folder, not the full path, which seems to be expected based on the updates to the example scripts. I changed the last line of the function to return os.path.join(folder, max(checkpoints, key=lambda x: int(_re_checkpoint.search(x).groups()[0])))
3. After I made those update, it was resuming from the oldest checkpoint, not the newest. I noticed the checkpoint regex was only capturing the final digit in the directory name. I changed it to _re_checkpoint = re.compile(r"^" + PREFIX_CHECKPOINT_DIR + r"\-(\d+)$") with the + inside the capture group, and now get_last_checkpoint is giving me the newest checkpoint as expected.

Who can review?

• trainer: @sgugger

[sgugger]

Thanks for all the fixes!

[sgugger]

Oh you need to run make style on your branch for the styling test to pass. Let me know if you run into any issue doing that!

[jncasey]

Sorry! I'm pretty rusty on the software dev stuff - college was 16 years ago. I think I've fixed it now.

[LysandreJik]

Great, thanks for fixing @jncasey!