how test the best checkpoint without fit

[Achazwl]

What should I do if I want to test the best checkpoint (using previous training checkpoint)? I would save the checkpoints in a quite complicated format, like {epoch:02d}-{val_loss:.2f}.ckpt. But when I run test, I don't want to specify the exact path of the checkpoint in my program, but let lightning tell me which checkpoint under the logging base_path is my best checkpoint.

Similar to resume training, setting the logging base_path to the resume_from_checkpoint attribute of Trainer would give it the checkpoint information. However, resume_from_checkpoint would matter only after fit #8796 , which is not called when only running test.

I have tried getting the trainer.checkpoint_callbacks[0].best_model_path before trainer.test and without fit, but it is empty.

I also tried getting the trainer.checkpoint_callbacks[0].best_model_path after trainer.fit when resume training, it is not empty, and it is just the best checkpoint path I need.

lightning version 1.4.6

Here's a code snippet:

trainer = PLTrainer(
    ...

    resume_from_checkpoint = os.path.join(os.path.join(log_path, "checkpoints"), "last.ckpt"),
    
    enable_checkpointing = True,
    callbacks = [
        ModelCheckpoint(
            monitor = "val_loss",
            save_top_k = 1,
            save_last = True,
            mode = "min",

            dirpath = os.path.join(log_path, "checkpoints"),
            filename = "{epoch:02d}-{val_loss:.2f}",
        ),
    ],
)

if test == True: # run test
    path = trainer.checkpoint_callbacks[0].best_model_path # empty here !
    model.load_from_checkpoint(path)
    res = trainer.test(model, dataloaders = test_dataloader)
else: # resume training
    trainer.fit(model, train_dataloaders = train_dataloader, val_dataloaders = valid_dataloader)
    path = trainer.checkpoint_callbacks[0].best_model_path # not empty here !
    res = trainer.test(dataloaders = test_dataloader, ckpt_path = 'best') # best can be used only after fit