How to see logged values after training

[erickrf]

During training and validation, my model logs accuracy and loss. These values are shown in the progress bar, but are lost once new ones replace them.

How can I access the history of logged metrics after training? I tried trainer.logged_metrics but it only has the last values of the metrics.

[awaelchli]

@erickrf It's best if you track these yourself in the LightningModule or perhaps with a logger (e.g. CSVLogger). Lightning doesn't track the history of all logged values because that would consume memory and potentially accumulate growing memory over a long time.

[rohitgr7]

@awaelchli do you think a callback like HistoryCallback or something else would be a good one to have? If the user needs these values then they can just add this without writing the same boilerplate code assuming that they are ok with memory overload? we can have flush_history_logs(int) as an argument that can limit the logged values by flushing these values at a certain interval? Also, it will be able to track values with self.log(logger=False).

[awaelchli]

hmm not sure
this callback would essentially be an in-memory logger? what would differentiate it from a logger inheriting our Logger interface? and what will it capture, the entire metrics?

[rohitgr7]

yep the entire metrics. But I guess CSVLogger is already good enough for this use case.

[VietTralala]

I think it would be nice to have an in memory logger for small experiments, so that one does not need to serialize and deserialize the logged object. Currently I am using the following Logger which is close to the dummy logger:

from pytorch_lightning.utilities import rank_zero_only
from pytorch_lightning.loggers.base import LightningLoggerBase

class MIPredictions(LightningLoggerBase):
    """From pytorch_lightning.loggers.base.DummyLogger and Tensorboard logger"""
    def __init__(
            self,
            name="MIPredictions",
            key2watch='train/mi'):
        super().__init__()
        self._experiment = None
        self._name = name or ""
        self.key2watch = key2watch
        self.logs_mi = []


    @property
    def experiment(self):
        return self._experiment

    @rank_zero_only
    def log_metrics(self, metrics, step=None) -> None:
        assert rank_zero_only.rank == 0, "experiment tried to log from global_rank != 0"

        for k, v in metrics.items():
            if isinstance(v, torch.Tensor):
                v = v.item()
            else:
                m = f"\n you tried to log {v} which is not currently supported. Try a tensor."
                ValueError(m) 

            if isinstance(v, float) and k == self.key2watch:
                self.logs_mi.append(v)
           

    def log_hyperparams(self, *args, **kwargs) -> None:
        pass

    @property
    def name(self) -> str:
        return self._name

    @property
    def version(self) -> str:
        return ""

    def finalize(self, status):
        print(status)