How to log data read time (disk io)?

[jmerkow]

I am trying to log disk read time with a callback. I do no think such a monitor exists. Looking at the GPUStatsMonitor, I might be able to infer this time from intra and inter step time, but its not clear to me when specifically each hook in this list occurs. Is there a diagram or something that shows what happens in between each hook?
I think the time from on_batch_end to on_batch_start should only include data io, but I wanted to confirm.

I was thinking of basically logging the time at on_epoch_start -> on_batch_start, on_batch_end -> on_batch_start

something like:

  def on_train_batch_end(self, trainer, *args, **kwargs):
        self.time = time.time()

  def on_train_epoch_start(self, trainer, *args, **kwargs):
        self.time = time.time()

  def on_train_batch_start(self, trainer, *args, **kwargs):
       elapsed = time.time() - self.time
       trainer.logger.log("train/data", elapsed, step=trainer.global_step)

[rohitgr7]

there is no diagram right now.

for now a pretty lazy hack to get the call order of these hooks 😅

import inspect
from unittest.mock import call, Mock

all_methods = [m[0] for m in inspect.getmembers(Callback, predicate=inspect.isfunction) if not m[0].startswith('_')]
cb = Callback()
tracker = Mock()

for meth in all_methods:
    setattr(cb, meth, Mock(getattr(cb, meth)))
    tracker.attach_mock(getattr(cb, meth), meth)

model = BoringModel()
dm = BoringDataModule()

trainer = Trainer(
    fast_dev_run=2,
    callbacks=[cb],
    num_sanity_val_steps=0,
)
trainer.fit(model, datamodule=dm)
trainer.test(model, datamodule=dm)

print([str(mc).split('.')[1].split('(')[0] for mc in tracker.mock_calls])

mind open an issue with a request for a diagram. I think it will be pretty useful for everyone :)

[jmerkow]

Will do.