Custom scheduler

[jw3126]

I would like to provide my own learning rate scheduler. What I would love is do is doing this in the lightning style, e.g. implementing some hooks:

class MyScheduler(pl.LightningScheduler):
    ...
    def on_step(self...):
        ...
    def on_epoch(self...):
        ...

Is something like this possible? How do other people handle custom schedulers?

PS: I asked this question before on the [deprecated forum.]
(https://forums.pytorchlightning.ai/t/custom-scheduler-class/1238)

[rohitgr7]

can you please elaborate on the usage more? like what do you want to do inside on_step and on_epoch methods?

[jw3126]

I want to experiment with different schedulers. The first thing I would like to try is combine cyclic learning rate with reduce on plateau.

[rohitgr7]

then you can subclass pytorch native scheduler, configure your custom logic there and pass it inside lightning. To configure whether to update it on step or epoch or some other thing you can have a look at lr_scheduler_config section.

[jw3126]

By pytorch native scheduler you mean torch.optim.lr_scheduler._LRScheduler? Since that starts with an underscore, I wonder if that could lead to trouble everytime my pytorch version slightly changes?

[rohitgr7]

yes, also it hasn't changed since a very long time, and don't see any possibility of it being changed anywhere in the near future. anyhow in case any update is made on the pytorch side, I believe it will be deprecated first. the core functionality of lr updates still relies within native pytorch, lightning just handles when & where to call it.

You can also subclass the exact scheduler too and update it's methods as per your requirement.

[jw3126]

Ok thanks for this clarification @rohitgr7