Request: Allow not using AMP even if it's available

[carbocation]

I'd like to be able to use this without AMP, even when AMP is available. From what I can tell, if AMP is available this library uses it. It would be nice to have an override in the constructor where I can choose to disable AMP even if AMP is available on my system.

[NaleRaphael]

If you are in a hurry, you might need to rewrite the following lines with IS_AMP_AVAILABLE = False

pytorch-lr-finder/torch_lr_finder/lr_finder.py

Lines 14 to 19 in acc5e7e

	try:
	from apex import amp
	IS_AMP_AVAILABLE = True
	except ImportError:
	IS_AMP_AVAILABLE = False

I'll summit a PR in next few days, because I need to take some items into consideration, including the earlier issue #67 in which you also commented. Sorry for the inactivity and inconvenience.

[NaleRaphael]

Hi @carbocation. Just come up with a question, are you trying to use LRFinder with AMP disabled while wrapping the model and optimizer with apex.amp.initialize()?
I'm curious about this use case. Because apex.amp.scale_loss() should do nothing if the optimizer is not wrapped by apex.amp.initialize() (see also code below).

pytorch-lr-finder/torch_lr_finder/lr_finder.py

Lines 383 to 387 in acc5e7e

	# Backward pass
	if IS_AMP_AVAILABLE and hasattr(self.optimizer, "_amp_stash"):
	# For minor performance optimization, see also:
	# https://nvidia.github.io/apex/advanced.html#gradient-accumulation-across-iterations
	delay_unscale = ((i + 1) % accumulation_steps) != 0

Anyway, I'm working on integrating torch.amp and apex.amp, so that we can determine which one to use by explicitly specifying the backend. And thanks for bringing this issue up.

[carbocation]

Apologies, I might have misunderstood the code. I am not wrapping my model or optimizer with any apex.amp initializer. (I use the built-in pytorch AMP sometimes, but that does not require wrapping the model.) So perhaps I interpreted this fully backwards: I thought AMP was always enabled by default and I was hoping for a way to disable that, but perhaps it was never enabled for me since I don't use apex.

[NaleRaphael]

It's okay, hope this helps clarify something if you ran into problems while using AMP.

[davidtvs]

Closing, I believe #91 addressed this issue