Refactor hyperparameter search backends

[alexmojaki]

Fixes #24379

The goal here is to clearly group the essential info/functionality about each backend together to make reading/changing things easier. For example if another backend integration is added it should be less likely for something to be forgotten as apparently happened with wandb.

@sgugger sorry I didn't get a full confirmation to go ahead with this, it just seemed easier to show what I meant with code rather than continue explaining in the issue. There's many other ways this could be done and I can change the approach but I hope that the general direction at least is clear from this PR.

I also think this would help move towards improving the user facing API since as mentioned in #24278 (comment) (cc @hugocool) the kwargs have no type hints and are not very easy to use. So maybe instead of:

          best_run = trainer.hyperparameter_search(
                direction="maximize",
                backend="ray",
                # this is just **kwargs, not so clear what's possible...
                storage_path="...",
                callbacks=...,
            )  

one could write:

          best_run = trainer.hyperparameter_search(
                direction="maximize",
                backend=RayTuneBackend(
                    # now more assistance is possible
                    storage_path="...",
                    callbacks=...,
                ),
            )  

[sgugger]

Thanks for your PR, this is not what we discussed on the issue which I recommended to just complete the error message to include wandb.

Using abstract classes like this is not really the way the Transformers library is designed and we don't heavily refactor everything as you did, preferring clearer code that can be more verbose but doesn;t require the reader to jump to superclasses all the time to get the behavior of a method. I also don't see how you would have benefits for IDE as you show in the PR description.

[alexmojaki]

Using abstract classes like this is not really the way the Transformers library is designed

That's fine, I was very unsure which approach to take. abc offers additional safety and IDE assistance as the standard way to ensure that all abstract methods are implemented, but it's probably overkill here and I also didn't like how heavy it was. I've pushed a much simpler strategy.

I recommended to just complete the error message to include wandb.

The point of this is that it's difficult to see all the missing bits. The current code isn't just missing wandb in the error message, it's also missing from default_hp_space (fixed in this PR) and the docstring (not enforced in this PR, although it could be, I just didn't want to jump there just yet).

[alexmojaki]

I also don't see how you would have benefits for IDE as you show in the PR description.

Sorry for the confusion, that's not part of this PR to keep the scope focused, but if this is merged I can follow it up with another which adds constructors to each backend class which accept the precise kwargs that the backend run supports.

[sgugger]

Thanks for bearing with me! I have additional comments but I think this is a better fit for the style of the library already.

[alexmojaki]

Opened huggingface/huggingface_hub#1526 in regards to the unrelated test failure.

[sgugger]

Thanks for iterating! One last nit and we should be good to merge. The error in the CI is indeed unrelated.

[sgugger]

Can just add a copyright here similar to all other files in the lib (potentially switching the year to 2023 if it's not)?

[sgugger]

Thanks a lot for your contribution!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.