Trainer.hyperparameter_search doesn't document wandb or offer it as a default backend

[alexmojaki]

Trainer.hyperparameter_search seems to prioritise the optuna/ray/sigopt backends, while wandb almost seems like a second-class citizen in the code. Specifically, the docstring explicitly mentions the first three backends multiple times in different contexts but not wandb, and default_hp_search_backend won't return wandb even if it's available. Is this intentional or accidental?

[sgugger]

Those are all integrations maintained by the authors of those libraries, we do not maintain them ourselves. It might be a bug, but it's up to the wandb folks to fix it in this case :-)

[alexmojaki]

Even the glue code in trainer.py that ties the various backends together?

Would you accept a PR to refactor this stuff? For example this code:

        if backend is None:
            backend = default_hp_search_backend()
            if backend is None:
                raise RuntimeError(
                    "At least one of optuna or ray should be installed. "
                    "To install optuna run `pip install optuna`. "
                    "To install ray run `pip install ray[tune]`. "
                    "To install sigopt run `pip install sigopt`."
                )
        backend = HPSearchBackend(backend)
        if backend == HPSearchBackend.OPTUNA and not is_optuna_available():
            raise RuntimeError("You picked the optuna backend, but it is not installed. Use `pip install optuna`.")
        if backend == HPSearchBackend.RAY and not is_ray_tune_available():
            raise RuntimeError(
                "You picked the Ray Tune backend, but it is not installed. Use `pip install 'ray[tune]'`."
            )
        if backend == HPSearchBackend.SIGOPT and not is_sigopt_available():
            raise RuntimeError("You picked the sigopt backend, but it is not installed. Use `pip install sigopt`.")
        if backend == HPSearchBackend.WANDB and not is_wandb_available():
            raise RuntimeError("You picked the wandb backend, but it is not installed. Use `pip install wandb`.")

contains a lot of repetition that I'd be happy to clean up, and it's easy to see how the wandb integration author missed a place to add a reference to wandb.

[sgugger]

The first bit with the runtime error is fine (though missing wandb). For the rest, it should be done in each integration which normally error very fast if the corresponding lib is not installed.