3. Hyperparameter Optimization

In case you'd like to let the machine scan the hyperparameters for you for the ones that perform the best on the validation set, in the embedding and prediction modules, there's a launch_hyperparameter_scan command that can be executed that will launch multiple jobs (in series on a local machine, in parallel on a high performance computing server using the -t option, add a GitHub issue and this can change soon).

The hyperparameter scan works by randomly adjusting the command line options that are hyperparameters for methylnet-embed and methylnet-predict. A default search grid of hyperparameters is supplied, but you can come add your own grids using the YAML format (see https://github.com/Christensen-Lab-Dartmouth/MethylNet/blob/master/example_scripts/example_embedding_hyperparameter_grid.yaml https://github.com/Christensen-Lab-Dartmouth/MethylNet/blob/master/example_scripts/example_prediction_hyperparameter_grid.yaml), referencing it with the -grid option. The neural network topology can also be randomly generated, and weight can be given to generating wide and deep or narrow and shallow networks using the (-mc) command line option. Once the hyperparameter search grid has been specified, use these commands to launch many jobs and obtain validation losses for each of the jobs (the results are tracked in the logs of the embeddings and predictions folders). Finally, the command offers ab option to select the top performing hyperparameters and train the final model again for both the embedding and prediction task. For every job launched, the hyperparameter scan also stores embedding visualizations in the visualizations directory so you can compare model results. See the supplementals and text in https://www.biorxiv.org/content/10.1101/692665v1 for more information about how this works.

Let's first do this using for finding the best embedding:

methylnet-embed launch_hyperparameter_scan -cu -sc Age -t -mc 0.84 -b 1. -g -j 400

This command spares you the expense of wasting a lot of time trying to figure out the interface of the previous sections, and handles the scan for you. Cuda/GPU usage is now specified using -cu. (-sc) Age is plotted in the final visualization. This does not have to be set, but you can enforce beta to take on a certain value by setting the -b option. (-g) should be set and just specifies to create and append the hyperparameter log. -j is the number of jobs to run. These 400 jobs with 400 random hyperparameters are run. Then, we use -n 1 to rerun the job with the top hyperparameter. Setting -n higher would rerun the top set of hyperparameters. The scan module also supports mixing of hyperparameters operations similar to a genetic algorithm, but this is beyond the scope of this tutorial or what is needed.

methylnet-embed launch_hyperparameter_scan -cu -sc Age -t -g -n 1 -b 1.

It is very easy to train the prediction model on the previous neural network. The only changes here are adding the -ic option, which replaces -sc. I did not mention this above, but if launching many jobs to PBS/Torque using the -t option, you can load custom libraries with your own commands embedded in the -a script.

methylnet-predict launch_hyperparameter_scan -cu -ic Age -t -mc 0.84 -g -j 200 -a "module load python/3-Anaconda && source activate methylnet_pro2"

And once again use the -n 1 command to get the model with the top prediction. It is important to always use the -n option, because when running these scans, MethylNet deletes previous jobs outputs to save space. The new job that is run is done so to produce the best results, but stochastically due to factors such as weight initialization, random seed, etc. the top -n job run may not perform as well as the previous best job and should be rerun until the user is satisfied with the results. -cat should be specified for categorical predictions.

methylnet-predict launch_hyperparameter_scan -cu -ic Age -t -g -n 1 -a "module load python/3-Anaconda && source activate methylnet_pro2"