Hyperparameter optimization

Attaching --multirun to your command will start the job in hpo mode. Note that multirun does not offer the continue (see continue) functionality!

Note

More details on the config options can be found at hpo.

Gridsearch

This is a very easy way to start a bunch of jobs with slightly varying setting. Find the details at hydra multirun. Multiple values for a parameter are simply given at once separated by a ,. You must specify hpo=grid for this variant. An example usage to vary learning rate is given below:

mml MY_MODE proj=MY_HPO_PROJ hpo=grid optimizer.lr=1.e-4,1.e-5 --multirun

Optuna

This hpo method is more involved. Details can be found at the hydra optuna plugin website as well as the website of the backend module Optuna.

usage

Create a config file defining your search space for the hyperparameters at configs/search_space. And afterwards call the program as follows (make sure your mode has a return value to optimize using AbstractBaseScheduler’s return_value).

mml MODE_WITH_RETURN_VALUE proj=MY_HPO_PROJ search_space=MY_SEARCH_SPACE --multirun

plugin

The mml-sql plugin extends this setup with a database backend to coordinate hyperparameter optimization trials across multiple nodes of a computing infrastructure. It further allows for results persistence in the storage.