Hyperparameter optimization

Attaching --multirun to your command will start the job in hpo mode. Note that multirun does not offer the continue (see Getting started) functionality!

Gridsearch

This is a very easy way to start a bunch of jobs with slightly varying setting. Find the details at hydra multirun. Multiple values for a parameter are simply given at once separated by a ,. You must specify hpo=grid for this variant. An example usage to vary learning rate is given below:

mml MY_MODE proj=MY_HPO_PROJ hpo=grid optimizer.lr=1.e-4,1.e-5 --multirun

Optuna

This hpo method is more involved. Details can be found at the hydra optuna plugin website as well as the website of the backend module Optuna.

usage

Create a config file defining your search space for the hyperparameters at configs/search_space. And afterwards call the program as follows (make sure your mode has a return value to optimize using AbstractBaseScheduler’s return_value).

mml MODE_WITH_RETURN_VALUE proj=MY_HPO_PROJ search_space=MY_SEARCH_SPACE --multirun

plugin

The mml-sql plugin extends this setup with a database backend to coordinate hyperparameter optimization trials across multiple nodes of a computing infrastructure. It further allows for results persistence in the storage.