hpo

The hpo config group controls hyperparameter optimization searches. It is triggered by the --multirun flag when invoking an mml experiment (see hydra’s documentation).

Note

If using an MMLJobDescription there is the multirun boolean flag.

To leverage the results of a hyperparameter run the CLI kwarg use_best_params can be used. See main config file for its documentation. Also note the overview given at Hyperparameter optimization.

default

hpo

• this config uses the optuna sweeper plugin of hydra

• the considered hyperparameters are given by the search_space config

• will sequentially run hpo.trials many experiments and sample hps according to hpo/sampler

• default sampler is tpe (https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.TPESampler.html)

• other samplers are hpo/sampler=random (https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.RandomSampler.html) and hpo/sampler=grid (https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.GridSampler.html)

• example: mml train proj=test search_space=example hpo/sampler=grid --multirun

• more details can be found in hydra’s optuna sweeper documentation

storage

default: null

• the optuna storage to use, if null only a temporary in-memory storage will be created

• see mml-sql for details on how to set up permanent storage

direction

default: minimize

• direction of optimization, either minimize or maximize

• must be set according to the kind of return value by the respective scheduler

pruning

default: False

• currently unused, intended for future pruning of hopeless runs

• see https://github.com/facebookresearch/hydra/issues/1954

trials

default: 100

• number of trials, each trial is one mml call with (potentially) different sampled hyperparameters

grid

storage

• this config corresponds to the default multirun variant of hydra

• use colons to provide multiple values via CLI (the search_space config is not used!)

• will sequentially run one exp for each possible combination of config values (using the colon syntax)

• example: mml train tasks=fake proj=test hpo=grid optimizer.lr=0.001,0.01 --multirun

• more details can be found in hydra’s multi-run documentation