callbacks

The callbacks config group determines the lightning callbacks to be used by mml, whenever lightning is invoked (training, testing, predicting). Note that callbacks are mostly deactivated during tuning. Lightning provides comes with some builtin callbacks but special care has to taken since some callbacks are handled by mml itself internally. Callback creation is handled within create_trainer(), there

• StopAfterKeyboardInterrupt will be added automatically and prevents lightning from catching keyboard interrupts

• MetricsTrackerCallback will also be added if metrics_callback=True and made accessible as metrics_callback

• MMLRichProgressBar or MMLTQDMProgressBar are used as progress bar modifications (depending on logging.render settings)

• MMLModelCheckpoint is added multiple times as described in create_trainer()

All other callbacks are coordinated through the callbacks config. Noteworthy though the yaml config files are within the callbacks folder the underlying config structure is cbs:{id:{**kwargs}}. This allows the following two features:

• stacking callbacks: callbacks=[early,swa] - here adding both early stopping and stochastic weight averaging to the callbacks

• modify callback kwargs: cbs.early.patience=5 - here setting the patience parameter of early stopping

The following callbacks configuration files are currently available:

default

cbs

lrm

• logs information on learning rate

• details: https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.LearningRateMonitor.html#learningratemonitor

early

cbs

early

• enables early stopping (by default based on validation loss, but configurable)

• many kwargs may be accessed e.g. via +cbs.early.min_delta=0.001, only some are documented here directly

• details: https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.EarlyStopping.html#earlystopping

monitor

default: val/loss

• the monitor parameter controls the metric that is observed and required to improve to prevent stopping

• may be set to any metric that is measured (see metrics), but the task name needs to be inserted (e.g. val/mml_fake_task/MulticlassAccuracy)

mode

default: min

• either min or max

• if adapting the monitor parameter it is important to also give the orientation of the metric

• if “the bigger, the better” this mode should be max

patience

default: 10

• controls the number of epochs awaiting improvement before training is stopped

• note that trainer.min_epochs overrules stopping too early

• if using lr_scheduler=plateau make sure EarlyStopping patience is larger than lr_scheduler.patience

none

cbs

• this configuration file will clear any callback (except the ones handled internally by mml)

mixup

cbs

mixup

• custom mixup callback MixUpCallback

• see docstring for kwargs

cutmix

cbs

cutmix

• custom cutmix callback CutMixCallback

• see docstring for kwargs

swa

cbs

swa

• activates stochastic weight averaging

• details: https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.StochasticWeightAveraging.html#stochasticweightaveraging

swa_lrs

default: 0.005

• the SWA learning rate to use

stats

cbs

stats

• collects statistics on the device

• details: https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.DeviceStatsMonitor.html#devicestatsmonitor