mode

The mode config group has a special role in mml and is required to be passed as the first argument (and without mode=...) e.g. as mml train ... or mml create .... Some examples can be found at Modes. Note that any additional configurations must be prepended by the mode keyword as usual (e.g. mode.subroutines=[x,y,z]). A mode basically maps to a subclass of AbstractBaseScheduler using the mode.scheduler._target_ config entry. This specific scheduler will be instantiated by mml and the run() method will be invoked to carry out any computations. Sometimes different configs may point to the same scheduler with slightly different default parameters. Such cases are marked below.

clean

mode
scheduler
_target_
default: CleanScheduler

  • the clean scheduler assists in getting rid of file artifacts, reducing disk usage by mml

subroutines
default: [temp,download]

  • “temp” routine removes temporary files that may remain as artefacts during task creation

  • “download” routine removes the downloads of a dataset

force
default: false

  • if true won’t ask for confirmation to delete files

all
default: false

  • if false only cares for tasks in task_list, otherwise all installed tasks will be considered

create

mode
scheduler
_target_
default: CreateScheduler

  • the clean scheduler assists in getting rid of file artifacts, reducing disk usage by mml

subroutines
default: [dataset,task]

  • “dataset” routine creates the necessary datasets, comprising potential download and extraction

  • “task” routine creates the task description files and necessary transformation, requires datasets

  • only datasets/tasks non-present will trigger a computation so it is safe to keep both subroutines

n_folds
default: 5

  • changes the default value into how many splits the generated tasks should be distributed to

  • must be an integer larger 1

  • only applies if the task creator uses the internal split_folds() method

  • does not apply if use_existing_folds() is used instead

ensure_balancing
default: true

  • changes the default value whether the splitting into folds should respect class distributions

  • only applies to classification tasks

  • only applies if the task creator uses the internal split_folds() method

  • does not apply if use_existing_folds() is used instead

downgrade

The downgrade config is a convenience entry to call the upgrade scheduler with the downgrade subroutine.

mode
subroutines
default: [downgrade]

  • “downgrade” undoes a compatibility upgrade back to mode.version

info

mode
scheduler
_target_
default: InfoScheduler

  • the info scheduler is intended to provide fast summarizing information on data / results

subroutines
default: [tasks,hpo,sample_grid,models]

  • “tasks” routine shows information on the tasks provided

  • “hpo” routine shows information on hpo results (see study_name parameter)

  • “samples” routine plots sample images for each task (and/or jointly)

  • “models” routine shows information on existing model descriptions for this project

study_name
default: null

  • provide a specific optuna study name to receive detailed information

  • required for the hpo subroutine

  • will only resolve in a warning if not provided

info_grid
default: True

  • generates a grid plot in tasks subroutine with each task corresponding to one cell

info_individual
default: True

  • generates a sample image in tasks subroutine for each task

post

mode
scheduler
_target_
default: PostprocessScheduler

  • the post scheduler postprocesses mml models by calibration and ensembling

subroutines
default: [calibrate,ensemble]

  • “calibrate” routine calibrates model predictions

  • “ensemble” routine ensembles (calibrated) predictions

prior
default: val

  • which prior to use for calibration

  • either test, val (based on respective splits), quantify (infers on unlabeled) or a list [p0, p1, …]

  • if test or val the TRUE underlying priors of these datasplits are used

  • quantify uses Adjusted Classify & Count on the unlabeled split

  • if a list of floats these will be used directly

eval_frac
default: 0.5

  • use this much of test data to select ensemble, remaining will be used as estimated performance

budget
default: 1000

  • how many ensemble candidates shall be tested

weights_budget
default: 10

  • how many weight variations shall be tested, set to zero to weigh all ensemble member equal

max_ensemble_size
default: 5

  • largest possible ensemble to test (up to this many base model are ensembled)

temperature
default: 0.3

  • between 0 and 1, determines search order, larger means more random combinations to be tested

weights_temperature
default: 0.5

  • between 0 and 1, determines deviation of weights to be inspected, larger means higher

pp

mode
scheduler
_target_
default: PreprocessScheduler

  • the pp scheduler preprocesses the data (of all splits) for faster training and inference

subroutines
default: [preprocess]

  • “preprocess” routine does the job and is the only subroutine for this scheduler

predict

The predict config is a convenience entry to call the train scheduler with the predict subroutine.

mode
subroutines
default: [predict]

  • “predict” routine predicts samples, requires appropriate models to be loaded

test

The test config is a convenience entry to call the train scheduler with the test subroutine.

mode
subroutines
default: [test]

  • “test” routine tests the loaded model on the test split of a task

tl

mode
scheduler
_target_
default: TransferScheduler

  • the tl scheduler enables transfer learning, aka finetuning from an existing pretrained model

pretrain_task
default: ???

  • the task the pretrained model was trained on

model_selection
default: performance

  • strategy to select the model among all models that have been pretrained on the given task

  • model must be loaded via the reuse functionality!

  • one of performance (=best loss), created (=latest), random

freeze
default: False

  • if True freeze the backbone of the model and only learn head parameters (also called linear probing)

train

mode
scheduler
_target_
default: TrainingScheduler

  • the train scheduler performs model training, as well as optionally testing and prediction

subroutines
default: [train]

  • “train” routine trains models on the target task(s)

  • “predict” produces logits on test and unlabeled split that will be stored, these can be used for postprocessing

  • “test” evaluates without any postprocessing

cv
default: true

  • whether to use cross validation

  • if true will train one model for each split in the data (of the pivot task)

  • predict and test routines will be performed for each split separately

nested
default: true

  • whether to nest the pivot task (setting aside a fraction for testing)

  • will automatically create a tagged variant for each fold (see cross validation)

  • the original validation split of that fold will be excluded as test data

  • the remaining train data is redistributed into same number of folds as the previous total number of splits

  • nesting allows to set aside the original test data and rely on a fraction of left-out training data during the development phase

store_parameters
default: true

  • set false to avoid exploding storage usage, incompatible with usage of predict or test

  • will deactivate any storing of model weights (except for the snaphots during training, see create_trainer())

  • you may alternatively set remove.parameters=true which will only delete model weights after the scheduler terminates successfully

store_best
default: true

  • if true selects the model state with the best validation score, otherwise uses final epoch model

multitask
default: false

  • whether to add additional tasks to be learned in parallel, set to false or any integer greater 1

  • will add a model head for each additional task

  • make sure potential additional task’s TaskType is supported by the model backbone

co_tasks
default: random

  • co tasks selection strategy, can be random, or an explicit list of co-tasks

  • (potential) co-tasks must be given in the task_list!

task_weights
default: null

  • wighting of tasks in the overall loss

  • if null all tasks are weighted equally, if a list of floats pivot is first and co_tasks next

eval_on
default: null

  • allows to predict/test on another task than the pivot one,

  • can also be a list of tasks, if so multiple predictions / tests are performed (one for each)

use_blueprint
default: False

  • whether to use a (previously generated) blueprint to override some configurations

  • blueprint must be loaded (via reuse.blueprint) and assigned to the pivot task

  • see the suggest plugin for an example how to generate blueprints

pipeline_keys
default: [arch,augmentations,cbs,loss,lr_scheduler,mode,optimizer,preprocessing,sampling,trainer,tta,tune]

  • these pipeline keys will be for one stored as a documentation of the training pipeline

  • on the other hand if using a blueprint, only these keys will be loaded to override the given sections

upgrade

mode
scheduler
_target_
default: UpgradeScheduler

  • the upgrade scheduler assists in migrating your database between different versions of mml

  • whenever a backward compatibility breaking version is released (with respect to data and/or results) this scheduler may resolve upcoming issues

subroutines
default: [upgrade]

  • “upgrade” does a compatibility upgrade from mode.version

version
default: ???

  • for the “upgrade” subroutine this marks the previous version

  • for the “downgrade” subroutine this marks the target version

  • is a required argument! Example mml upgrade mode.version=0.12.0