Baseline
This guide is a good beginning to familiarize with basic mml usage. The goal is to run some baseline models for an existing task. We will use the widely known MNIST dataset for demonstration purposes.
Step 1: Prepare data
MNIST is already available in mml through the mml-tasks plugin, which might be installed through pip easily.
!pip show mml-tasks
Name: mml-tasks
Version: 0.6.0
Summary: This is the MML tasks plugin, providing dataset and task implementation for a bunch of datasets.
Home-page: https://git.dkfz.de/imsy/ise/mml
Author: Patrick Godau
Author-email: patrick.godau@dkfz-heidelberg.de
License: MIT
Location: /home/scholzpa/miniconda3/envs/mml-dev/lib/python3.10/site-packages
Editable project location: /home/scholzpa/Documents/development/github/mml/plugins/tasks
Requires: mml-core
Required-by:
# available datasets
!mml-tasks
+------------------------------------------------------------+---------------------+---------------------------+-----------+-------------------------------+
| name | dataset | type | installed | license |
+------------------------------------------------------------+---------------------+---------------------------+-----------+-------------------------------+
| idle_action_recognition | idle_action | classification | True | CC_BY_NC_4_0 |
| bean_plant_disease_classification | ibean | classification | True | MIT |
| caltech101_object_classification | caltech101 | classification | True | UNKNOWN |
| cifar10_object_classification | cifar10 | classification | True | UNKNOWN |
| cifar100_object_classification | cifar100 | classification | True | UNKNOWN |
| sklin2_skin_lesions | sklin2 | classification | True | UNKNOWN |
| barretts_esophagus_diagnosis | barretts_esophagus | classification | True | UNKNOWN |
| mura_xr_wrist | mura | classification | True | UNKNOWN |
| mura_xr_shoulder | mura | classification | True | UNKNOWN |
| mura_xr_humerus | mura | classification | True | UNKNOWN |
| mura_xr_hand | mura | classification | True | UNKNOWN |
| mura_xr_forearm | mura | classification | True | UNKNOWN |
| mura_xr_finger | mura | classification | True | UNKNOWN |
| mura_xr_elbow | mura | classification | True | UNKNOWN |
| eye_condition_classification | cataract | classification | True | UNKNOWN |
| brain_tumor_classification | brain_tumor | classification | True | CC_BY_NC_SA_4_0 |
| derm7pt_skin_lesions | derm7pt | classification | True | UNKNOWN |
| covid-19-chest-ct-image-augmentation_raw | covid_chest_ct | classification | True | DATABASE_CONTENTS_LICENSE_1_0 |
| covid-19-chest-ct-image-augmentation_Aug | covid_chest_ct | classification | True | DATABASE_CONTENTS_LICENSE_1_0 |
| covid-19-chest-ct-image-augmentation_CGAN | covid_chest_ct | classification | True | DATABASE_CONTENTS_LICENSE_1_0 |
| covid-19-chest-ct-image-augmentation_Aug&CGAN | covid_chest_ct | classification | True | DATABASE_CONTENTS_LICENSE_1_0 |
| deep_drid_quality | deep_drid | classification | True | CC_BY_4_0 |
| deep_drid_dr_level | deep_drid | classification | True | CC_BY_4_0 |
| deep_drid_clarity | deep_drid | classification | True | CC_BY_4_0 |
| deep_drid_field | deep_drid | classification | True | CC_BY_4_0 |
| deep_drid_artifact | deep_drid | classification | True | CC_BY_4_0 |
| caltech256_object_classification | caltech256 | classification | True | UNKNOWN |
| emnist_digit_classification | emnist | classification | True | UNKNOWN |
| kvasir_capsule_anatomy | kvasir_capsule | classification | True | CC_BY_4_0 |
| kvasir_capsule_content | kvasir_capsule | classification | True | CC_BY_4_0 |
| kvasir_capsule_pathologies | kvasir_capsule | classification | True | CC_BY_4_0 |
| suncolondb-classification | SUNdatabase | classification | False | CUSTOM |
| brain_tumor_type_classification | brain_tumor_type | classification | True | CC_BY_4_0 |
| breast_cancer_classification_v2 | breast_ultrasound | classification | True | CC_0_1_0 |
| stanford_dogs_image_categorization | stanford_dogs | classification | True | UNKNOWN |
| hyperkvasir_anatomical-landmarks | hyperkvasir | classification | True | CC_BY_4_0 |
| hyperkvasir_pathological-findings | hyperkvasir | classification | True | CC_BY_4_0 |
| hyperkvasir_quality-of-mucosal-views | hyperkvasir | classification | True | CC_BY_4_0 |
| hyperkvasir_therapeutic-interventions | hyperkvasir | classification | True | CC_BY_4_0 |
| nerthus_bowel_cleansing_quality | nerthus | classification | True | UNKNOWN |
| cervix_type_classification | cervical_screening | classification | False | UNKNOWN |
| laryngeal_tissues | laryngeal | classification | True | CC_BY_NC_4_0 |
| laryngeal_tissues_original_folds | laryngeal | classification | True | CC_BY_NC_4_0 |
| pneumonia_classification | pneumonia | classification | True | CC_BY_4_0 |
| lapgyn4_anatomical_structures | lapgyn4 | classification | True | CC_BY_NC_4_0 |
| lapgyn4_surgical_actions | lapgyn4 | classification | True | CC_BY_NC_4_0 |
| lapgyn4_instrument_count | lapgyn4 | classification | True | CC_BY_NC_4_0 |
| lapgyn4_anatomical_actions | lapgyn4 | classification | True | CC_BY_NC_4_0 |
| shenzen_chest_xray_tuberculosis | shenzhen_xray_tb | classification | True | UNKNOWN |
| cholect45_triplet | cholect45 | multilabel_classification | True | CC_BY_NC_SA_4_0 |
| cholect45_instrument | cholect45 | multilabel_classification | True | CC_BY_NC_SA_4_0 |
| cholect45_verb | cholect45 | multilabel_classification | True | CC_BY_NC_SA_4_0 |
| cholect45_target | cholect45 | multilabel_classification | True | CC_BY_NC_SA_4_0 |
| cholect45_triplet_soft | cholect45 | multilabel_classification | True | CC_BY_NC_SA_4_0 |
| svhn | svhn | classification | True | UNKNOWN |
| cholec80_grasper_presence | cholec80 | classification | True | CC_BY_NC_SA_4_0 |
| cholec80_bipolar_presence | cholec80 | classification | True | CC_BY_NC_SA_4_0 |
| cholec80_hook_presence | cholec80 | classification | True | CC_BY_NC_SA_4_0 |
| cholec80_scissors_presence | cholec80 | classification | True | CC_BY_NC_SA_4_0 |
| cholec80_clipper_presence | cholec80 | classification | True | CC_BY_NC_SA_4_0 |
| cholec80_irrigator_presence | cholec80 | classification | True | CC_BY_NC_SA_4_0 |
| cholec80_specimenbag_presence | cholec80 | classification | True | CC_BY_NC_SA_4_0 |
| chexpert_enlarged_cardiomediastinum | chexpert | classification | True | UNKNOWN |
| chexpert_cardiomegaly | chexpert | classification | True | UNKNOWN |
| chexpert_lung_opacity | chexpert | classification | True | UNKNOWN |
| chexpert_lung_lesion | chexpert | classification | True | UNKNOWN |
| chexpert_edema | chexpert | classification | True | UNKNOWN |
| chexpert_consolidation | chexpert | classification | True | UNKNOWN |
| chexpert_pneumonia | chexpert | classification | True | UNKNOWN |
| chexpert_atelectasis | chexpert | classification | True | UNKNOWN |
| chexpert_pneumothorax | chexpert | classification | True | UNKNOWN |
| chexpert_pleural_effusion | chexpert | classification | True | UNKNOWN |
| chexpert_pleural_other | chexpert | classification | True | UNKNOWN |
| chexpert_fracture | chexpert | classification | True | UNKNOWN |
| chexpert_support_devices | chexpert | classification | True | UNKNOWN |
| identify_nbi_infframes | nbi_infframes | classification | True | CC_BY_NC_4_0 |
| identify_nbi_infframes_original_folds | nbi_infframes | classification | False | CC_BY_NC_4_0 |
| covid_xray_classification | covid_xray | classification | True | CC_0_1_0 |
| aptos19_blindness_detection | aptos_blindness | classification | True | UNKNOWN |
| isic20_melanoma_classification | isic20 | classification | True | CC_BY_NC_4_0 |
| mnist_digit_classification | mnist | classification | True | UNKNOWN |
| crawled_covid_ct_classification | covid_ct_crawled | classification | True | UNKNOWN |
| mednode_melanoma_classification | mednode | classification | True | UNKNOWN |
| ph2-melanocytic-lesions-segmentation | PH2 | semantic_segmentation | True | UNKNOWN |
| ph2-melanocytic-lesions-classification | PH2 | classification | True | UNKNOWN |
| pascal_voc_challenge_2012 | VOC12 | semantic_segmentation | True | UNKNOWN |
| endovissub18_robotic_instrument_seg | endovis18_rob_instr | semantic_segmentation | True | UNKNOWN |
| glenda_endometriosis_segmentation | glenda | semantic_segmentation | True | CC_BY_NC_4_0 |
| hyperkvasir_polyp_segmentation | hyperkvasir_seg | semantic_segmentation | True | CC_BY_4_0 |
| motion-based-segmentation | motion-based-rec | semantic_segmentation | True | UNKNOWN |
| crowdsourced-endoscopic-instrument-segmentation-crowd-only | crowdsourced-EIS | semantic_segmentation | True | UNKNOWN |
| endometrial_implants | enid | semantic_segmentation | True | CC_BY_NC_4_0 |
| image2image-raw | image2image | semantic_segmentation | True | UNKNOWN |
| image2image-rand | image2image | semantic_segmentation | True | UNKNOWN |
| image2image-cholec80 | image2image | semantic_segmentation | True | UNKNOWN |
+------------------------------------------------------------+---------------------+---------------------------+-----------+-------------------------------+
Overall 96 raw tasks available, from 48 datasets. Filter matches 95 raw tasks with additional 248 variants derived thereof.
# install data for MNIST (data already installed here, might take a couple of minutes depending on hardware and ethernet connection)
!mml create task_list=[mnist_digit_classification]
[2025-05-02 22:11:23,262][mml][INFO] - Started MML 1.0.4 on Python 3.10.17 with mode CREATE.
[2025-05-02 22:11:23,263][mml][INFO] - Plugins loaded: ['mml-tasks']
[2025-05-02 22:11:23,595][mml.core.scripts.schedulers.create_scheduler][INFO] - Skipping creation of task mnist_digit_classification because there already seems to be a RAW version of that.
[2025-05-02 22:11:23,596][mml][INFO] - MML init time was 0.0h 0.0m 0.33s.
[2025-05-02 22:11:23,600][mml.core.scripts.schedulers.create_scheduler][INFO] - Starting task creation!
[2025-05-02 22:11:23,600][mml.core.data_loading.file_manager][INFO] - A total of 0 paths have been created during this run.
[2025-05-02 22:11:23,601][mml.core.scripts.schedulers.base_scheduler][INFO] - Successfully finished all experiments!
[2025-05-02 22:11:23,601][mml][INFO] - MML run time was 0.0h 0.0m 0.00s.
# preprocess task
!mml pp task_list=[mnist_digit_classification] preprocessing=size224
/home/scholzpa/miniconda3/envs/mml/lib/python3.8/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[2024-11-22 15:18:20,420][mml][INFO] - Started MML 0.14.2 on Python 3.8.12 with mode PP.
[2024-11-22 15:18:20,420][mml][INFO] - Plugins loaded: ['mml-tasks', 'mml-dimensionality', 'mml-similarity', 'mml-tags', 'mml-sql', 'mml-drive', 'mml-lsf', 'mml-suggest', 'mml-prevalences', 'mml-tf']
[2024-11-22 15:18:20,649][mml.core.scripts.schedulers.base_scheduler][INFO] - Executing after init hook: check_lsf_workers
[2024-11-22 15:18:20,650][mml_lsf.workers][INFO] - LSF cluster plugin detected local system, no changes made to the number of workers.
[2024-11-22 15:18:20,657][mml][INFO] - MML init time was 0.0h 0.0m 0.24s.
[2024-11-22 15:18:20,659][mml.core.scripts.schedulers.base_scheduler][INFO] - Preparing experiment ...
[2024-11-22 15:18:20,660][mml.core.scripts.schedulers.base_scheduler][INFO] - Starting experiment!
[2024-11-22 15:18:20,660][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - Starting preprocessing data for task mnist_digit_classification
[2024-11-22 15:18:20,660][py.warnings][WARNING] - /home/scholzpa/Documents/development/gitlab/mml/src/mml/core/scripts/schedulers/preprocess_scheduler.py:101: UserWarning: THIS BEHAVIOUR CHANGED: Test data is now also to be preprocessed!
[2024-11-22 15:18:21,009][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - Preprocessing split: full_train
100%|███████████████████████████████████| 60000/60000 [00:23<00:00, 2564.80it/s]
[2024-11-22 15:18:44,669][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - Existing full_train files found:
image 0
[2024-11-22 15:18:44,670][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - Preprocessing split: test
[2024-11-22 15:18:44,670][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - No samples in split: test
[2024-11-22 15:18:44,670][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - Preprocessing split: unlabelled
[2024-11-22 15:18:44,670][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - No samples in split: unlabelled
[2024-11-22 15:18:45,041][mml.core.data_loading.file_manager][INFO] - Writing task description at /home/scholzpa/Pictures/datasets/mml_data/PREPROCESSED/size224/DSET_mnist/temp.json.
[2024-11-22 15:18:45,122][mml.core.data_preparation.utils][INFO] - Calculating mean, std and size. This may take a couple of minutes.
Gathering sizes: 100%|█████████████████| 60000/60000 [00:05<00:00, 11661.43it/s]
Gathering mean and std: 100%|█████████████████| 600/600 [02:12<00:00, 4.53it/s]
[2024-11-22 15:21:03,916][mml.core.data_loading.file_manager][INFO] - Writing task description at /home/scholzpa/Pictures/datasets/mml_data/PREPROCESSED/size224/DSET_mnist/TASK_mnist_digit_classification.json.
[2024-11-22 15:21:03,998][mml.core.data_preparation.task_creator][INFO] - Testing the loading of /home/scholzpa/Pictures/datasets/mml_data/PREPROCESSED/size224/DSET_mnist/TASK_mnist_digit_classification.json...
[2024-11-22 15:21:04,601][mml.core.data_preparation.task_creator][INFO] - Testing of /home/scholzpa/Pictures/datasets/mml_data/PREPROCESSED/size224/DSET_mnist/TASK_mnist_digit_classification.json finished, dataset loading time was 0.60 seconds, sample loading time was 0.00 seconds.
[2024-11-22 15:21:04,615][mml.core.scripts.schedulers.preprocess_scheduler][INFO] - Finished preprocessing the data for task mnist_digit_classification
[2024-11-22 15:21:04,630][mml.core.data_loading.file_manager][INFO] - A total of 0 paths have been created during this run.
[2024-11-22 15:21:04,630][mml.core.scripts.schedulers.base_scheduler][INFO] - Successfully finished all experiments!
[2024-11-22 15:21:04,630][mml][INFO] - MML run time was 0.0h 2.0m 43.97s.
Step 2: Use defaults for training
# to run a single training
!mml train pivot.name=mnist_digit_classification preprocessing=size224 proj=demo_baseline
# afterwards inspect the training via running tensorboard and inspecting through the browser
!tensorboard --logdir /path/to/mml/results/demo_baseline
Step 3: Optimizing hyperparameters
We will perform a simple grid search with a couple of architectures & augmentation strategies.
# to run a hyperparameter search (here: grid search with 3 models x 3 augmentation strategies),
# to speed things up: no cross-validation, only 1 epoch, deactivated learning rate tuning
!mml train pivot.name=mnist_digit_classification preprocessing=size224 arch.name=resnet18,tiny_vit_21m_224.dist_in22k_ft_in1k,tf_efficientnet_b0.ns_jft_in1k augmentations=basic,randaugment,load_imagenet_aa proj=demo_grid trainer.max_epochs=1 tune.lr=false sampling.batch_size=100 mode.cv=false mode.nested=False +hpo/sampler=grid --multirun
at the end it prints:
[HYDRA] Best parameters: {‘arch.name’: ‘tiny_vit_21m_224.dist_in22k_ft_in1k’, ‘augmentations’: ‘randaugment’} [HYDRA] Best value: 0.0211497992277145
# use the best hyperparameters in a full training run (the use_best_params is the name of our search, a folder inside the proj folder, inside "hpo", i.e. MML_RESULTS_PATH/demo_grid/hpo/2024-12-03_12-28-46_362374)
!mml train proj=demo_grid use_best_params=2024-12-03_12-28-46_362374 pivot.name=mnist_digit_classification
[mml][INFO] - ————————————————————————————- [mml][INFO] - Loaded hpo results from study 2024-12-03_12-28-46_362374 and merged 2 params into config. [mml][INFO] - > arch.name=tiny_vit_21m_224.dist_in22k_ft_in1k [mml][INFO] - > augmentations=randaugment [mml][INFO] - ————————————————————————————- [mml][INFO] - Started MML 0.14.2 on Python 3.10.9 with mode TRAIN. …
For more complex optimization setups (e.g. multi node optimization) one may use the mml-sql plugin. See the plugin documentation for installation and setup instructions.