{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# Adapt your dataset\n", "\n", "## NOTE:\n", "`mml` handles preprocessing the data internally. No need to manually preprocess any data in advance. See `preprocess mode`.\n", "\n", "Assume having used your dataset as a plain pytorch Dataset previously. Migration to `mml` is as easy as follows:\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Step 1: Write your DsetCreator and TaskCreator\n", "\n", "`mml` distinguishes the concepts of \"Datasets\" and \"Tasks\". Whereby \"Datasets\" contains all data (plus maybe more meta information, additional tasks on the same data, additional test samples, etc.) and the \"Task\" is only a description which samples and labels of the \"Dataset\" belong to that specific task. There are a lot of convenience functions to simplify this process.\n", "\n", "### Example: Reusing your previous dataset definition\n", "\n", "In this example we use some `torchvision` dataset to be integrated into `mml`, but it may be fully replaced with your existing dataset class." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2024-01-17T13:29:25.577217Z", "start_time": "2024-01-17T13:29:22.909528Z" }, "collapsed": false }, "outputs": [], "source": [ "from mml.api import (\n", " DSetCreator,\n", " TaskCreator,\n", " get_iterator_and_mapping_from_image_dataset,\n", " TaskType,\n", " Keyword,\n", " License,\n", " Modality,\n", " DataKind,\n", ")\n", "from torchvision.datasets import STL10" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2024-01-17T13:13:50.173543Z", "start_time": "2024-01-17T13:09:39.782179Z" }, "collapsed": false }, "outputs": [], "source": [ "REFERENCE = \"\"\"\n", "@inproceedings{Coates2011AnAO,\n", " title={An Analysis of Single-Layer Networks in Unsupervised Feature Learning},\n", " author={Adam Coates and A. Ng and Honglak Lee},\n", " booktitle={AISTATS},\n", " year={2011}\n", "}\"\"\"\n", "\n", "dset_creator = DSetCreator(dset_name=\"STL_10_DEMO\")\n", "train = STL10(root=dset_creator.download_path, split=\"train\", download=True)\n", "test = STL10(root=dset_creator.download_path, split=\"test\", download=True)\n", "dset_path = dset_creator.extract_from_pytorch_datasets(\n", " datasets={\"training\": train, \"testing\": test}, task_type=TaskType.CLASSIFICATION, class_names=train.classes\n", ")\n", "task_creator = TaskCreator(\n", " dset_path=dset_path,\n", " task_type=TaskType.CLASSIFICATION,\n", " name=\"STL_10_DEMO\",\n", " desc=\"STL-10 image recognition task\",\n", " ref=REFERENCE,\n", " url=\"https://cs.stanford.edu/~acoates/stl10/\",\n", " instr=\"downloaded via torchvision dataset (https://pytorch.org/vision/stable/generated/torchvision.datasets.STL10.html#torchvision.datasets.STL10)\",\n", " lic=License.UNKNOWN,\n", " release=\"2011\",\n", " keywords=[Keyword.NATURAL_OBJECTS],\n", ")\n", "train_iterator, idx_to_class = get_iterator_and_mapping_from_image_dataset(\n", " root=dset_path / \"training_data\", classes=train.classes\n", ")\n", "test_iterator, idx_to_class_2 = get_iterator_and_mapping_from_image_dataset(\n", " root=dset_path / \"testing_data\", classes=test.classes\n", ")\n", "assert all([a == b for a, b in zip(idx_to_class, idx_to_class_2)])\n", "task_creator.find_data(train_iterator=train_iterator, test_iterator=test_iterator, idx_to_class=idx_to_class)\n", "task_creator.auto_complete()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "That's it already! For the future you may reference your task with `stl10` (the value provided to `alias=` in the `TaskCreator`).\n", "\n", "### Example: DSetCreator when using public data\n", "\n", "In this case we recommend to implement the `DSetCreator` from scratch including the download of the data. This allows for better reproducibility. There are the following convenience functions so far:\n", "\n", " - `DSetCreator.download()` to download given a URL\n", " - `DSetCreator.kaggle_download()` to download given a kaggle dataset ID or competition ID\n", " - `DSetCreator.verify_pre_download()` if parts of the data have to be downloaded manually (e.g. access only after registration)\n", " - `DSetCreator.unpack_and_store()` simply call after any of the previous to extract the data from archive formats\n", " - `DSetCreator.transform_masks()` if necessary transform masks (e.g. from segmentation masks) to fit the `mml` requirements\n", "\n", "### Example: TaskCreator, writing your own data iterator\n", "\n", "If `get_iterator_and_mapping_from_image_dataset` does not fit your data structure, you may simply write an iterator yourself, as done with this example:\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2024-01-17T13:33:14.574381Z", "start_time": "2024-01-17T13:33:09.814012Z" }, "collapsed": false }, "outputs": [], "source": [ "dset_creator = DSetCreator(dset_name=\"laryngeal_DEMO\")\n", "dset_creator.download(\n", " url=\"https://zenodo.org/record/1003200/files/laryngeal%20dataset.tar?download=1\",\n", " file_name=\"laryngeal dataset.tar\",\n", " data_kind=DataKind.TRAINING_DATA,\n", ")\n", "dset_path = dset_creator.unpack_and_store()\n", "laryngeal_tissue = TaskCreator(\n", " dset_path=dset_path,\n", " task_type=TaskType.CLASSIFICATION,\n", " name=\"laryngeal_DEMO\",\n", " desc=\"Laryngeal dataset for patches of healthy and early-stage cancerous laryngeal tissues\",\n", " ref=\"...\",\n", " url=\"https://nearlab.polimi.it/medical/dataset/\",\n", " instr=\"download via zenodo.org/record/1003200/files/laryngeal%20dataset.tar?download=1\",\n", " lic=License.CC_BY_NC_4_0,\n", " release=\"2017\",\n", " keywords=[Keyword.MEDICAL, Keyword.LARYNGOSCOPY, Keyword.TISSUE_PATHOLOGY, Keyword.ENDOSCOPY],\n", ")\n", "classes = [\"Hbv\", \"He\", \"IPCL\", \"Le\"]\n", "folds = [\"FOLD 1\", \"FOLD 2\", \"FOLD 3\"]\n", "data_iterator = []\n", "for fold in folds:\n", " root = dset_path / \"training_data\" / \"laryngeal dataset\" / f\"{fold}\"\n", " folders = [p.name for p in root.iterdir() if p.is_dir()]\n", " assert all([cl in folders for cl in classes]), \"some class folder is not existent\"\n", " for class_folder in root.iterdir():\n", " assert class_folder.is_dir()\n", " if class_folder.name not in classes:\n", " continue\n", " for img_path in class_folder.iterdir():\n", " data_iterator.append(\n", " {\n", " Modality.SAMPLE_ID: img_path.stem,\n", " Modality.IMAGE: img_path,\n", " Modality.CLASS: classes.index(class_folder.name),\n", " }\n", " )\n", "idx_to_class = {classes.index(cl): cl for cl in classes}\n", "laryngeal_tissue.find_data(train_iterator=data_iterator, idx_to_class=idx_to_class)\n", "laryngeal_tissue.auto_complete()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "### Example: Multiple tasks per Dataset\n", "\n", "This example will be added later." ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Step 2: BONUS - automize the task creation\n", "\n", "`mml` has a `create` mode to generate tasks automatically. If set up correctly the above datasets would be downloaded and prepared automatically when calling `mml create tasks=example` (assuming `example.yaml` is already provided). This is much more convenient if using `mml` from within and not as a library - nevertheless possible and allows any other `mml` user that installed your package to quickly start on your data and code.\n", "\n", " - make your code installable via a package - you need a `pyproject.toml` and `setup.cfg` file for this\n", " - decorate the `DSetCreator` with `@register_dsetcreator` and your `TaskCreator` with `@register_taskcreator`\n", " - add an `activate.py` script to the root of your package's source code\n", " - import the module (file) that defines the creators within this file\n", " - in your `setup.cfg` (or `setup.py` or `pyproject.toml`, see [here](https://setuptools.pypa.io/en/latest/userguide/entry_point.html?highlight=entry_points#entry-points-syntax)) provide the correct entry point for `mml`\n", "\n", "```cfg\n", "[options.entry_points]\n", "mml.plugins =\n", " some_key = your_package:activate\n", "```\n", "\n", " - (replace some_key with a descriptive id and your_package with your package and your_module the module you want to refer to).\n", " - These tasks are now always linked when calling `mml task_list=[stl10,laryngeal_tissues]` 🎉\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "\n", "## Step 3: BONUS - add your task(s) to a tasks config file\n", "\n", "In order to refer to your task(s) later on create a tasks config file in a configs folder that is linked to `mml`\n", "\n", " - if you cloned `mml` just navigate into `configs/tasks`\n", " - if you are writing your own package, create a `configs` folder, best at your package root level, add a `tasks` folder inside\n", " - create a new file `example.yaml` with the following content\n", "\n", "```yaml\n", "# @package _global_\n", "\n", "tasks:\n", " - 'stl10'\n", " - 'laryngeal_tissues'\n", "\n", "pivot:\n", " name: False\n", " tags: ''\n", "\n", "tagging:\n", " all: False\n", " variants: []\n", "```\n", "\n", " - add something like the following to your `activate.py` (see step before)\n", "\n", "```python\n", "from hydra.core.config_search_path import ConfigSearchPath\n", "from hydra.core.plugins import Plugins\n", "from hydra.plugins.search_path_plugin import SearchPathPlugin\n", "\n", "\n", "# register plugin configs\n", "class MMLINSERTPLUGINNAMESearchPathPlugin(SearchPathPlugin):\n", " def manipulate_search_path(self, search_path: ConfigSearchPath) -> None:\n", " # Sets the search path for mml with copied config files\n", " search_path.append(\n", " provider=\"mml-???\", path=f\"pkg://mml_???.configs\"\n", " )\n", "\n", "\n", "Plugins.instance().register(MMLINSERTPLUGINNAMESearchPathPlugin)\n", "```\n", "\n", " - ofcourse you have to replace `INSERTPLUGINNAME` and `mml-???` / `mml_???`\n", "\n", "These tasks are now always linked when calling `mml tasks=example` 🎉" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }