Plugins

The concept of plugins is a general pillar of mml’s architecture. Plugins provide e.g. additional schedulers, tasks and config options. A list of available plugins can be found at plugins. This section is intended to describe the purpose and possibilities of plugins as well as their intended internal structure. It can thus also be seen as a guide to write your own plugin.

introduction

Plugins are actual python packages (and need to be such), and many are already available from the same registry as mml-core. Of course mml-core is a necessary dependency, so installing that one first is necessary (see Installation). In a similar fashion and within the same virtual environment one may install plugins. To make your code installable you have to add pyproject.toml and setup.cfg files ar your projects root level. See mml’s plugins/_template folder to see template files for those. More broadly the following plugin structure is recommended:

Note

root

├── src

│ └── my_plugin

│ ├── configs (optional)

│ │ └── config_group (e.g. foo)

│ │ └── bar.yaml (allows this config to be used with foo=bar)

│ ├── stuff (modules, assets, …)

│ ├── …

│ ├── __init__.py (should provide a __version__ string)

│ └── activate.py (contains all imports, adds the config search path)

├── tests

│ └── unit (put your tests in here)

├── README.md

├── MANIFEST.in (declare any assets to be shipped with your plugin)

├── pyproject.toml

└── setup.cfg

loading

Plugins are loaded during the initialization of mml during mml.cli.main(), but before compiling the hydra configuration of the experiment with the load_mml_plugins() function. This allows plugins to extend the search path of hydra for config files and provide config options themselves (you can check on these search path plugins and inspect the order by calling mml --info searchpath). Each plugin needs to provide an entry point as follows in their setup.cfg:

[options.entry_points]
mml.plugins =
    my-plugin = my_plugin.activate

This assure that my_plugin.activate is sourced before running the mml core routines and modify any behaviour before the experiment. The activate.py file may for example

• add the plugin’s config folder to the configs search path of mml to add new config options from cli

• add path assignments to the MMLFileManager via add_assignment_path()

• register DSetCreator’s or TaskCreator’s via register_dsetcreator() and register_taskcreator()

• add or overwrite methods of existing classes like AbstractBaseScheduler

• or simply import modules (from stuff) that do so themselves

testing

mml-core provides a pytest plugin so that tests (ideally within the tests folder) may use fixtures defined in mml.testing.fixtures.