Software engineering, development and digitalization
In 2020, an interdepartmental software task force (STF) was formed to serve as a forum for discussion on topics related to software development and digital workflows at the MPIE. A central goal was to facilitate interdepartmental collaboration by co-developing and integrating workflows, aligning internally developed software, and rolling out digital services. Trainings and workshops were organized in order to engage the entire MPIE community. Some highlights of the STF activities will be outlined in the following.
Each department presides over various independent software projects at various stages of development, the main ones being pyiron (CM) for atomistic simulations, DAMASK (MA) for modelling crystal plasticity, PARAPROBE (MA) for analysing atom probe tomography data, and TEMMETA (SN) for analysing transmission electron microscopy data. Since reproducibility is a central issue for all experimentalists and modellers, integration efforts centred on pyiron, which provides a platform for creating reproducible workflows. Mediated by interactions within the SFT, the developers of DAMASK and pyiron collaborated to create a common interface, thereby laying the foundations for direct coupling of atomistic simulations and continuum mechanics. Similarly, progress was made to integrate common TEMMETA workflows into pyiron, but for practical use cases graphical user interfaces must be developed, which is an ongoing effort.
The Jupyter notebooks employed here are becoming increasingly popular tools for creating such kind of advanced data analysis workflows. To reduce the barrier to experiment with these tools and to provide an environment of software packages for all researchers at the MPIE a jupyterhub server was launched in the CM department. The server allows MPIE members to perform digital workflows via a web browser. The deployment of the server utilizes in background a network of docker containers, which on the one hand ensures the reproducibility of the workflows via conda environments, and on the other hand takes care of the user authentication and http communications (Fig. 1).
In the SN department, an eLabFTW instance (an electronic lab notebook application) was launched to digitize the documentation of experimental workflows. With the help of QR codes that link to eLab entries, the history of samples and experiments performed on them is transparent and accessible to others after the creator of the sample leaves the institute. Through its python API, information on eLabFTW is accessible from jupyter notebooks and can be integrated with analyses. These workflows were presented outside the institute at multiple conferences and workshops.
Within the MPIE, STF organized three well received public events: (i) a seminar on electronic lab notebooks that featured an invited presentation by the developer of eLabFTW, (ii) a seminar demonstrating cmserver02 with a series of digital workflows featuring the simulation toolkits from the CM and MA departments, and (iii) a workshop on data visualization with python featuring a set of hands-on tutorials for plotting and image analysis (Fig. 2).
STF also organized internal meetings for knowledge exchange featuring talks on modern software development tools such as docker containerization, cython, E-Lab notebooks, Flask APIs, the numba and Dask packages, and CUDA programming in python. These meetings, as well as the workshops and hands-on-tutorials impressively demonstrated the practical potential of the developed hardware and the software infrastructure for interdepartmental collaborations.