Our Policy for Reproducibility & Transparency in Research

The ability to reproduce research results and the transparency of research data are crucial pillars of research. Yet scientists often point out limited reproducibility and transparency in their research areas, and AI and machine learning (ML) research are no exception, with some recent headlines including:

• The reproducibility issues that haunt health-care AI
• Artificial Intelligence Confronts a 'Reproducibility' Crisis
• AI research still has a reproducibility problem

In response to issues with reproducibility in ML research, there have even been dedicated "ML Reproducibility Challenges" running for multiple years, where researchers could submit reports documenting their attempts to reproduce research published at top ML conferences and journals.

To avoid being part of the problem, our research group has adopted action policies to ensure reproducibility and transparency in our research. In summary, every published paper must:

1. detail the experimental setup to enable reproduction of experiments and reported results (e.g. algorithm and architecture hyper-parameter values, environment parameters, dependencies, etc.);
2. provide access to implementation code used for the experiments along with documentation and instructions for running experiments;
3. provide access to datasets and results data reported in the paper along with documentation for how to use the data.

Besides ensuring reproducibility, making implementation code available has important additional benefits. If the code is readily available, then other researchers will more easily be able to (and, thus, more likely) use your algorithms as comparison baselines or to build new algorithms, thus increasing the impact of your work. It also gives a level of "protection" in that other researchers will use a correct implementation of your algorithm rather than a bad/buggy implementation of their own that could lead to incorrect results. And lastly, some people may find it helpful to understand your methods by reading your code in addition to your paper.

Releasing the results data allows other researchers to run hypothesis tests to verify your claims and to analyse your data further. This also includes any static datasets you used to train your models (such as for imitation learning of offline RL). However, uploading such data can be tricky if they require very large amounts of space, which is often the case in ML/RL research. Some universities provide a data sharing service where researchers affiliated with the university can create a data repository and upload large amounts of data (e.g. 100s of gigabytes). For example, we use the DataShare service provided by the University of Edinburgh. This service "freezes" each data repo and gives it a permanent public URL, meaning that users can be certain that the data in the repo will be in its original state and cannot be changed at a later time. There are also other free data sharing services on the web, such as UK Data Service, OpenML, Hugging Face, and others.

The action steps in our reproducibility policy are listed below in the form of a checklist that can be used by researchers in the field.

Give full details of the experimental setup

• In your paper, include all details needed to re-run your experiments, e.g.:
  • neural network architecture (layers, node types and numbers, activation functions, ...)
  • algorithm hyper-parameter values (learning and exploration rates, parameter scheduling, ...)
  • learning task parameters (world size, dynamics, observations, ...)
• If relevant for reproduction, also include:
  • code dependencies (i.e. packages and versions)
  • details of the computing hardware and distribution setup (e.g. parallel processes).
• If there is not enough space for all the details in the main paper, report the details in an appendix and link to the appendix in the main paper.
• When uploading to arXiv, append the appendix directly to the main paper PDF.

Upload the code

• Use a public code repository (e.g. we use a Github repo).
• Upload all implementation code for your algorithms, environments, and scripts to run experiments using an open-source license.
• Ensure that the code is clean and contains a basic documentation in the code files.
• Include a Readme file in your code repo that provides basic instructions for running the experiments.
• This should include instructions on creating an identical coding environment, for example using Python virtual environments with versioned packages or a Dockerfile.
• Optionally include troubleshooting instructions for issues that could be experienced when attempting to reproduce the results.
• Include the URL to your code in your main paper.

Upload the data

• Create a public data repository for your project with a permanent public URL.
• Upload all results data and used datasets.
• Include a Readme file in your data repo that explains the data format.
• Optionally include code files to help read or visualise the data. This could also include the code to generate the major plots in the paper.
• Include the URL to your data either in your paper or in the associated code repo.

Further reading

1. Odd Erik Gundersen, Yolanda Gil, David W. Aha: On Reproducible AI: Towards Reproducible Research, Open Science, and Digital Scholarship in AI Publications. AI Magazine, 2018.