The Popper Convention: Making Reproducible Systems Evaluation Practical

Abstract

Independent validation of experimental results in the field of systems research is a challenging task, mainly due to differences in software and hardware in computational environments. Recreating an environment that resembles the original is difficult and time-consuming. In this paper we introduce Popper, a convention based on a set of modern open source software (OSS) development principles for generating reproducible scientific publications. Concretely, we make the case for treating an article as an OSS project following a DevOps approach and applying software engineering best-practices to manage its associated artifacts and maintain the reproducibility of its findings. Popper leverages existing cloud-computing infrastructure and DevOps tools to produce academic articles that are easy to validate and extend. We present a use case that illustrates the usefulness of this approach. We show how, by following the Popper convention, reviewers and researchers can quickly get to the point of getting results without relying on the original author’s intervention.

Publication
4th International Workshop on Reproducibility in Parallel Computing (REPPAR) in conjunction with IPDPS 2017