Reproducible Evaluation of Systems

Last updated on Dec 6, 2024

Website: getpopper.io (archived)
Funding: NSF OAC-1450488, NSF OAC-1836650, CROSS
Overview: USENIX ;login: Winter 2016
Workshops and Conferences:

1st International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS 2018) held in conjunction with ACM HPDC 2018.
2nd International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS 2019) held in conjunction with ACM HPDC 2019. Annual forum for the ACM Emerging Interest Group for Reproducibility and Replicability (EIGREP).
3rd International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS 2020) held in conjunction with ACM HPDC 2020. Annual forum for the ACM Emerging Interest Group for Reproducibility and Replicability (EIGREP).
4th International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS 2021) held in conjunction with ACM HPDC 2021. Annual forum for the ACM Emerging Interest Group for Reproducibility and Replicability (EIGREP).
5th International Workshop on Practical Reproducible Evaluation of Computer Systems (P-RECS 2022) held in conjunction with ACM HPDC 2022. Annual forum for the ACM Emerging Interest Group for Reproducibility and Replicability (EIGREP).
2023 ACM Conference on Reproducibility and Replicability (ACM REP ‘23). Annual forum for the ACM Emerging Interest Group for Reproducibility and Replicability (EIGREP).
2024 ACM Conference on Reproducibility and Replicability (ACM REP ‘24). Annual forum for the ACM Emerging Interest Group for Reproducibility and Replicability (EIGREP).
2025 ACM Conference on Reproducibility and Replicability (ACM REP ‘25). Annual forum for the ACM Emerging Interest Group for Reproducibility and Replicability (EIGREP).

Independently validating experimental results in the field of computer systems research is a challenging task. Recreating an environment that resembles the one where an experiment was originally executed is a time-consuming endeavor. Popper is a convention (or protocol) for conducting experiments following a DevOps approach that allows researchers to make all associated artifacts publicly available with the goal of maximizing automation in the re-execution of an experiment and validation of its results.

Reproducibility

Carlos Maltzahn

Adjunct Professor, Sage Weil Presidential Chair for Open Source Software, Founder & Director of CROSS, OSPO

My research interests include programmable storage systems, big data storage & processing, scalable data management, distributed systems performance management, and practical reproducible research.

Publications

Farid Zakaria, Thomas R. W. Scogland, Todd Gamblin, Carlos Maltzahn. Mapping Out the HPC Dependency Chaos. SC22, 2022.

Stephen Lien Harrell, Scott Michael, Carlos Maltzahn. Advancing Adoption of Reproducibility in HPC: A Preface to the Special Section. IEEE Transactions on Parallel and Distributed Systems, 2022.

Jayjeet Chakraborty, Carlos Maltzahn, Ivo Jimenez. Enabling seamless execution of computational and data science workflows on HPC and cloud with the Popper container-native automation engine. CANOPIE HPC 2020 (at SC20), 2020.

Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez, Jan Rellermeyer, Carlos Maltzahn, Robert Ricci, Alexandru Iosup. Is Big Data Performance Reproducible in Modern Cloud Networks?. NSDI ‘20, 2020.

Jayjeet Chakraborty, Ivo Jimenez, Carlos Maltzahn, Arshul Mansoori, Quincy Wofford. Popper 2.0: A Container-native Workflow Execution Engine For Testing Complex Applications and Validating Scientific Claims. Poster at 2020 Exaxcale Computing Project Annual Meeting, Houston, TX, February 3-7, 2020, 2020.

Andrea David, Mariette Souppe, Ivo Jimenez, Katia Obraczka, Sam Mansfield, Kerry Veenstra, Carlos Maltzahn. Reproducible Computer Network Experiments: A Case Study Using Popper. P-RECS'19, 2019.

Ivo Jimenez, Carlos Maltzahn. Spotting Black Swans With Ease: The Case for a Practical Reproducibility Platform. 1st Workshop on Reproducible, Customizable and Portable Workflows for HPC (ResCuE-HPC'18, co-located with SC'18), 2018.

Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, Robert Ricci. Taming Performance Variability. 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI'18), 2018.

Michael A. Sevilla, Carlos Maltzahn. Popper Pitfalls: Experiences Following a Reproducibility Convention. P-RECS'18, 2018.

Ivo Jimenez, Noah Watkins, Michael Sevilla, Jay Lofstead, Carlos Maltzahn. quiho: Automated Performance Regression Testing Using Inferred Resource Utilization Profiles. 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018), 2018.

Ivo Jimenez, Michael Sevilla, Noah Watkins, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Andrea Arpac-Dusseau, Remzi Arpaci-Dusseau. The Popper Convention: Making Reproducible Systems Evaluation Practical. 4th International Workshop on Reproducibility in Parallel Computing (REPPAR) in conjunction with IPDPS 2017, 2017.

Ivo Jimenez, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Jay Lofstead, Carlos Maltzahn, Kathryn Mohror, Robert Ricci. PopperCI: Automated Reproducibility Validation. Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT'17) in conjunction with IEEE INFOCOM 2017, 2017.

Joshua P. Hacker, John Exby, David Gill, Ivo Jimenez, Carlos Maltzahn, Timothy See, Gretchen Mullendore, Kathryn Fossell. A Containerized Mesoscale Model and Analysis Toolkit to Accelerate Classroom Learning, Collaborative Research, and Uncertainty Quantification. Bull. Amer. Meteor. Soc., 2017.

Ivo Jimenez, Carlos Maltzahn, Jay Lofstead, Adam Moody, Kathryn Mohror, Remzi Arpaci-Dusseau, Andrea Arpaci-Dusseau. Characterizing and Reducing Cross-Platform Performance Variability Using OS-level Virtualization. VarSys'16, 2016.

Josh Hacker, John Exby, Nick Chartier, David Gill, Ivo Jimenez, Carlos Maltzahn, Gretchen Mullendore. Collaborative Research and Education with Numerical Weather Prediction Enabled by Software Containers. American Meteorological Society 32nd Conference on Environmental Processing Technologies, 2016.

Ivo Jimenez, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Adam Moody, Remzi Arpaci-Dusseau, Andrea Arpaci-Dusseau. Tackling the Reproducibility Problem in Storage Systems Research with Declarative Experiment Specifications. PDSW'15, 2015.

Ivo Jimenez, Carlos Maltzahn, Adam Moody, Kathryn Mohror, Jay Lofstead, Remzi Arpaci-Dusseau, Andrea Arpaci-Dusseau. The Role of Container Technology in Reproducible Computer Systems Research. First Workshop on Containers (WoC 2015) (Workshop co-located with IEEE International Conference on Cloud Engineering - IC2E 2015), 2015.