Reproducible Evaluation of Systems

Last updated on Feb 16, 2020

Website: falsifiable.us
Funding: NSF OAC-1450488, CROSS
Overview: USENIX ;login: Winter 2016
Workshops:

Independently validating experimental results in the field of computer systems research is a challenging task. Recreating an environment that resembles the one where an experiment was originally executed is a time-consuming endeavor. Popper is a convention (or protocol) for conducting experiments following a DevOps approach that allows researchers to make all associated artifacts publicly available with the goal of maximizing automation in the re-execution of an experiment and validation of its results.

reproducibility
Avatar
Carlos Maltzahn
Adjunct Professor, Founder & Director of CROSS

My research interests include programmable storage systems, big data storage & processing, scalable data management, distributed systems performance management, and practical reproducible research.

Related

Publications

Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez, Jan Rellermeyer, Carlos Maltzahn, Robert Ricci, Alexandru Iosup. Is Big Data Performance Reproducible in Modern Cloud Networks?. NSDI ‘20, 2020.

PDF Project Slides

Jayjeet Chakraborty, Ivo Jimenez, Carlos Maltzahn, Arshul Mansoori, Quincy Wofford. Popper 2.0: A Container-native Workflow Execution Engine For Testing Complex Applications and Validating Scientific Claims. Poster at 2020 Exaxcale Computing Project Annual Meeting, Houston, TX, February 3-7, 2020, 2020.

PDF Project

Andrea David, Mariette Souppe, Ivo Jimenez, Katia Obraczka, Sam Mansfield, Kerry Veenstra, Carlos Maltzahn. Reproducible Computer Network Experiments: A Case Study Using Popper. P-RECS’19, 2019.

PDF Project

Ivo Jimenez, Carlos Maltzahn. Spotting Black Swans With Ease: The Case for a Practical Reproducibility Platform. 1st Workshop on Reproducible, Customizable and Portable Workflows for HPC (ResCuE-HPC’18, co-located with SC’18), 2018.

PDF Project Slides

Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, Robert Ricci. Taming Performance Variability. 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18), 2018.

PDF Project

Michael A. Sevilla, Carlos Maltzahn. Popper Pitfalls: Experiences Following a Reproducibility Convention. P-RECS’18, 2018.

PDF Project Poster

Ivo Jimenez, Noah Watkins, Michael Sevilla, Jay Lofstead, Carlos Maltzahn. quiho: Automated Performance Regression Testing Using Inferred Resource Utilization Profiles. 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018), 2018.

PDF Project

Ivo Jimenez, Michael Sevilla, Noah Watkins, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Andrea Arpac-Dusseau, Remzi Arpaci-Dusseau. The Popper Convention: Making Reproducible Systems Evaluation Practical. 4th International Workshop on Reproducibility in Parallel Computing (REPPAR) in conjunction with IPDPS 2017, 2017.

PDF Project

Ivo Jimenez, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Jay Lofstead, Carlos Maltzahn, Kathryn Mohror, Robert Ricci. PopperCI: Automated Reproducibility Validation. Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT’17) in conjunction with IEEE INFOCOM 2017, 2017.

PDF Project

Joshua P. Hacker, John Exby, David Gill, Ivo Jimenez, Carlos Maltzahn, Timothy See, Gretchen Mullendore, Kathryn Fossell. A Containerized Mesoscale Model and Analysis Toolkit to Accelerate Classroom Learning, Collaborative Research, and Uncertainty Quantification. Bull. Amer. Meteor. Soc., 2017.

PDF Project

Ivo Jimenez, Carlos Maltzahn, Jay Lofstead, Adam Moody, Kathryn Mohror, Remzi Arpaci-Dusseau, Andrea Arpaci-Dusseau. Characterizing and Reducing Cross-Platform Performance Variability Using OS-level Virtualization. VarSys’16, 2016.

PDF Project

Josh Hacker, John Exby, Nick Chartier, David Gill, Ivo Jimenez, Carlos Maltzahn, Gretchen Mullendore. Collaborative Research and Education with Numerical Weather Prediction Enabled by Software Containers. American Meteorological Society 32nd Conference on Environmental Processing Technologies, 2016.

Project

Ivo Jimenez, Carlos Maltzahn, Jay Lofstead, Kathryn Mohror, Adam Moody, Remzi Arpaci-Dusseau, Andrea Arpaci-Dusseau. Tackling the Reproducibility Problem in Storage Systems Research with Declarative Experiment Specifications. PDSW’15, 2015.

PDF Project

Ivo Jimenez, Carlos Maltzahn, Adam Moody, Kathryn Mohror, Jay Lofstead, Remzi Arpaci-Dusseau, Andrea Arpaci-Dusseau. The Role of Container Technology in Reproducible Computer Systems Research. First Workshop on Containers (WoC 2015) (Workshop co-located with IEEE International Conference on Cloud Engineering - IC2E 2015), 2015.

PDF Project