Reproducibility

My schedule at Vault/FAST/NSDI 2020

Looking forward to meeting friends and colleagues this week. Here is my schedule.

Is Big Data Performance Reproducible in Modern Cloud Networks?

Performance variability has been acknowledged as a problem for over a decade by cloud practitioners and performance engineers. Yet, our survey of top systems conferences reveals that the research community regularly disregards variability when …

Popper 2.0: A Container-native Workflow Execution Engine For Testing Complex Applications and Validating Scientific Claims

MBWU: Benefit Quantification for Data Access Function Offloading

The storage industry is considering new kinds of storage de- vices that support data access function offloading, i.e. the ability to perform data access functions on the storage device itself as opposed to performing it on a separate compute system …

Reproducible Computer Network Experiments: A Case Study Using Popper

Computer network research experiments can be broadly grouped in three categories: simulated, controlled, and real-world experiments. Simulation frameworks, experiment testbeds and measurement tools, respectively, are commonly used as the platforms …

MBWU (MibeeWu): Quantifying benefits of offloading data management to storage devices

Spotting Black Swans With Ease: The Case for a Practical Reproducibility Platform

Advances in agile software delivery methodologies and tools (commonly referred to as DevOps) have not yet materialized in academic scenarios such as university, industry and government laboratories. In this position paper we make the case for Black …

Taming Performance Variability

The performance of compute hardware varies: software run repeatedly on the same server (or a different server with supposedly identical parts) can produce performance results that differ with each execution. This variation has important effects on …

Popper Pitfalls: Experiences Following a Reproducibility Convention

We describe the four publications we have tried to make reproducible and discuss how each paper has changed our workflows, practices, and collaboration policies. The fundamental insight is that paper artifacts must be made reproducible from the start …

quiho: Automated Performance Regression Testing Using Inferred Resource Utilization Profiles

We introduce quiho, a framework for profiling application performance that can be used in automated performance regression tests. quiho profiles an application by applying sensitivity analysis, in particular statistical regression analysis (SRA), …