Empowering Repeated Research Initiative
Overview
Repeated research is an important part of scientific methodology. Repeating or replicating research not only brings an independent perspective to the investigation, but also provides a basis for comparing different approaches and extending known results. The process of repeating others’ research also allows us to learn, not only from their insights but also their experimental methodology. Replication is thus an important facet of scientific debate.
Up until recently repeating computer science research, while desirable in principle, was often difficult in practice. This is due to several well-identified factors such as the lack of incentives (e.g., in publication venues) that value such effort, lack of common research platforms and tools, and the difficulty of packaging the requisite digital artifacts in an easily shareable form and the mechanisms to combine and publish them. Change is needed.
The objective of the “Empowering Repeated Research Initiative” is to create a forum that catalyzes such change. We envision two complementary activities to be undertaken in this initiative, in the short and longer term. A first set of nearer-term activities include a series of workshops (see below) educating students and involving faculties in the art of reproducible research, and identifying and developing a set of tools and best-practice-techniques for enabling reproducible research. We would like to invite submissions that repeat results published in past papers and provide a brief analysis of the results as well as the ability to replicate them, similar to “Xen and the Art of Repeated Research” by Clark et al. The repeated results can range over a variety of computer science systems topics such as: wireless, networking, operating systems, distributed systems, cloud computing, and others. With the emergence of generally accessible testbeds, technological innovations such as containers, virtual machines, or computational notebooks, repeatability can be both easier and an inherent part of the discovery process. Submission on these tools and experience with them are also welcome. Consensus best-practice techniques will emerge from these workshops. The longer-term objective is to to reach out to the main stakeholders, professional societies and conferences, with recommendations arguments and potential solutions for embedding reproducibility in the research life-cycle.
Empower will set up a series of workshop, with regular presentation, demos, panels and discussion. These will be organized virtually but additionally collocated with conferences and workshops when possible. Each event will be prepared by a small group of colleagues who will set up the agenda, organize the debate and synthesize recommendations. A continuous “challenge” will be organized with an award to recognize important “hands-on” contributions in the field with a strong emphasis on student involvement.
Topics will cover but are not limited to:
- Reproducible network and system research
- Computation and data reproducibility
- Educating the future researchers
-
Testbed Requirements to support reproducibility
- Best Practices for the 3M’s: metadata, measurement and monitoring
- Open Data
- How other sciences address reproducible research
- etc.
Organization and agenda
The EU Empower support action and the NSF PAWR Office will organize the initiative, provide the management and logistics. Participation is free but must be made here:
1st Virtual Workshop
- Tentative date: September/October 2020
- Topic: testbeds and tools
- Chair: Kate Keahey
2nd Virtual Workshop
- Tentative date: December 2020
3rd Physical workshop
- April 2021, Paris or Athens
Program Committee
Fdida, Serge | Sorbonne University, Co-chair
Keahey, Kate | University of Chicago, Co-chair
Bonaventure, Olivier | UC Louvain
Calvert, Ken | University of Kentucky
Carle, Georg | Technical University of Munich
Claffy, Kc | University of California, San Diego
Crowcroft, Jon | University of Cambridge
Dabbous, Walid | INRIA
Gosain Abhimanyu | Northeastern University
Hunold, Sascha | TU Wien
Kurose, Jim | University of Massachusetts Amherst
Legrand, Arnaud | CNRS
McKeown, Nick | Stanford University
Nakao, Aki | University of Tokyo
Rezende, Jose | Federal University of Rio de Janeiro & RNP
Ricci, Robert | University of Utah
Ruth Paul | RENCI
Seskar Ivan | Rutgers
Taufer, Michaela | University of Tennessee
Vermeulen, Brecht | imec / Ghent University
Zink, Mike | University of Massachusetts Amherst
Reproducibility Continuous Challenge
Chair: tbd
Sponsors
Contact & Registration
Contact: stavmag@the.forthnet.gr
Registration:https://docs.google.com/forms/d/1off8Ijo1_VRVkL–gg5pJ_biJUuKkAFkvyO2TMuFlMo/viewform?edit_requested=true
Information: http://localhost:8000/index.php/reproducibility/