skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Lessons Learned from the Chameleon Testbed
The Chameleon testbed is a case study in adapting the cloud paradigm for computer science research. In this paper, we explain how this adaptation was achieved, evaluate it from the perspective of supporting the most experiments for the most users, and make a case that utilizing mainstream technology in research testbeds can increase efficiency without compro- mising on functionality. We also highlight the opportunity inherent in the shared digital artifacts generated by testbeds and give an overview of the efforts we’ve made to develop it to foster reproducibility.  more » « less
Award ID(s):
1743358
PAR ID:
10195652
Author(s) / Creator(s):
; ; ; ; ; ; ; ; ; ; ; ; ; ;
Date Published:
Journal Name:
Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC '20)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. FABRIC is a unique national research infrastructure to enable cutting-edge andexploratory research at-scale in networking, cybersecurity, distributed computing andstorage systems, machine learning, and science applications. It is an everywhere-programmable nationwide instrument comprised of novel extensible network elementsequipped with large amounts of compute and storage, interconnected by high speed,dedicated optical links. It will connect a number of specialized testbeds for cloudresearch (NSF Cloud testbeds CloudLab and Chameleon), for research beyond 5Gtechnologies (Platforms for Advanced Wireless Research or PAWR), as well as productionhigh-performance computing facilities and science instruments to create a rich fabric fora wide variety of experimental activities. 
    more » « less
  2. Cyber-physical system (CPS) testbeds are critical research tools for advancing safety-critical technologies, from autonomous vehicles to smart grids. As CPS grow more complex, adaptive, and interconnected, testbeds must evolve in kind—yet the architectural assumptions guiding their design remain fragmented across domains. This paper introduces a morphological framework that enables comparison and synthesis across traditionally siloed CPS fields. Drawing on a structured review and a feature abstraction process, we derive 25 architectural dimensions and use them to classify 113 CPS testbeds spanning a wide range of use cases. Cluster analysis reveals three dominant design archetypes characterized by centralized control, reconfigurability, and virtualized operation, and highlights convergence in user interaction mechanisms and the decision-making architectures that govern testbed behavior. We quantify configurational rigidity, expose underexplored regions of the design space, and identify architectural opportunities in emerging CPS research domains. By reconceiving testbeds as design artifacts, this work opens new avenues for architectural innovation in CPS testbed research. 
    more » « less
  3. Given the highly empirical nature of research in cloud computing, networked systems, and related fields, testbeds play an important role in the research ecosystem. In this paper, we cover one such facility, CloudLab, which supports systems research by providing raw access to programmable hardware, enabling research at large scales, and creating as hared platform for repeatable research.We present our experiences designing CloudLab and operating it for four years, serving nearly 4,000 users who have run over 79,000 experiments on 2,250 servers, switches, and other pieces of datacenter equipment. From this experience,we draw lessons organized around two themes. The first set comes from analysis of data regarding the use of CloudLab:how users interact with it, what they use it for, and the implications for facility design and operation. Our second set of lessons comes from looking at the ways that algorithms used“under the hood,” such as resource allocation, have important—and sometimes unexpected—effects on user experience and behavior. These lessons can be of value to the designers and operators of IaaS facilities in general, systems testbeds in particular, and users who have a stake in understanding how these systems are built. 
    more » « less
  4. Middleware is required to support and interface multi-modal Dynamic Data Driven Application Systems (DDDAS) with back-end and other computing facilities. Middleware is also needed to support distributed simulations and emulations needed in earlier phases of system development. This work describes the Green Runtime Infrastructure (G-RTI), an energy-efficient client server based middleware developed to support distributed DDDAS simulation, emulation and deployment. G-RTI eases and accelerates the development and testing of multi-modal studies, testbeds and DDDAS systems. It serves as a platform for research in energy reduction techniques for middleware services. The services implemented by G-RTI are described and results of benchmarking studies are reported. Its application is demonstrated through a use-case for an end-to-end implementation of a connected vehicle application. G-RTI is open source. 
    more » « less
  5. Virtual testbeds are fundamental to the success of research on cognitive work in safety-critical domains. A testbed that can meet researchers' objectives and create a sense of reality for participants positively impacts the research process; they have the potential to allow researchers to address questions not achievable in physical environments. This paper discusses the development of a synthetic task environment (STE) for Urban Search and Rescue (USAR) to advance the boundaries of Human-Robot Teams (HRTs) using Roblox. Virtual testbeds can simulate USAR task environments and HRT interactions. After assessing alternative STE platforms, we discovered Roblox not only met our research capabilities but also would prove invaluable for research teams without substantial coding experience. This paper outlines the design process of creating an STE to meet our research team's objectives. 
    more » « less