Computational science today depends on complex, data-intensive applications operating on datasets from a variety of scientific instruments. A major challenge is the integration of data into the scientist's workflow. Recent advances in dynamic, networked cloud resources provide the building blocks to construct reconfigurable, end-to-end infrastructure that can increase scientific productivity. However, applications have not adequately taken advantage of these advanced capabilities. In this work, we have developed a novel network-centric platform that enables high-performance, adaptive data flows and coordinated access to distributed cloud resources and data repositories for atmospheric scientists. We demonstrate the effectiveness of our approach by evaluating time-critical, adaptive weather sensing workflows, which utilize advanced networked infrastructure to ingest live weather data from radars and compute data products used for timely response to weather events. The workflows are orchestrated by the Pegasus workflow management system and were chosen because of their diverse resource requirements. We show that our approach results in timely processing of Nowcast workflows under different infrastructure configurations and network conditions. We also show how workflow task clustering choices affect throughput of an ensemble of Nowcast workflows with improved turnaround times. Additionally, we find that using our network-centric platform powered by advanced layer2 networking techniques results in faster, more reliable data throughput, makes cloud resources easier to provision, and the workflows easier to configure for operational use and automation.
more »
« less
BRACELET: Edge-Cloud Microservice Infrastructure for Aging Scientific Instruments
Recent advances in cyber-infrastructure have enabled
digital data sharing and ubiquitous network connectivity
between scientific instruments and cloud-based storage infrastructure
for uploading, storing, curating, and correlating of large
amounts of materials and semiconductor fabrication data and
metadata. However, there is still a significant number of scientific
instruments running on old operating systems that are taken
offline and cannot connect to the cloud infrastructure, due to
security and network performance concerns. In this paper, we
propose BRACELET - an edge-cloud infrastructure that augments
the existing cloud-based infrastructure with edge devices and
helps to tackle the unique performance & security challenges that
scientific instruments face when they are connected to the cloud
through public network. With BRACELET, we put a networked
edge device, called cloudlet, in between the scientific instruments
and the cloud as the middle tier of a three-tier hierarchy. The
cloudlet will shape and protect the data traffic from scientific
instruments to the cloud, and will play a foundational role in
keeping the instruments connected throughout its lifetime, and
continuously providing the otherwise missing performance and
security features for the instrument as its operating system ages.
more »
« less
- Award ID(s):
- 1659293
- NSF-PAR ID:
- 10222900
- Date Published:
- Journal Name:
- IEEE International Conference on Computing, Networking, and Communications
- Page Range / eLocation ID:
- 692 to 696
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Current scientific experiments frequently involve control of specialized instruments (e.g., scanning electron mi- croscopes), image data collection from those instruments, and transfer of the data for processing at simulation centers. This process requires a “human-in-the-loop” to perform those tasks manually, which besides requiring a lot of effort and time, could lead to inconsistencies or errors. Thus, it is essential to have an automated system capable of performing remote instrumentation to intelligently control and collect data from the scientific instruments. In this paper, we propose a Remote Instrumentation Science Environment (RISE) for intelligent im- age analytics that provides the infrastructure to securely capture images, determine process parameters via machine learning, and provide experimental control actions via automation, under the premise of “human-on-the-loop”. The machine learning in RISE aids an iterative discovery process to assist researchers to tune instrument settings to improve the outcomes of experiments. Driven by two scientific use cases of image analytics pipelines, one in material science, and another in biomedical science, we show how RISE automation leverages a cutting-edge integration of cloud computing, on-premise HPC cluster, and a Python programming interface available on a microscope. Using web services, we implement RISE to perform automated image data collection/analysis guided by an intelligent agent to provide real-time feedback control of the microscope using the image analytics outputs. Our evaluation results show the benefits of RISE for researchers to obtain higher image analytics accuracy, save precious time in manually controlling the microscopes, while reducing errors in operating the instruments.more » « less
-
The adoption of big data analytics in healthcare applications is overwhelming not only because of the huge volume of data being analyzed, but also because of the heterogeneity and sensitivity of the data. Effective and efficient analysis and visualization of secure patient health records are needed to e.g., find new trends in disease management, determining risk factors for diseases, and personalized medicine. In this paper, we propose a novel community cloud architecture to help clinicians and researchers to have easy/increased accessibility to data sets from multiple sources, while also ensuring security compliance of data providers is not compromised. Our cloud-based system design configuration with cloudlet principles ensures application performance has high-speed processing, and data analytics is sufficiently scalable while adhering to security standards (e.g., HIPAA, NIST). Through a case study, we show how our community cloud architecture can be implemented along with best practices in an ophthalmology case study which includes health big data (i.e., Health Facts database, I2B2, Millennium) hosted in a campus cloud infrastructure featuring virtual desktop thin-clients and relevant Data Classification Levels in storage.more » « less
-
Mobile Edge Computing (MEC) creates new infrastructure at the edges of the mobile networks, thus providing transformative opportunities for applications seeking latency benefits by operating closer to end-users and devices. However, the reduced network distance between the application endpoints of the MEC flows causes pattern shifts in the packet bursts exchanged at the network edges. The longer and denser bursts create a new source of contention that is not considered by current solutions. As a result, naively collocating applications onto the MEC tier can negatively affect latency-critical workloads, resulting in up to 73% packets experiencing as much as 3.8x increased latency. This makes it impossible to support latency-centric SLOs in MEC, obviating its expected benefits from MEC. This paper is the first to describe this new contention point in mobile networks and its potentially crippling impact on the achievable latency benefit from MEC. We propose ShapeShifter, a new component in the MEC architecture which solves the MEC latency contention problem through adaptive latency-centric burst management of MEC flows. ShapeShifter is effective - it eliminates SLO violations for latency-critical applications and improves application performance in multi-tenant scenarios by up to 3.8 x – and practical – it can be deployed with minimal disruption to the current mobile network ecosystem.more » « less
-
As many robot automation applications increasingly rely on multi-core processing or deep-learning models, cloud computing is becoming an attractive and economically viable resource for systems that do not contain high computing power onboard. Despite its immense computing capacity, it is often underused by the robotics and automation community due to lack of expertise in cloud computing and cloud-based infrastructure. Fog Robotics balances computing and data between cloud edge devices. We propose a software framework, FogROS, as an extension of the Robot Operating System (ROS), the de-facto standard for creating robot automation applications and components. It allows researchers to deploy components of their software to the cloud with minimal effort, and correspondingly gain access to additional computing cores, GPUs, FPGAs, and TPUs, as well as predeployed software made available by other researchers. FogROS allows a researcher to specify which components of their software will be deployed to the cloud and to what type of computing hardware. We evaluate FogROS on 3 examples: (1) simultaneous localization and mapping (ORB-SLAM2), (2) Dexterity Network (Dex-Net) GPU-based grasp planning, and (3) multi-core motion planning using a 96-core cloud-based server. In all three examples, a component is deployed to the cloud and accelerated with a small change in system launch configuration, while incurring additional latency of 1.2 s, 0.6 s, and 0.5 s due to network communication, the computation speed is improved by 2.6x, 6.0x and 34.2x, respectively.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/ICCNC.2019.8685528
