We present a scalable, cloud-based science platform solution designed to enable next-to-the-data analyses of terabyte-scale astronomical tabular data sets. The presented platform is built on Amazon Web Services (over Kubernetes and S3 abstraction layers), utilizes Apache Spark and the Astronomy eXtensions for Spark for parallel data analysis and manipulation, and provides the familiar JupyterHub web-accessible front end for user access. We outline the architecture of the analysis platform, provide implementation details and rationale for (and against) technology choices, verify scalability through strong and weak scaling tests, and demonstrate usability through an example science analysis of data from the Zwicky Transient Facility’s 1Bn+ light-curve catalog. Furthermore, we show how this system enables an end user to iteratively build analyses (in Python) that transparently scale processing with no need for end-user interaction. The system is designed to be deployable by astronomers with moderate cloud engineering knowledge, or (ideally) IT groups. Over the past 3 yr, it has been utilized to build science platforms for the DiRAC Institute, the ZTF partnership, the LSST Solar System Science Collaboration, and the LSST Interdisciplinary Network for Collaboration and Computing, as well as for numerous short-term events (with over 100 simultaneous users). In a live demo instance, the deployment scripts, source code, and cost calculators are accessible.4
Accelerating the design and development of new advanced materials is one of the priorities in modern materials science. These efforts are critically dependent on the development of comprehensive materials cyberinfrastructures which enable efficient data storage, management, sharing, and collaboration as well as integration of computational tools that help establish processing–structure–property relationships. In this contribution, we present implementation of such computational tools into a cloud-based platform called BisQue (Kvilekval et al., Bioinformatics 26(4):554, 2010). We first describe the current state of BisQue as an open-source platform for multidisciplinary research in the cloud and its potential for 3D materials science. We then demonstrate how new computational tools, primarily aimed at processing–structure–property relationships, can be implemented into the system. Specifically, in this work, we develop a module for BisQue that enables microstructure-sensitive predictions of effective yield strength of two-phase materials. Towards this end, we present an implementation of a computationally efficient data-driven model into the BisQue platform. The new module is made available online (web address:
- Award ID(s):
- 1664172
- NSF-PAR ID:
- 10224257
- Publisher / Repository:
- Springer Science + Business Media
- Date Published:
- Journal Name:
- Integrating Materials and Manufacturing Innovation
- Volume:
- 8
- Issue:
- 1
- ISSN:
- 2193-9764
- Page Range / eLocation ID:
- p. 52-65
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract http://hub.astronomycommons.org/ -
more » « less
Abstract The availability and easy access of large-scale experimental and computational materials data have enabled the emergence of accelerated development of algorithms and models for materials property prediction, structure prediction, and generative design of materials. However, the lack of user-friendly materials informatics web servers has severely constrained the wide adoption of such tools in the daily practice of materials screening, tinkering, and design space exploration by materials scientists. Herein we first survey current materials informatics web apps and then propose and develop MaterialsAtlas.org, a web-based materials informatics toolbox for materials discovery, which includes a variety of routinely needed tools for exploratory materials discovery, including material’s composition and structure validity check (e.g. charge neutrality, electronegativity balance, dynamic stability, Pauling rules), materials property prediction (e.g. band gap, elastic moduli, hardness, and thermal conductivity), search for hypothetical materials, and utility tools. These user-friendly tools can be freely accessed at
http://www.materialsatlas.org . We argue that such materials informatics apps should be widely developed by the community to speed up materials discovery processes. -
The application of Materials Informatics to polymer nanocomposites would result in faster development and commercial implementation of these promising materials, particularly in applications requiring a unique combination of properties. This chapter focuses on a new data resource for nanocomposites — NanoMine — and the tools, models, and algorithms that support data-driven materials design. The chapter begins with a brief introduction to NanoMine, including the data structure and tools available. Critical to the ability to design nanocomposites, however, is developing robust structure–property–processing (s–p–p) relationships. Central to this development is the choice of appropriate microstructure characterization and reconstruction (MCR) techniques that capture a complex morphology and ultimately build statistically equivalent reconstructed composites for accurate modeling of properties. A wide range of MCR techniques is reviewed followed by an introduction of feature selection and feature extraction techniques to identify the most significant microstructure features for dimension reduction. This is then followed by examples of using a descriptor-based representation to create processing–structure (p–s) and structure–property (s–p) relationships for use in design. To overcome the difficulty in modeling the interphase region surrounding nanofillers, an adaptive sampling approach is presented to inversely determine the inter-phase properties based on both FEM simulations and physical experiment data of bulk properties. Finally, a case study for nanodielectrics in a capacitor is introduced to demonstrate the integration of the p–s and s–p relationships to develop optimized materials for achieving multiple desired properties.more » « less
-
Proteins and nucleic acids participate in essentially every biochemical process in living organisms, and the elucidation of their structure and motions is essential for our understanding how these molecular machines perform their function. Nuclear Magnetic Resonance (NMR) spectroscopy is a powerful versatile technique that provides critical information on the molecular structure and dynamics. Spin-relaxation data are used to determine the overall rotational diffusion and local motions of biological macromolecules, while residual dipolar couplings (RDCs) reveal local and long-range structural architecture of these molecules and their complexes. This information allows researchers to refine structures of proteins and nucleic acids and provides restraints for molecular docking. Several software packages have been developed by NMR researchers in order to tackle the complicated experimental data analysis and structure modeling. However, many of them are offline packages or command-line applications that require users to set up the run time environment and also to possess certain programming skills, which inevitably limits accessibility of this software to a broad scientific community. Here we present new science gateways designed for NMR/structural biology community that address these current limitations in NMR data analysis. Using the GenApp technology for scientific gateways (https://genapp.rocks), we successfully transformed ROTDIF and ALTENS, two offline packages for bio-NMR data analysis, into science gateways that provide advanced computational functionalities, cloud-based data management, and interactive 2D and 3D plotting and visualizations. Furthermore, these gateways are integrated with molecular structure visualization tools (Jmol) and with gateways/engines (SASSIE-web) capable of generating huge computer-simulated structural ensembles of proteins and nucleic acids. This enables researchers to seamlessly incorporate conformational ensembles into the analysis in order to adequately take into account structural heterogeneity and dynamic nature of biological macromolecules. ROTDIF-web offers a versatile set of integrated modules/tools for determining and predicting molecular rotational diffusion tensors and model-free characterization of bond dynamics in biomacromolecules and for docking of molecular complexes driven by the information extracted from NMR relaxation data. ALTENS allows characterization of the molecular alignment under anisotropic conditions, which enables researchers to obtain accurate local and long-range bond-vector restraints for refining 3-D structures of macromolecules and their complexes. We will describe our experience bringing our programs into GenApp and illustrate the use of these gateways for specific examples of protein systems of high biological significance. We expect these gateways to be useful to structural biologists and biophysicists as well as NMR community and to stimulate other researchers to share their scientific software in a similar way.more » « less
-
Data-driven methods have attracted increasingly more attention in materials research since the advent of the material genome initiative. The combination of materials science with computer science, statistics, and data-driven methods aims to expediate materials research and applications and can utilize both new and archived research data. In this paper, we present a data driven and deep learning approach that builds a portion of the structure–property relationship for polymer nanocomposites. Analysis of archived experimental data motivates development of a computational model which allows demonstration of the approach and gives flexibility to sufficiently explore a wide range of structures. Taking advantage of microstructure reconstruction methods and finite element simulations, we first explore qualitative relationships between microstructure descriptors and mechanical properties, resulting in new findings regarding the interplay of interphase, volume fraction and dispersion. Then we present a novel deep learning approach that combines convolutional neural networks with multi-task learning for building quantitative correlations between microstructures and property values. The performance of the model is compared with other state-of-the-art strategies including two-point statistics and structure descriptor-based approaches. Lastly, the interpretation of the deep learning model is investigated to show that the model is able to capture physical understandings while learning.more » « less
