skip to main content


Title: Compositionally restricted attention-based network for materials property predictions
Abstract In this paper, we demonstrate an application of the Transformer self-attention mechanism in the context of materials science. Our network, the Compositionally Restricted Attention-Based network (), explores the area of structure-agnostic materials property predictions when only a chemical formula is provided. Our results show that ’s performance matches or exceeds current best-practice methods on nearly all of 28 total benchmark datasets. We also demonstrate how ’s architecture lends itself towards model interpretability by showing different visualization approaches that are made possible by its design. We feel confident that and its attention-based framework will be of keen interest to future materials informatics researchers.  more » « less
Award ID(s):
1651668
NSF-PAR ID:
10248518
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
npj Computational Materials
Volume:
7
Issue:
1
ISSN:
2057-3960
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Conventional lithium-ion batteries are unable to meet the increasing demands for high-energy storage systems, because of their limited theoretical capacity. 1 In recent years, intensive attention has been paid to enhancing battery energy storage capability to satisfy the increasing energy demand in modern society and reduce the average energy capacity cost. Among the candidates for next generation high energy storage systems, the lithium sulfur battery is especially attractive because of its high theoretical specific energy (around 2600 W h kg-1) and potential cost reduction. In addition, sulfur is a cost effective and environmentally friendly material due to its abundance and low-toxicity. 2 Despite all of these advantages, the practical application of lithium sulfur batteries to date has been hindered by a series of obstacles, including low active material loading, poor cycle life, and sluggish sulfur conversion kinetics. 3 Achieving high mass loading cathode in the traditional 2D planar thick electrode has been challenged. The high distorsion of the traditional planar thick electrodes for ion/electron transfer leads to the limited utilization of active materials and high resistance, which eventually results in restricted energy density and accelerated electrode failure. 4 Furthermore, of the electrolyte to pores in the cathode and utilization ratio of active materials. Catalysts such as MnO 2 and Co dopants were employed to accelerate the sulfur conversion reaction during the charge and discharge process. 5 However, catalysts based on transition metals suffer from poor electronic conductivity. Other catalysts such as transition metal dopants are also limited due to the increased process complexities. . In addition, the severe shuttle effects in Li-S batteries may lead to fast failures of the battery. Constructing a protection layer on the separator for limiting the transmission of soluble polysulfides is considered an effective way to eliminate the shuttle phenomenon. However, the soluble sulfides still can largely dissolve around the cathode side causing the sluggish reaction condition for sulfur conversion. 5 To mitigate the issues above, herein we demonstrate a novel sulfur electrode design strategy enabled by additive manufacturing and oxidative vapor deposition (oCVD). Specifically, the electrode is strategically designed into a hierarchal hollow structure via stereolithography technique to increase sulfur usage. The active material concentration loaded to the battery cathode is controlled precisely during 3D printing by adjusting the number of printed layers. Owing to its freedom in geometry and structure, the suggested design is expected to improve the Li ions and electron transport rate considerably, and hence, the battery power density. The printed cathode is sintered at 700 °C at N 2 atmosphere to achieve carbonization of the cathode during which intrinsic carbon defects (e.g., pentagon carbon) as catalytic defect sites are in-situ generated on the cathode. The intrinsic carbon defects equipped with adequate electronic conductivity. The sintered 3D cathode is then transferred to the oCVD chamber for depositing a thin PEDOT layer as a protection layer to restrict dissolutions of sulfur compounds in the cathode. Density functional theory calculation reveals the electronic state variance between the structures with and without defects, the structure with defects demonstrates the higher kinetic condition for sulfur conversion. To further identify the favorable reaction dynamic process, the in-situ XRD is used to characterize the transformation between soluble and insoluble polysulfides, which is the main barrier in the charge and discharge process of Li-S batteries. The results show the oCVD coated 3D printed sulfur cathode exhibits a much higher kinetic process for sulfur conversion, which benefits from the highly tailored hierarchal hollow structure and the defects engineering on the cathode. Further, the oCVD coated 3D printed sulfur cathode also demonstrates higher stability during long cycling enabled by the oCVD PEDOT protection layer, which is verified by an absorption energy calculation of polysulfides at PEDOT. Such modeling and analysis help to elucidate the fundamental mechanisms that govern cathode performance and degradation in Li-S batteries. The current study also provides design strategies for the sulfur cathode as well as selection approaches to novel battery systems. References: Bhargav, A., (2020). Lithium-Sulfur Batteries: Attaining the Critical Metrics. Joule 4 , 285-291. Chung, S.-H., (2018). Progress on the Critical Parameters for Lithium–Sulfur Batteries to be Practically Viable. Advanced Functional Materials 28 , 1801188. Peng, H.-J.,(2017). Review on High-Loading and High-Energy Lithium–Sulfur Batteries. Advanced Energy Materials 7 , 1700260. Chu, T., (2021). 3D printing‐enabled advanced electrode architecture design. Carbon Energy 3 , 424-439. Shi, Z., (2021). Defect Engineering for Expediting Li–S Chemistry: Strategies, Mechanisms, and Perspectives. Advanced Energy Materials 11 . Figure 1 
    more » « less
  2. Abstract—Materials Genomics initiative has the goal of rapidly synthesizing materials with a given set of desired properties using data science techniques. An important step in this direction is the ability to predict the outcomes of complex chemical reactions. Some graph-based feature learning algorithms have been proposed recently. However, the comprehensive relationship between atoms or structures is not learned properly and not explainable, and multiple graphs cannot be handled. In this paper, chemical reaction processes are formulated as translation processes. Both atoms and edges are mapped to vectors represent- ing the structural information. We employ the graph convolution layers to learn meaningful information of atom graphs, and further employ its variations, message passing networks (MPNN) and edge attention graph convolution network (EAGCN) to learn edge representations. Particularly, multi-view EAGCN groups and maps edges to a set of representations for the properties of the chemical bond between atoms from multiple views. Each bond is viewed from its atom type, bond type, distance and neighbor environment. The final node and edge representations are mapped to a sequence defined by the SMILES of the molecule and then fed to a decoder model with attention. To make full usage of multi-view information, we propose multi-view attention model to handle self correlation inside each atom or edge, and mutual correlation between edges and atoms, both of which are important in chemical reaction processes. We have evaluated our method on the standard benchmark datasets (that have been used by all the prior works), and the results show that edge embedding with multi-view attention achieves superior accuracy compared to existing techniques. 
    more » « less
  3. The major focus of artificial intelligence (AI) research is made on biomimetic synaptic processes that are mimicked by functional memory devices in the computer industry [1]. It is urgent to find a memory technology for suiting with Brain-Inspired Computing to break the von Neumann bottleneck which limits the efficiency of conventional computer architectures [2]. Silicon-based flash memory, which currently dominates the market for data storage devices, is facing challenging issues to meet the needs of future data storage device development due to the limitations, such as high-power consumption, high operation voltage, and low retention capacity [1]. The emerging resistive random-access memory (RRAM) has elicited intense research as its simple sandwiched structure, including top electrode (TE) layer, bottom electrode (BE) layer, and an intermediate resistive switching (RS) layer, can store data using RS phenomenon between the high resistance state (HRS) and the low resistance state (LRS). This class of emerging devices is expected to outperform conventional memory devices [3]. Specifically, the advantages of RRAM include low-voltage operation, short programming time, great cyclic stability, and good scalability [4]. Among the materials for RS layer, indium gallium zinc oxide (IGZO) has attracted attention because of its abundance and high atomic diffusion property of oxygen atoms, transparency, and its easily modulated electrical properties by controlling the stoichiometric ratio of indium and gallium as well as oxygen potential in the sputter gas [5, 6]. Moreover, since the IGZO can be applied to both the thin-film transistor (TFT) channel and RS layer, the IGZO-based fully integrated transparent electronics are very promising [5]. In this work, we proposed transparent IGZO-based RRAMs. First, we chose ITO to serve as both TE and BE to achieve high transmittance in the visible regime of light. All three layers (TE, RS, BE layers) were deposited using a multi-target magnetron sputtering system on glass substrates to demonstrate fully transparent oxide-based devices. I-V characteristics were evaluated by a semiconductor parameter analyzer, and our devices showed typical butterfly curves indicating the bipolar RS property. And the IGZO-based RRAM can survive more than 50 continuous sweeping cycles. The optical transmission analysis was carried out via an UV-Vis spectrometer and the average transmittance around 80% out of entire devices in the visible-light wavelength range, implying high transparency. To investigate the thickness dependence on the properties of RS layer, 50nm, 100nm and 150nm RS layer of IGZO RRAM were fabricated. Also, the oxygen partial pressure during the sputtering of IGZO was varied to optimize the property because the oxygen vacancy concentration governs the RS and RRAM performance. Electrode selection is crucial and can impact the performance of the whole device [7]. Thus, Cu TE was chosen for our second type of device because the diffusion of Cu ions can be beneficial for the formation the conductive filament (CF). Finally, a ~5 nm SiO2 barrier layer was employed between TE and RS layers to confine the diffusion of Cu into the RS layer. At the same time, this SiO2 inserting layer can provide an additional interfacial series resistance in the device to lower the off current, consequently, improve the on/off ratio and whole performance. In conclusion, the transparent IGZO-based RRAMs were established. To tune the property of RS layer, the thickness layer and sputtering conditions of RS were adjusted. In order to engineer the diffusion capability of the TE material to the RS layer and the BE, a set of TE materials and a barrier layer were integrated in IGZO-based RRAM and the performance was compared. Our encouraging results clearly demonstrate that IGZO is a promising material in RRAM applications and overcoming the bottleneck of current memory technologies. 
    more » « less
  4. Borenstein, Elhanan (Ed.)
    Recurrent neural networks with memory and attention mechanisms are widely used in natural language processing because they can capture short and long term sequential information for diverse tasks. We propose an integrated deep learning model for microbial DNA sequence data, which exploits convolutional neural networks, recurrent neural networks, and attention mechanisms to predict taxonomic classifications and sample-associated attributes, such as the relationship between the microbiome and host phenotype, on the read/sequence level. In this paper, we develop this novel deep learning approach and evaluate its application to amplicon sequences. We apply our approach to short DNA reads and full sequences of 16S ribosomal RNA (rRNA) marker genes, which identify the heterogeneity of a microbial community sample. We demonstrate that our implementation of a novel attention-based deep network architecture, Read2Pheno , achieves read-level phenotypic prediction. Training Read2Pheno models will encode sequences (reads) into dense, meaningful representations: learned embedded vectors output from the intermediate layer of the network model, which can provide biological insight when visualized. The attention layer of Read2Pheno models can also automatically identify nucleotide regions in reads/sequences which are particularly informative for classification. As such, this novel approach can avoid pre/post-processing and manual interpretation required with conventional approaches to microbiome sequence classification. We further show, as proof-of-concept, that aggregating read-level information can robustly predict microbial community properties, host phenotype, and taxonomic classification, with performance at least comparable to conventional approaches. An implementation of the attention-based deep learning network is available at https://github.com/EESI/sequence_attention (a python package) and https://github.com/EESI/seq2att (a command line tool). 
    more » « less
  5. null (Ed.)
    Point set is a major type of 3D structure representation format characterized by its data availability and compactness. Most former deep learning-based point set models pay equal attention to different point set regions and channels, thus having limited ability in focusing on small regions and specific channels that are important for characterizing the object of interest. In this paper, we introduce a novel model named Attention-based Point Network (AttPNet). It uses attention mechanism for both global feature masking and channel weighting to focus on characteristic regions and channels. There are two branches in our model. The first branch calculates an attention mask for every point. The second branch uses convolution layers to abstract global features from point sets, where channel attention block is adapted to focus on important channels. Evaluations on the ModelNet40 benchmark dataset show that our model outperforms the existing best model in classification tasks by 0.7% without voting. In addition, experiments on augmented data demonstrate that our model is robust to rotational perturbations and missing points. We also design a Electron Cryo-Tomography (ECT) point cloud dataset and further demonstrate our model’s ability in dealing with fine-grained structures on the ECT dataset. 
    more » « less