Search for: All records

Deep learning models have achieved remarkable accuracy for structural response modeling. However, these models heavily depend on having a sufficient amount of training data, which can be challenging and time-consuming to collect. Moreover, data-driven models sometimes struggle to adhere to physics constraints. Therefore, in this study, a physics-informed long short-term memory (PI-LSTM) network was applied to structural response modeling by incorporating physics constraints into deep learning. The physics constraints were modified to accommodate the characteristics of both linear and nonlinear cases. The PI-LSTM network, inspired by and compared with existing physics-informed deep learning models (PhyCNN and PhyLSTM), was validated using the numerical simulation results of the single-degree-of-freedom (SDOF) system and the experimental results of the six-story building. Additionally, the PI-LSTM network underwent thorough investigation and validation across the four cases of the SDOF system and numerical simulation results of the six-story building with the comparison of the regular LSTM. The results indicate that the PI-LSTM network outperformed the regular LSTM models in terms of accuracy. Furthermore, the PI-LSTM network exhibited a more concentrated and higher accuracy range when analyzing the results of both the SDOF system and the six-story building. These findings demonstrate that the PI-LSTM network presents a reliable and efficient approach for structural response modeling. 

Abstract El Niño–Southern Oscillation (ENSO) exhibits highly asymmetric temporal evolutions between its warm and cold phases. While El Niño events usually terminate rapidly after their mature phase and show an already established transition into the cold phase by the following summer, many La Niña events tend to persist throughout the second year and even reintensify in the ensuing winter. While many mechanisms were proposed, no consensus has been reached yet and the essential physical processes responsible for the multiyear behavior of La Niña remain to be illustrated. Here, we show that a unique ocean physical process operates during multiyear La Niña events. It is characterized by rapid double reversals of zonal ocean current anomalies in the equatorial Pacific and exhibits a fairly regular near-annual periodicity. Mixed-layer heat budget analyses reveal comparable contributions of the thermocline and zonal advective feedbacks to the SST anomaly growth in the first year of multiyear La Niña events; however, the zonal advective feedback plays a dominant role in the reintensification of La Niña events. Furthermore, the unique ocean process is identified to be closely associated with the preconditioning heat content state in the central to eastern equatorial Pacific before the first year of La Niña, which has been shown in previous studies to play an active role in setting the stage for the future reintensification of La Niña. Despite systematic underestimation, the above oceanic process can be broadly reproduced by state-of-the-art climate models, providing a potential additional source of predictability for the multiyear La Niña events. 

Deep transfer learning (TL) has great potential for a wide range of applications in civil engineering. This work aims to propose a deep transfer learning-based method for vehicle classification by asphalt pavement vibration. This work first used the pavement vibration IoT monitoring system to collect raw vibration signals and performed the wavelet transform to obtain denoised vibration signals. The vibration signals were then represented in two different ways, including the time-domain graph and the time-frequency graph. Finally, two deep transfer learning-based methods, namely Method Ⅰ (Time-domain & TL) and Method Ⅱ (Time-frequency & TL), were applied for vehicle classification according to the two different representations of vibration signals. The results show that the CNN model had a satisfactory performance in both methods with the accuracy of Method Ⅰ exceeding 0.94 and Method Ⅱ exceeding 0.95. The CNN model in Method Ⅱ performed better in the accuracy metrics with considering label imbalance, but worse in the accuracy metrics without considering label imbalance than that in Method Ⅰ. The differences between these two methods have been investigated and discussed in detail in terms of input types, accuracy metrics, and application prospects. The CNN model with deep transfer learning could be an effective, accurate, and reliable technique for vehicle classification based on asphalt pavement vibration. 

Abstract Objective . Neural decoding is an important tool in neural engineering and neural data analysis. Of various machine learning algorithms adopted for neural decoding, the recently introduced deep learning is promising to excel. Therefore, we sought to apply deep learning to decode movement trajectories from the activity of motor cortical neurons. Approach . In this paper, we assessed the performance of deep learning methods in three different decoding schemes, concurrent, time-delay, and spatiotemporal. In the concurrent decoding scheme where the input to the network is the neural activity coincidental to the movement, deep learning networks including artificial neural network (ANN) and long-short term memory (LSTM) were applied to decode movement and compared with traditional machine learning algorithms. Both ANN and LSTM were further evaluated in the time-delay decoding scheme in which temporal delays are allowed between neural signals and movements. Lastly, in the spatiotemporal decoding scheme, we trained convolutional neural network (CNN) to extract movement information from images representing the spatial arrangement of neurons, their activity, and connectomes (i.e. the relative strengths of connectivity between neurons) and combined CNN and ANN to develop a hybrid spatiotemporal network. To reveal the input features of the CNN in the hybrid network that deep learning discovered for movement decoding, we performed a sensitivity analysis and identified specific regions in the spatial domain. Main results . Deep learning networks (ANN and LSTM) outperformed traditional machine learning algorithms in the concurrent decoding scheme. The results of ANN and LSTM in the time-delay decoding scheme showed that including neural data from time points preceding movement enabled decoders to perform more robustly when the temporal relationship between the neural activity and movement dynamically changes over time. In the spatiotemporal decoding scheme, the hybrid spatiotemporal network containing the concurrent ANN decoder outperformed single-network concurrent decoders. Significance . Taken together, our study demonstrates that deep learning could become a robust and effective method for the neural decoding of behavior. 

Deep neural networks are often overparameterized and may not easily achieve model generalization. Adversarial training has shown effectiveness in improving generalization by regularizing the change of loss on top of adversarially chosen perturbations. The recently proposed sharpness-aware minimization (SAM) algorithm conducts adversarial weight perturbation, encouraging the model to converge to a flat minima. SAM finds a common adversarial weight perturbation per-batch. Although per-instance adversarial weight perturbations are stronger adversaries and can potentially lead to better generalization performance, their computational cost is very high and thus it is impossible to use per-instance perturbations efficiently in SAM. In this paper, we tackle this efficiency bottleneck and propose sharpness-aware minimization with dynamic reweighting (delta-SAM). Our theoretical analysis motivates that it is possible to approach the stronger, per-instance adversarial weight perturbations using reweighted per-batch weight perturbations. delta-SAM dynamically reweights perturbation within each batch according to the theoretically principled weighting factors, serving as a good approximation to per-instance perturbation. Experiments on various natural language understanding tasks demonstrate the effectiveness of delta-SAM. 

Abstract Many previous studies have shown that an Indian Ocean basin warming (IOBW) occurs usually during El Niño–Southern Oscillation (ENSO) decaying spring to summer seasons through modifying the equatorial zonal circulation. Decadal modulation associated with the interdecadal Pacific oscillation (IPO) is further investigated here to understand the nonstationary ENSO–IOBW relationship during ENSO decaying summer (July–September). During the positive IPO phase, significant warm sea surface temperature (SST) anomalies are observed over the tropical Indian Ocean in El Niño decaying summers and vice versa for La Niña events, while these patterns are not well detected in the negative IPO phase. Different decaying speeds of ENSO associated with the IPO phase, largely controlled by both zonal advective and thermocline feedbacks, are suggested to be mainly responsible for these different ENSO–IOBW relationships. In contrast to ENSO events in the negative IPO phase, the ones in the positive IPO phase display a slower decaying speed and delay their transitions both from a warm to a cold state and a cold to a warm state. The slower decay of El Niño and La Niña thereby helps to sustain the teleconnection forcing over the equatorial Indian Ocean and corresponding SST anomalies there can persist into summer. This IPO modulation of the ENSO–IOBW relationship carries important implications for the seasonal prediction of the Indian Ocean SST anomalies and associated summer climate anomalies. 

Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model’s penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.