More Like this

1. Longitudinal Deep Kernel Gaussian Process Regression

   Liang, Junjie; Wu, Yanting; Xu, Dongxuan; Honavar, Vasant (January 2021, Proceedings of the AAAI Conference on Artificial Intelligence)

   null (Ed.)

   Gaussian processes offer an attractive framework for predictive modeling from longitudinal data, i.e., irregularly sampled, sparse observations from a set of individuals over time. However, such methods have two key shortcomings: (i) They rely on ad hoc heuristics or expensive trial and error to choose the effective kernels, and (ii) They fail to handle multilevel correlation structure in the data. We introduce Longitudinal deep kernel Gaussian process regression (L-DKGPR) to overcome these limitations by fully automating the discovery of complex multilevel correlation structure from longitudinal data. Specifically, L-DKGPR eliminates the need for ad hoc heuristics or trial and error using a novel adaptation of deep kernel learning that combines the expressive power of deep neural networks with the flexibility of non-parametric kernel methods. L-DKGPR effectively learns the multilevel correlation with a novel additive kernel that simultaneously accommodates both time-varying and the time-invariant effects. We derive an efficient algorithm to train L-DKGPR using latent space inducing points and variational inference. Results of extensive experiments on several benchmark data sets demonstrate that L-DKGPR significantly outperforms the state-of-the-art longitudinal data analysis (LDA) methods. 

   more » « less
2. Conditional Deep Gaussian Processes: Empirical Bayes Hyperdata Learning

   https://doi.org/10.3390/e23111387  

   Lu, Chi-Ken; Shafto, Patrick (November 2021, Entropy)

   null (Ed.)

   It is desirable to combine the expressive power of deep learning with Gaussian Process (GP) in one expressive Bayesian learning model. Deep kernel learning showed success as a deep network used for feature extraction. Then, a GP was used as the function model. Recently, it was suggested that, albeit training with marginal likelihood, the deterministic nature of a feature extractor might lead to overfitting, and replacement with a Bayesian network seemed to cure it. Here, we propose the conditional deep Gaussian process (DGP) in which the intermediate GPs in hierarchical composition are supported by the hyperdata and the exposed GP remains zero mean. Motivated by the inducing points in sparse GP, the hyperdata also play the role of function supports, but are hyperparameters rather than random variables. It follows our previous moment matching approach to approximate the marginal prior for conditional DGP with a GP carrying an effective kernel. Thus, as in empirical Bayes, the hyperdata are learned by optimizing the approximate marginal likelihood which implicitly depends on the hyperdata via the kernel. We show the equivalence with the deep kernel learning in the limit of dense hyperdata in latent space. However, the conditional DGP and the corresponding approximate inference enjoy the benefit of being more Bayesian than deep kernel learning. Preliminary extrapolation results demonstrate expressive power from the depth of hierarchy by exploiting the exact covariance and hyperdata learning, in comparison with GP kernel composition, DGP variational inference and deep kernel learning. We also address the non-Gaussian aspect of our model as well as way of upgrading to a full Bayes inference. 

   more » « less
3. RoeNet: Predicting discontinuity of hyperbolic systems from continuous data

   https://doi.org/10.1002/nme.7406  

   Tong, Yunjin; Xiong, Shiying; He, Xingzhe; Yang, Shuqi; Wang, Zhecheng; Tao, Rui; Liu, Runze; Zhu, Bo (December 2023, International Journal for Numerical Methods in Engineering)

   Abstract Predicting future discontinuous phenomena that are unobservable from training data sets has long been a challenging problem in scientific machine learning. We introduce a novel paradigm to predict the emergence and evolution of various discontinuities of hyperbolic partial differential equations (PDEs) based on given training data over a short window with limited discontinuity information. Our method is inspired by the classical Roe solver [P. L. Roe, J Comput Phys., vol. 43, 1981], a basic tool for simulating various hyperbolic PDEs in computational physics. By carefully designing the computing primitives, the data flow, and the novel pseudoinverse processing module, we enable our data‐driven predictor to satisfy all the essential mathematical criteria of a Roe solver and hence deliver accurate predictions of hyperbolic PDEs. We demonstrate through various examples that our data‐driven Roe predictor outperforms original human‐designed Roe solver and deep neural networks with weak priors in terms of accuracy and robustness. 

   more » « less
4. A Kernel Multiple Change-point Algorithm via Model Selection

   Arlot, Sylvain; Celisse, Alain; Harchaoui, Zaid (January 2019, Journal of machine learning research)

   We consider a general formulation of the multiple change-point problem, in which the data is assumed to belong to a set equipped with a positive semidefinite kernel. We propose a model-selection penalty allowing to select the number of change points in Harchaoui and Cappe's kernel-based change-point detection method. The model-selection penalty generalizes non-asymptotic model-selection penalties for the change-in-mean problem with univariate data. We prove a non-asymptotic oracle inequality for the resulting kernel-based change-point detection method, whatever the unknown number of change points, thanks to a concentration result for Hilbert-space valued random variables which may be of independent interest. Experiments on synthetic and real data illustrate the proposed method, demonstrating its ability to detect subtle changes in the distribution of data. 

   more » « less
5. Spatiotemporal modeling and prediction in cellular networks: A big data enabled deep learning approach

   https://doi.org/10.1109/INFOCOM.2017.8057090  

   Wang, Jing; Tang, Jian; Xu, Zhiyuan; Wang, Yanzhi; Xue, Guoliang; Zhang, Xing; Yang, Dejun (May 2017, IEEE INFOCOM2017)

   In this paper, we propose to leverage the emerging deep learning techniques for spatiotemporal modeling and prediction in cellular networks, based on big system data. First, we perform a preliminary analysis for a big dataset from China Mobile, and use traffic load as an example to show non-zero temporal autocorrelation and non-zero spatial correlation among neighboring Base Stations (BSs), which motivate us to discover both temporal and spatial dependencies in our study. Then we present a hybrid deep learning model for spatiotemporal prediction, which includes a novel autoencoder-based deep model for spatial modeling and Long Short-Term Memory units (LSTMs) for temporal modeling. The autoencoder-based model consists of a Global Stacked AutoEncoder (GSAE) and multiple Local SAEs (LSAEs), which can offer good representations for input data, reduced model size, and support for parallel and application-aware training. Moreover, we present a new algorithm for training the proposed spatial model. We conducted extensive experiments to evaluate the performance of the proposed model using the China Mobile dataset. The results show that the proposed deep model significantly improves prediction accuracy compared to two commonly used baseline methods, ARIMA and SVR. We also present some results to justify effectiveness of the autoencoder-based spatial model. 

   more » « less