1. Introduction. Materials informatics has fundamentally changed materials science research (e.g., Himanen et al. (2019)). To design or select a material for a particular function, researchers have traditionally relied on intuition and costly experimentation. This process is now supplemented by machine learning to predict a candidate material's properties and triage materials for further experimentation. The addition of machine learning has been shown to improve efficiency, especially when using multiple data sources (e.g., Batra (2021)). We aim to build a predictive model for a material's band gap, defined as the energy differential between the lowest-unoccupied and highest-occupied electronic states (Kittel, McEuen and Wiley (2019)). The band gap governs desirable properties that are useful in industrial sectors such as electric and photovoltaic conductivity. Traditionally, band gap size is computed using methods in quantum mechanics such as density functional theory (DFT; Kohn and Sham (1965)). However, these methods require running costly computer simulations, and so it is not feasible to exhaustively search over a broad class of materials. Our objective is to build a statistical model (sometimes called a meta-model, surrogate model, or emulator; O 'Hagan (2006)) along with an accompanying transfer learning methodology that is both effective at predicting the output of the DFT simulation and is able to optimize experimentation on future test data sets from related data sources.
Deep neural network (DNN) architectures have emerged as leading models in materials informatics. Beyond materials science DNNs have revolutionized the field of machine learning with significant breakthroughs in a wide variety of applications including computer vision (Voulodimos et al. (2018)), natural language processing (Young et al. (2018)), and protein folding (Senior et al. (2020)); see Dargan et al. (2020) for further references to modern applications. Their ability to model complex nonlinear processes enables them to handle a large class of prediction problems. Training a neural network, however, is not an easy task, often requiring intensive computational cost, large quantities of training data, and careful hyperparameter selection (Bengio (2012)). As an overparameterized expressive model, DNNs are prone to overfitting, especially in the case of small data sets. Tan et al. (2018) conclude that the number of parameters in a DNN and the size of the data required for good generalization performance have an almost linear relationship. Since these models tend to have thousands (sometimes millions) of parameters, the size of the data required becomes quickly prohibitive. This is where transfer learning plays a central role.
Broadly defined, transfer learning describes a machine learning approach for augmenting the training of a learning task on a target population data set with a learning algorithm that has already been trained on a closely related data set from a source population, particularly for scenarios where the target and source populations are not identical; see Weiss, Khoshgoftaar and Wang (2016) for a recent survey on transfer learning. By developing effective transfer learning strategies, it is possible to reduce the computational burden and the need for large training data sets for training DNN models in applications where complex DNNs have already been trained for similar data sets. For example, Raghu et al. (2019) and Ahishakiye et al. (2021) investigated the use of a pretrained DNN on ImageNet (Deng et al. (2009)) to improve the accuracy in medical imaging tasks where labeled data is usually scarce. Although the potential benefits for transfer learning are significant, there are no generally accepted procedures for how to construct or evaluate a transfer-learning strategy, and this is especially true from the Bayesian perspective. Dube et al. (2020) provides a review of the current approaches to address a variety of important questions of concern for developing transfer-learning methods. For instance, what aspects of the source tasks can be leveraged to improve performance on the target task? In the case of DNN source models, this commonly boils down to isolating the layers that are task agnostic versus those that are task dependent.
1.1. Data and motivation. Motivated by the predictive materials science application of estimating the band gap of a material based on its structural properties, we propose a principled statistical approach to transfer learning within a Bayesian framework. In materials science, the band gap governs desirable properties of candidate materials. The conductivity of a solid is closely related to the band gap size. For example, materials with large band gaps are generally insulators, while materials with low band gaps are semiconductors (Van de Walle ( 2012)). Band gap size (measured in electronvolts, eV) is typically computed using different quantum mechanics methods like the DFT. DFT calculation can be expensive for large molecules and crystals. This, combined with the large possible number of materials, has driven the need for fast screening methods. According to Olsthoorn et al. (2019), there has been a growing interest in the development of accurate machine learning models for material sciences and quantum systems. For small organic molecules, Olsthoorn et al. (2019) reported that DNN models have reached chemical accuracy. For large molecular crystals, however, the task of band gap prediction seems to be much harder. In addition, DFT calculations are more expensive to perform on molecular crystals. For example, Ma et al. (2021) found that the algorithmic complexity of DFT is cubic in the number of particles for a given molecule; hence, it is easier to collect and compute band gap values for molecules than it is for molecular crystals. Given the difficulty of band gap calculation for molecular crystals, we propose to use transfer-learning methods to borrow information from molecules (source) to molecular crystals (target) to improve prediction. The target data considered here will consist of 50,000 molecular crystals and their band gap values computed using DFT. This data set was provided by the Material Science Research Group at Carnegie Mellon University and required a sustained effort over several years to produce. The source data will be the OE62 data set (Stuke et al. (2019)), which consists of 62,000 molecules and their band gap values. The molecules in the source data were all extracted from organic crystals, and the band gap values were also computed using DFT.
1.2. Literature review. Bayesian neural networks have attracted considerable attention in recent years as a result of computational advances (e.g., Wilson (2020)). One of the earliest works on Bayesian transfer learning in DNNs was Wohlert et al. (2018), where a single source task is considered and a DNN with feed forward layers is trained using mean field variational Bayes (VB; Zhang et al. (2019)). The approximate posterior learned on the source task is used as a prior on the parameters for the target task. Wohlert et al. (2018) did not, however, address the problem of freezing or training the transferred layers and surprisingly gave worse performance than training using only the target data. Chandra and Kapoor (2020) introduced a new method for Bayesian multisource transfer learning where they proposed a new Markov chain Monte Carlo (MCMC) sampling scheme with parameter proposals for the target task based on proposals for all the source tasks. At first, all source task parameters are sampled, and then a single source for transfer of knowledge is used for the proposals for the target task. This method requires storing all source tasks data in memory together with the target task and does not fall under the two-stage inductive transfer-learning framework considered in our paper. Bueno et al. (2019) used Bayesian transfer learning for volcanoseismic prediction and uncertainty quantification using variational inference. Yang, Jiao and Sun (2020) focused on Bayesian hyperparameter optimization for transfer learning. Zhou et al. (2020) proposed a new form of regularization that can be viewed as a Gaussian prior on the network weights where the Fisher information from the source task is used as the precision matrix. More generally, Bayesian transfer learning for models other than DNNs has been considered by other articles in the literature, most commonly attributable to the fact that it is natural to borrow strength across data sets via prior distributions (e.g., Karbalayghareh, Qian and Dougherty (2018)).
The Bayesian transfer-learning methodology that we develop sequentially trains a DNN on a target data set by first specifying a prior distribution concentrated on parameter estimates for the DNN trained from a source data set. While this approach is sensible, the primary challenge is that constructing prior densities from pretrained DNN layers is not trivial because individual parameters are not identifiable nor do they have inherent meaning. Accordingly, to solve this issue we extend the fundamental notion of penalized complexity priors (PCPs) from Simpson et al. (2017) to the deep-learning setting and specify our transfer-learning prior distribution in terms of the Kullback-Leibler divergence between the source and target model. This prior is constructed with the intuition that the predictive model is likely shared (to some extent) across tasks but is flexible enough to disregard the source task if appropriate.
The target data set in our materials science application is small, and so we implement a fully Bayesian analysis with computations via MCMC sampling strategies. However, since the source data set is large, a fully Bayesian analysis is prohibitive, and so we analyze the source data using both optimization and variational-inference (VI; Blundell et al. (2015)) methods. Both methods provide point estimates for each parameter, but VI also provides (approximate) uncertainty quantification. Therefore, in addition to applying to cases where data sets from both the target and the source are available, our proposed method can be applied using source data DNNs trained by other users, such as ImageNet (Deng et al. ( 2009)). In our exposition we develop our transfer-learning methods using the optimization-based source model and the VI-based source model in parallel, and we show that both provide improvements over traditional transfer-learning methods for both synthetic and real data.
The remainder of the paper proceeds as follows. Section 2 reviews background material on DNNs and transfer learning. Section 3 introduces the proposed PCP priors for transfer learning in DNNs. The proposed methods are then evaluated using simulated and real data in Sections 4 and 5, respectively. Section 6 concludes. The code for reproducing our empirical results is available at https://github.ncsu.edu/mabba/Pybayes.git.
2.1. Deep-learning model. We begin by briefly reviewing the DNN model; for a more comprehesive review, see Goodfellow, Bengio and Courville (2016). In general, DNNs are comprised of successive layers, starting with an input layer, followed by one or more hidden layers, and finally an output layer. Each layer contains a number of nodes that are connected to the next layer through weights that control the impact of a given layer on the next. For concreteness we describe the Bayesian transfer-learning model in the simple case of a continuous response and fully-connected feed forward neural network. However, the ideas proposed in this paper translate to other structures, including recurrent (Medsker and Jain (2001)) and convolutional (LeCun, Bengio et al. (1995)) networks (see Section 6 for further discussion).
A dense feed forward layer is a function l : R d -→ R k characterized by a weight matrix W ∈ R k×d , a bias vector b ∈ R k , and a real-valued activation function a applied elementwise to a vector argument. For x ∈ R d , the dense layer applies an affine transformation to x followed by the activation function, l(x) = a(W x + b). A DNN with L hidden layers first applies an input layer, typically l 0 (x) = x, then L -1 hidden layers, and one output layer; each layer is assigned unique weights, biases, and (potentially) activation functions; that is, l i (x) = a i (W i x + b i ) for layer i ∈ {1, . . . , L}. The composition of these L layers defines a function f from the input space X to the output space Y,
where θ := {θ 1 , . . . , θ L } and θ i := {b i , W i } are the parameters for layer i. For a regression task, we consider f (x; θ) to be the mean response given x, whereas for a classification task, f (x; θ) represents a vector of probabilities.
Before discussing transfer learning for DNNs, we review general definitions and concepts of transfer learning. We adopt notation from Pan and Yang (2009) and Weiss, Khoshgoftaar and Wang (2016) whose early work has been widely adopted in the literature. A domain is composed of two parts, a feature space X and a sampling distribution p(X), where X denotes a set of instances X = {x i |x i ∈ X , i = 1, . . . , n}. Accordingly, D = {X , p}. Next, a task (e.g., classification or prediction) consists of a label space Y and a decision function f ; that is, T = {Y, f }.
Transfer learning considers two domains: the source and the target. The main objective is to perform the target task in the target domain, and the analysis are supplemented by the related data from the source task (or multiple source tasks, e.g., Maurer, Pontil and Romera-Paredes (2016)). For a source domain and task, denoted D S and T S , respectively, an observed data set is the collection {(x i , y i )|x i ∈ X S , y i ∈ Y S , 1 ≤ i ≤ n S }, and f S is understood as a conditional distribution or decision function for the instances y i , given
as the domain, task, and data for the target domain. DNNs typically require inductive transfer learning where the feature space is the same across tasks (Pan and Yang (2009)), i.e., X S = X T . Therefore, we will drop the subscript on the feature space and simply use X . Note that this does not imply that the conditional decision functions or the sampling distributions of the features are the same for the source and target.
As discussed in Section 2.1, DNNs are composed of many layers, each performing a transformation of the data. These transformations can be viewed as learned representations. It has been argued that different layers may learn different concepts relating to the task (Dube et al. (2020)). Specifically, there is evidence that early layers learn general representations of the data, while the the deeper layers are more task specific and learn specialized representations. In Zeiler and Fergus (2013), the authors studied the activation of early layers of DNNs in computer vision problems, where convolutional layers are typically used, and noted that early convolutional layers extract highlevel features with the potential to generalize to different image domains. Using this logic, it follows that transfer learning for DNNs amounts to determining how many early layers to share across tasks and how many to assign as task specific.
As defined in (1), in DNNs the tasks are defined by the weight parameters for each layer of the network. Assuming the same network architecture for both tasks, define θ S = {θ S1 , . . . , θ SL } and θ T = {θ T 1 , . . . , θ T L } as the parameters for the source and target tasks, respectively. A transfer-learning strategy with DNNs can be defined as the process of learning simultaneously the parameters θ S to augment the learning of θ T on the target task. The transferred layers can either be held fixed on the target task (referred to as freezing) or finetuned. Recently, Dube et al. (2020) investigated the empirical performances of freezing and fine-tuning. In their work they focused on a computer vision task with pretrained layers from different architectures that were fit to the Imagenet data set. The conclusion was that freezing pretrained layers, although not optimal, was always better than random initialization. The best performance was obtained when the learning rate for the pretrained layers was significantly lower than the rate of the other layers in the model. They advocate for a factor of 10% as a default choice.
For the target task, let θ T ≡ θ = {θ 1 , . . . , θ L }, and denote v i as the number of parameters in θ i ; for the source task, denote θ = { θ 1 , . . . , θL } as the estimated values for θ that are trained from the source data, and when available, let i be the covariance matrix for θ i (e.g., the posterior covariance, given the source data). Our goal is to leverage these estimated parameters to construct an informative prior for the target task. If the tasks were the same, the natural solution would be to set θ = θ . On the other hand, if the tasks were not at all related, θ does not encode any information for the target task and would probably be worse than random initialization. In between these two extreme cases lies the core of transfer learning, where there is a reason to assume that tasks are related but we do not know to which degree.
The focus of our analysis is a judicious choice of the prior distribution for θ . Constructing a prior distribution that uses θ requires, first, the choice of a method for computing θ from the source data, and second, requires specification of the prior distribution for θ , given θ . As described in the next subsection, we use the PCP of Simpson et al. (2017) for the prior distribution of θ , given θ . The remainder of this section provides an overview the PCP idea of Simpson et al. (2017) and then describes two methods we propose for constructing θ . The PCP is derived for both methods of constructing θ .
3.1. Penalized complexity priors. In the framework of Simpson et al. (2017), the problem of prior choice is considered from a model complexity perspective. The prior on the parameter of interest θ is controlled by a flexibility parameter τ . The base (or simplest) model for θ , which we will denote by p 0 (θ ), corresponds to the case where τ = 0, and for larger values of τ , the prior deviates from the base model. Following Occam's razor, simple models should be preferred until there is evidence for more complex ones; in other words, the hyperprior on τ should favor the base model. PCPs, as defined in Simpson et al. (2017), give a principled way of choosing a prior on τ that controls the deviation from the base model by penalizing a scaled version of the Kullback-Liebler divergence from p 0 (θ ) to p(θ|τ ),
(2)
and notice that h(τ ) = 0 if and only if τ = 0. Furthermore, if h(•) is strictly increasing, penalising h(τ ) for large values induces a prior that penalizes large values of τ and, consequently, puts more mass close to the base model. Simpson et al. (2017) recommend using an exponential distribution, h(τ ) ∼ Exp(λ) to ensure a mode at τ = 0 and a constant rate of penalization for larger τ . Let
where N is the multivariate Gaussian density function, g is an increasing function, the flexibility parameter τ i controls the deviation from the base model in layer i, and τ = (τ 1 , . . . , τ L ).
The case τ = 0 is used to define the base model. This layerwise specification reflects the fact that some layers can be more transferable than others and hence require more penalization.
If we marginalize over τ , the prior model becomes a scale mixture of Gaussian distributions, where the mixing distribution will be the prior on τ . This latter distribution will be implicitly defined by placing an exponential distribution on h(τ ) = d KL (p(θ T |τ )||p 0 (θ T )). When L > 1, the implicit prior on τ is not identifiable since the mapping h : R L + -→ R + cannot be bijective and hence is not invertible. To circumvent this issue, Simpson et al. (2017) extend the PCP prior to a multivariate parameter τ by having the prior on τ uniform on each level set of h(•). Instead, we opt for the simpler approach of constraining τ to the (L -1)simplex given by any weighted combination of its elements. This is equivalent to τ = τ c, where c ∼ Dirichlet(1 L ) and τ ∈ R + . In this formulation τ becomes a global flexibility parameter, and the weights c i control how much deviation each layer is allowed since τ i = τ c i . Thus, given the vector of weights c, we have a one-to-one mapping h c ( τ ) through which we have an explicit penalising prior on the global parameter τ .
If the source data are analyzed using VB methods, then θ i is an approximate posterior mean and i is an approximate v i × v i posterior covariance matrix. Because standard VB methods approximate the posterior as independent across parameters, i is a diagonal matrix for each i. We incorporate this information in the prior by setting g(τ ) = 1 + τ so that the base model with τ = 0 uses the posterior of the source data directly as the prior for the target data, which would be optimal Bayesian learning if the two tasks are the same. If τ i > 0, then the prior variance increases to reflect uncertainty about the relationship between source and target tasks. Under this prior distribution, the KL divergence is
, the one-to-one mapping between d and τ is
where φ(x) = xlog(1 + x). Furthermore, φ is continuously differentiable and strictly increasing; hence, given c, we have a unique prior on τ corresponding to the desired prior on the KL divergence from the base model. The induced prior is 4), coupled with the Dirichlet prior on the weight c, completely specifies a prior distribution on the vector of scales τ . Using numerical inversion of the mapping in (3), sampling from this prior is straightforward.
It can be shown that the induced prior on the global flexibility parameter in (4) has a mode at zero with a strictly decreasing density, as τ increases. Also, the tail of the prior behaves like a modified Weibull distribution (Almalki and Nadarajah (2014)) with rate λ v i c i and shape 0.5. Figure 1 plots the density when λ = 1.
Assume that a point estimate θ is derived from the source data but no measure of uncertainty is provided. In this case the source data does not define a base distribution for θ , and thus we cannot compute the KL divergence between base and full models for θ without further assumptions. Since DNNs are primarily used for prediction, we place a prior on the KL divergence between the predictive model under the source and target models. Let f (x|θ ) denote the target DNN output for a given input x and parameter values (θ). Let the regression model likelihood be p(y) ∼ Normal(f (x|θ), σ 2 ) and θ i ∼ Normal( θ i , τ i I), which induces a prior distribution on the function f . Unfortunately, for the DNN model it is not available in analytical form. To overcome this, we approximate the DNN output using a first-order Taylor approximation around θ
For small values of τ (i.e., the most relevant values for the prior, assuming the target and source are similar), a first-order approximation is arguably tight enough. Furthermore, if the activation functions used in the model are piecewise linear, like the widely used ReLU function (Agarap (2018)), the output function f (x|θ ) will also be piecewise linear, and we cannot go beyond a first-order Taylor expansion.
Based on the approximation, for the base model with τ = 0, we get the prior distribution
and for the flexible model, the prior on f (y| τ , c) is
where α(x) = ( ∇ θ i f (x| θ , ) 2 ) 1≤i≤L . Now that we have two normal models, an analytic expression for the KL distance is
In equation ( 6) α(x) depends on the gradient of f (x| θ ) for a given input point x. Assuming that the inputs are identically distributed and we have N target data points, the marginal KL divergence can be approximated by
Using (7), we have a one-to-one mapping between d = √ 2KL(p||p 0 ) and the global flexibility parameter τ . Hence, an exponential distribution on d induces a prior on τ , which can be obtained with the same simple change of variables as in (4). Let
The form of the prior in ( 8) is similar to (4): in both cases the prior induced on τ has mode at zero, is a strictly decreasing density as τ increases, and has the same tail behavior.
We evaluate the approximation used in (5) to verify that the derived prior on τ will result in an approximately exponential distribution on the KL divergence between the two models. We assume L = 3 layers with {64, 32, 1} nodes, respectively. The source-data estimates θ i are generated following the Glorot initialization scheme (Glorot and Bengio (2010)). Since we cannot compute the KL divergence analytically, the distribution of KL divergence is approximated with Monte Carlo sampling. The approximation is evaluated for 50 different target data sets generated, as described in Section 4. For each of those data sets, we compute the gradients in (5) and first draw c and τ from their priors for different values of the rate parameter λ, then sample θ i |τ ∼ Normal( θ i , τ i I). We then sample 20,000 replications of the model output f (x; θ ) and approximate the KL divergence from the base model f (x; θ ) for each replication to get the empirical distribution of the KL divergence. Figure 2 plots the cumulative hazard function of the scaled KL divergence d, which should be linear with slope λ if the distribution is Exponential(λ). The approximation holds near the origin for all λ and is tighter for the entire distribution for larger values of λ.
and the target data are generated, for constant k 2 , as
We set p = 30 covariates, so x i ∈ R p , ρ = 0.5, σ = 1, k 1 = 15, and k 2 = k 1 + k with k ∈ {0, 5, 10, 15}, and we generate 1000 synthetic source training observations with a varying number of target training observations n ∈ {30, 50, 70}. The constant c is set so that the signal-to-noise ratio is 4:1 for the target data (i.e., so the variance of cμ 2 (x i ) is 4). If k = 0, then the two data-generating processes are identical, and if k is large, they are dissimilar. For each scenario we simulate 500 data sets and compare predictions on a test set of size n test = 200.
Both the source and target data are analyzed using the fully-connected model with four hidden layers having widths 24, 16, 12, and 8, respectively. The source data are fit using the optimization scheme provided in Section 2 of the Supplementary Material (Abba et al. (2023)); this produces estimates θ and their variances in the case of VI. We then fit several models to each target data set. The first group are non-Bayesian optimization-based transferlearning "TL" methods. We consider two methods that fix some layers using the source fit and tune the remaining layers using the target data. The first model "TL1" shares no layers and is fit to the target data with no connection to the source data. The second model "TL2" proceeds in two stages; first, the output layer only is trained on the target data while the others are frozen, and next, in the second stage all layers are fine tuned.
The second group of methods are Bayesian. The first Bayesian model "BNN" ignores the source data by setting the prior θ i |τ ∼ Normal(0, τ i I) and τ i ∼ InvGamma(2, 1) so that the marginal standard deviation is 1. Although there is no consensus on the default prior choice in Bayesian deep learning, priors with zero mean and unit standard deviation are the most common (Fortuin (2022)). The second method, "BTL-PCP-PE," fits the Bayesian transfer-learning model with the priors centered at point estimates θ alone, as described in Section 3.3. The third model, "Bayes-PCP-VI," uses the PCP prior based on VI, as described in Section 3.2. For both PCP methods, we assume c ∼ Dirichlet(1 L ) and set λ so that the correlation between f (x i ; θ) and Y i is 0.5. All Bayesian method are fit using MCMC, as described in Section 3 of the Supplementary Material (Abba et al. (2023)).
The results are compared using MSE, and for the Bayesian methods, the coverage and width of 95% prediction intervals for μ 2 (x i ) over a target data test set of 200 observations is provided. The results are reported in Table 1. BLT-PCP-PE outperforms TL1 and TL2 in all cases, perhaps due to the added stability of prior distributions. When the source and target are similar (i.e., k ∈ {0, 5}), all the transfer-learning methods work well, especially if the number of training samples is small. When the discrepancy between the source and target domain is high, however, the transfer-learning methods do not improve the performance of the models. In fact, the point estimate method "TL-1" outperforms "TL-2," and the Bayesian methods are comparable. An interesting result is the poor performance of the PCP prior for the VI case compared to the other Bayesian methods. It seems that building a PCP prior, based on the mean-field VI posterior approximation, does not encode enough information for the model to learn from small data sets. However, while VI does not perform as well as MCMC, it is much faster than MCMC and still outperforms TL1 and TL2 in most cases and is thus a viable option for large data sets. The average computational time for each method is reprted in Table 2. Furthermore, when k is large, "BNN" outperforms all the other methods. This is expected since the prior distribution in the PCP has a sharp mode at zero which results in a strong penalizing effect. Thus, when the data sets are dissimilar and the number of target training samples is small, BNN performs better. Perhaps choosing a different distribution for the prior, with heavier tails, on the scaled Kullback-Leibler divergence could reduce this dependence on the source in such cases. That being said, this could also lead to a less sharp mode at the origin and hence less concentration around the true solution when the data sets are similar.
To mimic the the real data analysis in Section 5, we conduct a second set of simulations with the same generative model as above but with p = 1000 covariates, ρ = 0.5, σ = 0.32, and the constant c = 1 to approximate the 3:1 signal-to-noise in the data analysis. The number of active covariates is determined by k 1 = 900 and k 2 = k 1 + k with k ∈ {0, 100}; if k = 0, then the two data-generating processes are identical, and if k is large, they are similar but not identical. We generate 10,000 synthetic source training observations and n = 500 target training observations. For each scenario we simulate 100 data sets and compare predictions on a test set of size n test = 10,000. For these larger data sets, we also consider the VI approach of Zhou et al. (2020). The results in Table 3 resemble those in Table 1; that is, the BTL-PCP-PE approach has the smallest MSE and coverage near the nominal level.
TABLE 1 Simulation study results: The non-Bayesian methods ("TL1-TL2") use transfer learning by fixing different numbers of layers of the network using source data, the Bayesian methods are the Bayesian model without transfer learning ("BNN") or full Bayesian transfer-learning model with point estimate ("BTL-PCP-PE") and VI ("BTL-PCP-VI") in the first stage. Table (a) gives the mean square prediction error (Monte-Carlo standard errors) over the 500 replicated data sets by the difference between the number of active covariates over tasks, k, and size of the target data set, n.
We apply the proposed transfer-learning methods for band gap prediction of the molecular crystal target data set. For training we consider only subsets of the data with cardinality 1000, 2500, or 5000 samples to represent a typical data set size, a validation set of 10,000 samples, and finally a testing set of 20,000 samples. Since we are dealing with molecules, we first need to preprocess the data and compute descriptors x that can be fed into the DNN. We use the MBTR descriptor method (Huo and Rupp (2018)) to compute p = 1260 descriptors for every data point in both the source and target data sets. These descriptors are functions of the structure of the material, for example, the types and configuration of its atoms. The architecture of the model will have two hidden layers with 128 and 64 neurons, respectively. First, the model is trained on the source data with 40,000 samples for training set and 10,000 for the validation. The parameters of the best performing model in terms of mean squared error (MSE) are saved and transferred to the target task as θ .
On the target task, we consider four different transfer-learning methods. The non-Bayesian methods differ in the treatment of the hidden layers of the model; we can either freeze the hidden layers and treat them as feature extractors and only fit the output layer to the target task "TL-freeze," or we can fine-tune the hidden layers along with the output layer on the target task "TL-fine-tune." For both Bayesian methods, the priors are θ i |τ ∼ Normal( θ i , τ i I). The two methods differ in the priors on τ = (τ 1 , τ 2 , τ 3 ); we compare the PCP with point estimate approach "BTL-PCP-PE" from Section 3.3 with c ∼ Dirichlet(1, 1, 1) to the uninformative prior approach with τ i ∼ InvGamma(2, 1) "BTL-BNN." Since the point estimate approach outperformed the VI approach in the simulation study, we consider only the point estimate approach in this data analysis. To verify that transfer learning provides benefit over using only the target data, we compare to the state-of-the-art DNN model used in materials science where DNNs have been developed as end-to-end models in the sense that they take as input the molecular representation and learn their own embedding. Two of the leading methods are SHNET (Schütt et al. (2017)) and MEGNET (Chen et al. (2019)).
The target data are split into groups of size 5000, 10,000, and 20,000 for training so that each group contains materials of different classes and the band gap distribution stays roughly the same. We train the model for samples of size n ∈ {1000, 2500, 5000}. Figure 3 plots the MSE on a test set of size 20,000 for each method and training set, and Figure 4 plots the observed and predicted values for the BTL-PCP-PE method. As expected, MSE decreases for each method as the size training set increases, and in all cases the BTL-PCP-PE method gives the smallest MSE.
With a fully Bayesian treatment of all the parameters, we can look at the summary statistics of c, the parameter that controls which layers contain the useful information from the source data and which are irrelevant. For training set size 5000, the posterior means (standard deviation) for c 1 , c 2 , and c 3 are 0.05 (0.02), 0.05 (0.03), and 0.90 (0.15), respectively. Therefore, the first two layers are shrunk toward the source data fits while the output layer varies from the source-model fit. The Bayesian methods also provide prediction intervals. The empirical coverage and average width of 95% prediction intervals for the test set are, respectively, 0.91 and 1.24 for BTL-BNN and 0.94 and 1.16 for BTL-PCP-PE.
These results are competitive with state-of-the-art methods in the materials community. For a training set of size 5000 (the same 5000 as in Figure 3), the MSE for SHNET and MEGNET are 0.48 eV 2 and 1.01 eV 2 , respectively. In fact, using the same data except with a much larger training set of 30,000 observations, the methods achieved MSE of 0.59 eV 2 for SHNET and 0.34 eV 2 for MEGNET.
For molecular crystals the computational cost of band gap values can be orders of magnitude higher in comparison to organic molecules. Recently, DNN models reached chemical accuracy levels for band gap prediction on small organic molecules and can now be reliably used as a fast screening method. However, these models often require large data sets for training and can have unstable performance on small data sets. Using transfer learning, we leverage the accuracy of neural networks models on organic molecules to build a data efficient model with accurate predictions and calibrated uncertainty quantification for molecular crystals band gap prediction. Our new transfer-learning methods provide regularization by centering the prior distribution on estimates from an auxiliary source data set, that is, organic molecules. We use the PCP framework to ensure that the prior concentrates around the source model but is allowed to deviate if appropriate. We develop two methods for cases where the source data analysis does and does not provide uncertainty measures for the parameter estimates. We show via simulation that the proposed methods reduce prediction error, compared to standard transfer-learning methods, and unlike standard methods, the Bayesian approach gives reasonable coverage for prediction intervals.
The accuracy of our method for large molecular crystals matches state-of-the-art models while only requiring a sixth of the training data. The performance on both simulated and real data sets indicate that the methods can be used to guide and optimize experimentation and selection of materials with specific properties without the need for a large training sample, hence reducing the cost associated with DFT calculations. The potential scope of the application of the developed transfer-learning methods goes beyond material science and includes areas of applied statistical learning where data labelling is costly and auxiliary source tasks are available.
There are many areas of future work. We have developed our method in the simplest case of a continuous response and feed forward network. The prior based on VI in Section 3.2 would apply directly to other networks and response distributions, as the prior is only a function of the approximate Gaussian posterior distribution from the source data analysis. Similarly, the prior based on point estimates in Section 3.3 can be applied without modification to non-Gaussian responses and richer architecture. Changing the network architecture would change the form of the gradients in (5), but the method itself can be used without modification. The method also applies for noncontinuous responses if we view f as a process that spans the real line that is related to the response distribution. For example, for binary data, f in (5) could be the logistic function of the success probability, which spans the real line, and thus the proposed methods could be applied on this scale. It may also be possible to extend the PCP prior from neural networks to other frameworks such as tree-based methods (e.g., Chen and Guestrin (2016), Chipman, George and McCulloch (2010)).
Next, our method applies an informative prior on the overall variance parameter but a uniform prior on the distribution of the variance across levels, c. If prior information about the transferability of different layers is available, it could be incorporated in the prior for c. For example, it may be reasonable to assume the prior mean of c i increases from the input to the output layer. Lastly, another interesting area of future work is to compare the two-stage analysis that sequentially analyzes the source then target data with a simultaneous analysis of all data sources. Here we have assumed that building a hierarchical model for all data sources is computationally prohibitive, but it would be interesting to compare the efficiency of these two approaches when possible, especially when the source data set is small and thus the parameter estimates are uncertain.