is a prominent strategy to address a machine learning task of interest using information and parameters already learned and/or available for a related task. Such designs significantly aid training of overparameterized models like deep neural networks (e.g., [4,24,16]), which are inherently challenging due to the vast number of parameters compared to the number of training data examples. There are various ways to integrate the previously-learned information from the source task in the learning process of the target task; often this is done by taking subsets of parameters (e.g., layers in neural networks) learned for the source task and plugging them in the target task model as parameter subsets that can be set fixed, finely tuned, or serve as non-random initialization for a thorough learning process. Obviously, transfer learning is useful only if the source and target tasks are sufficiently related with respect to the transfer mechanism utilized, e.g., [23,29,13]. Moreover, finding a successful transfer learning setting for deep neural networks was shown in [22] to be a delicate engineering task. The importance of transfer learning in contemporary practice should motivate fundamental understanding of its main aspects via analytical frameworks that may consider linear structures (see, e.g., [14]).
In general, the impressive success of overparameterized architectures for supervised learning have raised fundamental questions on the classical role of the bias-variance tradeoff that guided the traditional designs towards seemingly-optimal underparameterized models [5]. Em-pirical studies from recent years [25,9,2] have demonstrated the phenomenon that overparameterized supervised learning corresponds to a generalization error curve with a double descent trend (with respect to the number of parameters in the learned model). This double descent shape means that the generalization error peaks when the learned model starts to interpolate the training data (i.e., to achieve zero training error), but then the error continuously decreases as the overparmeterization increases, often arriving to a global minimum that outperforms the best underparameterized solution. This phenomenon has been studied theoretically from the linear regression perspective in an extensive series of papers, e.g., in [3,11,28,17,1,19]. The next stage is to provide corresponding fundamental understanding to learning problems beyond a single fully-supervised regression problem (see, for example, the study in [7] on overparameterized linear subspace fitting in unsupervised and semi-supervised settings).
In this paper we study the fundamentals of the natural meeting point between overparameterized models and the transfer learning concept. Our analytical framework is based on the least squares solutions to two related linear regression problems: the first is a source task whose solution has been found independently, and the second is a target task that is addressed using the solution already available for the source task. Specifically, the target task is carried out while keeping a subset of its parameters fixed to values transferred from the source task solution. Accordingly, the target task includes three types of parameters: free to-be-optimized parameters, transferred parameters set fixed to values from the source task, and parameters fixed to zeros (which in our case correspond to the elimination of input features). The mixture of the parameter types defines the parameterization level (i.e., the relation between the number of free parameters and the number of examples given) and the transfer-learning level (i.e., the portion of transferred parameters among the solution layout).
We conduct a non-asymptotic statistical analysis of the generalization errors in this transfer learning structure where the minimum 2 -norm solutions are used when the source and/or target tasks are overparameterized. Clearly, since the source task is solved independently, its generalization error follows a regular (one-dimensional) double descent shape with respect to the number of examples and free parameters available in the source task. Hence, our main contribution and interest are in the characterization of the generalization error of the target task that is carried out using the transfer learning approach described above. We show that the generalization error of the target task follows a double descent trend that depends on the double descent shape of the source task and on the transfer learning factors such as the number of parameters transferred and the relation between the source and target tasks. We also examine the generalization error of the target task as a function of two quantities: the number of free parameters in the source task and the number of free parameters in the target task. This interpretation presents the generalization error of the target task as having a two-dimensional double descent trend.
We also show how the generalization error of the target task is affected by the specific set of transferred parameters and its delicate interplay with the forms of the true solution and the source-target task relation. By that, we provide an analytical theory to the fragile nature of successful transfer learning designs. We demonstrate that the practical approach of arbitrary selection of transferred parameters signifies the fragile nature of successful parameter transfer settings, especially when the true (unknown) parameters have a sparse form in the feature space.
We characterize the settings where transferring a set of parameters is more beneficial (in the sense of improved generalization in the target task) than defining them as additional free parameters or zeroing them. We also prove that the transfer of parameters from an overparameterized solution of a source task is not necessarily optimal when the source task is closer or identical to the target task.
The majority of this paper is focused on the analytical and empirical study of the minimum 2 -norm solution in our transfer learning setting. Yet, we also empirically examine the utilization of the minimum 1 -norm solution and the ridge regression approach in our transfer learning framework. The minimum 1 -norm (interpolating) solution also induces generalization errors that follow a double descent shape, and can outperform the minimum 2 -norm solution especially if the true parameters are sparse. The ridge regression method includes a regularization term that prevents interpolation and, when properly tuned, eliminates the double descent peak. Ridge regression can outperform the interpolating solutions for a wide range of overparameterization levels. However, at the proximity of maximal overparameterization, the minimum 2 -norm solution performs comparably to the ridge regression. This demonstrates that interpolation at extreme overparameterization levels can substitute properly tuned regularization in our transfer learning setting.
1.1. Related Work. Despite the prevalence of transfer learning in contemporary practice, there are only few analytical theories for this topic. First, Lampinen and Ganguli [14] studied the optimization dynamics in transfer learning of multi-layer linear models, which is a different research objective than our focus on double descent phenomena. Interestingly, when we posted the first version of our work on arXiv in June 2020, there was no literature on double descent phenomena and interpolating solutions in transfer learning. Later on, Dhifallah and Lu [8] studied single-layer nonlinear models that are suitable for both regression and classification problems, where the source task is ridge regularized and therefore prevents interpolation and double descent phenomena. Gerace at al. [10] studied a binary classification problem that is addressed by transfer learning of the first layer in a two-layer model that includes nonlinearities. The learning settings in [10] include regularization on both the source and target task and therefore attenuate some of the double descent behavior. The analysis in [10] requires numerical optimizations and empirical estimation of covariance matrices as inputs for their asymptotic formulations.
Remotely from overparamterized learning but related to transfer learning theory, Obst et al. [20] studied fine-tuning (based on gradient descent) between linear regression problems in underparameterized settings.
To summarize, each of the existing transfer learning theories employs different analytical tools, assumptions, and accordingly presents a unique analytical perspective on the topic. We focus on transfer learning between linear regression problems for Gaussian data and without regularization. This allows us to consider the minimum 2 -norm solution and explicitly characterize the generalization error and its double descent behavior in a detailed, completely closed-form formulation that is unavailable elsewhere. This also contributes to our discussion on beneficial transfer learning and lets us to analytically characterize the optimal source task to transfer from.
1.2. Paper Organization. This paper is organized as follows. In Section 2 we define the transfer learning architecture examined in this paper. In Section 3 we analytically characterize the double descent phenomenon in our transfer learning setting. In Section 4 we analyze the conditions for beneficial transfer of parameters compared to the alternatives of free and zeroed parameters. In Section 5 we characterize the optimal source task to transfer from. Sections 3-5 are focused on the minimum 2 -norm solution in the overparameterized regime of our transfer learning setting; in Section 6 we examine transfer learning in conjunction with the minimum 1 -norm solution and the ridge regression method. Section 7 concludes the paper. The Supplementary Materials include all of the proofs and mathematical developments as well as additional details and results for the empirical part of the paper.
Tasks: Problem Definition.
2.1. Source Task: Data Model and Solution Form. We start with the source data model, where a d-dimensional Gaussian input vector z ∼ N (0, I d ) is connected to a response value v ∈ R via the noisy linear model
where ξ ∼ N 0, σ 2 ξ is a Gaussian noise component independent of z, σ ξ > 0, and θ ∈ R d is an unknown vector. The data user is unfamiliar with the distribution of (z, v), however gets a dataset of n independent and identically distributed (i.i.d.) draws of (z, v) pairs denoted
. The n data samples can be rearranged as Z [z (1) , . . . , z ( n) ] T and v [v (1) , . . . , v ( n) ] T that satisfy the relation v = Zθ + ξ where ξ [ξ (1) , . . . , ξ ( n) ] T is an unknown noise vector that its i th component ξ (i) participates in the relation
underlying the i th data sample.
The source task is defined for a new (out of sample) data pair z (test) , v (test) drawn from the distribution induced by (2.1) independently of the n examples in D. For a given z (test) , the source task is to estimate the response value v (test) by the value v that minimizes the corresponding out-of-sample squared error (i.e., the generalization error of the source task)
where the second equality stems from the data model in (2.1) and the corresponding linear form of v = z (test),T θ where θ estimates θ based on D.
To address the source task based on the n examples, one should choose the number of free parameters in the estimate θ ∈ R d . Consider a predetermined layout where p out of the d components of θ are free to be optimized, whereas the remaining d -p components are constrained to zero values. The coordinates of the free parameters are specified in the set S {s 1 , . . . , s p } where 1 ≤ s 1 < • • • < s p ≤ d and the complementary set S c {1, . . . , d} \ S contains the coordinates constrained to be zero valued. We define the |S|×d matrix Q S as the linear operator that extracts from a d-dimensional vector its |S|-dimensional subvector of components residing at the coordinates specified in S. Specifically, the values of the (k, s k ) components (k = 1, . . . , |S|) of Q S are ones and the other components of Q S are zeros. The definition given here for Q S can be adapted also to other sets of coordinates (e.g., Q S c for S c ) as denoted by the subscript of Q. We now turn to formulate the source task using the linear regression form of θ = arg min
where Z + S is the pseudoinverse of Z S ZQ T S . Note that Z S is a n × p matrix that its i th row is formed by the p components of z (i) specified by the coordinates in S, namely, only p out of the d features of the input data vectors are utilized. Hence, in the underparameterized case of p ≤ n the solution (2.4) almost surely reduces to the unique least squares form of (2.5)
Moreover, in both (2.4)-(2.5), θ is a d-dimensional vector that may have nonzero values only in the p coordinates specified in S (this can be easily observed by noting that for an arbitrary w ∈ R |S| , the vector u = Q T S w is a d-dimensional vector that its components satisfy u s k = w k for k = 1, ..., |S| and u j = 0 for j / ∈ S). While the specific optimization form in (2.3) was not explicit in previous studies of non-asymptotic settings, e.g., [5,3], the solution in (2.4) coincides with theirs and, thus, the formulation of the generalization error of our source task (which is a linear regression problem that, by itself, does not have any transfer learning aspect) is available from [5,3] and provided in Appendix A.2 in our notations for completeness of presentation. where H ∈ R d×d is a deterministic matrix and η ∼ N 0, σ 2 η I d is a Gaussian noise vector with σ η ≥ 0. Here η, x, , z and ξ are independent. The data user does not know the distribution of (x, y) but receives a small dataset of n i.i.d. draws of (x, y) pairs denoted as
. The n data samples can be organized in a n×d matrix of input variables X [x (1) , . . . , x (n) ] T and a n × 1 vector of responses y [y (1) , . . . , y (n) ] T that together satisfy the relation y = Xβ + where [ (1) , . . . , (n) ] T is an unknown noise vector that its i th component (i) is involved in the connection y (i) = x (i),T β + (i) underlying the i th example pair.
The target task considers a new (out of sample) data pair x (test) , y (test) drawn from the model in (2.6) independently of the training examples in D. Given x (test) , the goal is to establish an estimate y of the response value y (test) such that the out-of-sample squared error, i.e., the generalization error of the target task, (2.8)
is minimized, where y = x (test),T β, and the second equality stems from the data model in (2.6).
The target task is addressed via linear regression that seeks for an estimate β ∈ R d with a layout including three disjoint sets of coordinates F , T , Z that satisfy F ∪ T ∪ Z = {1, . . . , d} and correspond to three types of parameters:
• p parameters are free to be optimized and their coordinates are specified in F .
• t parameters are transferred from the co-located coordinates of the estimate θ already formed for the source task. Only the free parameters of the source task are relevant to be transferred to the target task and, therefore, T ⊆ S and t ∈ {0, . . . , p}. The transferred parameters are taken as is from θ and set fixed in the corresponding coordinates of β, i.e., for k ∈ T , β k = θ k where β k and θ k are the k th components of β and θ, respectively. • parameters are set to zeros. Their coordinates are included in Z and effectively correspond to ignoring features in the same coordinates of the input vectors. Clearly, the layout should satisfy p+t+ = d. Then, the constrained linear regression problem for the target task is formulated as
where Q T and Q Z are the linear operators extracting the subvectors corresponding to the coordinates in T and Z, respectively, from d-dimensional vectors. Here θ ∈ R d is the precomputed estimate for the source task and considered a constant vector for the purpose of the target task. The examined transfer learning structure includes a single computation of the source task (2.3), followed by a single computation of the target task (2.9) that produces the eventual estimate of interest β using the given θ. The minimum 2 -norm solution of the target task in (2.9) is (see details in Appendix A.3)
In the underparameterized case of p ≤ n the solution (2.10) almost surely reduces to the unique least squares form of (2.11)
Note that the desired layout is indeed implemented by the β forms in (2.10), (2.11): the components corresponding to Z are zeros, the components corresponding to T are taken as is from θ, and only the p coordinates specified in F are adjusted for the purpose of minimizing the in-sample error in the optimization cost of (2.9) while considering the transferred parameters. In this paper we study the generalization ability of overparameterized solutions (i.e., when p > n) to the target task formulated in (2.9): In Sections 3-5 we analyze the minimum 2 -norm solution (2.10) and in Section 6 we examine other solutions.
3. The Double Descent Phenomenon in Transfer Learning.
3.1. Analytical Characterization of the Double Descent Phenomenon. Consider an overall layout of coordinate subsets L {S, F , T , Z}. The following theorem formulates the generalization error of the target task that is solved using a set of t parameters that are transferred as is from their co-located coordinates (indicated by T ) at the source task solution.
Theorem 3.1. Let L = {S, F , T , Z} be a deterministic (i.e., non-random) coordinate layout. Then, the out-of-sample error of the target task has the form of
where
is the error in the transferred coordinates in the target task solution.
The last theorem is proved using non-asymptotic properties of Wishart matrices (see Appendix B).
The error formulation in (3.1) expresses a double descent form in terms of p and n, similarly to the formulation given in [3] for a linear regression task without transfer learning. However, here (3.1) introduces a new term, E (T ,S) transfer , that encapsulates the transfer learning aspect of our setting.
The formulation of the transfer error term in (3.2) demonstrates a bias-variance decomposition where the bias in the transferred parameters is measured with respect to the true parameters of the target task, and the variance depends only on the source task. This biasvariance decomposition is affected by the coordinates of the transferred parameters (i.e., T ) and also indirectly (through θ T ) by the coordinates of the free parameters in the source task (i.e., S), as we will see next.
The following corollary explicitly formulates the transfer bias and variance terms and their detailed dependency on the parameterization level of the source task, i.e., the p, n pair, and the set S of free parameters in the source task (see proof in Appendix B.3).
Corollary 3.2. The transfer bias term from (3.2) can be written as
where r 1 for p ≤ n, n p
for p > n.
The transfer variance term from (3.2) can be formulated as
)
Note that the out-of-sample error formulation in (3.1) depends on the parameterization level of the target task (i.e., the p, n pair) and also on the parameterization level of the source task (i.e., the p, n pair) via the transfer bias and variance terms that are formulated in (3.3)- (3.4). The formulations in Corollary 3.2 imply that the target error peaks not only around p = n, but also around p = n when transfer learning is applied (t > 0). This induces the double double descent behavior that will be demonstrated in the next subsection. Later, in Sections 4-5 we will analyze the formulations in Theorem 3.1 and Corollary 3.2 in more detail and characterize the conditions for beneficial transfer learning.
3.2. On-Average Analysis of Arbitrarily Selected Parameters. In this subsection, we consider the overall layout of coordinate subsets L {S, F , T , Z} as a random structure. This will let us to formulate the expected value (with respect to L) of the generalization error of interest. The simplified setting in this subsection provides useful insights towards Section 3.3 where we return to analyze the transfer of a set of parameters which is induced by a single layout L (i.e., in Section 3.3 we will return to use Theorem 3.1 and Corollary 3.2 where there is no expectation over a random L).
For given d, p, p, and t, we consider a uniform distribution of the coordinate layout L which is defined as follows.
Definition 3.3. A coordinate subset layout L = {S, F , T , Z} that is { p, p, t}-uniformly distributed, for p ∈ {1, . . . , d} and (p, t) ∈ {0, . . . , d} × {0, . . . , p} such that p + t ≤ d, satisfies: S is uniformly chosen at random from all the subsets of p unique coordinates of {1, . . . , d}. Given S, the target-task coordinate layout {F , T , Z} is uniformly chosen at random from all the layouts where F , T , Z are three disjoint sets of coordinates that satisfy F ∪ T ∪ Z = {1, . . . , d} such that |F|= p, |T |= t and T ⊆ S, and |Z|= d -p -t.
Then, the following formulates the out-of-sample error of the target task under expectation with respect to a uniformly distributed L (more details are provided in Appendix B.4).
Corollary 3.4. Let L = {S, F , T , Z} be a coordinate subset layout that is { p, p, t}-uniformly distributed. Then, the expected out-of-sample error of the target task has the form of
where
where r 1 for p ≤ n,
and ζ Hβ 2 2 . Figure 1 presents the curves of E L [E out ] with respect to the number of free parameters p in the target task, whereas the source task has p = d free parameters. In Fig. 1, the solid-line curves correspond to analytical values induced by Corollary 3.4, and the respective empirically computed values are denoted by circles (all the presented results are for d = 120, n = 20, n = 50, β 2 2 = d, σ 2 = 0.05 • d, σ 2 ξ = 0.025 • d. See additional details in Appendix C. The number of free parameters p is upper bounded by d -t that gets smaller for a larger number of transferred parameters t (see, in Fig. 1, the earlier stopping of the curves when t is larger). Observe that the generalization error peaks at p = n and, then, decreases as p grows in the overparameterized range of p > n + 1. We identify this behavior as a double descent phenomenon, but without the first descent in the underparameterized range (double descent curves without the first descent are common when the parameters are selected arbitrarily, for example, see the results in [3,7]).
The error of a trivial solution in the form of the null estimate, i.e., the estimate β is all zeros, is presented as black dotted horizontal lines in the figures. In various settings where the source and target tasks are sufficiently related and the number of parameters is sufficiently far from the peak of the double descent error curve, the examined transfer learning method (with t > 0 arbitrarily selected parameters) outperforms the null estimate. We can interpret the results in Figure 1 as examples for important cases of transfer learning settings. Figs. 1a,1e correspond to transfer learning between two highly related tasks, therefore, transfer learning is beneficial in the sense that for a given p / ∈ {n-1, n, n+1} the error decreases as t increases (i.e., as more parameters are transferred instead of being omitted). Figs. 1b,1f correspond to transfer learning between two moderately related tasks, hence, transfer learning is still beneficial, but less than in the former case of highly related tasks. Figs. 1c,1g correspond to transfer learning between two unrelated tasks (although not extremely different), hence, transfer learning is useless, but not harmful (i.e., for a given p, the number of transferred Figure 2 demonstrates the empirical standard deviations of the generalization errors (denoted as shaded areas around the curves and markers that denote the average errors). The results show that the empirical errors are more concentrated around their (theoretical and empirical) expectations as the dimensions of the problem (e.g., input dimension, number of data examples and parameters) are proportionally increased. For this example, we consider the setting of Fig. 1b in three proportional scales of the dimensions and dimension-dependent quantities:
• By considering the generalization error formula from Theorem 3.1 as a function of p and p (i.e., the number of free parameters in the source and target tasks, respectively) we receive a two-dimensional double descent behavior as presented in Fig. 3 and its extended version Fig. 10 in Appendix C.1 that presents results for additional pairs of t and σ 2 η . The results show a double descent trend along the p axis (with a peak at p = n) and also, when parameter transfer is applied (i.e., t > 0), a double descent trend along the p axis (with a peak at p = n). Our solution structure implies that p ∈ {t, . . . , d} and p ∈ {0, . . . , d -t}, hence, a larger number of transferred parameters t eliminates a larger portion of the underparameterized range of the source task and also eliminates a larger portion of the overparameterized range of the target task (see in Fig. 3 the white eliminated regions that grow with t). When t is high, the wide elimination of portions from the ( p, p)-plane hinders the complete form of the two-dimensional double descent phenomenon (see, e.g., Fig. 3d).
Conceptually, we can observe a tradeoff between overparamterized learning and transfer learning where parameters are transferred as is from their co-located coordinates of the source task solution: an increased transfer of parameters limits the level of overparameterization applicable in the target task and, in turn, this may limit the overall potential gains from the transfer learning. Yet, when the source task is sufficiently related to the target task (see, e.g., Figs. 1a,1b), the parameter transfer can compensate (sometimes only partially) for an insufficient number of free parameters (in the target task). The last claim is also evident in Figs. 1a,1b,1e,1f where, for p > n + 1, there is a range of generalization error values that is achievable by several settings of (p, t) pairs (i.e., specific error levels can be attained by curves of different colors in the same subfigure). E.g., in Fig. 1b the error achieved by p = 112 free parameters and no parameter transfer can be also achieved using p = 70 free parameters and t = 48 parameters transferred from the source task.
Selected Parameters. Section 3.2 considers an on-average error for a random coordinate layout. We now turn to discuss the generalization behavior of a single coordinate layout that was formed arbitrarily (i.e., without using any knowledge on the problem setting). Hence, we return to consider the generalization error E (L) out for a given layout L using Theorem 3.1 and Corollary 3.2 from Section 3.1.
Figure 4 shows the curves of E (L) out for specific coordinate layouts L that evolve with respect to the number of free parameters p in the target task (for more examples see Figures 1314in the Appendices). The excellent fit of the analytical results (that were computed using Theorem 3.1 and Corollary 3.2) to the empirical values (computed by averaging over 250 experiments with the same evolution of L) is evident. The effect of the specific coordinate layout is clearly visible by the less-smooth curves (compared to the on-average results in Fig. 1). We examine two different cases for the true β: a linearly-increasing (Fig. 4a) and a sparse (Fig. 4e) layout of values, both have the same 2 norm. The difference in the true β forms yields error curves that significantly differ despite the use of the same sequential construction of the coordinate layouts with respect to p (e.g., compare Figs. 4f and4b). The operator H in the task relation model greatly affects the generalization error curves as evident from comparing our results for different types of H: an identity, local averaging (with neighborhood size 11), and discrete derivative operators (e.g., compare subfigures within the first row of Fig. 4). The results clearly show that the interplay among the structures of H, β, and the coordinate layout significantly affects the generalization performance.
Our results also exhibit that an arbitrarily selected set T of t transferred parameters can be the best setting for a given set F of p free parameters but not necessarily for an extended set F ⊃ F of p > p free parameters. This is especially evident when the true parameters have a sparse form over an unknown support in the feature domain (i.e., true parameters with nonzero values are scarce and their coordinates are unknown). For example, see Fig. 4g where the green colored curve does not consistently maintain its relative vertical order with the red color curve at the overparameterized range of solutions; as a second example see Fig. 4h and observe that the blue and red curves do not maintain their vertical order in the overparameterized range. This exemplifies that, when arbitrary selection of parameters is employed due to unknown task relation, transfer learning settings can be fragile and hence finding a successful setting may require delicate, trial and error engineering. Therefore, our theory qualitatively explains similar practical aspects in deep neural networks (see, for example, [22]).
4. When is Transfer Learning Beneficial?. The formulation of the generalization error in the target task (Theorem 3.1 and Corollary 3.2) shows that the benefits from parameter transfer depend on various aspects of the learning setting. In this section we characterize the conditions for beneficial transfer from the viewpoint of the following question: Given a setting where T is the intended set of coordinates for parameter transfer, can avoiding this parameter transfer improve generalization? out for an arbitrary coordinate layout L. In the first row of subfigures the true β has feature-domain values that follow a linear form , see (a). In the second row of subfigures the true β has a sparse form in the feature domain with non-zero values at 30 coordinates selected randomly out of the d = 120, see (e). In each row there are three error plots for different circulant forms of the operator H (here the local averaging is defined for a neighborhood of 11 samples). Here transfer due to transferring an arbitrarily selected parameter. The error difference is presented with respect to the number p of free parameters in the source task and the variance σ 2 η of the noise in the task relation model. Each subfigure in Fig. 5 shows results for a different definition of H. All the subfigures demonstrate that transferring an arbitrarily selected parameter is more beneficial when the solution to the source task is highly overparameterized; indeed, then the generalization error in the source task itself is lower due to its own double descent phenomena (e.g., recall the behavior of the error along the vertical axis in Fig. 3b-3d). As expected, a low level of noise in the task relation is also important for beneficial transfer.
The examples in Fig. 5 also show a somewhat unexpected behavior: parameter transfer from an underparameterized source task can be more beneficial as the tasks are less related, e.g., compare the lower left corners in Figures 5a and5b. To understand this observation mathematically one may recall the transfer bias and variance formulations in Corollary 3.2 and notice the following effects of H, which in Figs. 5a-5b has the form of H = cI d where c ∈ (0, 1). First, the transfer bias increases as c ∈ (0, 1) gets smaller; also note that the transfer bias sums only over the |T | = t transferred coordinates. Second, the (underparameterized) transfer variance decreases as c ∈ (0, 1) gets smaller; here note that the transfer variance is affected by |S c | = d -p coordinates of H = cI d in addition to the number t of transferred parameters. In the experiment setting of Fig. 5a, the source task is underparameterized with a sufficiently large d -p value such that the reduction in the transfer variance dominates the increase in the transfer bias, resulting in beneficial transfer. More generally, the lesson here is that beneficial transfer learning can be achieved also in non-intuitive settings where the tasks are not necessarily highly related. In Section 5 we will provide additional insights into counter-intuitive beneficial cases.
Free Parameters. Now we turn to discuss the question of when the transfer of the parameters in T is more beneficial than defining them as free for optimization.
Consider the coordinate layout L = {S, F , T , Z} where, as usual, |S| = p, |F| = p, |T | = t. Here we also define a second coordinate layout L = {S, F , T , Z} which is a modified version of L without transferred parameters, specifically, T = ∅ and F = F ∪ T . Hence, the consideration of L reflects the case where all the transferred parameters in L are replaced by free parameters. Then, we define the following error difference term due to transferring the parameters in T instead of allocating them as free parameters:
where
out and E (L ) out are the out-of-sample errors in the target task (recall Theorem 3.1) for the coordinate layouts L and L , respectively. The definition in (4.4) implies that transferring the parameters in T is beneficial (over allocating these coordinates as t free parameters in addition to the p free parameters in F ) if ∆E (L) TvsF < 0. We now turn to examine the effects of p and t on beneficial transfer in the sense of ∆E (L) TvsF < 0. For this, note that L includes p = p + t free parameters and hence is not necessarily in the same parameterization regime as L that includes p free parameters. The parameterization regimes of L and L affect the behavior of the benefits from parameter transfer, as described by the following corollaries of Theorem 3.1 (these corollaries are simply proved by setting Eq. (3.1) in (4.4) and reorganizing the formulations).
Corollary 4.1. For p + t ≤ n -2, namely, when both L and L correspond to underparameterized settings, the error difference due to transferred versus free parameters is
Accordingly, the transfer of the t > 0 parameters in T ⊆ S is more beneficial than allocating them as free parameters (i.e., ∆E (T ,S)
Corollary 4.2. For p + t ≥ n + 2, namely, when L corresponds to an overparameterized setting, the error difference due to transferred versus free parameters is
where the coefficients γ 0 ,γ 1 ,γ 2 ,γ 3 depend on the parameterization level of L:
• For an underparameterized L with p ≤ n -2:
. Accordingly, the transfer of the t > 0 parameters in T ⊆ S is more beneficial than allocating them as free parameters (i.e., ∆E
Clearly, the formulation of ∆E
TvsF is more intricate in the case of overparameterized L in Corollary 4.2 than in the case of underparameterized L and L in Corollary 4.1. Specifically, the set of free parameters F appears in (4.7)-(4.8) in addition to the sets T and Z.
To intuitively understand the behavior of ∆E
TvsF , consider expectation over coordinates that are selected uniformly at random (recall Definition 3.3). Specifically, we define ∆E TvsF
TvsF while noting that L is deterministically related to L and hence it is sufficient to consider the expectation over L. Figure 6 shows the value of ∆E TvsF as a function of p and t for several operators H in the task relation. Figure 6 demonstrates the following typical behaviors:
• For underparameterized L and L (i.e., p + t ≤ n -2, which corresponds to the region to the left of the dotted black lines in the bottom subfigures in Fig. 6): Parameter transfer is more likely to be beneficial, and with larger gains, as p + t increases towards n -2. Moreover, when (4.6) is satisfied, it is beneficial to have more transferred than free parameters (i.e., having t > p for a fixed sum of p + t). The intuition here is that, when we are constrained to the underparameterized regime in both options (i.e., L and L ), transferring parameters instead of having more free parameters can keep the error farther away from the peak of the double descent curve, which yields a beneficial transfer. Interestingly, due to the significant peak of the generalization error around the interpolation threshold, this behavior also occurs when the source and target tasks are quite different (see Fig. 6c). • For underparameterized L and overparameterized L (i.e., p ≤ n -2 and p + t ≥ n + 2, which correspond to the region to the right of the dotted black lines in the bottom subfigures in Fig. 6): Parameter transfer is more likely to degrade as p increases towards n -2 while p + t ≥ n + 2. Moreover, increasing the number t of transferred parameters degrades the benefits from transfer learning. The intuition here is that, when the transfer option L is underparameterized and the no-transfer option L is overparameterized, any increase in p + t makes L more overparameterized and hence farther away from the double descent peak (hence having an improved generalization ability).
• For overparameterized L and L (i.e., p ≥ n + 2, which corresponds to the upper subfigures in Fig. 6): Here, the important behavior is that, for a given T , parameter transfer improves as the number of free parameters p increases. Of course that this improvement becomes beneficial (at high overparameterization levels) only when the source task is sufficiently related to the target task (see, e.g., Figures 6a and6b). The intuition here is that, when both options (i.e., L and L ) are in the overparameterized regime then the increase in p moves both L and L away from the peak of the double descent. The transfer learning option L is closer to the double descent peak and hence typically improves more (due to the steeper slope at the corresponding part of the generalization error curve, e.g., see in Fig. 1) than the no-transfer case L . for a given coordinate layout L. Namely, we characterize the best source task to transfer from when the other aspects of the parameter transfer are fixed.
Consider an operator H that has the diagonal form of (5.1)
H , . . . , λ
H }
where {λ
H } d j=1 as the eigenvalues of H due to the fact that applying the same orthonormal rotation on the feature spaces of the source and target tasks can induce a non-diagonal H that its eigenvalues are the same {λ
where θ (j) and β (j) are the j th components of the true parameter vectors of the source and target tasks, respectively, and η (j) is the j th noise component in η. Then, we can further develop the expressions from Corollary 3.2. First, the transfer bias term from (3.3) can be written as
where r 1 for p ≤ n, n p for p > n.
Second, the transfer variance term Var T ,S from (3.4) can be also rewritten using
Consequently, we get the following result (proof is provided in Appendix E.1). For j ∈ {1, . . . , d} where β (j) = 0, λ
H can have any value. Let us interpret the meaning of Theorem 5.1 for the case of β (j) = 0 for j ∈ {1, . . . , d}.
The theorem shows that the linear operator in the optimal task relation has two different characterizations depending on whether the source task is under or over parameterized:
• For an underparameterized source task it is best that the true parameters in the transferred coordinates of the source task are the same (up to the additive noise terms from η) as the corresponding true parameters of the target task (see (5.5) and recall (5.2)). • For an overparameterized source task it is best that the true parameters in the transferred coordinates of the source task are amplified versions by a factor of (5.5)) of the corresponding true parameters of the target task. This amplification intends to partially compensate for the increased transfer bias in the case of overparameterized source task (see the p > n case in (5.3)). Indeed, the optimal amplification increases as the source task is more overparameterized (i.e., as p increases towards d). While this amplification reduces the transfer bias, it increases the transfer variance (see (3.4) and (5.4)) and hence the amplification is somewhat restrained. Also, the optimal amplification p 2 -1 n p-1+t( p-n) decreases as the number t of transferred parameters increases. Specifically, when t = p (i.e., all the free parameters of the source task are transferred to the target task) the optimal λ (j) H values for j ∈ T become 1 and there is no amplification. Eq. (5.6) considers the coordinates of the free parameters of the source task that are not transferred (i.e., S \ T ) and shows that the optimal true parameters of the source task in these coordinates are zeros in the overparameterized regime due to the dependency of the transfer variance on the true parameters of the source task at S \ T (see (3.4) and (5.4)). In the underparameterized regime of the source task, there is no such dependency on θ S\T and hence these true parameters may have any value without affecting the minimization of E (T ,S) transfer .
According to Eq. (5.7), in both the under and over parameterized cases, it is best that the true parameters of the source task are all zeros in the coordinates that do not participate in the source task solution (i.e., S c ). This optimal form reduces the misspecification in the solution of the source task to originate only in the task relation noise η S c . Avoiding these misspecifications makes the estimate of each free source parameter closer to its true value (i.e., instead of also trying to compensate for omitted parameters that are informative) and, hence, this improves the relevance of the parameters θ T that are transferred "as is" to the TvsZ is required for understanding whether the optimal eigenvalues of H induce beneficial transfer and, if so, how much can the eigenvalues deviate from their optimal values while still having a beneficial transfer.
In Fig. 7 we show the analytical values of ∆E (T ,S) transfer (recall the definition in (4.3)) as a function of p and the eigenvalues of H that correspond to the transferred parameters (see Fig. 20 in Appendix E.2 for the corresponding empirical evaluations). Specifically, we consider a family of H operators that are diagonal in the feature domain and have the following stepshaped structure for their eigenvalues (see Fig. 7d): the first 8 eigenvalues are all equal to a value that varies among the H operators in this family; the next 112 eigenvalues are 0.1 for any H operator in this family. Here we consider transfer of the first eight parameters, i.e., T = {1, . . . , 8} that correspond to the eigenvalues with a tunable value in our evaluations (this is reflected by the horizontal axes in Figs. 7a-7c). The results in Fig. 7 demonstrate that the range of eigenvalues that induce beneficial transfer increases together with overparameterization (see the wider blue regions around the green dashed line that denotes the optimal eigenvalue from Theorem 5.1). Fig. 7a shows that benefits can be also obtained in underparameterized settings where the eigenvalues (in T ) are around 1, however, these eigenvalue regions are smaller and produce lower benefits than their overparameterized alternatives. The effect of the noise in the task model is also apparent by comparing Fig. 7a to Fig. 7b that shows results for the same settings but with a higher σ 2 η . Specifically, the higher noise level in the task relation reduces the size of the beneficial regions and their gains in the overparameterized range, and leaves the underparameterized range without any beneficial regions. Next, compare Fig. 7a to Fig. 7c that shows results for the same settings but with β of the less favorable structure in Fig. 7f instead of the structure in Fig. 7e. This shows how the structure of the true parameters in β can also affect the parameter transfer performance.
The previous sections presented analytical and empirical results on transfer learning between two least squares solutions that take their minimum 2 -norm forms (2.4), (2.10) in the overparameterized regime. In this section we examine the same transfer learning mechanism (i.e., transferring co-located parameters from the already-learned source model to the to-be-learned target model) for two more kinds of solutions to linear regression: least squares with the minimum 1 -norm form in the overparameterized regime, and ridge regression.
6.1. The Minimum 1 -Norm Interpolator. This setting also emerges from the optimization problems formulated in (2.3) and (2.9) for the source and target tasks, respectively. In the underparameterized regime, the source and target tasks almost surely have the unique least squares solutions as provided in (2.5) and (2.11).
For an overparameterized source task, i.e., p > n, the problem (2.3) has infinite interpolating solutions among which we choose here the minimum 1 -norm solution:
This solution equals to θ where θ S c = 0 and θ S = arg min k∈R p k 1 subject to Z S k = v, which is a basis pursuit problem [6] that can be addressed via linear programming methods.
For an overparameterized target task, i.e., p > n, the problem (2.9) has infinite interpolating solutions, from which we choose here the minimum 1 -norm solution
that can be formulated also as β where β Z = 0, β T = θ T and
which is a basis pursuit problem that can be solved via linear programming.
The empirical out-of-sample errors of the minimum 1 -norm transfer learning are shown in Figure 8 as blue curves where the shade of blue denotes the number of transferred parameters. The results demonstrate that the double descent behavior occurs also for the minimum 1norm solution. Double descent phenomena for minimum 1 -norm solutions to linear regression (without transfer learning) were studied in [19,18]. The asymptotic analyses in [15,27] show that a triple descent can be observed if the true parameters are sufficiently sparse. Here we do not clearly observe an additional error peak in the overparameterized regime, which could possibly be due to the misspecification strategy (i.e., zeroing parameters in predetermined coordinates) that we use for controlling the parameterization levels in our non-asymptotic setting. For β of a linear shape the minimum 2 -norm solution is better than the minimum 1 -norm solution in the entire overparameterized range (see Figs. 8a-8c). Nevertheless, for a sparse β the minimum 1 -norm solution can provide the best performance in the high overparameterization levels due to its sparsity-promoting nature (see Figs. 8d, 8g-8i).
Regression. Now we turn to formulate our transfer learning approach for ridge regression. So far we considered the optimization problems (2.3) and (2.9) that do not include explicit regularization terms in their cost functions. Hence, the ridge regression extension of (2.3) is
where α > 0 determines the level of ridge regularization. The formulation in (6.4) is equivalent to θ where θ S c = 0 and
The optimization in (6.5) has a standard ridge regression form and, thus, its closed-form solution is In appendix F.1 we formulate the out-of-sample error of the ridge regression solution to the source task, and show that optimal tuning is provided by α =
. In our experiments we assume that θ S 2 2 , θ S c 2 2 are unknown and only θ 2 2 , σ 2 ξ are known about the source task. Thus, we assume θ S
2 that let us to approximate the optimal tuning using α =
Proceeding to the target task, the ridge regression extension of (2.9) is
for α > 0. The solution of (6.7) has the closed form of β where β Z = 0, β T = θ T , and
In appendix F.2 we provide further details on (6.8), the corresponding out-of-sample error, and show that optimal tuning is given by α =
. In our experiments we approximate the optimal tuning by setting α = dσ 2
2 for more details. The empirical out-of-sample errors for the ridge regression transfer learning appear in Figure 8 as red curves where the shade of red denotes the number of transferred parameters. As expected, the ridge regularization (which is approximately optimally tuned) resolves the error peak and eliminates the double descent shape of the minimum norm interpolating solutions. In the case of suboptimally tuned ridge regularization the double descent peak may not be fully resolved (see Fig. 9). For β with a linear shape, the ridge regularization suits best among the examined methods at any parameterization level, except for the maximal overparameterization levels where the minimum 2 -norm solution performs comparably (see Figs. 8a-8c). However, for a sparse β the minimum 1 -norm outperforms the ridge solution for high overparameterization levels (see Figs. 8d-8f).
In this work we have established an analytical framework for the fundamental study of transfer learning in conjunction with overparameterized models. We used least squares solutions to linear regression problems for shedding clarifying light on the generalization performance induced for a target task addressed using parameters transferred from an already completed source task. We formulated the generalization error of the target task and presented its two-dimensional double descent shape as a function of the number of free parameters individually available in the source and target tasks. We characterized the conditions for a beneficial transfer of parameters and demonstrated its high sensitivity to the delicate interaction among crucial aspects such as the source-target task relation, the specific choice of transferred parameters, and the form of the true solution. We importantly showed that overparameterized transfer learning is not necessarily improved by using a source task which is closer or identical to the target task. Our focus was mainly on the analytical and empirical study of the minimum 2 -norm solution to overparameterized transfer learning. We where Z S ZQ T S and Z S c ZQ T S c . By setting the equality constraint in the optimization cost, the problem in (A.1) becomes θ = arg min
Without the equality constraint, (A.2) is just an unconstrained least squares problem that its minimum 2 -norm solution is
where Z + S is the Moore-Penrose pseudoinverse of Z S . Note that θ in (A.3) satisfies the equality constraint in (A.1) and, therefore, (A.3) is also the solution for the constrained optimization problems in (A.1), (A.2), and (2.3).
A.2. The Double Descent Formulation for the Generalization Error of the Source Task. The generalization error of a single linear regression problem (that includes noise) in non-asymptotic settings is provided in [3] for a given coordinate subset (i.e., deterministic S in our terms). The result from [3] can be written in our notations as
for p ≥ n + 2.
In case that the coordinate subset S is uniformly chosen at random from all the subsets of p ∈ {1, . . . , d} unique coordinates of {1, . . . , d}, then we get that
2 . Accordingly, the expectation over S of the generalization error of the source task leads to the following result
The formulation in (A.5) considers θ as a deterministic vector. For the analysis of the target task, where the task relation model (2.7) is assumed to hold, it is also useful to formulate the expectation of the out-of-sample error of the source task with respect to both S and the noise vector η from the task relation model. This leads us to to consider θ as a random vector and to formulate the following expectation.
where
A.3. The Estimate β in Eq. (2.10). The optimization problem in (2.9), given for the target task, can be addressed using the relation
where X F XQ T F , X T XQ T T , and X Z XQ T Z . By setting the equality constraints of (A.7) in its optimization cost, the problem (A.7) can be translated into the form of
The last optimization is a restricted least squares problem that can be solved using the method of Lagrange multipliers to show that
where θ T Q T θ and X + F is the Moore-Penrose pseudoinverse of X F . Appendix B. Proofs for Section 3. In this section we outline the proof of Theorem 3.1 for the generalization error of the target task in the setting where a specific coordinate subset layout L determines the transferred set of parameters. We start in Section B.1 by providing auxiliary results that use non-asymptotic properties of Gaussian and Wishart matrices. Then, in Section B.2 we prove Theorem 3.1, in Section B.3 we prove Corollary 3.2, and in Section B.4 we prove Corollary 3.4.
B.1. Auxiliary Results using Non-Asymptotic Properties of Gaussian and Wishart Matrices. The random matrix X F XQ T F is of size n×p and all its components are i.i.d. standard Gaussian variables. Then, almost surely,
where X + F X F is the p × p projection operator onto the range of X F . Accordingly, let a ∈ R p be a random vector independent of the matrix X F and, then,
The components of X F are i.i.d. standard Gaussian variables, hence X T F X F ∼ W p (I p , n) is a p × p Wishart matrix with n degrees of freedom, and
Wishart matrix with p degrees of freedom. The pseudoinverse of the n × n Wishart matrix (almost surely) satisfies
where the result for p ≥ n + 2 corresponds to the common case of inverse Wishart matrix with more degrees of freedom than its dimension, and the result for p ≤ n -2 is based on constructions provided in the proof of Theorem 1.3 in [5].
Following (B.3), let u ∈ R n be a random vector independent of X F . Then,
The results in (B.1)-(B.5) are presented using notions of the target task, specifically, using the data matrix X and the coordinate subset T . One can obtain the corresponding results for the source task by updating (B.1)-(B.5) by replacing X, T , n and p with Z, S, n and p, respectively. For example, the result corresponding to (B.1) is
where Z + S Z S is the p × p projection operator onto the range of Z S . The next auxiliary results consider a coordinate subset layout L = {S, F , T , Z} which is specific, i.e., non random, and therefore the induced operators such as Q S , Q F , Q T , Q Z are also fixed and do not have any random aspect. Recall that Q T S Q S is a d × d diagonal matrix with its j th diagonal component equals 1 if j ∈ S and 0 otherwise. Similarly holds for the other coordinate subsets. Accordingly, here the norms of vector forms such as
for p > n.
(B.7)
In our case we have the n × p matrix Z S that its components are i.i.d. standard Gaussian variables, thus, Z S can be decomposed into a form that involves an independent Haardistributed matrix, i.e., a random orthonormal matrix that is uniformly distributed over the set of orthonormal matrices of the relevant size. This lets us to prove the results in (B.7)-(B.10) using some algebra and the non-asymptotic properties of random Haar-distributed matrices, see examples for such properties in Lemma 2.5 in [26] and also in Proposition 1.2 in [12]. B.2. Proof Outline of Theorem 3.1. The generalization error E out of the target task was expressed in its basic form in Eq. (2.8) for a specific coordinate subset layout L = {S, F , T , Z}. Please note that the expectations below do not include any expectation with respect to L or its components, which are non-random here.
We start with the relevant decomposition of the error expression, namely,
Then, we use the expression for the estimate θ given in (2.4) and the relation y = Xβ + , to decompose the third term in (B.11) as follows
We further develop the last expression using the result in (B.4) and that X T T X T ∼ W t (I t , n) is a Wishart matrix with mean E X T T X T = nI t , and get
We proceed to the fourth error term in (B.11), i.e., the error in the subvector induced by the specific T of interest, and develop its formulation as follows:
(B. 16) where the first term can be developed using (B.2) into and the second term in (B. 16) can be rewritten using the result in (B.4) and that Now we turn to prove the transfer variance formulation from Corollary 3.2. For a start, note that For S that is uniformly chosen at random from all the subsets of p unique coordinates of {1, . . . , d}, we get that the mean of the projection operator
where we used the structure of Q T S Q S that is a d × d diagonal matrix with its j th diagonal component equals 1 if j ∈ S and 0 otherwise. Definition 3.3 also specifies that, given S, the target-task coordinate layout {F, T , Z} is uniformly chosen at random from all the layouts where F , T , and Z are three disjoint sets of coordinates that satisfy F ∪ T ∪ Z = {1, . . . , d} such that |F|= p, |T |= t and T ⊆ S, and |Z|= d -p -t. Accordingly, 25) and similarly
Another useful auxiliary result, based on the relation Q S Q T S = I p (carefully note the transpose appearance), is provided by
The results in (B.25)-(B.27) imply that C.1. Additional Results for the On-Average Analysis in Section 3.2. In Fig. 11 we present the empirically computed values of the out-of-sample squared error of the target task, E L [E out ], with respect to the number of free parameters p and p (in the source and target tasks, respectively). The empirical values in Fig. 11 (and also the values denoted by circle markers in Fig. 1 in Section 3.2) were obtained by averaging over 250 experiments where each experiment was carried out based on new realizations of the data matrices, noise components, and the sequential order of adding coordinates to subsets (such as S) for the gradual increase of p and p within each experiment. Note that the results in Fig. 4 do not include averaging over the sequential order of adding coordinates to subsets. Each single evaluation of the expectation of the squared error for an out-of-sample data pair x (test) , y (test)
Results for the Single-Layout Analysis in Section 3.3. The following results are for two different forms of the true solution β: the first is a form with linearly increasing values (Fig. 4a), the second is a form with sparse values where only 25% of coordinates have non-zero value (Fig. 4e). Note that both forms satisfy β 2 2 = d. The three types of linear operator H in the evaluations in Section 3.3 are as follows. First, H = I d that is the identity operator. Second, is the circulant matrix H that corresponds to a shift-invariant local averaging operator that uniformly considers 11-coordinates neighborhood around the computed coordinate (note that in other parts of this paper we consider also averaging operators with neighborhood sizes other than 11). Third, is the circulant matrix H that corresponds to discrete derivative operator based on the convolution kernel [-0.5, 0.5].
Figures 13-14 present the analytical and empirical values of the generalization error of the target task with respect to specific coordinate layouts L that evolve with respect to the value of p (this evolution of L is the same in each of the subfigures and it is not particularly designed to any of the combinations of the true β, H, and σ 2 η ). It is clear from Figures 13-14 that the increase in σ 2 η , which by its definition corresponds to less related source and target tasks, reduces the benefits or even increases the harm due to transfer of parameters (one can observe that in Figs. 13-14 by comparing the error curves among subfigures in the same row).
The effect of H with respect to the true β is also evident. First, the identity operator H = I d does not reduce the relation between the source and target tasks and therefore does not degrade the parameter transfer performance by itself (i.e., for H = I d , only the additive noise level σ 2 η can reduce the relation between the tasks). Second, when H is a local averaging operator it does not reduce the benefits from transfer learning (e.g., compare second to first row of subfigures in Figs. [13][14] in the case of linearly-increasing β shape (because local averaging does not affect a linear function, except to the few first and last coordinates where the periodic averaging is applied), in contrast, the local averaging operator significantly degrades the parameter transfer performance in the case of the sparse β form. Lastly, when H is a discrete derivative operator it renders transfer learning harmful in the case of linearlyincreasing β shape (e.g., compare third to first row of subfigures in Fig. 13). In the case of the sparse β form the discrete derivative reduces the potential benefits of the parameter transfer but does not eliminate them completely in case these benefits exist for H = I d (e.g., compare third to first row of subfigures in Fig. 14). is the out-ofsample error of the target task that is empirically computed for m transferred parameters, p free parameters in the target task, and p free parameters in the source task. Correspondingly,
is the empirically computed error induced by avoiding parameter transfer. Therefore, the formula in (D.1) empirically measures the average error difference for a single transferred parameter by averaging over the various settings induced by different values of p while p is kept fixed. To obtain a good numerical accuracy with averaging over a moderate number of experiments we use the value m = 5.
Each empirical evaluation of E L E ( p,p,t) out , for a specific set of values p, p, t corresponds to averaging over 500 experiments where each experiment was conducted for new realizations of the data matrices, noise components, and the sequential order of adding coordinates to subsets. Each single evaluation of the expectation of the squared error for an out-of-sample data pair x (test) , y (test) was empirically computed by averaging over 1000 out-of-sample realizations of data pairs. D.2. Results for n < d. In Figures 1617we present analytical and empirical values of 1 t ∆E transfer induced by settings where n < d (specifically, n = 50 and d = 80), which naturally enable the corresponding overparameterized (i.e., n < p < d) and underparameterized (i.e., p < n < d) settings of the source task. In Fig. 16 we provide the analytical and empirical results for cases where H is local averaging and discrete derivative operators. In the main text only the analytical results were provided and here we show them again near their empirical counterparts that excellently match them (up to the resolution of the empirical settings).
In Figure 17 we additional results for cases where the operator H is a scaled identity matrix.
Here we provide in Fig. 18 the analytical and empirical evaluations of ∆E transfer that correspond to settings where n > d, we specifically consider n = 150 and d = 80. Note that n > d implies that, by the definition of p, the corresponding settings (of the source task) are underparameterized with p ≤ d < n. Like in Fig. 17, the results in Fig. 18 show the excellent match between the analytical and empirical results. D.4. Empirical Results for Benefits in Transferred versus Free Parameters. We provide in Figure 19 the empirical evaluation of ∆E TvsF that corresponds to the analytical evaluation in Figure 6 in the main text. The empirical evaluation was conducted by averaging over 750 experiments where, in each of them, ∆E TvsF was evaluated based on its definition in (4.4) and using a different realization of L (from the uniform distribution that we use) and its corresponding L . In the case of an underparameterized source task where 1 ≤ p ≤ n -2, we get (E.1) In the case of an overparameterized source task where p ≥ n + 2, we get
where the three sums refer to disjoint sets of coordinates. Accordingly, in case of an overparameterized source task and β (k) = 0, the condition
transfer due to convexity. In both cases (E.1) and (E.2), if k ∈ {1, . . . , d} corresponds to β (k) = 0 then λ
The Optimal H in a Componentwise Task Relation. In Figure 20 we provide the empirical evaluations that correspond to the analytical results in Figure 7. The empirical values were computed by averaging over 500 experiments.
Appendix F. Additional Details and Results for Section 6. F.1. Ridge Regression: Error Expression and Optimal Tuning for the Source Task. The out-of-sample error of the ridge solution (6.5)-(6.6) to the source task is developed as follows. First, due to the layout of free parameters in θ we get that
Now we set the closed-form ridge solution from (6.6) to develop the third term in (F.1):
Also, we use the eigendecomposition Z T S Z S = Φ Z S Λ Z S Φ T Z S where Φ Z S is a p × p orthonormal matrix and Λ Z S diag{λ Z S ,1 , . . . , λ Z S , p } is the p× p diagonal matrix formed by the eigenvalues of Z T S Z S . By using this eigendecomposition and (F.3), we develop (F.2) into the form of
The optimal tuning is achieved by the α value that satisfies ∂ Eout ∂ α = 0, which by using (F.1) and (F.4) is α =
. In the main text we explain how we approximate the optimal α in our experiments.
F.2. Ridge Regression: Error Expression and Optimal Tuning for the Target Task. Let us start by explaining in more detail the ridge solution in (6.8). For this, note that the optimization constraints in (6.7) imply y -
. Hence, the solution of (6.7) is equivalent to β where β Z = 0, β T = θ T , and
The optimization in (F.5) has a standard ridge regression form and, accordingly, its closedform solution is provided in (6.8).
Let us develop the out-of-sample error expression of the target task. Based on the layout of free, transferred, and zeroed parameters in β we have
Then, similar to the proof given for the source task in Section F.1, we get that
where we use the eigendecomposition X T F X F = Φ X F Λ X F Φ T X F where Φ X F is a p×p orthonormal matrix and Λ X F diag{λ X F ,1 , . . . , λ X F ,p } is the p × p diagonal matrix of the eigenvalues of X T F X F .
Here, the optimal tuning is obtained by the α value that provides ∂Eout ∂α = 0. Then, (F.6) and (F.7) imply that the optimal tuning is given by (F.8)
.
In our experiments we assume that only β Here, all the subfigures correspond to a β vector with a piecewise-constant form (see Fig. 15). Here, all the subfigures correspond to a β vector with a linear form (see Fig. 4a). Here, all the subfigures correspond to a β vector with a linear form (see Fig. 4a). Note that the results in this figure are for n > d.