Knowledge-Guided Wasserstein Distributionally Robust Optimization (2025)

Zitao WangDepartment of Statistics, Columbia University.Ziyuan WangDepartment of Industrial Engineering and Management Sciences, Northwestern University.Molei LiuThese authors contributed equally to this work.Department of Biostatistics, Columbia Mailman School of Public Health.Nian Si^∗Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology.

Abstract

Transfer learning is a popular strategy to leverage external knowledge and improve statistical efficiency, particularly with a limited target sample. We propose a novel knowledge-guided Wasserstein Distributionally Robust Optimization (KG-WDRO) framework that adaptively incorporates multiple sources of external knowledge to overcome the conservativeness of vanilla WDRO, which often results in overly pessimistic shrinkage toward zero. Our method constructs smaller Wasserstein ambiguity sets by controlling the transportation along directions informed by the source knowledge. This strategy can alleviate perturbations on the predictive projection of the covariates and protect against information loss. Theoretically, we establish the equivalence between our WDRO formulation and the knowledge-guided shrinkage estimation based on collinear similarity, ensuring tractability and geometrizing the feasible set. This also reveals a novel and general interpretation for recent shrinkage-based transfer learning approaches from the perspective of distributional robustness. In addition, our framework can adjust for scaling differences in the regression models between the source and target and accommodates general types of regularization such as lasso and ridge. Extensive simulations demonstrate the superior performance and adaptivity of KG-WDRO in enhancing small-sample transfer learning.

Keywords: Wasserstein distributionally robust optimization; Knowledge-guided learning; Difference-of-convex optimization; Shrinkage-based transfer learning.

1 Introduction

Traditional machine learning methods or empirical risk minimization often suffer from overfitting and a lack of generalization power, particularly in high-dimensional and small-sample-size settings. In recent years, distributionally robust optimization (DRO) has emerged as a powerful framework for mitigating the effects of model misspecification and enhancing robustness in machine learning generalizations. Among various DRO formulations, Wasserstein-DRO (WDRO) gained more attention due to its tractability and generalizability. Specifically, in WDRO, one optimizes over worst-case distributions within an ambiguity set defined by a Wasserstein ball centered at an empirical measure.

However, one persistent challenge with WDRO is its tendency to be overly conservative, which can lead to suboptimal performance in practice as found in Liu etal., (2024). In many real-world scenarios, prior knowledge can be leveraged to improve model performance and robustness. For example, in electronic healthcare record data, prior knowledge might come from predictive models trained on existing large, population-wide datasets. In such a context, transfer learning has proven to be a versatile approach for improving performance on a target task. Despite its successes, the integration of prior knowledge into WDRO frameworks has remained an open question.

In this work, we introduce Knowledge-Guided Wasserstein Distributionally Robust Optimization (KG-WDRO), a novel framework that adapts the Wasserstein ambiguity set using external knowledge (parameters). We assume access to prior predictors of pre-trained models, which can guide the predictive model in the target dataset. By constraining the transport cost along directions informed by prior knowledge, our approach addresses the conservativeness of vanilla WDRO while preserving robustness. Intuitively, this strategy allows the model to focus its uncertainty on regions where prior knowledge is less reliable, effectively robustify knowledge-guided generalization.

1.1 Related Works

1.1.1 Wasserstein DRO

Wasserstein DRO has recently garnered significant attention due to its tractability (Blanchet and Murthy,, 2019; MohajerinEsfahani and Kuhn,, 2018; Gao and Kleywegt,, 2023) and generalizability (Blanchet etal., 2019a, ; Gao etal.,, 2022). Notably, Blanchet etal., 2019a and Gao etal., (2022) demonstrate that Wasserstein DRO with mean square loss is equivalent to the square root lasso Belloni etal., (2011). Similarly, Shafieezadeh-Abadeh etal., (2015, 2019); Blanchet etal., 2019a ; Gao etal., (2022) establish that Wasserstein DRO with logistic loss and hinge loss corresponds to their regularized counterparts. Moreover, the statistical properties of the WDRO estimator have also been investigated in Blanchet etal., (2021, 2022); Gao, (2023). However, leveraging external knowledge in Wasserstein DRO has been an open problem.

1.1.2 Transfer Learning

Improving prediction accuracy for target populations by integrating diverse source datasets has driven methodological advances in transfer learning. Contemporary approaches aim to address challenges including distributional heterogeneity and limited labeled target data. A common assumption is that the target outcome model aligns partially with source models, enabling knowledge transfer. For example, recent frameworks employ selective parameter reduction to identify transferable sources and sparse or ridge shrinkage to leverage their knowledge (Bastani,, 2020; Li etal.,, 2021; Tian and Feng,, 2023). Subsequent works tackle covariate distribution mismatches and semi-supervised scenarios, enhancing robustness when labeled target data is scarce (Cai etal.,, 2024; He etal.,, 2024; Zhou etal.,, 2024). Further innovations include geometric or profile-based adaptations, where the target model is represented as a weighted combination of source coefficients (Gu etal.,, 2024; Lin etal.,, 2024).

Methods	Ridge-type	Lasso-type	ScaleAdjustment	Continuousoutcome	Binaryoutcome	PartialTransfer	Multi-Sourceensemble
KG-WDRO	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Bastani, (2020)	$\checkmark$			$\checkmark$
Li etal., (2021)		$\checkmark$		$\checkmark$
Tian and Feng, (2023)		$\checkmark$		$\checkmark$	$\checkmark$
Gu etal., (2024)	$\checkmark$		$\checkmark$	$\checkmark$			$\checkmark$
Lin etal., (2024)		$\checkmark$	$\checkmark$	$\checkmark$			$\checkmark$

1.2 Our Contribution

Our contributions are fourfold. Framework: We introduce KG-WDRO, a principled and flexible framework that integrates prior knowledge into WDRO for linear regression and binary classification. This framework mitigates the conservativeness of standard WDRO, enables automated covariate scaling adjustments, and prevents negative transfer. Theory: We establish the equivalence between KG-WDRO and shrinkage-based estimation methods, offering a novel perspective that unifies and interprets a broad range of knowledge transfer learning approaches through the lens of distributional robustness. Table 1 provides an overview of them, highlighting their key capabilities and advantages and comparing them with our framework. Technicalities: Leveraging Toland’s Duality (Theorem A6), we reformulate the innermost maximization in WDRO’s strong duality (Proposition 1) into a univariate optimization problem (B). This reformulation enhances tractability while accommodating more general cost functions. Empirical Validation: Through extensive experiments, we demonstrate the effectiveness of KG-WDRO in improving small-sample transfer learning.

Below is an overview of our main results for the linear regression case.

Example 1.

Suppose $\theta$ is an accessible prior predictor for a linear model parameterized with $\beta$ . We show that the shrinkage-based transfer-learning regression problem, which estimates a target predictor $\beta$ by solving

\inf_{\beta,\kappa}\|\mathbf{y}-\mathbf{X}\beta\|_{2}+\sqrt{\delta}\|\beta-%\kappa\theta\|_{p},

can be interpreted as a Wasserstein distributionally robust optimization (WDRO) problem of the form (WDRO), where the loss function is least squares, $\ell(X,Y;\beta)=(Y-\beta^{\scriptscriptstyle\sf T}X)^{2},$ and the ambiguity set $\mathcal{B}_{\delta}(\mathbb{P}_{N};c_{2,\infty})$ is defined as a ball around the empirical measure. The cost function $c_{2,\infty}$ augments the standard transport cost by the constraint $x^{\scriptscriptstyle\sf T}\theta=u^{\scriptscriptstyle\sf T}\theta$ so that

\displaystyle c_{2,\infty}\big{(}(x,y),(u,v)\big{)}=\|x-u\|_{q}^{2}+\infty%\cdot|y-v|+\infty\cdot|(x-u)^{\scriptscriptstyle\sf T}\theta|.

This establishes a distributionally robust optimization (DRO) perspective on a broad class of transfer-learning methods as will be discussed in Section 3.

1.3 Notations & Organizations

We summarize the mathematical notations used in this work. The positive integers $N$ , $M$ , and $d$ denote, respectively, the target sample size, the number of sources, and the dimension of the support of the covariate $X$ . The integers $p$ and $q\in[1,\infty]$ are reserved for pairs of Hölder conjugates, satisfying $p^{-1}+q^{-1}=1$ for $p,q\in(1,\infty)$ , as well as the pair $1$ and $\infty$ . For a distribution $\mathbb{P}$ supported on the Euclidean space $\mathbb{R}^{d}$ , we use $\mathbb{P}_{N}$ to denote the empirical measure of $\mathbb{P}$ with sample size $N$ . In modeling the target-covariate relationship, the distribution is often factorized as $\mathbb{P}=\mathbb{P}^{Y|X}\times\mathbb{P}^{X}$ . For a vector $v\in\mathbb{R}^{d}$ , $\|v\|_{p}$ denotes the $p$ -norm, where $p\in[1,\infty]$ , and $v^{\scriptscriptstyle\sf T}$ denote the transpose of $v$ . For any two vectors $u,v\in\mathbb{R}^{d}$ , the notation $\cos{(u,v)}$ denote the cosine of the angle between $u$ and $v$ , calculated by $\cos{(u,v)}\|u\|_{2}\|v\|_{2}=u^{\scriptscriptstyle\sf T}v$ . All vectors are assumed to be column vectors. Other specialized notations are defined in context as needed.

The remainder of the paper is organized as follows. Section 2 provides a review of the WDRO framework, including the strong duality result. In Section 3, we introduce our KG-WDRO framework and demonstrate its equivalence to shrinkage-based estimations in both linear regression and binary classification. Section 4 presents comprehensive results from our numerical simulations. All proofs and detailed descriptions of the numerical simulation setups are provided in the appendix.

2 Preliminaries

We first begin with a short overview of the distributionally robust framework on statistical learning.

2.1 Optimal Transport Cost

Let $\mathbb{P}$ and $\mathbb{Q}$ denote two probability distributions supported on $\mathbb{R}^{d}$ , and we use $\mathcal{P}(\mathbb{R}^{d}\times\mathbb{R}^{d})$ to label the set of all probability measures on the product space $\mathbb{R}^{d}\times\mathbb{R}^{d}$ . We say that an element $\pi\in\mathcal{P}(\mathbb{R}^{d}\times\mathbb{R}^{d})$ has first marginal $\mathbb{P}$ and second marginal $\mathbb{Q}$ if

\pi(A\times\mathbb{R}^{d})=\mathbb{P}(A),\,\,\,\pi(\mathbb{R}^{d}\times B)=%\mathbb{Q}(B),

for all Borel measurable sets $A,B\in\mathbb{R}^{d}$ . The class of all such measures $\pi$ is collected as $\Pi(\mathbb{P},\mathbb{Q})$ , and is called the set of transport plans, which is always non-empty. Choose a non-negative, lower semi-continuous function $c:\mathbb{R}^{d}\times\mathbb{R}^{d}\to[0,\infty]$ such that $c(u,v)=0$ whenever $u=v$ , then the Kantorovich’s formulation of optimal transport is defined as

\mathcal{D}_{c}(\mathbb{P},\mathbb{Q})\coloneqq\inf_{\pi\in\Pi(\mathbb{P},%\mathbb{Q})}\mathbb{E}_{\pi}\left[c(U,V)\right].

It is well-known that (Villani,, 2009, Theorem 4.1) there exists an optimal coupling $\pi^{\dagger}$ that solves the Kantorovich’s problem $\inf_{\pi\in\Pi(\mathbb{P},\mathbb{Q})}\mathbb{E}_{\pi}\left[c(U,V)\right]$ . Intuitively, we may think of the value $c(u,v)$ as the cost of transferring one unit of mass from $u\in\mathbb{R}^{d}$ to $v\in\mathbb{R}^{d}$ , then $\mathbb{E}_{\pi}[c(U,V)]$ gives the average cost of transferring under the plan $\pi$ . The optimal transport cost $\mathcal{D}_{c}(\mathbb{P},\mathbb{Q})$ gives a measure of discrepancy between probability distributions on $\mathbb{R}^{d}$ .

If $c(u,v)$ defines a metric on $\mathbb{R}^{d}$ , then for any $p\in[1,\infty)$ the optimal transport cost,

\mathcal{D}_{c}^{1/p}(\mathbb{P},\mathbb{Q})\coloneqq\left(\inf_{\pi\in\Pi(%\mathbb{P},\mathbb{Q})}\mathbb{E}_{\pi}\left[c(U,V)^{p}\right]\right)^{1/p},

defines a metric between probability distributions and metrizes weak convergence under moment assumptions. It is called the $p$ -Wasserstein distance. We direct the interested readers to (Villani,, 2009, Chapter 6) for more details. It is worth mentioning that none of our judiciously chosen cost functions qualify as metrics on the support of the data.

2.2 Distributionally Robust Optimization

In standard statistical learning framework, one generally assumes that the target-covariate pair $(X,Y)\in\mathbb{R}^{d}\times\mathbb{R}\cong\mathbb{R}^{d+1}$ follows a data-generating distribution $\mathbb{P}\coloneqq\mathbb{P}_{X,Y}$ on the support $\mathbb{R}^{d+1}$ . One then seeks to find a ‘best’ parameter $\beta$ that relates $Y$ to $X$ through a parameterized model by solving the stochastic optimization,

\inf_{\beta}\mathbb{E}_{\mathbb{P}}\left[\ell{(X,Y;\beta})\right].

The loss function $\ell{(x,y;\beta)}$ provides a quantification of the goodness-of-fit in the parameter $\beta$ given the realized observation $(x,y)$ . Since only samples $\{(x_{i},y_{i})\}_{i=1,\ldots,N}$ are observed, we can typically only solve the empirical objective,

\inf_{\beta}\mathbb{E}_{\mathbb{P}_{N}}[\ell(X,Y;\beta)]=\inf_{\beta}\dfrac{1}%{N}\sum_{i=1}^{N}\ell(x_{i},y_{i};\beta).

Therefore the distribution $\mathbb{P}$ that underlies the data-generating mechanism is uncertain to the decision-maker. This motivated the distributionally robust optimization (DRO) framework, which entails solving the following minimax stochastic program:

\inf_{\beta}\sup_{\mathbb{P}\in\mathcal{P}_{\rm amb}}\mathbb{E}_{\mathbb{P}}[%\ell(X,Y;\beta)],

where the ambiguity set $\mathcal{P}_{\rm amb}$ represents a class of probability measures supported on $\mathbb{R}^{d+1}$ that are candidates to the true data-generating distributions. In Wasserstein-DRO, the ambiguity set is constructed by forming a ‘ $\delta$ -ball’ around the canonical empirical measure $\mathbb{P}_{N}$ associated to the decision-maker-defined transport cost $c$ , i.e. we let the ambiguity set $\mathcal{P}_{\rm amb}$ be chosen as:

\displaystyle\mathcal{B}_{\delta}(\mathbb{P}_{N};c)\coloneqq\{\mathbb{P}\in%\mathcal{P}(\mathbb{R}^{d+1})|\mathcal{D}_{c}(\mathbb{P},\mathbb{P}_{N})\leq%\delta\}.

(WDRO)

This ambiguity set captures probability measures that are close to the observed empirical measure in the transport cost $\mathcal{D}_{c}$ , which may be taken as a class of candidates of measures perturbed from $\mathbb{P}_{N}$ . The solution $\beta_{\rm DRO}$ to (2.2) that solves the worst case expected loss should perform well over the entire set of perturbations in the ambiguity set. This is in contrast to $\beta_{\rm ERM}$ that solves (2.2) only performs well on the training samples. This adds a robustness layer to the WDRO problem (WDRO). For a comprehensive overview of different constructions of ambiguity sets, we direct the interested reader to (Kuhn etal.,, 2024, Section 2).

2.3 Strong Duality of Wasserstein DRO

The Wasserstein DRO problem involves an inner maximization over an infinite-dimensional set, which appears computationally intractable. However, the distribution $\mathbb{P}_{n}$ is discrete, strong duality of the Wasserstein DRO reformulates it as a simple univariate optimization.

Proposition 1 (Strong Duality, (Blanchet etal., 2019a, , Proposition 1)).

Let $c:\mathbb{R}^{d+1}\times\mathbb{R}^{d+1}\to[0,\infty]$ be a lower semi-continuous cost function satisfying $c\big{(}(x,y),(u,v)\big{)}=0$ whenever $(x,y)=(u,v)$ . Then the distributionally robust regression problem

\inf_{\beta\in\mathbb{R}^{d}}\sup_{\mathbb{P}:\mathcal{B}_{\delta}(\mathbb{P}_%{N})}\mathbb{E}_{\mathbb{P}}\left[\ell(X,Y;\beta)\right],

is equivalent to,

\inf_{\beta\in\mathbb{R}^{d}}\inf_{\gamma\geq 0}\left\{\gamma\delta+\dfrac{1}{%n}\sum_{i=1}^{N}\phi_{\gamma}(x_{i},y_{i};\beta)\right\},

where $\phi_{\gamma}(x_{i},y_{i};\beta)$ is given by,

\sup_{(u,v)\in\mathbb{R}^{d+1}}\big{\{}\ell(u,v;\beta)-\gamma c\big{(}(u,v),(x%_{i},y_{i})\big{)}\big{\}}.

For more general results, see (Blanchet and Murthy,, 2019, Theorem 1) and (Gao etal.,, 2022, Section 2). The exchangeability of $\sup$ and $\inf$ in Wasserstein-DRO is also established by (Blanchet etal., 2019a, , Lemma 1).

3 Knowledge-Guided Wasserstein DRO

In this section, we propose new cost functions for the Wasserstein DRO framework that leverage prior knowledge for transfer learning. For linear regression and binary classification, these cost functions act as regularizers, encouraging collinearity with prior knowledge.

3.1 Knowledge-Guided Transport Cost

It is shown in (Blanchet etal., 2019a, , Theorem 1) that using the squared $q$ -norm on the covariates as the cost function

c_{2}\big{(}(x,y),(u,v)\big{)}=\|x-u\|_{q}^{2}+\infty\cdot|y-v|,

(1)

equates Wasserstein distributionally robust linear regression with $p$ -norm regularization on the root mean squared error (RMSE). The cost function $c_{2}$ perturbs only the observed covariates $\{x_{i}\}_{i=1}^{N}$ , while keeping the observed targets $\{y_{i}\}_{i=1}^{N}$ fixed. Keeping the observed target $Y$ as fixed often leads to more mathematically tractable reformulation, another intuition is that we trust the mechanism by which the target $Y$ is generated once $X$ is known.

In the presence of prior knowledge $\theta$ that may aid in inferring $\beta$ , we aim to control the extent of perturbation along the direction of $\theta$ .

Specifically, we constrain the size of the prediction discrepancy $\theta^{\scriptscriptstyle\sf T}x-\theta^{\scriptscriptstyle\sf T}u=\theta^{%\scriptscriptstyle\sf T}\Delta$ , where $\Delta\coloneqq x-u$ .To achieve this goal, we henceforth augment the cost function $c_{2}$ with an additional penalty term that accounts for the size of the perturbation in the direction of $\theta$ :

\displaystyle c_{2,\lambda}\big{(}(x,y),(u,v)\big{)}=\|\Delta\|_{q}^{2}+\infty%\cdot|y-v|+\lambda h(|\theta^{\scriptscriptstyle\sf T}\Delta|),

(2)

where $\lambda>0$ and $h(x):\mathbb{R}\to\mathbb{R}^{+}\cup\{0\}$ is a non-negative, monotone increasing function of $|x|$ such that $h(0)=0$ . Recall that in the cost function $c_{2}(\cdot)$ , the targets $y$ remain fixed. Intuitively, the new cost function (2) encourages the Wasserstein ambiguity set to include distributions whose marginals in $X$ generate predictions that align with the data based on the prior predictor $\theta$ . The parameter $\lambda$ controls the level of confidence in the prior knowledge. We call this kind of cost functions knowledge-guided. Since $c_{2,\lambda}$ upper bounds the cost function $c_{2}$ , we have $\mathcal{B}_{\delta}(\mathbb{P}^{X}_{N};c_{2,\lambda_{2}})\subseteq\mathcal{B}%_{\delta}(\mathbb{P}^{X}_{N};c_{2,\lambda_{1}})\subseteq\mathcal{B}_{\delta}(%\mathbb{P}^{X}_{N};c_{2})$ whenever $\lambda_{2}>\lambda_{1}$ .

The corresponding optimal transport problem given by:

\inf_{\pi\in\Pi(\mathbb{Q}^{X},\mathbb{P}_{N}^{X})}\mathbb{E}_{\pi}[c_{2,%\lambda}(X,U)],

can also be expressed as:

\inf_{\pi\in\Pi(\mathbb{Q}^{X},\mathbb{P}_{N}^{X})}\mathbb{E}_{\pi}[c_{2}(X,U)%]+\lambda\mathbb{E}_{\pi}[h(|\theta^{\scriptscriptstyle\sf T}\Delta|)].

This formulation regularizes the original optimal transport problem by penalizing large values of the expectation $\mathbb{E}_{\pi}[h(|\theta^{\scriptscriptstyle\sf T}\Delta|)]$ .

For any user-defined function $h$ that measures the discrepancy in generalization with respect to the prior knowledge $\theta$ , we refer to it as weak-transferring of knowledge if $\lambda<+\infty$ , and strong-transferring of knowledge if $\lambda=+\infty$ . In the case of strong-transferring, to ensure the finiteness of the optimal transport problem, the minimizing transport plan $\pi^{\dagger}$ must satisfy the orthogonality condition $\theta^{\scriptscriptstyle\sf T}\Delta=0$ , $\pi^{\dagger}$ -almost surely. Consequently, the value of $\theta^{\scriptscriptstyle\sf T}X$ remains unchanged after perturbing $\mathbb{P}_{N}^{X}$ within $\mathcal{B}_{\delta}(\mathbb{P}_{N}^{X};c_{2,\infty})$ . As a result, this should promote $\beta_{\mathrm{DRO}}\to\theta$ as $\delta\to\infty$ .

Remark 1.

The above framework extends to incorporate multi-sites prior knowledge, meaning that instead of a single prior knowledge coefficient $\theta_{1}$ , we consider a set of coefficients $\{\theta_{1},\theta_{2},\ldots,\theta_{M}\}$ . Let $\Theta\coloneqq\operatorname{span}\{\theta_{1},\theta_{2},\ldots,\theta_{M}\}$ represent the linear span of these prior knowledge coefficients. In the case of strong-transferring, we must ensure that $\operatorname{rank}(\Theta)<d$ ; otherwise, the set of orthogonality conditions $\{\theta_{m}^{\scriptscriptstyle\sf T}\Delta=0;m\in[M]\}$ would imply that the perturbation $\Delta$ is identically zero ( $\Delta=\mathbf{0}$ ). This would render the ambiguity set redundant and reduce the WDRO problem (WDRO) to the ERM problem (2.2). This result is confirmed by the statements of Theorems 1 and 3.

3.2 Linear Regression

We begin by examining the WDRO problem (WDRO) for linear regression within the strong-transferring domain. Following this, we present a specific case within the weak-transferring domain. Let $\Theta\coloneqq\operatorname{span}\{\theta_{1},\ldots,\theta_{M}\}$ represent the linear span of the prior knowledge.

3.2.1 Strong-Transferring

Define the cost function $c_{2,\infty}\big{(}(x,y),(u,v)\big{)}\coloneqq\|x-u\|_{q}^{2}+\infty\cdot|y-v|%+\infty\cdot|\theta_{1}^{\scriptscriptstyle\sf T}x-\theta_{1}^{%\scriptscriptstyle\sf T}u|+\ldots+\infty\cdot|\theta_{M}^{\scriptscriptstyle%\sf T}x-\theta_{M}^{\scriptscriptstyle\sf T}u|$ , and for a set of observed samples $\{(x_{i},y_{i})\}_{i\in[N]}$ , we use ${\rm MSE}_{N}(\beta)\coloneqq N^{-1}\sum_{i=1}^{N}(y_{i}-\beta^{%\scriptscriptstyle\sf T}x_{i})^{2}$ . Without making any additional distributional assumptions on $(X,Y)$ , we obtain the following finite-dimensional representation.

Theorem 1 (Linear Regression with Strong-Transferring).

Consider the least-squared loss $\ell(X,Y;\beta)=(Y-\beta^{\scriptscriptstyle\sf T}X)^{2}$ , then for any $q\in[1,\infty]$ we have

		$\displaystyle\inf_{\beta\in\mathbb{R}^{d}}\sup_{\mathbb{P}:\mathcal{B}_{\delta%}(\mathbb{P}_{N};c_{2,\infty})}\mathbb{E}_{\mathbb{P}}\left[(Y-\beta^{%\scriptscriptstyle\sf T}X)^{2}\right]$
	$\displaystyle=$	$\displaystyle\inf_{\beta\in\mathbb{R}^{d},\vartheta\in\Theta}\left(\sqrt{{\rmMSE%}_{N}(\beta)}+\sqrt{\delta}\\|\beta-\vartheta\\|_{p}\right)^{2},$

where $p$ is such that $p^{-1}+q^{-1}=1$ .

From the above result, we observe that the knowledge-guided WDRO problem for linear regression is equivalent to regularizing the RMSE with a $p$ -norm distance to the linear span $\Theta$ . The regularization parameter is entirely determined by the size (or radius) of the Wasserstein ambiguity set. Importantly, the penalty term focuses on the collinearity with the prior knowledge rather than their algebraic difference or angular proximity.

Consider the case when there is only a single prior knowledge $\theta_{1}$ , the penalty term does not constrain the solution $\beta_{\rm DRO}$ to be close to $\theta_{1}$ , but rather to $\kappa\cdot\theta_{1}$ for some $\kappa\in\mathbb{R}$ to be optimized. Consequently, this knowledge transfer automatically robustify solution against scaling of covariates. Furthermore, it can prevent negative transfer by adapting its sign to be positively correlated with $\beta^{*}$ , which is the solution to population objective (2.2). When $\delta\to\infty$ , the penalty term becomes dominant, forcing $\beta$ to lie in $\Theta$ for any $p\geq 1$ . This reduces the WDRO problem to a simple constrained regression problem,

\inf_{\beta\in\Theta}{\rm MSE}_{N}(\beta),

reflecting the complete reliance on the prior knowledge and prevents excessive shrinkage towards the null estimator.

Remark 2.

We now discuss two special cases of the penalty term, $p=2$ (ridge-type regularization) and $p=1$ (lasso-type regularization). For simplicity, we consider the case of a single prior knowledge vector $\theta$ .

Ridge-type. The penalty term can be explicitly calculated as

\min_{\kappa\in\mathbb{R}}\|\beta-\kappa\theta\|_{2}=\left\|\beta-\dfrac{\beta%^{\scriptscriptstyle\sf T}\theta}{\|\theta\|_{2}^{2}}\theta\right\|_{2}=\|%\beta^{\perp\theta}\|_{2},

where $\beta^{\perp\theta}$ is the component of $\beta$ orthogonal to $\theta$ . This penalty term shrinks distance to the line in the direction of $\theta$ . Furthermore, note that

\|\beta^{\perp\theta}\|_{2}=\|\beta\|_{2}\sin(\beta,\theta)=\|\beta\|_{2}\sqrt%{1-\cos^{2}(\beta,\theta)},

which represents a trade-off between the magnitude of $\beta$ and its angular proximity to the prior knowledge $\theta$ . This trade-off is illustrated in the leftmost figure of Fig.1, drawing the feasibility set of the regularization as a constraint. This regularization is closely related but different to the one proposed in Gu etal., (2024), where they penalize large values of a computational relaxation of $\sin{(\beta,\theta)}$ .

Lasso-type. When the prior knowledge $\theta$ is sparse, the penalty term $\min_{\kappa}\|\beta-\kappa\theta\|_{1}$ promotes sparse representation learning. Consider a simple example where the dimension is $d=3$ and $\theta=(1,0,0)^{\scriptscriptstyle\sf T}$ . In this case, we have:

	$\displaystyle\min_{\kappa}\\|\beta-\kappa\theta\\|_{1}$	$\displaystyle=\min_{\kappa}\big{(}\|\beta_{1}-\kappa\|+\|\beta_{2}\|+\|\beta_{3}\|%\big{)}$
		$\displaystyle=\|\beta_{2}\|+\|\beta_{3}\|\eqqcolon\\|\beta_{\shortminus 1}\\|_{1},$

where $\beta_{\shortminus 1}=(\beta_{2},\beta_{3})^{\scriptscriptstyle\sf T}$ . This formulation enforces sparsity only on the last two components of $\beta$ , reflecting the sparsity pattern of $\theta$ .

3.2.2 Weak Transferring

For the special case of $q=p=2$ , we define the weak-transferring cost function $c_{2,\lambda}\big{(}(x,y),(u,v)\big{)}=\|x-u\|_{2}^{2}+\lambda(\theta^{%\scriptscriptstyle\sf T}x-\theta^{\scriptscriptstyle\sf T}u)^{2}+\infty\cdot|y%-v|$ with $0<\lambda<+\infty$ . Here, we select $h(x)=x^{2}$ as the user-defined function on controlling the size of perturbation in $\theta$ . For simplicity, we consider a single prior knowledge vector $\theta$ in this setup. This result can be straightforwardly extended to a multi-source setup with different values of $\lambda$ ’s.

Theorem 2 (Linear Regression with Weak Transferring).

Consider the least-squared loss $\ell(X,Y;\beta)=(Y-\beta^{\scriptscriptstyle\sf T}X)^{2}$ , then for $p=q=2$ we have

		$\displaystyle\inf_{\beta\in\mathbb{R}^{d}}\sup_{\mathbb{P}:\mathcal{B}_{\delta%}(\mathbb{P}_{N};c_{2,\lambda})}\mathbb{E}_{\mathbb{P}}\left[(Y-\beta^{%\scriptscriptstyle\sf T}X)^{2}\right]$
	$\displaystyle=$	$\displaystyle\inf_{\beta\in\mathbb{R}^{d}}\left(\sqrt{{\rm MSE}_{N}(\beta)}+%\sqrt{\delta}\left\\|\beta\right\\|_{\Psi_{\lambda^{\shortminus 1}}}\right)^{2}.$

With $\Psi_{\lambda}=I_{d}-\dfrac{1}{\|\theta\|_{2}^{2}+\lambda}\theta\theta^{%\scriptscriptstyle\sf T}$ and $\|\beta\|_{\Psi_{\lambda}}^{2}=\beta^{\scriptscriptstyle\sf T}\Psi_{\lambda}\beta$ .

Write $P_{\lambda}={\theta\theta^{\scriptscriptstyle\sf T}}/({\lVert\theta\|_{2}^{2}+%\lambda})$ , we note that as $\lambda\to\infty$ , we have $P_{\lambda^{\shortminus 1}}\to P_{0}={\theta\theta^{\scriptscriptstyle\sf T}}/%{\|\theta\|_{2}^{2}}$ recovering the projection matrix onto the prior knowledge $\theta$ . Consequently, $\|\beta\|_{\Psi_{\lambda^{\shortminus 1}}}\to\|\beta^{\perp\theta}\|_{2}$ .

We observe that the action

P_{\lambda^{\shortminus 1}}\beta=\dfrac{\beta^{\scriptscriptstyle\sf T}\theta}%{\|\theta\|_{2}^{2}+\lambda^{-1}}\theta

is exactly the ridge regression of $\beta$ onto $\theta$ with a regularization parameter $\lambda^{-1}$ . Thus, the finiteness of $\lambda$ , which can reflect a caution in the prior knowledge $\theta$ , induces a shrinkage effect on the component of $\beta$ explainable by $\theta$ in the dot product geometry. Since $\Psi_{\lambda}\succ I_{d}-P$ , we have $\|\beta\|_{\Psi_{\lambda^{\shortminus 1}}}>\|\beta^{\perp\theta}\|_{2}$ for any finite $\lambda>0$ , this implies the inclusion of feasibility set

\{\beta:\|\beta\|_{\Psi_{\lambda^{\shortminus 1}}}\leq K\}\subset\{\beta:\|%\beta^{\perp\theta}\|_{2}\leq K\},

as plotted in Fig.1 for an illustration on $\mathbb{R}^{2}$ . The contour $\{\beta\in\mathbb{R}^{2}:\|\beta\|_{\Psi_{\lambda^{\shortminus 1}}}=K\}$ forms an ellipse centered around the origin $\boldsymbol{0}$ . The ellipse has a major axis of half length $K\sqrt{\dfrac{\|\theta\|_{2}^{2}+\lambda^{-1}}{\lambda^{-1}}}$ aligned with the direction of $\theta$ , and a minor axis with half-length $K$ aligned with the direction of $\theta^{\perp}$ . As $\lambda\to 0$ , representing no-confidence in $\theta$ , the half-length of the major axis converges to $K$ , resulting in a perfect circle as in ridge regression.

The two-dimensional hyper-parameters $(\delta,\lambda^{-1})$ enable the use of data-driven methods, such as grid-search cross-validation, for hyper-parameter tuning. Unlike the strong-transferring domain, the inclusion of $\lambda^{-1}$ allows the data to self-determine the informativeness of the source samples.

3.3 Binary Classifications

In this section, we focus on the context of binary classification, where the goal is to predict the discrete label $Y\in\{-1,1\}$ based on the covariates $X\in\mathbb{R}^{d}$ . Unlike the previous section, we use the $q$ -norm, rather than its square, to account for distributional ambiguity in the covariate distribution. Define the strong-transferring cost function $c_{1,\infty}\big{(}(x,y),(u,v)\big{)}\coloneqq\|x-u\|_{q}+\infty\cdot|y-v|+%\infty\cdot|\theta_{1}^{\scriptscriptstyle\sf T}x-\theta_{1}^{%\scriptscriptstyle\sf T}u|+\ldots+\infty\cdot|\theta_{M}^{\scriptscriptstyle%\sf T}x-\theta_{M}^{\scriptscriptstyle\sf T}u|.$ We consider two loss functions here. The logistic loss function is given by

\ell(X,Y;\beta)=\log{\left(1+e^{-Y\beta^{\scriptscriptstyle\sf T}X}\right)},

which is the negative log-likelihood of the model that postulates

\log{\dfrac{\mathbb{P}(Y=1|X=x)}{\mathbb{P}(Y=-1|X=x)}}=\beta^{*}{{}^{%\scriptscriptstyle\sf T}}x.

The hinge loss is given by

\ell{(X,Y;\beta)}=(1-Y\beta^{\scriptscriptstyle\sf T}X)^{+},

which is typically used for training classifiers that look for ‘maximum-margins’ in class boundaries, most notably support vector machines.

Suppose $Y\in\{-1,1\}$ is binary and without any distributional assumptions on $X$ , we have the following result which recovers regularized logistic regressions and support vector machines.

Theorem 3 (Binary Classification with Strong Transferring).

Suppose the loss function $\ell(X,Y;\beta)$ is either the logistic loss $\log{\left(1+e^{-Y\beta^{\scriptscriptstyle\sf T}X}\right)}$ or the hinge loss $(1-Y\beta^{\scriptscriptstyle\sf T}X)^{+}$ , then for any $q\in[1,\infty]$ we have

		$\displaystyle\inf_{\beta\in\mathbb{R}^{d}}\sup_{\mathbb{P}:\mathcal{B}_{\delta%}(\mathbb{P}_{N};c_{1,\infty})}\mathbb{E}_{\mathbb{P}}\left[\ell(X,Y;\beta)\right]$
	$\displaystyle=$	$\displaystyle\inf_{\beta\in\mathbb{R}^{d},\vartheta\in\Theta}\dfrac{1}{N}\sum_%{i=1}^{N}\ell(x_{i},y_{i};\beta)+\delta\\|\beta-\vartheta\\|_{p},$

where $p$ is such that $p^{-1}+q^{-1}=1$ .

3.4 Sub-Coefficient-Vector Transferring

In this subsection, we generalize the statements of Theorems 1 and 3 for $p=2$ to arbitrary norms induced by positive-definite quadratic forms. Let $\Lambda\in\mathbb{R}^{d\times d}$ be a positive-definite symmetric matrix. The norm $\|x\|_{\Lambda}=\sqrt{x^{\scriptscriptstyle\sf T}\Lambda x}$ induces a metric on $\mathbb{R}^{d}$ , defined as $d_{\Lambda}(x,u)=\|x-u\|_{\Lambda}$ , known as the Mahalanobis distance. Since $\Lambda$ is positive definite, it admits a decomposition $\Lambda=\Gamma^{\scriptscriptstyle\sf T}\Gamma$ with $\Gamma$ invertible, and the norm $\|x\|_{\Lambda}=\|\Gamma x\|_{2}$ measures length in the geometry distorted by $\Gamma$ . By (Blanchet etal., 2019b, , Lemma 1), the dual norm of $\|\cdot\|_{\Lambda}$ is $\|\cdot\|_{\Lambda^{-1}}$ . Using Proposition A4, the statements of Theorems 1 and 3 can be easily generalized. Define the space of positive-definite symmetric matrices as $\mathbb{S}_{+}^{d\times d}$ and the cost function: $c_{2,\infty}^{\Lambda}\big{(}(x,y),(u,v)\big{)}\coloneqq\|x-u\|_{\Lambda}^{2}+%\infty\cdot|y-v|+\infty\cdot\sum_{m=1}^{M}|\theta_{m}^{\scriptscriptstyle\sf T%}x-\theta_{m}^{\scriptscriptstyle\sf T}u|$ .

Corollary 1 (Theorem 1).

For the least-squares loss $\ell(X,Y;\beta)=(Y-\beta^{\scriptscriptstyle\sf T}X)^{2}$ and any $\Lambda\in\mathbb{S}_{+}^{d\times d}$ :

		$\displaystyle\inf_{\beta\in\mathbb{R}^{d}}\sup_{\mathbb{P}:\mathcal{B}_{\delta%}(\mathbb{P}_{N};c_{2,\infty}^{\Lambda})}\mathbb{E}_{\mathbb{P}}\left[(Y-\beta%^{\scriptscriptstyle\sf T}X)^{2}\right]$
	$\displaystyle=$	$\displaystyle\inf_{\beta\in\mathbb{R}^{d},\vartheta\in\Theta}\left(\sqrt{{\rmMSE%}_{N}(\beta)}+\sqrt{\delta}\\|\beta-\vartheta\\|_{\Lambda^{-1}}\right)^{2}.$

This formulation enables the use of metric learning methods to determine $\Lambda$ directly from the data, as detailed in Blanchet etal., 2019b . For example, if the two-dimensional prior $\theta=[\theta_{1},\theta_{2}]$ is known to primarily influence the first component of the truth $\beta=[\theta_{1}+\epsilon,\beta_{2}]$ , we can select $\Lambda=\mathrm{diag}(d_{1},d_{2})$ with $d_{1}\ll d_{2}$ . This imposes a weaker penalty on perturbations in the first direction, resulting in a weighted penalty term: $\min_{\kappa}\big{(}{(\beta_{1}-\kappa\theta_{1})}/{d_{1}}+{(\beta_{2}-\kappa%\theta_{2})}/{d_{2}}\big{)}$ , which prioritizes aligning $\beta_{1}$ with $\theta_{1}$ , while $\beta_{2}$ is determined more flexibly based on the data. We call this sub-coefficient-vector transferring, or the ability to partially transfer prior knowledge. A similar corollary applies to Theorem 3, as stated in Corollary 2.

Finally, we again draw the reader’s attention to Table 1, which compares several transfer learning methods discussed in Section 1.1.2. Notably, our proposed KG-WDRO framework brings together a broad range of desirable capabilities within a single, unified approach to transfer learning.

4 Numerical Results

Knowledge-Guided Wasserstein Distributionally Robust Optimization (4)

In this section, we present numerical simulations to validate the effectiveness of the proposed KG-WDRO method. We compare learners across different settings, including high-dimensional sparse models, correlated covariates, and multi-source prior knowledge, for either linear regression or binary classification tasks. Performance is evaluated using out-of-sample classification error for binary classifiers and out-of-sample $R^{2}$ for linear regressors.

For the single-source experiments, target-source coefficient pairs $(\beta,\theta)$ are generated from a multivariate normal distribution:

(\beta_{j},\theta_{j})\sim N\left(\begin{pmatrix}0\\0\end{pmatrix},\begin{pmatrix}\sigma^{2}&\rho\sigma^{2}\\\rho\sigma^{2}&\sigma^{2}\end{pmatrix}\right),

where $\rho$ is the correlation between $\beta$ and $\theta$ , and the expected length of $\theta$ is approximately $\sigma\sqrt{d-0.5}$ . We scale $\beta$ as $\beta\leftarrow s\beta$ with $s\in(0,1]$ to study the stabilizing effects of strong prior knowledge in small-sample settings. The dimension-to-sample ratio $d/N$ is varied by fixing $d$ and increasing $N$ . Performance is averaged over 100 simulations. Each dataset consists of three parts: data = (train, val, test). The (train, val) pair shares the same size, and hyperparameters are selected based on validation performance. The source data contains 800 samples, with source truth $\theta$ estimated accordingly. Out-of-sample performance is measured on the test set of 5000 data points.

4.1 Simulation 1: Logistic with $\ell_{1}$ -Strong Transferring

In the first experiment, we compare two learners for binary classification tasks with high-dimensional sparse coefficients against our proposed KG-WDRO learner, $\beta_{\rm KG}$ , derived using Theorem 3 with $p=1$ . The competing learners are the target-only vanilla WDRO learner $\beta_{\rm WDRO}$ (Blanchet etal., 2019a, , Theorem 2) and $\beta_{\rm TransG}$ , obtained via the $\mathcal{A}$ -Trans-GLM algorithm (Tian and Feng,, 2023, Algorithm 1). The target-source pair $(\beta,\theta)$ is generated using (4) with a dimension of 50 and augmented with 100 zeros for sparsity, resulting in a total dimension of 150. We test six settings, varying the sample size $N\in\{20,50,80\}$ , signal strength $s\in\{0.5,1\}$ , and truth-prior correlation $\rho\in\{0.3,0.5,0.7,0.8,0.9,0.95\}$ .

The comparison between $\beta_{\rm KG}$ and $\beta_{\rm TransG}$ is highly competitive, with $\beta_{\rm KG}$ consistently outperforming $\beta_{\rm TransG}$ by up to $2\%$ in accuracy when the sample size is small ( $N=20$ ) across all values of $\rho$ , as shown in the upper-left plot of Figure 2. In larger sample size scenarios, both learners perform similarly (see Table A1 for detailed results). Both transfer learning methods, $\beta_{\rm KG}$ and $\beta_{\rm TransG}$ , significantly outperform the target-only learner, $\beta_{\rm WDRO}$ .

4.2 Simulation 2: Linear Regression with $\ell_{2}$ -Weak Transferring

In this simulation, we compare two learners on high-dimensional linear regression with correlated covariates against our proposed learners, $\beta_{\rm KGweak}$ (Theorem 2) and $\beta_{\rm KGstrong}$ (Theorem 1), both using $p=2$ . There is no sparsity in the regression coefficients. The competing learners are the target-only vanilla WDRO learner $\beta_{\rm WDRO}$ (Blanchet etal., 2019a, , Theorem 1) and the Trans-Ridge algorithm adapted from (Li etal.,, 2021, Algorithm 1), denoted as $\beta_{\rm TransR}$ . The covariates are fixed at dimension 100, with a pairwise correlation of 0.3. The experiment is conducted across six settings, varying the sample size $N\in\{50,70,90\}$ , signal strength $s\in\{0.8,1\}$ , and truth-prior correlation $\rho\in\{0.3,0.5,0.7,0.8,0.9,0.95\}$ .

As shown in the upper-right plot of Figure 2, the performance of $\beta_{\rm TransR}$ lags significantly behind both $\beta_{\rm KGstrong}$ and $\beta_{\rm KGweak}$ until the correlation $\rho$ becomes sufficiently high. Across all settings, $\beta_{\rm KGstrong}$ and $\beta_{\rm KGweak}$ consistently outperform $\beta_{\rm TransR}$ when $\rho$ is moderate or low, as documented in Table A2. Furthermore, all three transfer learning methods demonstrate superior performance compared to the target-only learner, $\beta_{\rm WDRO}$ .

4.3 Simulation 3: Transfer Learning with Multiple Sites

In the final set of experiments, we validate our methods in a multi-source transfer learning setting with high-dimensional sparse linear regression. The significant components of the three source coefficients are generated using (4) with correlation $\varrho$ and dimension 50, denoted as $\{\theta_{1},\theta_{2},\theta_{3}\}$ . We construct a linear combination, $\theta_{S}=a\theta_{1}+b\theta_{2}+c\theta_{3}$ , and generate $\beta=\rho\theta_{S}+\varepsilon$ , where $\varepsilon\sim N(0,(1-\rho^{2})\mathrm{Var}(\theta_{S}))$ , ensuring $\mathrm{Corr}(\beta,\theta_{S})=\rho$ . $\beta$ is then scaled to match the magnitude of $\theta_{S}$ , and all vectors are augmented with 100 zeros, yielding a total dimension of 150. Our proposed method, $\beta_{\rm KG}$ (Theorem 1, $p=1$ ), is compared against the oracle Trans-Lasso algorithm (Li etal.,, 2021, Algorithm 1) ( $\beta_{\rm TransL}$ ) and the vanilla WDRO learner $\beta_{\rm WDRO}$ . The experiment spans six settings: $[a,b,c]=[1,-0.5,0.2]$ and $[1,1,1]$ , with $\varrho=0.9$ and $0.6$ , respectively. Sample sizes vary in $N\in\{50,60,70\}$ . The truth-prior correlation ranges in $\rho\in\{0.7,0.75,0.8,0.85,0.9,0.95\}$ .

When $[a,b,c]=[1,-0.5,0.2]$ , the contributions of the $\theta$ ’s to the generation of $\theta_{S}$ are unequal. In this case, it is not surprising that $\beta_{\rm KG}$ outperforms $\beta_{\rm TransL}$ , as shown in the bottom-left plot of Figure 2. When $\theta_{S}$ is an equal-weighted average of the $\theta$ ’s ( $[a,b,c]=[1,1,1]$ ), the performance of $\beta_{\rm KG}$ and $\beta_{\rm TransL}$ becomes similar. However, $\beta_{\rm KG}$ still demonstrates superior performance in larger sample sizes and higher correlations, as documented in Table A3.

5 Conclusion

We propose the knowledge-guided Wassersteindistributionally robust optimization (KG-WDRO) framework, which utilizes prior knowledge of predictors to mitigate the over-conservativeness of conventional DRO methods. We establish tractable reformulations and demonstrate their superior performance compared to other methods. For future work, we aim to provide statistical guarantees of our proposed estimators. Furthermore, based on these statistical properties, we plan to develop a principled approach for selecting hyperparameters such as $\delta$ and $\lambda$ .

References

Bastani, (2020)Bastani, H. (2020).Predicting with proxies: Transfer learning in high dimension.Management Science, 67(5):2964–2984.
Belloni etal., (2011)Belloni, A., Chernozhukov, V., and Wang, L. (2011).Square-root lasso: Pivotal recovery of sparse signals via conic programming.Biometrika, 98(4):791–806.
(3)Blanchet, J., Kang, Y., and Murthy, K. (2019a).Robust wasserstein profile inference and applications to machine learning.Journal of Applied Probability, 56(3):830–857.
(4)Blanchet, J., Kang, Y., Murthy, K., and Zhang, F. (2019b).Data-driven optimal transport cost selection for distributionally robust optimization.In 2019 Winter Simulation Conference (WSC), pages 3740–3751.
Blanchet and Murthy, (2019)Blanchet, J. and Murthy, K. (2019).Quantifying distributional model risk via optimal transport.Mathematics of Operations Research, 44(2):565–600.
Blanchet etal., (2021)Blanchet, J., Murthy, K., and Nguyen, V.A. (2021).Statistical analysis of wasserstein distributionally robust estimators.In Tutorials in Operations Research: Emerging optimization methods and modeling techniques with applications, pages 227–254. INFORMS.
Blanchet etal., (2022)Blanchet, J., Murthy, K., and Si, N. (2022).Confidence regions in wasserstein distributionally robust estimation.Biometrika, 109(2):295–315.
Cai etal., (2024)Cai, T., Li, M., and Liu, M. (2024).Semi-supervised triply robust inductive transfer learning.Journal of the American Statistical Association, pages 1–14.
Gao, (2023)Gao, R. (2023).Finite-sample guarantees for wasserstein distributionally robust optimization: Breaking the curse of dimensionality.Operations Research, 71(6):2291–2306.
Gao etal., (2022)Gao, R., Chen, X., and Kleywegt, A.J. (2022).Wasserstein distributionally robust optimization and variation regularization.Operations Research, 72(3):1177–1191.
Gao and Kleywegt, (2023)Gao, R. and Kleywegt, A. (2023).Distributionally robust stochastic optimization with wasserstein distance.Mathematics of Operations Research, 48(2):603–655.
Gu etal., (2024)Gu, T., Han, Y., and Duan, R. (2024).Robust angle-based transfer learning in high dimensions.Journal of the Royal Statistical Society Series B: Statistical Methodology.
He etal., (2024)He, Z., Sun, Y., and Li, R. (2024).Transfusion: Covariate-shift robust transfer learning for high-dimensional regression.In International Conference on Artificial Intelligence and Statistics, pages 703–711. PMLR.
Kuhn etal., (2024)Kuhn, D., Shafiee, S., and Wiesemann, W. (2024).Distributionally robust optimization.arXiv preprint arXiv: 2411.02549.
Li etal., (2021)Li, S., Cai, T.T., and Li, H. (2021).Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality.Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):149–173.
Lin etal., (2024)Lin, Z., Zhao, J., Wang, F., and Wang, H. (2024).Profiled transfer learning for high dimensional linear model.arXiv preprint arXiv:2406.00701.
Liu etal., (2024)Liu, J., Wang, T., Cui, P., and Namkoong, H. (2024).Rethinking distribution shifts: Empirical analysis and inductive modeling for tabular data.arXiv preprint arXiv: 2307.05284.
Luenberger and Ye, (2008)Luenberger, D.G. and Ye, Y. (2008).Linear and Nonlinear Programming.International Series in Operations Research & Management Science. Springer New York, NY.
MohajerinEsfahani and Kuhn, (2018)MohajerinEsfahani, P. and Kuhn, D. (2018).Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations.Mathematical Programming, 171(1):115–166.
Rockafellar, (1970)Rockafellar, R.T. (1970).Convex Analysis.Princeton University Press, Princeton, N.J.
Shafieezadeh-Abadeh etal., (2015)Shafieezadeh-Abadeh, S., Esfahani, P.M., and Kuhn, D. (2015).Distributionally robust logistic regression.In Advances in Neural Information Processing Systems, volume28.
Shafieezadeh-Abadeh etal., (2019)Shafieezadeh-Abadeh, S., Kuhn, D., and Esfahani, P.M. (2019).Regularization via mass transportation.Journal of Machine Learning Research, 20(103):1–68.
Tian and Feng, (2023)Tian, Y. and Feng, Y. (2023).Transfer learning under high-dimensional generalized linear models.Journal of the American Statistical Association, 118(544):2684–2697.
Toland, (1978)Toland, J.F. (1978).Duality in nonconvex optimization.Journal of Mathematical Analysis and Applications, 66(2):399–415.
Toland, (1979)Toland, J.F. (1979).A duality principle for non-convex optimisation and the calculus of variations.Archive for Rational Mechanics and Analysis, 71:41–61.
Villani, (2009)Villani, C. (2009).Optimal Transport: Old and New, volume 338 of Grundlehren der mathematischen Wissenschaften.Springer Berlin, Heidelberg, 1 edition.
Zhou etal., (2024)Zhou, D., Li, M., Cai, T., and Liu, M. (2024).Model-assisted and knowledge-guided transfer regression for the underrepresented population.arXiv preprint arXiv: 2410.06484.

Appendix A Additional Details in Numerical Results

This section provides details to supplement Section 4. We outline the data-generating distributions for all three sets of experiments, the hyperparameter grids, and the learners used to identify prior knowledge. We present the exact numerical results for all three sets of experiments. Recall that the notation $s\in(0,1]$ represents the signal strength of the true parameter $\beta$ , which works by rescaling the magnitude of $\beta$ such that $\beta\leftarrow s\beta$ . The notation $d$ is the dimension of the covariates, and $N$ is the sample size. Finally, the symbol $\rho$ represents the correlation between the true $\beta$ and the prior $\theta$ .

A.1 Simulation Results

A.1.1 Simulation 1: Logistic Regression

Setting		$\rho=0.3$	$\rho=0.5$	$\rho=0.7$	$\rho=0.8$	$\rho=0.9$	$\rho=0.95$	WDRO
$s=1$	KG-WDRO	0.587	0.647	0.714	0.748	0.794	0.817	0.565
$N=20$	Trans-GLM	0.585	0.641	0.702	0.735	0.778	0.800	-
$s=1$	KG-WDRO	0.586	0.647	0.713	0.751	0.797	0.823	0.619
$N=50$	Trans-GLM	0.586	0.645	0.710	0.752	0.792	0.823	-
$s=1$	KG-WDRO	0.583	0.646	0.713	0.751	0.798	0.823	0.654
$N=80$	Trans-GLM	0.584	0.646	0.714	0.755	0.800	0.826	-
$s=0.5$	KG-WDRO	0.581	0.634	0.690	0.721	0.762	0.787	0.549
$N=20$	Trans-GLM	0.579	0.626	0.674	0.708	0.748	0.760	-
$s=0.5$	KG-WDRO	0.580	0.635	0.689	0.728	0.768	0.794	0.588
$N=50$	Trans-GLM	0.579	0.633	0.693	0.723	0.769	0.789	-
$s=0.5$	KG-WDRO	0.581	0.637	0.700	0.732	0.775	0.790	0.617
$N=80$	Trans-GLM	0.581	0.638	0.702	0.737	0.779	0.799	-

A.1.2 Simulation 2: Linear Regression

Setting		$\rho=0.3$	$\rho=0.5$	$\rho=0.7$	$\rho=0.8$	$\rho=0.9$	$\rho=0.95$	WDRO
$s=1$	KG-WDRO (Strong)	0.585	0.645	0.740	0.801	0.870	0.912	0.108
$N=50$	KG-WDRO (Weak)	0.583	0.646	0.741	0.800	0.871	0.910	-
	Trans-Ridge	0.391	0.548	0.706	0.786	0.870	0.915	-
$s=1$	KG-WDRO (Strong)	0.707	0.745	0.803	0.843	0.894	0.924	0.513
$N=70$	KG-WDRO (Weak)	0.704	0.743	0.803	0.842	0.892	0.923	-
	Trans-Ridge	0.599	0.692	0.788	0.838	0.893	0.925	-
$s=1$	KG-WDRO (Strong)	0.806	0.827	0.859	0.881	0.911	0.932	0.758
$N=90$	KG-WDRO (Weak)	0.804	0.825	0.857	0.880	0.910	0.930	-
	Trans-Ridge	0.762	0.802	0.849	0.877	0.910	0.932	-
$s=0.8$	KG-WDRO (Strong)	0.563	0.621	0.716	0.777	0.850	0.894	0.030
$N=50$	KG-WDRO (Weak)	0.561	0.622	0.716	0.777	0.849	0.892	-
	Trans-Ridge	0.213	0.405	0.600	0.700	0.803	0.858	-
$s=0.8$	KG-WDRO (Strong)	0.673	0.713	0.774	0.818	0.872	0.905	0.361
$N=70$	KG-WDRO (Weak)	0.670	0.710	0.774	0.816	0.869	0.903	-
	Trans-Ridge	0.470	0.585	0.704	0.768	0.837	0.875	-
$s=0.8$	KG-WDRO (Strong)	0.768	0.791	0.826	0.851	0.886	0.911	0.703
$N=90$	KG-WDRO (Weak)	0.765	0.788	0.825	0.851	0.885	0.909	-
	Trans-Ridge	0.671	0.724	0.785	0.821	0.863	0.890	-

A.1.3 Simulation 3: Multi-Sites

Here, recall that the notation $\varrho$ denote the correlation of generating the three prior knowledge under the scheme (4).

Setting		$\rho=0.7$	$\rho=0.75$	$\rho=0.8$	$\rho=0.85$	$\rho=0.9$	$\rho=0.95$	WDRO
$[1,-0.5,0.2]$	KG-WDRO	0.560	0.640	0.713	0.783	0.850	0.916	-0.584
$\varrho=0.9,N=50$	Trans-Lasso	0.578	0.625	0.673	0.723	0.767	0.815	-
$[1,-0.5,0.2]$	KG-WDRO	0.674	0.728	0.776	0.825	0.875	0.926	0.027
$\varrho=0.9,N=60$	Trans-Lasso	0.666	0.697	0.732	0.770	0.808	0.850	-
$[1,-0.5,0.2]$	KG-WDRO	0.793	0.820	0.848	0.878	0.907	0.939	0.375
$\varrho=0.9,N=70$	Trans-Lasso	0.756	0.779	0.805	0.832	0.857	0.882	-
$[1,1,1]$	KG-WDRO	0.565	0.642	0.715	0.785	0.852	0.916	-2.837
$\varrho=0.6,N=50$	Trans-Lasso	0.628	0.680	0.735	0.790	0.838	0.889	-
$[1,1,1]$	KG-WDRO	0.673	0.729	0.778	0.829	0.877	0.928	-0.015
$\varrho=0.6,N=60$	Trans-Lasso	0.708	0.744	0.786	0.826	0.863	0.902	-
$[1,1,1]$	KG-WDRO	0.797	0.825	0.852	0.880	0.911	0.942	0.354
$\varrho=0.6,N=70$	Trans-Lasso	0.794	0.820	0.844	0.868	0.894	0.919	-

A.2 Simulation Setup

Let ${\rm Ber}(p)$ denote a bernoulli distribution with probability parameter $p$ , $\mathcal{U}[a,b]$ denote a uniform distribution supported on $[a,b]$ , and $\mathcal{N}(\mu,\sigma^{2})$ denote a univariate normal distribution with mean $\mu$ and variance $\sigma^{2}$ .

A.2.1 Simulation 1: Logistic Regression

In this simulation, the coefficients are generated in a high-dimensional sparse setting. The dimension of the nonzero components is set to 50, which is then augmented with 100 zero components to introduce sparsity. The nonzero components of the true coefficient-prior pair $(\beta,\theta)$ are generated using the multivariate normal scheme in (4), with component variance $\sigma^{2}=0.4$ and $\rho\in\{0.3,0.5,0.7,0.8,0.9,0.95\}$ . The target labels are generated as $Y_{\rm target}\sim{\rm Ber}\left(1/(1+\exp{(\shortminus\beta^{%\scriptscriptstyle\sf T}X)}\right)$ , and the source labels are generated as $Y_{\rm source}\sim{\rm Ber}\left(1/(1+\exp{(\shortminus\theta^{%\scriptscriptstyle\sf T}X)}\right)$ , where $X\sim\mathcal{U}[-2,2]^{150}$ . The sample size $N$ for $(X_{\rm target},Y_{\rm target})$ is varied across $\{20,50,80\}$ , while the sample size for the source data $(X_{\rm source},Y_{\rm source})$ is fixed at 800. Each dataset is paired with a validation set of the same size for hyperparameter selection.

Let $\rm grid_{1}$ denote a hyperparameter grid ranging from $0.0001$ to $1$ with $10$ log-spaced values, and let $\rm grid_{2}$ denote a hyperparameter grid ranging from $0.0001$ to $2$ with $20$ log-spaced values. The $\beta_{\rm WDRO}$ estimator is learned by selecting the best-performing hyperparameter on $\rm grid_{1}$ using validation data. For the $\mathcal{A}$ -Trans-GLM learner (Tian and Feng,, 2023, Algorithm 1), the transferring step is optimized using $\rm grid_{1}$ , and the debiasing step is optimized using $\rm grid_{2}$ . For the KG-WDRO learner $\beta_{\rm KG}$ proposed in Theorem 3 with $p=1$ , the prior $\theta$ is first learned from the source data using the vanilla WDRO method on $\rm grid_{1}$ , followed by learning $\beta_{\rm KG}$ on $\rm grid_{2}$ with the learned $\theta_{\rm WDRO}$ as input.

The simulations are conducted on the parameter grid

N\in\{20,50,80\}\times\rho\in\{0.3,0.5,0.7,0.8,0.9,0.95\}\times s\in\{0.5,1\},

with each configuration repeated $100$ times. The average results are reported.

A.2.2 Simulation 2: Linear Regression

In this simulation, the coefficients are generated in a high-dimensional correlated setting. The dimension of the coefficients is set to 100 and the components of the true coefficient-prior pair $(\beta,\theta)$ are generated using the multivariate normal scheme in (4), with component variance $\sigma^{2}=0.1$ and $\rho\in\{0.3,0.5,0.7,0.8,0.9,0.95\}$ . The target labels are generated as $Y_{\rm target}\sim\mathcal{N}(\beta^{\scriptscriptstyle\sf T}X,\sqrt{0.5})$ , and the source labels are generated as $Y_{\rm source}\sim\mathcal{N}(\theta^{\scriptscriptstyle\sf T}X,\sqrt{0.5})$ , where $X\sim\mathcal{N}(\mathbf{0},\Sigma)$ with

\Sigma_{i,j}=\begin{cases}1&\text{if }i=j,\\0.3&\text{if }i\neq j,\end{cases}\quad\text{for all }i,j=1,2,\dots,100.

The sample size $N$ for $(X_{\rm target},Y_{\rm target})$ is varied across $\{50,70,90\}$ , while the sample size for the source data $(X_{\rm source},Y_{\rm source})$ is fixed at 800. Each dataset is paired with a validation set of the same size for hyperparameter selection.

Let $\rm grid_{1}$ denote a hyperparameter grid ranging from $0.0001$ to $1$ with $10$ log-spaced values, and let $\rm grid_{2}$ denote a hyperparameter grid ranging from $0.0001$ to $1.5$ with $20$ log-spaced values. The $\beta_{\rm WDRO}$ estimator is learned by selecting the best-performing hyperparameter on $\rm grid_{1}$ using validation data. For the Trans-Ridge learner adapted from (Li etal.,, 2021, Algorithm 1), the transferring step is optimized using $\rm grid_{1}$ , and the debiasing step is optimized using $\rm grid_{2}$ . For the KG-WDRO learner $\beta_{\rm KGstrong}$ proposed in Theorem 1 with $p=2$ , and the $\beta_{\rm KGweak}$ learner proposed in Theorem 2, the prior $\theta$ is first learned from the source data using the vanilla WDRO method on $\rm grid_{1}$ , followed by learning $\beta_{\rm KGstrong}$ on $\rm grid_{2}$ with the learned $\theta_{\rm WDRO}$ as input. The $\lambda^{-1}$ grid for $\beta_{\rm KGweak}$ is 0.0001 to 8 with 20 log-spaced values.

The simulations are conducted on the parameter grid

N\in\{50,70,90\}\times\rho\in\{0.3,0.5,0.7,0.8,0.9,0.95\}\times s\in\{0.8,1\},

with each configuration repeated $100$ times. The average results are reported.

A.2.3 Simulation 3: Multiple Sites

In this simulation, the coefficients are generated in a high-dimensional sparse setting. The dimension of the nonzero components is set to 50, which is then augmented with 100 zero components to introduce sparsity. The number of external source is 3, we generate the their coefficients $\theta_{1},\theta_{2},\theta_{3}$ using the scheme (4). We construct a linear combination, $\theta_{S}=a\theta_{1}+b\theta_{2}+c\theta_{3}$ , and generate the target coefficient $\beta=\rho\theta_{S}+\varepsilon$ , where $\varepsilon\sim N(0,(1-\rho^{2})\mathrm{Var}(\theta_{S}))$ , ensuring $\mathrm{Corr}(\beta,\theta_{S})=\rho$ . The target coefficient $\beta$ is then scaled to match the magnitude of $\theta_{S}$ .

The target labels are generated as $Y_{\rm target}\sim\mathcal{N}(\beta^{\scriptscriptstyle\sf T}X,\sqrt{0.5})$ , and the source labels are generated as $Y_{{\rm source},m}\sim\mathcal{N}(\theta_{m}^{\scriptscriptstyle\sf T}X,\sqrt{%0.5})$ for $m\in[3]$ , where $X\sim\mathcal{N}(\mathbf{0},\Sigma)$ with

\Sigma_{i,j}=\begin{cases}1&\text{if }i=j,\\0.1&\text{if }i\neq j,\end{cases}\quad\text{for all }i,j=1,2,\dots,150.

The sample size for the target data ranges in $\{50,60,70\}$ .

Let $\rm grid_{1}$ denote a hyperparameter grid ranging from $0.0001$ to $1$ with $15$ log-spaced values, and let $\rm grid_{2}$ denote a hyperparameter grid ranging from $0.0001$ to $3$ with $20$ log-spaced values. The $\beta_{\rm WDRO}$ estimator is learned by selecting the best-performing hyperparameter on $\rm grid_{1}$ using validation data. For the oracle Trans-Lasso learner (Li etal.,, 2021, Algorithm 1), the transferring step is optimized using $\rm grid_{1}$ , and the debiasing step is optimized using $\rm grid_{2}$ using all three source data. For the KG-WDRO learner $\beta_{\rm KG}$ proposed in Theorem 1 with $p=1$ , the priors $\theta_{1},\theta_{2},\theta_{3}$ are first learned from the three source data using the vanilla WDRO method on $\rm grid_{1}$ , followed by learning $\beta_{\rm KG}$ on $\rm grid_{2}$ with the learned $\theta_{1,\rm WDRO},\theta_{2,\rm WDRO},\theta_{3,\rm WDRO}$ as input.

The simulations are conducted on the parameter grid

N\in\{50,60,70\}\times\rho\in\{0.3,0.5,0.7,0.8,0.9,0.95\}\times[a,b,c]\in\{[1,%-0.5,0.2],[1,1,1]\},

with each configuration repeated $100$ times. The average results are reported.

Appendix B Proof of Results in Regression.

Lemma A1.

Let $f_{\beta}:\mathbb{R}^{d}\to\mathbb{R}$ be defined as $\Delta\in\mathbb{R}^{d}\mapsto(\beta^{\scriptscriptstyle\sf T}\Delta)^{2}-2r(%\beta)\beta^{\scriptscriptstyle\sf T}\Delta$ depending on some $\beta\in\mathbb{R}^{d}$ and let $r(\beta)$ be a non-negative real-valued function in $\beta$ . Then the convex conjugate $f_{\beta}^{*}(\Delta^{*}):\mathbb{R}^{d}\to\mathbb{R}$ is given by

f_{\beta}^{*}(\Delta^{*})=\begin{cases}\dfrac{(\beta^{\scriptscriptstyle\sf T}%\Delta^{*}+2r(\beta)\lVert\beta\rVert_{2}^{2})^{2}}{4\lVert\beta\rVert_{2}^{4}%}&\text{if }\Delta^{*}\in\operatorname{span}{\beta},\\+\infty&\text{otherwise}.\end{cases}

Therefore the biconjugate $f_{\beta}^{**}(\Delta):\mathbb{R}^{d}\to\mathbb{R}$ of $f_{\beta}(\Delta)$ has representation:

f_{\beta}^{**}(\Delta)=\sup_{\alpha\in\mathbb{R}}\left(\alpha(\beta^{%\scriptscriptstyle\sf T}\Delta)-\dfrac{(\alpha+2r(\beta))^{2}}{4}\right).

Proof.

The convex conjugate $f_{\beta}^{*}(\Delta^{*})$ is defined as

f_{\beta}^{*}(\Delta^{*})\coloneqq\sup_{\Delta\in\mathbb{R}^{d}}\big{(}\Delta^%{*}{{}^{\scriptscriptstyle\sf T}}\Delta-(\beta^{\scriptscriptstyle\sf T}\Delta%)^{2}+2r(\beta)(\beta^{\scriptscriptstyle\sf T}\Delta)\big{)},

where $\Delta^{*},\beta\in\mathbb{R}^{d}$ and $r(\beta)\in\mathbb{R}$ are taken as fixed values. Orthogonalize $\Delta=a\beta+\omega$ in the direction of $\beta$ with $a\in\mathbb{R}$ , and $\omega\in\mathbb{R}^{d}$ such that $\beta^{\scriptscriptstyle\sf T}\omega=0$ . Then , we have $\Delta^{*}{{}^{\scriptscriptstyle\sf T}}\Delta=a\Delta^{*}{{}^{%\scriptscriptstyle\sf T}}\beta+\Delta^{*}{{}^{\scriptscriptstyle\sf T}}\omega$ , and the convex conjugate becomes

	$\displaystyle f^{}(\Delta^{})$	$\displaystyle=\sup_{a,\omega}\big{(}a(\Delta^{}{{}^{\scriptscriptstyle\sf T}}%\beta)+\Delta^{}{{}^{\scriptscriptstyle\sf T}}\omega-a^{2}\\|\beta\\|_{2}^{4}+2%ar(\beta)\\|\beta\\|_{2}^{2}\big{)}$
	$\displaystyle\rm{s.t}$	$\displaystyle\quad\beta^{\scriptscriptstyle\sf T}\omega=0.$

Fixing $\omega$ , the objective is a negative quadratic function in $a$ , hence the objective in $a$ is bounded from above by a finite value. Now, if $\Delta^{*}$ is not orthogonal to $\omega$ , the term $\sup_{\omega}\Delta^{*}{{}^{\scriptscriptstyle\sf T}}\omega$ is unbounded, and the convex conjugate $f^{*}(\Delta^{*})=+\infty$ . If $\Delta^{*}$ is orthogonal to $\omega$ , then the convex conjugate attains finite value. Note that $\Delta^{*}{{}^{\scriptscriptstyle\sf T}}\omega=0\iff\Delta^{*}\in\operatorname%{span}{\beta}$ . Hence condition on $\{\Delta^{*}=\alpha\beta\,;\alpha\in\mathbb{R}\}$ , we have

	$\displaystyle f^{}(\Delta^{})$	$\displaystyle=\sup_{a}\big{(}a(\Delta^{*}{{}^{\scriptscriptstyle\sf T}}\beta)-%a^{2}\\|\beta\\|_{2}^{4}+2r(\beta)a\\|\beta\\|_{2}^{2}\big{)}$
		$\displaystyle=\dfrac{\big{(}\Delta^{*}{{}^{\scriptscriptstyle\sf T}}\beta+2r(%\beta)\\|\beta\\|_{2}^{2}\big{)}^{2}}{4\\|\beta\\|_{2}^{4}},$

where the optimal solution is $a^{*}=\dfrac{\alpha+2r(\beta)}{2\|\beta\|_{2}^{2}}$ , and the coefficient $\alpha$ is given by the projection scalar $\alpha=\dfrac{\Delta^{*}{{}^{\scriptscriptstyle\sf T}}\beta}{\|\beta\|_{2}^{2}}$ .

The biconjuagte

f^{**}(\Delta)=\sup_{\Delta^{*}}\big{(}\Delta^{\scriptscriptstyle\sf T}\Delta^%{*}-f^{*}(\Delta^{*})\big{)},

is therefore bounded from below if and only if $\Delta^{*}\in\operatorname{span}{\beta}$ . Let $\Delta^{*}=\alpha\beta$ for some $\alpha\in\mathbb{R}$ , then substituting we get the representation,

	$\displaystyle f^{**}(\Delta)$	$\displaystyle=\sup_{\alpha}\left(\Delta^{\scriptscriptstyle\sf T}(\alpha\beta)%-\dfrac{\big{(}\beta^{\scriptscriptstyle\sf T}(\alpha\beta)+2r(\beta)\\|\beta\\|%_{2}^{2}\big{)}^{2}}{4\\|\beta\\|_{2}^{4}}\right)$
		$\displaystyle=\sup_{\alpha}\left(\alpha(\Delta^{\scriptscriptstyle\sf T}\beta)%-\dfrac{(\alpha+2r(\beta))^{2}}{4}\right).$

It can be readily verified that $f^{**}(\Delta)=f(\Delta)$ as promised by the Fenchel-Moreau Theorem (Theorem A4).∎

Lemma A2.

Let $g_{\theta}(\Delta):\mathbb{R}^{d}\to\mathbb{R}$ be defined as $\Delta\in\mathbb{R}^{d}\mapsto|\theta^{\scriptscriptstyle\sf T}\Delta|$ for some $\theta\in\mathbb{R}^{d}$ . Then the convex conjugate $g_{\theta}^{*}(\Delta^{*})$ is given by

g_{\theta}^{*}(\Delta^{*})=\begin{cases}0&\text{if }\Delta^{*}=\alpha\theta%\text{ and }|\alpha|\leq 1,\\+\infty&\text{otherwise.}\end{cases}

Therefore the convex conjugate of the function $g(\Delta)\coloneqq\gamma\sum_{m=1}^{M}g_{\theta_{m}}(\Delta)$ for some $\gamma>0$ is given by

g^{*}(\Delta^{*})=\begin{cases}0&\text{if }\Delta^{*}=\sum_{m=1}^{M}\alpha_{m}%\theta_{m}\text{ and },|\alpha_{m}|\leq\gamma\text{ for each }m,\\+\infty&\text{otherwise.}\end{cases}

Proof.

The convex conjugate is defined as

g_{\theta}^{*}(\Delta^{*})=\sup_{\Delta}\big{(}\Delta^{*}{{}^{%\scriptscriptstyle\sf T}}\Delta-|\theta^{\scriptscriptstyle\sf T}\Delta|\big{)},

again, orthogonalize $\Delta=a\theta+\omega$ , where $a=\dfrac{\theta^{\scriptscriptstyle\sf T}\Delta}{\|\theta\|_{2}^{2}}$ and $\theta^{\scriptscriptstyle\sf T}\omega=0$ . Now by the change of variable $u\coloneqq\theta^{\scriptscriptstyle\sf T}\Delta$ , the convex conjugate is now

	$\displaystyle g^{}_{\theta}(\Delta^{})$	$\displaystyle=\sup_{u,\omega}\left(\dfrac{u}{\\|\theta\\|_{2}^{2}}(\Delta^{}{{}%^{\scriptscriptstyle\sf T}}\theta)+\Delta^{}{{}^{\scriptscriptstyle\sf T}}%\omega-\|u\|\right)$
	s.t	$\displaystyle\theta^{\scriptscriptstyle\sf T}\omega=0.$

Thus the convex conjugate $g_{\theta}^{*}(\Delta^{*})=+\infty$ if $\Delta^{*}\not\in\operatorname{span}{\theta}$ . If $\Delta^{*}=\alpha\theta$ for some $\alpha\in\mathbb{R}$ , then

	$\displaystyle g^{}_{\theta}(\Delta^{})=g^{*}_{\theta}(\alpha\theta)$	$\displaystyle=\sup_{u}\left(\dfrac{u}{\\|\theta\\|_{2}^{2}}\alpha\\|\theta\\|_{2}^%{2}-\|u\|\right)$
		$\displaystyle=\sup_{u}\big{(}\alpha u-\|u\|\big{)}$
		$\displaystyle=\begin{cases}0&\text{if }\|\alpha\|\leq 1,\\+\infty&\text{otherwise,}\end{cases}$

where the last equality holds by noting that $\sup_{u}\alpha u-|u|=|\cdot|^{*}(\alpha)$ is the convex conjugate of the absolute value function (Proposition A2). This proofs the convex conjugate of $g_{\theta}^{*}(\Delta^{*})$ . Now $g(\Delta)=\gamma\sum_{m=1}^{M}g_{\theta_{m}}(\Delta)=\gamma\bar{g}(\Delta)$ , the convex conjugate of $\bar{g}(\Delta)$ is

	$\displaystyle\bar{g}^{}(\Delta^{})$	$\displaystyle=(g_{\theta_{1}}+\ldots+g_{\theta_{M}})^{}(\Delta^{})$
		$\displaystyle=\inf_{\Delta^{}}\big{(}g^{}_{\theta_{1}}(\Delta^{}_{1})+%\ldots+g_{\theta_{M}}^{}(\Delta_{M}^{})\big{)}\,\,\,\text{ s.t}\,\,\,\Delta_%{1}^{}+\ldots+\Delta_{M}^{}=\Delta^{},$

where the second line follows from the infimal convolution property of sum of convex conjugates (Theorem A5). We know that $\bar{g}^{*}$ is finite if and only if $g^{*}_{\theta_{m}}(\Delta_{m}^{*})=0$ for all $m\in[M]$ , that is $\Delta_{m}^{*}=\alpha_{m}\theta_{m}$ for some $\alpha_{m}\in[-1,1]$ for all $m\in[M]$ . Hence $\bar{g}^{*}(\Delta^{*})=0$ if and only if $\Delta^{*}=\sum_{m=1}^{M}\alpha_{m}\theta_{m}$ and $\alpha_{m}\in[-1,1]$ for all $m\in[M]$ . Finally we can calculate the convex conjugate $g^{*}(\Delta^{*})=(\gamma\bar{g})^{*}(\Delta^{*})=\gamma\bar{g}^{*}\left(%\dfrac{\Delta^{*}}{\gamma}\right)$ by the scaling law of convex conjugates (Proposition A3) given $\gamma>0$ . This concludes the proof.∎

We now give the proof to Theorem 1.

Proof of Theorem 1.

Let $r(\beta)\coloneqq y-\beta^{\scriptscriptstyle\sf T}x$ . Then first consider the cost function

c_{2}\big{(}(x,y),(u,v)\big{)}\coloneqq\|x-u\|_{q}^{2}+\infty\cdot|y-v|+d(%\theta_{1}^{\scriptscriptstyle\sf T}x-\theta_{1}^{\scriptscriptstyle\sf T}u)+%\ldots+d(\theta_{M}^{\scriptscriptstyle\sf T}x-\theta_{M}^{\scriptscriptstyle%\sf T}u).

where we replaced the transferring strength from $+\infty$ to a finite-valued distance function $d(x):\mathbb{R}\to\mathbb{R}$ that is a monotone function in $|x|$ , with $d(0)=0$ . We will then let $d(x)\to\infty$ except at $x=0$ . Then the supremum function

\displaystyle\phi_{\gamma}(x,y;\beta)=\sup_{(u,v)\in\mathbb{R}^{d+1}}\big{\{}%\ell(u,v;\beta)-\gamma c\big{(}(u,v),(x,y)\big{)}\big{\}},

is finite if and only if $v=y$ . Then, we have

		$\displaystyle l\left(u,v;\beta\right)-\gamma c\left(\left(u,v\right),(x,y%\right))$
	$\displaystyle=$	$\displaystyle\left(y-\beta^{\scriptscriptstyle\sf T}u\right)^{2}-\gamma\left\\|%x-u\right\\|_{q}^{2}-\gamma d\left(\theta_{1}^{\scriptscriptstyle\sf T}x-\theta%_{1}^{\scriptscriptstyle\sf T}u\right)-\ldots-\gamma d(\theta_{M}^{%\scriptscriptstyle\sf T}x-\theta_{M}^{\scriptscriptstyle\sf T}u).$

Denote by $\Delta\coloneqq u-x$ , we get

		$\displaystyle l\left(u,v;\beta\right)-\gamma c\left(\left(u,v\right),(x,y%\right))$
	$\displaystyle=$	$\displaystyle r(\beta)^{2}+\big{\{}(\beta^{\scriptscriptstyle\sf T}\Delta)^{2}%-2r(\beta)\beta^{\scriptscriptstyle\sf T}\Delta-\gamma\\|\Delta\\|_{q}^{2}-%\gamma d(\theta_{1}^{\scriptscriptstyle\sf T}\Delta)-\ldots-\gamma d(\theta_{M%}^{\scriptscriptstyle\sf T}\Delta)\big{\}}.$

Consider the objective in $\Delta$

		$\displaystyle\sup_{\Delta}\big{\{}(\beta^{\scriptscriptstyle\sf T}\Delta)^{2}-%2r(\beta)\beta^{\scriptscriptstyle\sf T}\Delta-\gamma\\|\Delta\\|_{q}^{2}-\gammad%(\theta_{1}^{\scriptscriptstyle\sf T}\Delta)-\ldots-\gamma d(\theta_{M}^{%\scriptscriptstyle\sf T}\Delta)\big{\}}$
	$\displaystyle\coloneqq$	$\displaystyle\sup_{\Delta}\big{\{}f_{\beta}(\Delta)-g(\Delta)\big{\}},$

where we let $f_{\beta}(\Delta)\coloneqq(\beta^{\scriptscriptstyle\sf T}\Delta)^{2}-2r(\beta%)\beta^{\scriptscriptstyle\sf T}\Delta$ and $g(\Delta)\coloneqq\gamma\|\Delta\|_{q}^{2}+\gamma d(\theta_{1}^{%\scriptscriptstyle\sf T}\Delta)+\ldots+\gamma d(\theta_{M}^{\scriptscriptstyle%\sf T}\Delta)$ . This is a convex + concave optimization, we express the convex part of $f_{\beta}(\Delta)$ as a supremum of infinitely many affine functions. Then by Lemma A1, we have $f_{\beta}(\Delta)=f_{\beta}^{**}(\Delta)=\sup_{\alpha\in\mathbb{R}}\left(%\alpha(\beta^{\scriptscriptstyle\sf T}\Delta)-\dfrac{(\alpha+2r(\beta))^{2}}{4%}\right)$ , then we may write

	$\displaystyle\sup_{\Delta}\big{\{}f_{\beta}(\Delta)-g(\Delta)\big{\}}$
$\displaystyle=$	$\displaystyle\sup_{\Delta}\left\{\sup_{\alpha\in\mathbb{R}}\left(\alpha(\beta^%{\scriptscriptstyle\sf T}\Delta)-\dfrac{(\alpha+2r(\beta))^{2}}{4}\right)-g(%\Delta)\right\}$
$\displaystyle=$	$\displaystyle\sup_{\alpha}\left\{\sup_{\Delta}\big{(}\Delta^{%\scriptscriptstyle\sf T}(\alpha\beta)-g(\Delta)\big{)}-\dfrac{\big{(}\alpha+2r%(\beta)\big{)}^{2}}{4}\right\}$
$\displaystyle=$	$\displaystyle\sup_{\alpha}\left\{g^{*}(\alpha\beta)-\dfrac{\big{(}\alpha+2r(%\beta)\big{)}^{2}}{4}\right\},$	(Toland’s Duality)

where $g^{*}$ is the convex conjugate of $g$ . Let $g(\Delta)\coloneqq g_{1}(\Delta)+g_{\theta}(\Delta)$ , with $g_{1}(\Delta)=\gamma\|\Delta\|_{q}^{2}$ and $g_{\theta}(\Delta)\coloneqq\gamma\sum_{m=1}^{M}d(\theta_{m}^{%\scriptscriptstyle\sf T}\Delta)$ . Then we can compute the convex conjugate of $g$ using the infimal convolution property (Theorem A5). Then

\displaystyle g^{*}(\Delta^{*})

\displaystyle=\inf_{\Delta^{*}_{1}+\Delta^{*}_{2}=\Delta^{*}}\big{(}g_{1}^{*}(%\Delta_{1}^{*})+g_{\theta}^{*}(\Delta_{2}^{*})\big{)}.

We know that $g_{1}^{*}(\Delta_{1}^{*})=\dfrac{1}{4\gamma}\|\Delta_{1}^{*}\|_{p}^{2}$ , where $p^{-1}+q^{-1}=1$ (Proposition A2). Now suppose $d(x)=\lambda|x|$ for some $\lambda>0$ , by Lemma A2, we have,

g_{\theta}^{*}(\Delta_{2}^{*})=\begin{cases}0&\text{if }\Delta_{2}^{*}=\sum_{m%=1}^{M}\alpha_{m}\theta_{m}\text{ and },|\alpha_{m}|\leq\gamma\lambda\text{ %for each }m,\\+\infty&\text{otherwise.}\end{cases}

Then the convex conjugate $g^{*}(\Delta^{*})$ is

	$\displaystyle g^{}(\Delta^{})$	$\displaystyle=\inf_{\Delta_{2}^{}}g_{1}^{}(\Delta^{}-\Delta_{2}^{}),$
	$\displaystyle\text{s.t }\,\Delta_{2}^{*}=\sum_{m=1}^{M}\alpha_{m}$	$\displaystyle\theta_{m}\text{ and },\|\alpha_{m}\|\leq\gamma\lambda\text{ for %each }m,$

which is equivalently,

	$\displaystyle g^{}(\Delta^{})$	$\displaystyle=\dfrac{1}{4\gamma}\inf_{\boldsymbol{\alpha}}\left\\|\Delta^{*}-%\sum_{m=1}^{M}\alpha_{m}\theta_{m}\right\\|_{p}^{2},$
		$\displaystyle\text{s.t }\,\|\alpha_{m}\|\leq\gamma\lambda\text{ for each }m.$

Letting $\lambda\to\infty$ , we recover the cost function $c_{2,\infty}$ , and when $\lambda\to\infty$ , each $\alpha_{m}$ is now free in $\mathbb{R}$ . Then we have $g^{*}(\Delta^{*})=\dfrac{1}{4\gamma}\inf_{\vartheta\in\Theta}\|\Delta^{*}-%\vartheta\|_{p}^{2}$ , with $\Theta\coloneqq\operatorname{span}{\{\theta_{1},\ldots,\theta_{M}\}}$ , the validity of this tactic follows from (Luenberger and Ye,, 2008, Theorem 1, Section 13.1). Then we have $g^{*}(\alpha\beta)=\dfrac{1}{4\gamma}\inf_{\vartheta\in\Theta}\|\alpha\beta-%\vartheta\|_{p}^{2}.$ Suppose $\alpha\neq 0,$ then dividing by $\alpha$ , we get

g^{*}(\alpha\beta)=\dfrac{\alpha^{2}}{4\gamma}\inf_{\vartheta\in\Theta}\|\beta%-\vartheta\|_{p}^{2}.

If $\alpha=0$ , then $g^{*}(\alpha\beta)=g^{*}(\boldsymbol{0})=\dfrac{1}{4\gamma}\inf_{\vartheta}\|%\vartheta\|_{p}^{2}=0$ , so the representation $g^{*}(\alpha\beta)=\dfrac{\alpha^{2}}{4\gamma}\inf_{\vartheta\in\Theta}\|\beta%-\vartheta\|_{p}^{2},$ is valid for all $\alpha\in\mathbb{R}$ . Therefore following the proof of (Blanchet etal., 2019a, , Theorem 1),

	$\displaystyle\phi_{\gamma}(x,y;\beta)$	$\displaystyle=r(\beta)^{2}+\dfrac{1}{4}\sup_{\alpha}\left\{\dfrac{\alpha^{2}}{%\gamma}\inf_{\vartheta\in\Theta}\\|\beta-\vartheta\\|_{p}^{2}-\big{(}\alpha+2r(%\beta)\big{)}^{2}\right\}$
		$\displaystyle=\dfrac{1}{4}\sup_{\alpha}\left\{\left(\dfrac{\inf_{\vartheta}\\|%\beta-\vartheta\\|_{p}^{2}}{\gamma}-1\right)\alpha^{2}-4r(\beta)\alpha\right\}$
		$\displaystyle=\begin{cases}\dfrac{r(\beta)^{2}\gamma}{\gamma-\inf_{\vartheta}%\\|\beta-\vartheta\\|_{p}^{2}}&\text{if }\inf_{\vartheta}\\|\beta-\vartheta\\|_{p}%^{2}\leq\gamma,\\+\infty&\text{otherwise.}\end{cases}$

Then the minimization objective can be simplified as

		$\displaystyle\inf_{\beta\in\mathbb{R}^{d}}\min_{\gamma\geq 0}\left\{\gamma%\delta+\dfrac{1}{n}\sum_{i=1}^{N}\phi_{\gamma}(x_{i},y_{i};\beta)\right\}$
	$\displaystyle=$	$\displaystyle\inf_{\beta}\inf_{\gamma\geq\inf_{\vartheta}\lVert\beta-\vartheta%\rVert_{p}^{2}}\left\{\gamma\delta+\dfrac{1}{n}\sum_{i=1}^{N}\dfrac{r_{i}(%\beta)^{2}\gamma}{\gamma-\inf_{\vartheta}\lVert\beta-\vartheta\rVert_{p}^{2}}\right\}$
	$\displaystyle=$	$\displaystyle\inf_{\beta}\inf_{\gamma\geq\inf_{\vartheta}\lVert\beta-\vartheta%\rVert_{p}^{2}}\left\{\gamma\delta+\text{MSE}(\beta)\dfrac{\gamma}{\gamma-\inf%_{\vartheta}\lVert\beta-\vartheta\rVert_{p}^{2}}\right\}$
	$\displaystyle=$	$\displaystyle\inf_{\beta}\left(\sqrt{\text{MSE}(\beta)}+\sqrt{\delta}\inf_{%\vartheta}\\|\beta-\vartheta\\|_{p}\right)^{2},$

where the last equality follows because $\gamma\delta+\dfrac{1}{n}\text{MSE}(\beta)\dfrac{\gamma}{\gamma-\inf_{%\vartheta}\lVert\beta-\vartheta\rVert_{p}^{2}}$ is a convex function in $\gamma$ that tends to $+\infty$ approaching the boundaries $\inf_{\vartheta}\|\beta-\vartheta\|_{p}^{2}$ and $+\infty$ , so the optimization over $\gamma$ can be solved using first-order condition. Then by Proposition 1, strong duality holds and,

\inf_{\beta}\sup_{\mathbb{P}:\mathcal{D}_{c_{2}}(\mathbb{P},\mathbb{P}_{n})%\leq\delta}\mathbb{E}_{\mathbb{P}}\left[(Y-\beta^{\scriptscriptstyle\sf T}X)^{%2}\right]=\inf_{\beta,\vartheta}\left(\sqrt{\text{MSE}(\beta)}+\sqrt{\delta}\|%\beta-\vartheta\|_{p}\right)^{2}.

This reduces the infinite-dimensional optimization to a finite-dimensional problem, where we interchanged $\inf_{\vartheta}$ and the quadratic function, since the quadratic function is monotone increasing on the positive reals.∎

The next proof is to Theorem 2 with the weak transferring cost function $c_{2,\lambda}\big{(}(x,y),(u,v)\big{)}=\|x-u\|_{2}^{2}+\lambda(\theta^{%\scriptscriptstyle\sf T}x-\theta^{\scriptscriptstyle\sf T}u)^{2}+\infty\cdot|y%-v|$ with some $\lambda>0$ . The statements generalizes to multi-sites by first considering orthogonalizing the prior knowledge $\{\theta_{1},\ldots,\theta_{M}\}$ .

Proof of Theorem 2.

Following the proof of Theorem 1, we solve the optimization problem

\sup_{\Delta\in\mathbb{R}^{d}}\big{(}(\beta^{\scriptscriptstyle\sf T}\Delta)^{%2}-2r(\beta)\beta^{\scriptscriptstyle\sf T}\Delta-\gamma\|\Delta\|_{2}^{2}-%\gamma\lambda(\theta^{\scriptscriptstyle\sf T}\Delta)^{2}\big{)},

where we recall that $\gamma$ is the dual-variable in statement of Proposition 1, $\lambda>0$ is the transferring strength, $\theta\in\mathbb{R}^{d}$ is the prior knowledge, and $r(\beta)=(y-\beta^{\scriptscriptstyle\sf T}x)^{2}$ is the residual in $\beta$ .

Then let $\mathbb{O}$ be an orthogonal matrix, whose first column is $\theta/\|\theta\|_{2}$ , then use $\widetilde{\Delta}\coloneqq\mathbb{O}^{-1}\Delta.$ The objective function now becomes

(\beta^{\scriptscriptstyle\sf T}\mathbb{O}\widetilde{\Delta})^{2}-2r(\beta)%\beta^{\scriptscriptstyle\sf T}\mathbb{O}\widetilde{\Delta}-\gamma\|\widetilde%{\Delta}\|_{2}^{2}-\gamma\lambda\|\theta\|_{2}^{2}\widetilde{\Delta}_{1}^{2},

where the last term follows because $\theta^{\scriptscriptstyle\sf T}\mathbb{O}=(\|\theta\|_{2},0,\ldots,0)$ , and $\widetilde{\Delta}_{1}$ denotes the first component of $\widetilde{\Delta}$ . Now define

D={\rm diag}{\left\{\sqrt{\lambda\|\theta\|_{2}^{2}+1},1,\ldots,1\right\}},

and consider the change of variable $\bar{\Delta}=D\widetilde{\Delta}$ , then the last two terms become

\displaystyle\|\widetilde{\Delta}\|_{2}^{2}+\lambda\|\theta\|_{2}^{2}%\widetilde{\Delta}_{1}^{2}=\|D^{-1}\bar{\Delta}\|_{2}^{2}+\lambda\|\theta\|_{2%}\dfrac{\bar{\Delta}_{1}^{2}}{{\lambda\|\theta\|_{2}^{2}+1}}=\sum_{i=1}^{d}%\bar{\Delta}_{d}^{2}=\|\bar{\Delta}\|_{2}^{2}.

Therefore, the objective becomes

		$\displaystyle\sup_{\bar{\Delta}}\big{(}(\beta^{\scriptscriptstyle\sf T}\mathbb%{O}D^{-1}\bar{\Delta})^{2}-2r(\beta)\beta^{\scriptscriptstyle\sf T}\mathbb{O}D%^{-1}\bar{\Delta}-\gamma\\|\bar{\Delta}\\|_{2}^{2}\big{)}$
	$\displaystyle=$	$\displaystyle\sup_{\bar{\Delta}}\big{(}\\|\beta^{\scriptscriptstyle\sf T}%\mathbb{O}D^{-1}\\|_{2}^{2}\\|\bar{\Delta}\\|_{2}^{2}-2r(\beta)\\|\beta^{%\scriptscriptstyle\sf T}\mathbb{O}D^{-1}\\|_{2}\\|\bar{\Delta}\\|_{2}-\gamma\\|%\bar{\Delta}\\|_{2}^{2})$
	$\displaystyle=$	$\displaystyle\sup_{\bar{\Delta}}\big{(}(\\|\beta\\|_{\Psi_{\lambda}}-\gamma)\\|%\bar{\Delta}\\|_{2}^{2}-2r(\beta)\\|\beta\\|_{\Psi_{\lambda}}\\|\bar{\Delta}\\|_{2}%\big{)}$

which has finite optimal value $\dfrac{r(\beta)^{2}\|\beta\|_{\Psi_{\lambda}}^{2}}{\gamma-\|\beta\|_{\Psi_{%\lambda}}^{2}}$ whenever $\gamma\geq\|\beta\|_{\Psi_{\lambda}}$ , with $\Psi_{\lambda}$ denoting the positive-definite symmetric matrix,

\Psi_{\lambda}=I_{d}-\dfrac{1}{\|\theta\|_{2}^{2}+\lambda^{-1}}\theta\theta^{%\scriptscriptstyle\sf T},

that is independent of the choice of $\mathbb{O}$ . The first equality follows because we applied Cauchy-Schwarz inequality and since $\bar{\Delta}\in\mathbb{R}^{d}$ is free, there is some $\bar{\Delta}$ that achieves equality. The rest of the proof follows exactly along the proof of Theorem 1 by completing the optimization over the dual problem using Proposition 1.∎

Appendix C Proof of Results in Classification.

Lemma A3.

Consider the convex function $h_{\beta}(x):\mathbb{R}^{d}\to\mathbb{R}$ by $x\in\mathbb{R}^{d}\mapsto\log{(1+\exp{(\shortminus\beta^{\scriptscriptstyle\sfT%}x)})}$ , for some $q>0$ and $x^{\prime}\in\mathbb{R}$ . Then for every $\gamma>0$ , the constraint optimization problem $H_{\beta}(x^{\prime})$ defined as,

	$\displaystyle\sup_{x\in\mathbb{R}^{p}}\quad$	$\displaystyle h_{\beta}(x)-\gamma\lVert x^{\prime}-x\rVert_{q},$
	$\displaystyle\rm s.t\quad$	$\displaystyle\theta^{\scriptscriptstyle\sf T}(x^{\prime}-x)=0,$

has optimal objective value,

H_{\beta}(x^{\prime})=\begin{cases}h_{\beta}(x^{\prime})&\text{if }\inf_{%\kappa\in\mathbb{R}}\lVert\beta-\kappa\theta\rVert_{p}\leq\gamma,\\+\infty&\text{otherwise,}\end{cases}

where $p,q\in[1,\infty)$ with $p^{-1}+q^{-1}=1$ .

Proof.

This lemma is a simple extension of (Shafieezadeh-Abadeh etal.,, 2015, Lemma 1). Following their proof, it is shown that

h_{\beta}(x)=h_{\beta}^{**}(x)=\sup_{0\leq\alpha\leq 1}\big{(}(\alpha\beta)^{%\scriptscriptstyle\sf T}x-\bar{h}^{*}(\alpha)\big{)},

where

\bar{h}^{*}(\alpha)=\begin{cases}\alpha\log{(\alpha)}+(1-\alpha)\log{(1-\alpha%)}&\text{if }\alpha\in[0,1],\\+\infty&\text{otherwise},\end{cases}

is the convex conjugate of the function $\log{\big{(}1+e^{\shortminus x}\big{)}}$ (Proposition A2). Then it is shown that the objective $H_{\beta}$ must has representation

	$\displaystyle\sup_{0\leq\alpha\leq 1}\inf_{\\|q\\|_{p}\leq\gamma}\sup_{x}$	$\displaystyle\big{(}(\alpha\beta+q)^{\scriptscriptstyle\sf T}x-\bar{h}^{*}(%\alpha)-q^{\scriptscriptstyle\sf T}x^{\prime}\big{)},$
		$\displaystyle\text{s.t}\,\,\,\theta^{\scriptscriptstyle\sf T}(x-x^{\prime})=0.$

Fixing $\alpha$ and $q$ , then the inner maximization in $x$

	$\displaystyle\sup_{x}$	$\displaystyle\big{(}(\alpha\beta+q)^{\scriptscriptstyle\sf T}x-q^{%\scriptscriptstyle\sf T}x^{\prime}\big{)},$
		$\displaystyle\text{s.t}\,\,\,\theta^{\scriptscriptstyle\sf T}(x-x^{\prime})=0,$

has solution $(\alpha\beta)^{\scriptscriptstyle\sf T}x^{\prime}$ subject to $\alpha\beta+q=\mu\theta$ for some $\mu\in\mathbb{R}$ derived using the first-order condition of the Lagrangian duality or $+\infty$ otherwise. Then condition on $\{\alpha\beta+q=\mu\theta|\mu\in\mathbb{R}\}$ , the objective has representation

	$\displaystyle H_{\beta}(x^{\prime})$	$\displaystyle=\sup_{0\leq\alpha\leq 1}\inf_{\\|q\\|_{p}\leq\gamma}\big{(}(\alpha%\beta)^{\scriptscriptstyle\sf T}x^{\prime}-\bar{h}^{*}(\alpha)\big{)}\,\,\,%\text{s.t}\,\,\,q=\mu\theta-\alpha\beta$
		$\displaystyle=\sup_{0\leq\alpha\leq 1}\inf_{\mu,\\|\mu\theta-\alpha\beta\\|_{p}%\leq\gamma}\big{(}(\alpha\beta)^{\scriptscriptstyle\sf T}x^{\prime}-\bar{h}^{*%}(\alpha)\big{)}.$

Consider the constraint $\|\mu\theta-\alpha\beta\|_{p}\leq\gamma$ over $\mu$ . Suppose $\alpha>0$ , then dividing by $-\alpha$ , we get the equivalent constraint $\left\{|\alpha|\left\|\beta-\dfrac{\mu}{\alpha}\theta\right\|_{p}\right\}\leq\gamma$ over $\mu$ . Defining the change of variable $\kappa\coloneqq\dfrac{\mu}{\alpha}$ , then since the Lagrange multiplier $\mu\in\mathbb{R}$ is free, we have $\kappa$ is free, and the constraint becomes $\inf_{\kappa}|\alpha|\|\beta-\kappa\theta\|_{p}\leq\gamma$ over $\kappa\in\mathbb{R}$ . If $\alpha=0$ , then $\inf_{\mu}\|\mu\theta-0\|=0=0\cdot\inf_{\kappa}\|\beta-\kappa\theta\|_{p}$ . So the equivalent constraint $\inf_{\kappa}|\alpha|\|\beta-\kappa\theta\|_{p}\leq\gamma$ is valid for all $\alpha\in[0,1]$ . Then condition on $\{\alpha\beta+q=\mu\theta|\mu\in\mathbb{R}\}$ , the objective becomes,

	$\displaystyle H_{\beta}(x^{\prime})$	$\displaystyle=\sup_{0\leq\alpha\leq 1}\big{(}(\alpha\beta)^{\scriptscriptstyle%\sf T}x^{\prime}-\bar{h}^{*}(\alpha)\big{)}\text{ s.t }\sup_{0\leq\alpha\leq 1%}\|\alpha\|\inf_{\kappa}\\|\beta-\kappa\theta\\|_{p}\leq\gamma,$
		$\displaystyle=\sup_{0\leq\alpha\leq 1}\big{(}(\alpha\beta)^{\scriptscriptstyle%\sf T}x^{\prime}-\bar{h}^{*}(\alpha)\big{)}\text{ s.t }\inf_{\kappa}\\|\beta-%\kappa\theta\\|_{p}\leq\gamma,$

Recognizing that

\sup_{0\leq\alpha\leq 1}\big{(}(\alpha\beta)^{\scriptscriptstyle\sf T}x^{%\prime}-\bar{h}^{*}(\alpha)\big{)}=\sup_{0\leq\alpha\leq 1}\alpha(\beta^{%\scriptscriptstyle\sf T}x^{\prime})-\bar{h}^{*}(\alpha)=\bar{h}^{**}(\beta^{%\scriptscriptstyle\sf T}x^{\prime})=h_{\beta}(x^{\prime}),

we get

H_{\beta}(x^{\prime})=\begin{cases}h_{\beta}(x^{\prime})&\text{if }\inf_{%\kappa}\|\beta-\kappa\theta\|_{p}\leq\gamma,\\+\infty&\text{otherwise.}\end{cases}

∎

The above Lemma A3 is easily generalized to incorporate multiple orthogonality constraints $\{\theta_{m}^{\scriptscriptstyle\sf T}(x^{\prime}-x)=0\,;m\in[M]\}$ using the exact same Lagrangian formulation. Again, recall $\Theta=\operatorname{span}{\{\theta_{1},\ldots,\theta_{M}\}}$ . Thus the optimal objective value under multiple constraints becomes

H_{\beta}(x^{\prime})=\begin{cases}h_{\beta}(x^{\prime})&\text{if }\inf_{%\vartheta\in\Theta}\|\beta-\vartheta\|_{p}\leq\gamma,\\+\infty&\text{otherwise.}\end{cases}

We now give the proof to Theorem 3.

Proof of Theorem 3 for Logistic Loss.

Using Proposition 1, we apply the strong duality, and consider the inner optimization problem

		$\displaystyle\sup_{\mathbb{P}:\mathcal{D}_{c_{1},\infty}(\mathbb{P},\mathbb{P}%_{n})\leq\delta}\mathbb{E}_{\mathbb{P}}\left[\log{\left(1+e^{-Y\beta^{%\scriptscriptstyle\sf T}X}\right)}\right]$
	$\displaystyle=$	$\displaystyle\begin{cases}\inf_{\gamma\geq 0}\left\{\gamma\delta+\dfrac{1}{n}%\sum_{i=1}^{N}\sup_{u\in\mathbb{R}^{d}}\left(\log{\left(1+e^{-y_{i}\beta^{%\scriptscriptstyle\sf T}u}\right)}-\gamma\\|x_{i}-u\\|_{q}\right)\right\},&\\\text{s.t }\,\theta_{m}^{\scriptscriptstyle\sf T}(x_{i}-u)=0,\text{ for all }m%\in[M]\text{ and }i\in[N].\end{cases}$

For each $i\in[N]$ , we apply Lemma A3 to the maximization problem,

H_{\beta}(x_{i})=\begin{cases}\sup_{u\in\mathbb{R}^{d}}\left(\log{\left(1+e^{-%y_{i}\beta^{\scriptscriptstyle\sf T}u}\right)}-\gamma\|x_{i}-u\|_{q}\right),&%\\\text{s.t }\,\theta_{m}^{\scriptscriptstyle\sf T}(x_{i}-u)=0,\text{ for all }m%\in[M].\end{cases}

which has solution

\begin{cases}\log{\left(1+e^{-y_{i}\beta^{\scriptscriptstyle\sf T}x_{i}}\right%)}&\text{if }\inf_{\vartheta\in\Theta}\|\beta-\vartheta\|_{p}\leq\gamma,\\+\infty&\text{otherwise.}\end{cases}

Therefore, the maximization problem $\sup_{\mathbb{P}:\mathcal{D}_{c_{1},\infty}(\mathbb{P},\mathbb{P}_{n})\leq%\delta}\mathbb{E}_{\mathbb{P}}\left[\log{\left(1+e^{-Y\beta^{%\scriptscriptstyle\sf T}X}\right)}\right]$ is bounded from above if and only if $\gamma\geq\inf_{\vartheta}\|\beta-\vartheta\|_{p}$ . Under this condition, this reduces the inner optimization problem,

	$\displaystyle\sup_{\mathbb{P}:\mathcal{D}_{c_{1},\infty}(\mathbb{P},\mathbb{P}%_{n})\leq\delta}\mathbb{E}_{\mathbb{P}}\left[\log{\left(1+e^{-Y\beta^{%\scriptscriptstyle\sf T}X}\right)}\right]$	$\displaystyle=\inf_{\gamma\geq\inf_{\vartheta}\\|\beta-\vartheta\\|_{p}}\left\{%\gamma\delta+\dfrac{1}{n}\sum_{i=1}^{N}\log{\left(1+e^{-y_{i}\beta^{%\scriptscriptstyle\sf T}x_{i}}\right)}\right\}$
		$\displaystyle=\dfrac{1}{n}\sum_{i=1}^{N}\log{\left(1+e^{-y_{i}\beta^{%\scriptscriptstyle\sf T}x_{i}}\right)}+\delta\inf_{\vartheta}\\|\beta-\vartheta%\\|_{p}.$

This concludes the proof.∎

We now give the proof to the maximum margin classifier using the hinge loss.

Proof of Theorem 3 for Hinge Loss.

As in the case to the proof of Theorem 1, we first consider the relaxed cost function

c_{1}((x,y),(u,v))=\|x-u\|_{q}+\infty\cdot|y-v|+\lambda\cdot\sum_{m=1}^{M}|%\theta_{m}^{\scriptscriptstyle\sf T}x-\theta_{m}^{\scriptscriptstyle\sf T}u|,

where we relaxed the transferring strength from $+\infty$ to some finite value $\lambda>0$ . We will then let $\lambda\to+\infty$ . Again, by strong duality, we can solve the worst case hinge loss by solving the dual problem

\inf_{\gamma\geq 0}\left\{\gamma\delta+\dfrac{1}{n}\sum_{i=1}^{N}\sup_{u}\left%((1-y_{i}\beta^{\scriptscriptstyle\sf T}u)^{+}-\gamma\|u-x_{i}\|_{q}-\gamma%\lambda\sum_{m=1}^{M}|\theta_{m}^{\scriptscriptstyle\sf T}(x_{i}-u)|\right)%\right\}.

Let $\Delta\coloneqq u-x$ , then we have

		$\displaystyle\sup_{u}\left((1-y\beta^{\scriptscriptstyle\sf T}u)^{+}-\gamma\\|u%-x\\|_{q}-\gamma\lambda\sum_{m=1}^{M}\|\theta_{m}^{\scriptscriptstyle\sf T}(x-u)%\|\right)$
	$\displaystyle=$	$\displaystyle\sup_{\Delta}\left((1-y\beta^{\scriptscriptstyle\sf T}(\Delta+x))%^{+}-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=1}^{M}\|\theta_{m}^{%\scriptscriptstyle\sf T}\Delta\|\right)$
	$\displaystyle=$	$\displaystyle\sup_{\Delta}\sup_{0\leq\alpha\leq 1}\left(\alpha(1-y\beta^{%\scriptscriptstyle\sf T}(\Delta+x))-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=%1}^{M}\|\theta_{m}^{\scriptscriptstyle\sf T}\Delta\|\right)$
	$\displaystyle=$	$\displaystyle\sup_{0\leq\alpha\leq 1}\sup_{\Delta}\left(-\alpha y\beta^{%\scriptscriptstyle\sf T}\Delta-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=1}^{M%}\|\theta_{m}^{\scriptscriptstyle\sf T}\Delta\|+\alpha(1-y\beta^{%\scriptscriptstyle\sf T}x)\right).$

Where in the second equality we used $x^{+}=\sup_{0\leq\alpha\leq 1}\alpha x$ . Fixing $\alpha$ , consider the inner minimization in $\Delta$ ,

\displaystyle\sup_{\Delta}\left(-\alpha y\beta^{\scriptscriptstyle\sf T}\Delta%-\gamma\|\Delta\|_{q}-\gamma\lambda\sum_{m=1}^{M}|\theta_{m}^{%\scriptscriptstyle\sf T}\Delta|\right)=g^{*}(-\alpha y\beta),

where $g^{*}(\Delta^{*})$ is the convex conjugate of $g(\Delta)\coloneqq\gamma\|\Delta\|_{q}+\gamma\lambda\sum_{m=1}^{M}|\theta_{m}^%{\scriptscriptstyle\sf T}\Delta|$ . Set $\gamma\|\Delta_{1}\|_{q}\eqqcolon g_{1}(\Delta_{1})$ and $\gamma\lambda\sum_{m=1}^{M}|\theta_{m}^{\scriptscriptstyle\sf T}\Delta_{2}|%\eqqcolon g_{2}(\Delta_{2})$ , then by the infimal convolution property of convex conjugates (Theorem A5), we know that

g^{*}(\Delta^{*})=\inf_{\Delta^{*}_{1}+\Delta^{*}_{2}=\Delta^{*}}\big{(}g_{1}^%{*}(\Delta^{*}_{1})+g^{*}_{2}(\Delta^{*}_{2})\big{)}.

From Lemma A2, we know that if $g^{*}(\Delta^{*})$ is finite, then $g_{2}^{*}(\Delta_{2}^{*})=0$ subject to $\Delta_{2}^{*}=\sum_{m=1}^{M}\alpha_{m}\theta_{m}$ and $|\alpha_{m}|\leq\lambda\gamma$ for all $m\in[M]$ . Now it is well known that (Proposition A2),

g_{1}^{*}(\Delta_{1}^{*})=(\gamma\|\,\cdot\,\|_{q})^{*}(\Delta_{1}^{*})=I_{\{%\|\Delta_{1}^{*}\|_{p}\leq\gamma\}}(\Delta_{1}^{*}),

where $I_{C}(x)$ denotes the convex indicator on the set $C$ . Therefore, letting $\lambda\to\infty$ , the constraints $\{|\alpha_{m}|\leq\lambda\gamma|m\in[M]\}$ is redundant, and we have

g^{*}(\Delta^{*})=\begin{cases}0&\text{if }\inf_{\vartheta\in\Theta}\|\Delta^{%*}-\vartheta\|_{p}\leq\gamma,\\+\infty&\text{otherwise,}\end{cases}

where we let $\Theta\coloneqq\operatorname{span}{\{\theta_{1},\ldots,\theta_{M}\}}$ . Therefore, $g^{*}(-\alpha y\beta)$ is finite if and only if $\inf_{\vartheta}\|-\alpha y\beta-\vartheta\|_{p}\leq\gamma$ . Now $y=\pm 1$ , so we can remove $-y$ , and this leaves us the condition that $\inf_{\vartheta}\|\alpha\beta-\vartheta\|_{p}\leq\gamma$ , which is equivalent to $\alpha\inf_{\vartheta}\|\beta-\vartheta\|_{p}\leq\gamma$ for all $\alpha\in[0,1]$ , including $\alpha=0$ . Taking supremum over $\alpha\in[0,1]$ , the final condition is $\inf_{\vartheta}\|\beta-\vartheta\|_{p}\leq\gamma$ . Therefore, assuming the dual problem is bounded from above, it reduces as

		$\displaystyle\sup_{0\leq\alpha\leq 1}\sup_{\Delta}\left(-\alpha y\beta^{%\scriptscriptstyle\sf T}\Delta-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=1}^{M%}\|\theta_{m}^{\scriptscriptstyle\sf T}\Delta\|+\alpha(1-y\beta^{%\scriptscriptstyle\sf T}x)\right)$
	$\displaystyle=$	$\displaystyle\sup_{0\leq\alpha\leq 1}\left(I_{\{\inf_{\vartheta}\\|\beta-%\vartheta\\|_{p}\leq\gamma\}}+\alpha(1-y\beta^{\scriptscriptstyle\sf T}x)\right)$
	$\displaystyle=$	$\displaystyle(1-y\beta^{\scriptscriptstyle\sf T}x)^{+}\quad\text{given}\quad%\inf_{\vartheta}\\|\beta-\vartheta\\|_{p}\leq\gamma.$

Finally, the dual form of the distributionally robust optimization problem is

		$\displaystyle\inf_{\beta}\inf_{\gamma\geq 0}\left\{\gamma\delta+\dfrac{1}{n}%\sum_{i=1}^{N}\sup_{u}\left((1-y_{i}\beta^{\scriptscriptstyle\sf T}u)^{+}-%\gamma\\|u-x_{i}\\|_{q}-\gamma\lambda\sum_{m=1}^{M}\|\theta_{m}^{%\scriptscriptstyle\sf T}(x_{i}-u)\|\right)\right\}$
	$\displaystyle=$	$\displaystyle\inf_{\beta}\inf_{\gamma\geq\inf_{\vartheta}\\|\beta-\vartheta\\|_{%p}}\left\{\gamma\delta+\dfrac{1}{n}\sum_{i=1}^{N}(1-y_{i}\beta^{%\scriptscriptstyle\sf T}x_{i})^{+}\right\}$
	$\displaystyle=$	$\displaystyle\inf_{\beta,\vartheta}\left\{\dfrac{1}{n}\sum_{i=1}^{N}(1-y_{i}%\beta^{\scriptscriptstyle\sf T}x_{i})^{+}+\delta\\|\beta-\vartheta\\|_{p}\right\}.$

This completes the proof.∎

Appendix D Proof of Results in Mahalanobis Norm Regularization

Proof of Corollary 1.

This is a direct consequence of the convex conjugate of $\|x\|_{\Lambda}^{2}$ given in Proposition A4.∎

Define the cost function $c_{1,\infty}^{\Lambda}\big{(}(x,y),(u,v)\big{)}\coloneqq\|x-u\|_{\Lambda}+%\infty\cdot|y-v|+\infty\cdot\sum_{m=1}^{M}|\theta_{m}^{\scriptscriptstyle\sf T%}x-\theta_{m}^{\scriptscriptstyle\sf T}u|$ .

Corollary 2 (Theorem 3).

		$\displaystyle\inf_{\beta\in\mathbb{R}^{d}}\sup_{\mathbb{P}:\mathcal{B}_{\delta%}(\mathbb{P}_{N}^{X};c^{\Lambda}_{1,\infty})}\mathbb{E}_{\mathbb{P}}\left[\ell%(X,Y;\beta)\right]$
	$\displaystyle=$	$\displaystyle\inf_{\beta\in\mathbb{R}^{d},\vartheta\in\Theta}\dfrac{1}{N}\sum_%{i=1}^{N}\ell(x_{i},y_{i};\beta)+\delta\\|\beta-\vartheta\\|_{\Lambda^{%\shortminus 1}}.$

Proof.

For the logistic loss case, this is a direct consequence of the dual norm of $\|x\|_{\Lambda}$ , for the hinge loss case this is a direct consequence of the convex conjugate of $\|x\|_{\Lambda}$ . Both given by Proposition A4.∎

Appendix E Useful Results on Convex Conjugation

In this section we review some results on the concept of convex conjugates that repeatedly come up in the proofs. For more details on convex conjugations, the interested readers can consult (Rockafellar,, 1970, Sections 12 & 16).

Definition A1 (Convex Conjugate).

Let $f:\mathbb{R}^{d}\to\mathbb{R}$ be a real-valued function on the Euclidean space, then the convex conjugate of $f$ is the function $f^{*}:\mathbb{R}^{d}\to\mathbb{R}$ that evaluates $x^{*}\in\mathbb{R}^{n}$ by

f^{*}(x^{*})=\sup_{x\in\operatorname{\textbf{dom}}{(f)}}\big{(}x^{*}{{}^{%\scriptscriptstyle\sf T}}x-f(x)\big{)}.

This is also called the Legendre transformation of $f$ , and the Legendre-Fenchel transformation for $f$ defined on arbitrary real topological vector spaces. Here we collect some examples of convex conjugates that appeared in the appendix. These are well-known.

Proposition A2.

Let $p,q\geq 1$ be such that $\dfrac{1}{p}+\dfrac{1}{q}=1$ .

1.
The convex conjugate of the absolute value function $f(x)=|x|$ on $\mathbb{R}$ is given by $|\cdot|^{*}(x^{*})=I_{|x^{*}|\leq 1}(x^{*})$ , the convex indicator function on the set $\{|x^{*}|\leq 1|x^{*}\in\mathbb{R}\}$ .
2.
The convex conjugate of the $q$ -norm $\|x\|_{q}$ on $\mathbb{R}^{d}$ is given by $\|\cdot\|_{q}^{*}(x^{*})=I_{\|x^{*}\|_{p}\leq 1}(x^{*})$ , the convex indicator function on the set $\{\|x^{*}\|_{p}\leq 1|x^{*}\in\mathbb{R}^{d}\}$ .
3.
The convex conjugate of $\dfrac{1}{2}\|x\|_{q}^{2}$ on $\mathbb{R}^{d}$ is given by $\left(\dfrac{1}{2}\|\cdot\|_{q}^{2}\right)^{*}(x^{*})=\dfrac{1}{2}\|x^{*}\|_{p%}^{2}$ .

The convex conjugate of $\log{\big{(}1+e^{\shortminus x}\big{)}}$ on $\mathbb{R}$ is given by

\begin{cases}x^{*}\log{(x^{*})}+(1-x^{*})\log{(1-x^{*})}&\text{if }x^{*}\in(0,%1)\\0&\text{if }x^{*}=0,1\\+\infty&\text{otherwise}.\end{cases}

Another easy consequence from the definition of convex conjugation is the below scaling laws.

Proposition A3 (Scaling Laws).

Let $f^{*}(x^{*})$ be the convex conjugate of $f(x)$ on $\mathbb{R}^{d}$ . Then we have,

1.
the convex conjugate of $f(ax)$ whenever $a\neq 0$ is given by $f^{*}(x^{*}/a)$ .
2.
the convex conjugate of $af(x)$ whenever $a>0$ is given by $af^{*}(x^{*}/a)$ .

Let $\Gamma\left(\mathbb{R}^{d}\right)$ denote the class of proper convex lower-semi continuous functions on $\mathbb{R}^{d}$ , the next statement says that this conjugation induces an one-to-one symmetric correspondence on the class $\Gamma\left(\mathbb{R}^{d}\right)$ . It is a cornerstone of modern convex analysis and used in the proof of Theorem 1 and Lemma A3.

Theorem A4 (Fenchel-Moreau).

Let $f$ be a proper convex, lower semi-continuous function on $\mathbb{R}^{d}$ , then

1.
the convex conjugation $f\mapsto f^{*}$ is a bijection on $\Gamma\left(\mathbb{R}^{d}\right)$ ;
2.
$f\in\Gamma\left(\mathbb{R}^{d}\right)\iff f^{**}=f$ .

Proof.

For a proof please consult (Rockafellar,, 1970, Section 12).∎

The next statement concerns the commutativity of convex conjugation and function summation. Its usefulness is profound, and applied to the proof of Theorem 1 and Theorem 3.

Theorem A5 (Infimal Convolution Property of Convex Conjugation).

Let $f_{1},\ldots,f_{M}$ be proper convex functions on $\mathbb{R}^{d}$ , then

(f_{1}\square\ldots\square f_{M})^{*}=f_{1}^{*}+\ldots f_{M}^{*},

and

(f_{1}+\ldots+f_{M})^{*}(x^{*})=\inf_{x^{*}}\{f_{1}^{*}(x_{1}^{*})+\ldots f_{M%}^{*}(x_{M}^{*})\,|\,x_{1}^{*}+\ldots+x_{M}^{*}=x^{*}\}.

Proof.

For a proof please consult (Rockafellar,, 1970, Theorem 16.4).∎

Proposition A4.

Let $\Lambda\in\mathbb{S}_{+}^{d\times d}$ , then the dual norm of $\|x\|_{\Lambda}$ is $\|x\|_{\Lambda^{-1}}.$ The Cauchy-Schwarz inequality $x^{\scriptscriptstyle\sf T}u\leq\|x\|_{\Lambda}\|u\|_{\Lambda^{\shortminus 1}}$ holds, and equality is attainable. The convex conjugate of $\|x\|_{\Lambda}$ is given by $I_{\|x^{*}\|_{\Lambda^{\shortminus 1}}\leq 1}(x^{*})$ , and the convex conjugate of $\|x\|_{\Lambda}^{2}$ is given by $\|x^{*}\|_{\Lambda^{\shortminus 1}}^{2}/4$ .

Proof.

The dual norm of $\|x\|_{\Lambda}$ , the Cauchy-Schwarz inequality and attainability of equality follows from (Blanchet etal., 2019b, , Lemma 1). Now to compute the convex conjugate of $\|x\|_{\Lambda}^{2}$ , we want to evaluate

\sup_{x\in\mathbb{R}^{d}}(x^{*}{{}^{\scriptscriptstyle\sf T}}x-\|x\|_{\Lambda}%^{2}).

By the Cauchy-Schwarz inequality we have $x^{*}{{}^{\scriptscriptstyle\sf T}}x\leq\|x\|_{\Lambda}\|x^{*}\|_{\Lambda^{%\shortminus 1}}$ , and so we have

x^{*}{{}^{\scriptscriptstyle\sf T}}x-\|x\|_{\Lambda}^{2}\leq\|x\|_{\Lambda}\|x%^{*}\|_{\Lambda^{\shortminus 1}}-\|x\|_{\Lambda}^{2}.

Hence

\sup_{x\in\mathbb{R}^{d}}(x^{*}{{}^{\scriptscriptstyle\sf T}}x-\|x\|_{\Lambda}%^{2})\leq\sup_{t\geq 0}(t\|x^{*}\|_{\Lambda^{\shortminus 1}}-t^{2})=\dfrac{1}{%4}\|x^{*}\|_{\Lambda^{\shortminus 1}}^{2}.

By attainability of equality in the Cauchy-Schwarz inequality, the supremum are equal, and we have

\sup_{x\in\mathbb{R}^{d}}(x^{*}{{}^{\scriptscriptstyle\sf T}}x-\|x\|_{\Lambda}%^{2})=\dfrac{1}{4}\|x^{*}\|_{\Lambda^{\shortminus 1}}^{2}.

This proofs the convex conjugate of $\|x\|_{\Lambda}^{2}$ . Now consider the convex conjugate of $\|x\|_{\Lambda}$ , then we need to evaluate

\sup_{x\in\mathbb{R}^{d}}(x^{*}{{}^{\scriptscriptstyle\sf T}}x-\|x\|_{\Lambda}),

again, by Cauchy-Schwarz and the attainability of equality, we have

	$\displaystyle\sup_{x\in\mathbb{R}^{d}}(x^{*}{{}^{\scriptscriptstyle\sf T}}x-\\|%x\\|_{\Lambda})$	$\displaystyle=\sup_{x\in\mathbb{R}^{d}}(\\|x\\|_{\Lambda}\\|x^{*}\\|_{\Lambda^{%\shortminus 1}}-\\|x\\|_{\Lambda})$
		$\displaystyle=\sup_{x\in\mathbb{{R}^{d}}}(\\|x\\|_{\Lambda}(\\|x^{*}\\|_{\Lambda^{%\shortminus 1}}-1))$
		$\displaystyle=\begin{cases}0&\text{if }\\|x^{*}\\|_{\Lambda^{\shortminus 1}}\leq1%,\\+\infty&\text{otherwise}.\end{cases}$

This completes the proof.

∎

Appendix F Toland’s Duality

The duality theory of Toland’s (Toland,, 1978, 1979) concerns the minimization of nonconvex functions, in particular, applies to the minimization of the difference of convex functions (DC problems). The duality holds under minimal conditions, and one tries to see if the DC problem can be transformed into something more manageable.

Theorem A6 (Toland’s Duality).

Let $f$ and $g$ be functions on $\mathbb{R}^{d}$ , if $f\in\Gamma\left(\mathbb{R}^{d}\right)$ , then we have

\inf_{x\in\mathbb{R}^{d}}\left\{f(x)-g(x)\right\}=\inf_{x^{*}\in\mathbb{R}^{d}%}\left\{g^{*}(x^{*})-f^{*}(x^{*})\right\}.

Toland’s duality is implicitly used in the proof to Theorem 1 and Lemma A3 which also sketches a proof to the above duality theorem.

	$\displaystyle g^{}_{\theta}(\Delta^{})=g^{*}_{\theta}(\alpha\theta)$	$\displaystyle=\sup_{u}\left(\dfrac{u}{\\|\theta\\|_{2}^{2}}\alpha\\|\theta\\|_{2}^%{2}-\|u\|\right)$
		$\displaystyle=\sup_{u}\big{(}\alpha u-\|u\|\big{)}$
		$\displaystyle=\begin{cases}0&\text{if }\|\alpha\|\leq 1,\\+\infty&\text{otherwise,}\end{cases}$

	$\displaystyle\phi_{\gamma}(x,y;\beta)$	$\displaystyle=r(\beta)^{2}+\dfrac{1}{4}\sup_{\alpha}\left\{\dfrac{\alpha^{2}}{%\gamma}\inf_{\vartheta\in\Theta}\\|\beta-\vartheta\\|_{p}^{2}-\big{(}\alpha+2r(%\beta)\big{)}^{2}\right\}$
		$\displaystyle=\dfrac{1}{4}\sup_{\alpha}\left\{\left(\dfrac{\inf_{\vartheta}\\|%\beta-\vartheta\\|_{p}^{2}}{\gamma}-1\right)\alpha^{2}-4r(\beta)\alpha\right\}$
		$\displaystyle=\begin{cases}\dfrac{r(\beta)^{2}\gamma}{\gamma-\inf_{\vartheta}%\\|\beta-\vartheta\\|_{p}^{2}}&\text{if }\inf_{\vartheta}\\|\beta-\vartheta\\|_{p}%^{2}\leq\gamma,\\+\infty&\text{otherwise.}\end{cases}$

		$\displaystyle\sup_{\bar{\Delta}}\big{(}(\beta^{\scriptscriptstyle\sf T}\mathbb%{O}D^{-1}\bar{\Delta})^{2}-2r(\beta)\beta^{\scriptscriptstyle\sf T}\mathbb{O}D%^{-1}\bar{\Delta}-\gamma\\|\bar{\Delta}\\|_{2}^{2}\big{)}$
	$\displaystyle=$	$\displaystyle\sup_{\bar{\Delta}}\big{(}\\|\beta^{\scriptscriptstyle\sf T}%\mathbb{O}D^{-1}\\|_{2}^{2}\\|\bar{\Delta}\\|_{2}^{2}-2r(\beta)\\|\beta^{%\scriptscriptstyle\sf T}\mathbb{O}D^{-1}\\|_{2}\\|\bar{\Delta}\\|_{2}-\gamma\\|%\bar{\Delta}\\|_{2}^{2})$
	$\displaystyle=$	$\displaystyle\sup_{\bar{\Delta}}\big{(}(\\|\beta\\|_{\Psi_{\lambda}}-\gamma)\\|%\bar{\Delta}\\|_{2}^{2}-2r(\beta)\\|\beta\\|_{\Psi_{\lambda}}\\|\bar{\Delta}\\|_{2}%\big{)}$

		$\displaystyle\sup_{u}\left((1-y\beta^{\scriptscriptstyle\sf T}u)^{+}-\gamma\\|u%-x\\|_{q}-\gamma\lambda\sum_{m=1}^{M}\|\theta_{m}^{\scriptscriptstyle\sf T}(x-u)%\|\right)$
	$\displaystyle=$	$\displaystyle\sup_{\Delta}\left((1-y\beta^{\scriptscriptstyle\sf T}(\Delta+x))%^{+}-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=1}^{M}\|\theta_{m}^{%\scriptscriptstyle\sf T}\Delta\|\right)$
	$\displaystyle=$	$\displaystyle\sup_{\Delta}\sup_{0\leq\alpha\leq 1}\left(\alpha(1-y\beta^{%\scriptscriptstyle\sf T}(\Delta+x))-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=%1}^{M}\|\theta_{m}^{\scriptscriptstyle\sf T}\Delta\|\right)$
	$\displaystyle=$	$\displaystyle\sup_{0\leq\alpha\leq 1}\sup_{\Delta}\left(-\alpha y\beta^{%\scriptscriptstyle\sf T}\Delta-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=1}^{M%}\|\theta_{m}^{\scriptscriptstyle\sf T}\Delta\|+\alpha(1-y\beta^{%\scriptscriptstyle\sf T}x)\right).$

		$\displaystyle\sup_{0\leq\alpha\leq 1}\sup_{\Delta}\left(-\alpha y\beta^{%\scriptscriptstyle\sf T}\Delta-\gamma\\|\Delta\\|_{q}-\gamma\lambda\sum_{m=1}^{M%}\|\theta_{m}^{\scriptscriptstyle\sf T}\Delta\|+\alpha(1-y\beta^{%\scriptscriptstyle\sf T}x)\right)$
	$\displaystyle=$	$\displaystyle\sup_{0\leq\alpha\leq 1}\left(I_{\{\inf_{\vartheta}\\|\beta-%\vartheta\\|_{p}\leq\gamma\}}+\alpha(1-y\beta^{\scriptscriptstyle\sf T}x)\right)$
	$\displaystyle=$	$\displaystyle(1-y\beta^{\scriptscriptstyle\sf T}x)^{+}\quad\text{given}\quad%\inf_{\vartheta}\\|\beta-\vartheta\\|_{p}\leq\gamma.$

Knowledge-Guided Wasserstein Distributionally Robust Optimization (2025)

Abstract

1 Introduction

1.1 Related Works

1.1.1 Wasserstein DRO

1.1.2 Transfer Learning

1.2 Our Contribution

Example 1.

1.3 Notations & Organizations

2 Preliminaries

2.1 Optimal Transport Cost

2.2 Distributionally Robust Optimization

2.3 Strong Duality of Wasserstein DRO

Proposition 1 (Strong Duality, (Blanchet etal., 2019a, , Proposition 1)).

3 Knowledge-Guided Wasserstein DRO

3.1 Knowledge-Guided Transport Cost

Remark 1.

3.2 Linear Regression

3.2.1 Strong-Transferring

Theorem 1 (Linear Regression with Strong-Transferring).

Remark 2.

3.2.2 Weak Transferring

Theorem 2 (Linear Regression with Weak Transferring).

3.3 Binary Classifications

Theorem 3 (Binary Classification with Strong Transferring).

3.4 Sub-Coefficient-Vector Transferring

Corollary 1 (Theorem 1).

4 Numerical Results

4.1 Simulation 1: Logistic with ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-Strong Transferring

4.2 Simulation 2: Linear Regression with ℓ2subscriptℓ2\ell_{2}roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT-Weak Transferring

4.3 Simulation 3: Transfer Learning with Multiple Sites

5 Conclusion

References

Appendix A Additional Details in Numerical Results

A.1 Simulation Results

A.1.1 Simulation 1: Logistic Regression

A.1.2 Simulation 2: Linear Regression

A.1.3 Simulation 3: Multi-Sites

A.2 Simulation Setup

A.2.1 Simulation 1: Logistic Regression

A.2.2 Simulation 2: Linear Regression

A.2.3 Simulation 3: Multiple Sites

Appendix B Proof of Results in Regression.

Lemma A1.

Proof.

Lemma A2.

Proof.

Proof of Theorem 1.

Proof of Theorem 2.

Appendix C Proof of Results in Classification.

Lemma A3.

Proof.

Proof of Theorem 3 for Logistic Loss.

Proof of Theorem 3 for Hinge Loss.

Appendix D Proof of Results in Mahalanobis Norm Regularization

Proof of Corollary 1.

Corollary 2 (Theorem 3).

Proof.

Appendix E Useful Results on Convex Conjugation

Definition A1 (Convex Conjugate).

Proposition A2.

Proposition A3 (Scaling Laws).

Theorem A4 (Fenchel-Moreau).

Proof.

Theorem A5 (Infimal Convolution Property of Convex Conjugation).

Proof.

Proposition A4.

Proof.

Appendix F Toland’s Duality

Theorem A6 (Toland’s Duality).

4.1 Simulation 1: Logistic with $\ell_{1}$ -Strong Transferring

4.2 Simulation 2: Linear Regression with $\ell_{2}$ -Weak Transferring