Machine Learning (ML) can bring bias and discrimination even with good intentions [1,2,3,4,5,6]. Fair ML has been developed to counteract unfairness, but the practical use of fair ML models is limited by predictive uncertainty. Predictive uncertainty is the extent to which ML can confidently predict the future. Over-or under-confidence can cause an ML model to be unaware of its own knowledge gaps and make inaccurate predictions [7,8,9]. This can lead to unfairness in decision-making. To address this, we can produce predicted intervals for each sample and incorporate uncertainty into fairness to make decisions more reliable and trustworthy.
The idea of "equalized coverage" -an uncertainty-aware notion of demographic parity [10] -was introduced in a study [11] as a way to ensure that every group receives the same level of prediction certainty. It works by generating prediction intervals that cover the true label Y with a specified probability (e.g., 90%), while also reflecting uncertainty through interval width. However, even with this approach, there are still disparities in coverage rates across groups when conditioning on Y . For example, we observe from the empirical results (Fig. 1) for Adult dataset [4] that low-income women are less likely to be covered than men in the same income bracket, and high-income men are less likely to be predicted to earn as much as high-income women. Consequently, the widths of prediction intervals for different groups, as an indicator of the uncertainty, are not comparable under different coverage rates. These disparities can lead to unfair risk assessment for domains like bank loans and taxation. Further, equalized coverage may sacrifice the efficiency of uncertainty estimation for ensuring coverage rate as it produces wider prediction intervals for the group it intends to protect (See Section 5).
Figure 1: Evaluating equalized coverage [11] on the Adult dataset. The protected attribute is gender. The test data is partitioned into 10 equal-mass bins based on the ascending order of salaries. Significant variations in the coverage rates can be observed among different groups within the head and tail bins.
To address the limitations of equalized coverage, we propose a novel uncertainty-based fairness notion Equal Opportunity of Coverage (EOC), extending from the standard fairness notion of equal opportunity [12]. EOC aims to achieve two properties: (1) similar coverage rates for different groups (e.g., female and male) with similar outcomes Y (e.g., salaries), ( 2) achieve a desired level of coverage rate (e.g., 90%) for the entire population. Ideally, to provide informative predictions, intervals should be as narrow as possible while still satisfying EOC.
We consider the regression task as it is a more general fair ML setting [13,14] with minimal assumptions about the underlying data distribution.
Achieving EOC confronts various challenges. Firstly, the majority of prior fair ML approaches are developed for classification problems [12,15], with only a few for regression [14,16]. This necessitates developing effective techniques to measure and improve EOC in a regression setting. Secondly, ensuring EOC and marginal coverage rate for test data is difficult when true labels are unknown. Finally, prior works [11,17,18,19] have primarily focused on improving fairness and satisfying coverage rate guarantees, but often neglect the width of prediction intervals during the optimization process. This often results in generating wider intervals that limit the amount of decisionmaking information available. However, optimizing interval width with fairness constraints is a non-convex problem, and coverage rates are difficult to guarantee on noisy data, making it challenging to consider all these factors simultaneously.
To address these challenges, we propose Binned Fair Quantile Regression (BFQR), a distribution-free post-processing method that improves EOC while maintaining a desired marginal coverage rate and a narrow prediction interval. It consists of three major steps: First, a hold-out (calibration) dataset is calibrated to improve EOC based on true label Y within discretized bins; second, we leverage conformal prediction [20,21] to achieve EOC for test data from the calibration results in the first step; and finally, an efficient and robust optimization technique is developed to minimize the mean width of prediction intervals. Experiments on both synthetic and real-world data show that BFQR is more effective in improving EOC fairness than state-of-the-art methods.
Related Work. In regression settings, equal opportunity [12] has been studied mostly by adversarial training [22,23]. For quantile regression, models with equal opportunity constraint [24,25] are proposed. However, all mentioned are in-processing methods and have trade-offs between accuracy and fairness [26,27]. There are several works on uncertainty-aware fairness in regression. The pioneering work equalized coverage [11], is built on the validity of conditional coverage [28], and has several follow-ups [17,18]. [19] also proposes conformalized fair regression, imposing demographic parity fairness on prediction interval bounds. However, neither of these methods considers the finegrained group (e.g., low-income females) fairness conditional on true labels as well as the increased width of prediction intervals that provide little information.
Conformal Prediction extends traditional ML by providing a set of prediction intervals around the predicted value, which can be used to assess the level of confidence or uncertainty in the prediction [20]. It is distribution-free and has a rigorous statistical guarantee. A commonly used approach is split conformal prediction [20,21] Equalized Coverage. Let V 2 V be the indicator of whether Y is covered in the prediction set Ĉ(X), i.e., V = [Y 2 Ĉ(X)]. V = {0, 1}. Given a desired error rate µ 2 [0, 1], equalized coverage is satisfied [11] when 8a 2 A, P r{V |A = a} = P r{V } 1 µ and P r{V } 1 µ. Equalized coverage guarantees equal conditional coverage, i.e., V is independent of A (denoted as V ? ? A), but fails to ensure equal coverage conditional on Y , especially for extreme values of Y (Fig. 1). This is problematic as it can perpetuate discrimination against marginalized groups (e.g., females with low income) in risk assessment.
Equal Opportunity of Coverage addresses the limitation of equalized coverage and is defined based on the equal opportunity [12]: Definition 3.1 (Equal Opportunity of Coverage (EOC)). EOC is satisfied when P r{V |A = a, Y = y} = P r{V |Y = y}, i.e., V ? ? A|Y and P r{V } 1 µ, 8y 2 Y and 8a 2 A.
The definition of EOC requires that (1) conditioned on the target variable Y , whether a sample is covered in its prediction interval should be independent of its sensitive attribute A; (2) the marginal coverage rate is above the desired level. Interestingly, the difference between equalized coverage and EOC in their mathematical forms share similarities with the difference between demographic parity and equalized odds. Note that our focus here is whether the true label is covered in the prediction interval since even though the prediction interval contains false labels, V = I[Y 2 Ĉ(X)] = 1 is still valid. Therefore, while EOC has a similar formation to equalized odds, it in fact describes equal opportunity which focuses on Y = 1.
Preferably, both EOC and equalized coverage should be guaranteed. However, the mutual exclusivity theorem below suggests that there is an inherent trade-off between EOC and equalized coverage: Theorem 3.1 (Mutual Exclusivity). If A 6? ? Y and V 6? ? Y , then either equalized coverage or equal opportunity of coverage holds but not both.
Proof. If V ? ? A and V ? ? A|Y , then either A ? ? Y or V ? ? Y .
Unfairness often arises from the fact that features predictive of Y are also correlated to the protected attribute due to e.g., historical bias in the data [29]. This indicates V ? ? A. For the second condition, though V 6? ? Y is possible, V needs to depend on Y to ensure prediction intervals with reasonable width. Predicting certain values (e.g., extreme values) of Y can be challenging due to representative bias. If we enforce the predictor to provide high coverage for marginalized groups with these values, it is highly likely to result in extremely wide intervals that offer little guidance in decision-making.
Meauring EOC. Given the underlying distribution p 2 ∆(VåAåY), where V, A, Y is the domain of v, a, y respectively. We can determine how likely p satisfies EOC by measuring its distance to p 0 , the closest distribution that perfectly achieves EOC. Formally, we denote P EOC as the property of EOC, the set of all distributions in defined space that satisfy EOC, i.e., P EOC := {p 2 ∆(V å A å Y) : (V, A, Y ) á p, V ? ? A|Y } . p 0 2 P EOC is the distribution with minimum total variation (TV) distance to p, i.e., 8q 2 P EOC , d T V (p, p 0 ) ÿ d T V (p, q). The TV distance between p and p 0 is formally defined as
where k•k 1 denotes the l 1 norm of a distribution. Instead of the commonly-used Kolmogorov-Smirnov (KS) distance [12,30], TV distance is chosen as the measure due to the significant drawback of the KS distance -its insensitivity to the deviations between p and p 0 at the tails [31]. Whereas in real-world data, we often confront such significant deviation (see e.g., Fig. 1).
It is challenging to directly measure the distance between p and p 0 from observed data when the target variable Y is continuous due to data sparsity issues for each value of Y . However, drawing from previous works [32,33], we can instead use an easy-to-compute statistic T as a surrogate of d T V (p, p 0 ). T is the measure of independence, assessing the weighted summed violation of independence within each discretized bin. The expectation of T , E[T ], has a fixed upper bound when p satisfies EOC, and a lower bound that increases with the TV distance between p and p 0 . The intuition here is that when Y is divided into sufficient bins and p Y := P r{V, A|Y }, the conditional distribution of p on Y , is Lipschitz continuous, enforcing independence within each bin generates a distribution that does not deviate far from P EOC in terms of TV distance. Lemma 3.2. [33] Let d = dn 2/5 e be the number of bins Y is discretized into, with each bin having an equal size of samples. Let T be a sum of independence measures within all discretized bins. Given the Lipschitz continuity of p Y in Assumption 2, L is the Lipschitz constant, we have:
Lemma 3.2 indicates that we can approximately evaluate EOC for p by measuring the independence within each bin. If V ? ? A almost holds in each bin, such that E[T ] is small enough, then it is highly possible that V ? ? A|Y , i.e., p satisfies EOC. Moreover, an increasing function of / is upper bounded by E[T ]. As E[T ] increases, it is highly likely that p is further away from any p 0 that satisfies EOC. This lemma serves as a theoretical foundation for the proposed method below to improve EOC and for adopting T as an evaluation metric in experiments.
In order to calculate E[T ] from data, we introduce unbiased estimators of T in Appendix 8.1.1.
According to the central limit theorem, we could estimate E[T ] through a sufficient number of random samplings. Whereas for the sake of efficiency, we prefer to construct T with bounded variance such that E[T ] could converge through limited repeated samplings.
In this section, we introduce a post-processing approach where we have a trained ML model, a calibration dataset, and a test dataset for which we aim to improve EOC. It consists of three steps. First, we enforce the independence of A and V within each interval of Y (i.e., EOC) for the calibration data, and then leverage conformal prediction to achieve EOC for test data. Lastly, we describe an efficient and robust optimization approach that optimizes both EOC and widths of prediction intervals.
The first step aims to improve EOC on the calibration data where we have ground-truth labels. As the target variable Y is continuous, the number of samples with certain values of Y can be extremely small, thus calibrating for each distinct value of Y is almost impossible. Meanwhile, commonly used methods designed for continuous variables such as adversarial learning [23,22], which intend to learn a near-optimal p, are computationally inefficient for post-processing approaches [34].
According to Lemma 3.2, E[T ] is an upper bound of an increasing function of d T V (p, p 0 ). Thus, this indicates that if E[T ] decreases, the maximum distance between p and p 0 is reduced, resulting in an improvement in EOC. Informed by this idea, we first focus on enhancing EOC on the calibration data D c . For simplicity, we employ the framework introduced in [35] as the conformal prediction model, though our post-processing method is applicable to any base model. Specifically, let qα denotes the µ-th conditional quantile regression function, i.e., for i-th sample (X i , Y i ), qα (X i ) := inf{y 2 ∆Y : P r{Y i ÿ y|X = X i } µ}. Fix the lower and upper quantiles as µ lo = µ/2 and µ hi = 1 µ/2, then qα lo (X i ) and qα hi (X i ) denote lower and upper quantile regression functions, respectively. The base model is trained on the training data and used for inference on calibration and test data. ). To enhance EOC on D c , we can minimize E[T ] by enforcing the independence of A and V within all discretized bins. In particular, we fix the coverage rates within each bin as m 2 [0, 1], therefore V ? ? A|Y 2 B m . Note that, although the coverage rates are equal across groups within the same bin, the quantile value at m are computed separately for each group, as illustrated in Figure 2. We first calculate the vanilla prediction intervals Ĉ(X i ) = [q α lo (X i ), qα hi (X i )] obtained from the trained model, and get the conformity score
EOC is enhanced and the marginal coverage rate is guaranteed on the calibration data.
This subsection seeks to preserve EOC and marginal coverage rate for test data based on results for calibration data through conformal prediction. The key is to find out, for a test sample with A n+1 = a n+1 , which bin B m it belongs to. We can then calibrate the test sample with the quantile G a,m ( ). However, direct calibration based on the predicted bin would not improve EOC since the prediction result can be biased due to, e.g., skewed distributions for different groups [29].
To address this issue, we propose Binned Fair Quantile Regression (BFQR) (see Algorithm 1 in Appendix 8.2.1). Our method could be treated as a variant of Mondrian conformal prediction [36,37], where the confidence in each bin is evaluated independently. Suppose that a new data point with feature X i and protected attribute a falls into a certain bin B m , we calibrate it with the corresponding quantile value G a,m ( m ). Then, we obtain a sub-interval of prediction within bin
) is then the prediction interval of X i . Under a mild exchangeability assumption similar to [11], our algorithm provides both the marginal coverage guarantee and fair coverage guarantees within each bin. Assumption 1 (Exchangeability). All calibration data (X i , Y i ), i = 1, ..., n and a sample of test data (X n+1 , Y n+1 ) are exchangeable conditioned on A n+1 = a and Y n+1 2 B m , and conformity scores {S(X i , Y i ), Here we sketch the proofs. Take (X i , A i , Y i ) as new sample from test data, i 2 D t . Suppose Y i 2 B m , then the p-value of the null hypothesis Y i 2 B m is given by ûAi,m = 1+|j2Dc(Ai,m):S(Xj ,Yj )ÿS(Xi,Yi)| 1+|Dc(Ai,m)|
, which is the proportion of conformity scores that are less than the score of the new sample among all calibration data that have the same protected attribute and fall into the same bin. The prediction interval Ĉm (X i ) in B m is the intersection of B m and the prediction interval calibrated by G a,m ( m ), which includes the part of B m in the prediction interval where the p-value ûAi,m is greater than m . This guarantees the bin coverage of B m at level m in Theorem 4.1. The complete proofs can be found in Appendix 8.2.2.
With improved EOC and coverage guarantee, we now need to identify m for each bin so that the mean width of prediction intervals for D t is the smallest. As such, our goal is to solve a constrained optimization problem, where the decision variables are the coverage rates in each bin m , m = 1, . . . , M , and the objective function is the mean width of prediction intervals in test data:
An easy solution to the optimization problem Eq. 1 can be obtained by adjusting Ĉ(X i ), X i 2 D c with split conformal prediction described in Section 2. However, it is not optimal since m is determined by Q1 α (S, D c ) calculated on all calibration data, but different bins have varying costs of width associated with changes in their coverage rates. For instance, if a bin has a coverage rate of 0.98, increasing it to 0.99 would lead to a significant increase in its width, whereas increasing the coverage rate of a bin from 0.50 to 0.51 would result in only a minor increase in width. Therefore, we use the solution of split conformal prediction as the initialization and then optimize it.
Solving Eq. 1 is challenging in that it is computationally expensive or even infeasible to compute the value and the gradient of the objective function. First, the computation of prediction interval C(X i ) involves multiple intersection and union operations, which is a complex step function of m . Second, prediction interval C(X i ) is related to quantile value G a,m , which is estimated from data with noise. Directly using slopes of G a,m as the gradient methods could result in over-fitting to noise, as shown in Section 5.2.4. Third, We cannot assume the objective function's convexity or differentiability as the data may come from any possible distribution, and sorting is involved in calculating G a,m [38]. To address those obstacles, we propose an efficient and robust optimization algorithm (detailed in Algorithm 2 in Appendix 8.3.2) that utilizes a relaxed upper bound and optimizes through approximated subgradients [39].
The steps of our approach are described below. First, to accelerate computing, we use a dummy continuous prediction interval C d (X i ) = Convex( S m C m (X i )), i.e., the convex hull of all subintervals, as an upper bound to substitute the original prediction interval C(X i ) in the objective function. However, C d (X i ) is still related to noisy G a,m . To address this, in each iteration, we compute the slope for each bin m in increasing and decreasing directions, denoted as t+ m and t m , respectively. Since the objective function aims to decrease without changing the marginal coverage rate, we take a greedy strategy by moving up a step ; in the bin with the steepest descendent direction max m { t m }, meanwhile taking a step ; down the slowest ascendant direction min m { t+ m }. We stop until max m { t+ m } min m { t m } + 2", where " is an appropriate estimation error bound related with |D c | [40,41]. More details can be found in Appendix 8.3.1.
The proposed method can be viewed as a subgradient method, incorporating considerations for quantile value estimation errors and maintaining a constant mean coverage rate. Combining the three steps, we could get prediction intervals with improved EOC, guaranteed marginal coverage, and decreased average width of prediction intervals.
In this section, we conduct three sets of experiments on both synthetic and real-world data to evaluate (1) the effectiveness of the proposed approach for achieving EOC (Section 5.2) ; (2) the impact of the key parameter M , the number of bins (Section 5.2.3); and (3) the effectiveness and efficiency of the proposed optimization framework (Section 5.2.4).
We propose two metrics to evaluate the first property of EOC, i.e., whether coverage rates for different groups with similar outcomes are close. With the M discretized bins in Section 4.1, the first metric is the average of maximum difference in coverage rates between groups for all bins, similar to the definition of conditional KS distance in [12]. In main experiments, M is set as 20. The second metric T is introduced in Lemma 3.2, where T is calculated on d = d|D t | 2/5 e bins. Specifically, d 6 = M for fairness of evaluation, e.g., d = 100 for synthetic data. A smaller T implies a closer distance to the ideal EOC distribution and, therefore a better EOC. Furthermore, for the second property of EOC, we need to ensure the desired marginal coverage rates. The efficiency of uncertainty estimation is measured by the width of the prediction interval. In addition, we check the conditional coverage rates on groups to measure equalized coverage. All metrics related to coverage rates are multiplied by 100 in tables to exhibit significant differences. Every experiment is repeated 100 times on random divisions of data with different seeds, with
We compare our method with the following state-of-the-art methods: 1) Split Conformalized Quantile Regression (CQR) [35,21] with only marginal coverage guarantee; 2) Group-conditional Conformalized Quantile Regression (GCQR) [11] with both marginal and conditional coverage guarantee; 3) MultiValid Predictor (MVP) [18] with both marginal and conditional coverage guarantee. Note that the base model for this algorithm is trained on the union of training and calibration data as MVP does not require any calibration data; 4) Conformal Fair Quantile Prediction (CFQR) [19], which guarantees marginal coverage and demographic parity on both upper and lower bounds of the predicted intervals; 5) Label-conditional Conformalized Quantile Regression (LCQR) [42], which is designed for classification problems to provide marginal coverage and equalized coverage for each class. We adapt it to regression tasks. The base model for all compared conformal prediction methods is set as the QR model at the level of 0.05 and 0.95, and the desired marginal coverage is set to 0.9. Considering that for some real-world applications like scoring [43], disjoint prediction intervals make little sense, we evaluate prediction intervals C(X) along with their dummy prediction intervals C d (X) in Section 4.3, represented as BFQR and BFQR*. In the comparison tables, the best results and the second-best results are highlighted in bold and underlined, and undercovered groups who fail to reject the null hypothesis at 0.05 level in one-tailed t-tests are emphasized in Italian.
We generate ten independent and exponentially distributed features with the scale of 1, X = (X 1 , . . . , X 10 ); protected attribute A is randomly selected from {0, 1, 2} with a probability of 0.1, 0.2, 0.7, respectively. The labels Y for A = 1 follow a random distribution, thus impossible to predict; labels Y for the other two groups are the linear summations of X and A, plus noises that increase with Y . A size of 100,000 samples are generated from this distribution and the data generating process is detailed in Appendix 8.4.1.
We have the following observations from the results in Table 1: 1) All methods have marginal coverage guarantee, which is attributed to statistical guarantee from conformal prediction. 2) Our proposed methods, BFQR with disjoint intervals and BFQR* with joint intervals achieve the best trade-off between EOC and equalized coverage. In particular, our method achieves the second-best EOC (i.e., Mean Max Coverage Gap and T ), meanwhile, the conditional coverage rates are almost equal across different groups. While CFQR has significantly better performance w.r.t. EOC, the conditional coverage for A = 1 (the most challenging case) is extremely low compared to A = 0 and A = 2. This result aligns with the mutual exclusivity between EOC and equalized coverage formulated in Theorem 3.1. 3) Among all methods, the average interval width of our method is the smallest, validating the effectiveness of the optimization process in Section 4.3. One of the main advantages of BFQR and BFQR* is optimizing through bin coverages. In this process, bins that sacrifice interval width for an over-coverage rate are adjusted to a lower but satisfactory coverage rate. Therefore, we are able to guarantee a smaller average interval width while improving the EOC.
We further evaluate our method on two benchmark datasets: Adult [44,45] where gender is the protected attribute and the outcome is salary; MEPS (Medical Expenditure Panel Survey) data [46,11] where race is the protected attribute and the outcome is the health care system utilization score.
We observe similar results: For Adult data (shown in Table 2), all methods achieve marginal coverage.
Our methods achieve the best EOC and smallest mean width of prediction interval while maintaining competitive conditional coverage rates. CFQR with the best EOC on synthetic data does not have consistently good performance, and the mean interval width is greatly larger compared with other methods. The increased width of LCQR implies that as some bins are difficult to predict, enforcing all bins to reach the same high coverage rates generates prediction results with little useful information.
For MEPS data (shown in Table 3), the results are similar to those in synthetic data: our method is the second-best w.r.t. EOC, with marginal coverage rates guaranteed and a significantly smaller average prediction width. The number of bins, M , is the only primary parameter in our method. Using synthetic data that ideally satisfies Assumption 1, we evaluate how M influences EOC. In our experiment, M varies among {1, 5, 10, 20, 50}. When bin size is 1, our method degenerates to GCQR. The results are shown in Fig. 3, where the best EOC for BFQR and BFQR* is achieved with 10 bins. When the number of bins is small, the sample size of D a,m increases. According to Theorem 4.1, the upper bound of the expectation of max group coverage rate gap is decreased. However, this only suggests a decrease in the expectation, not for every Y within bins. When the quantile value is calculated on large samples, it fails to characterize the conformity scores of individuals. BFQR* with continuous intervals exhibits better EOC compared to BFQR as its coverage gaps are primarily influenced by the quantile values of the first and last bins, involving less randomness. As M greatly affects the performance of our methods, it should be cautiously chosen for various problem settings. To demonstrate the efficiency of the proposed optimization method, we compare it with three recent optimization methods: 1) BFGS-SQP [47,48], a constrained nonsmooth optimization method, 2) Augmented Lagrange (AL) [49], another method to solve constrained nonsmooth optimization problems, and 3) Bayesian optimization [50,51], a global optimization method for solving problems with noisy objective functions. We evaluate these methods based on their performance in terms of EOC, width, and running times. The results on synthetic data are in Table 4. The Bayesian method has extremely slow computational speed and cannot guarantee marginal coverage. In comparison to BFGS-SQP and AL, our method achieves similar EOC performance but with slightly lower prediction interval widths, which confirms the effectiveness of our subgradient approximation. Additionally, our method is more efficient as it significantly reduces running times. EOC or equalized coverage? The results in Section 5.2 clearly demonstrate that perfect EOC and equalized coverage are mutually exclusive, empirically verifying Theorem 3.1. In real-world applications, it becomes crucial to trade off between EOC and equalized coverage. We recommend placing a higher emphasis on EOC when certain labels are considered more favorable, e.g., when individuals labeled as low-salary are the most important subpopulation for decision-making. Under EOC, each group with the same label is treated equally, not only in terms of equal coverage rates (i.e., equal probability of being included in the prediction result) but also in terms of comparable prediction interval widths as a measure of uncertainty. For example, under equalized coverage, the coverage rate for men is around 0.65 but 0.55 for women in the first bin, low-income population. We cannot conclude that the model is more confident in the prediction results for low-income men solely based on the larger prediction intervals of men: this discrepancy might arise from the over-coverage of men. In this sense, our designed metric EOC, not only contributes to evaluating and enhancing fairness under uncertainty but also serves for fair uncertainty quantification, discovering model bias in uncertainty quantification. A possible application of EOC is for guiding sample selection in active learning [52,53], allowing the applied model to label the most uncertain samples in a fair manner.
Discretizition into equal-mass bins is a favorable strategy to avoid excessively small sizes of D(a, m). While this guarantees a tighter upper bound for the expectation of the maximum coverage gap within bins according to Theorem 4.1, our method can be extended beyond equal-mass bins. When the number of samples in each bin is unequal, we adjust the constraints in Eq. 1 by incorporating a weighted sum of m , i.e., P
Consequently, in Algorithm 2, the gradient in the direction of each bin also should be weighted by |D(m)|. These adaptations allow our method to maintain its effectiveness when the sample sizes in each class are uneven, e.g., in imbalanced classification problems. Moreover, our method is able to address certain coverage rate concerns within particular bins, such as low-income females, by assigning specific coverage rates to the corresponding bins while allowing flexibility in adjusting the coverage rates of other bins.
Why Better EOC? Our methods have been empirically shown to have better EOC than methods designed for equalized coverage, and here we elucidate the reasons behind this improvement. Given the underlying distribution of the µ-quantile conformity scores in group a conditioned on y as Q a,y (µ). For every y and a, calibrating with Q a,y (µ) would generate perfect EOC. However, due to the limitations of available calibration samples, the estimation of Q a,y (µ) is subject to bias. Since the base model cannot perfectly capture the underlying distribution, the conformity score, as a measure of disagreement, must correlate with Y . While equalized coverage methods generally ignore this correlation, our method tries to handle it. In Section 4.1, we used G a,m ( m ) to estimate Q a,y ( m ) for all y 2 B m , which is the m -quantile value of conformity scores whose y belong to bin B m . The localized estimation within discretized bins is upper-bounded by the largest m -quantile value of score conditioned on any y and lower-bounded by the smallest one, regardless of the unknown distribution of Y . Consequently, we could better characterize the conditional distribution within each bin. This enhancement is especially evident when the Lipschitiz constant L in Lemma 3.2 is small, such that the deviation from perfect EOC to the distribution generated by bins is negligible.
Possible Extension. For future work, we aim to further improve our method by seeking a tighter bound to achieve optimal EOC. Possible direction may follow [14,54], by discretizing Y into finite values and bounding deviation in EOC of the discretized predictor. Additionally, our conformalized method leverages statistical properties to ensure coverage rates in each bin and optimize prediction interval widths, making it particularly appropriate for large datasets. With a better discretization strategy that better utilizes the calibration samples, e.g., ensemble sampling, our method may maintain its effectiveness on a small calibration set.
Potential Exploration beyond the Current Scope. The context of this paper is situated in postprocessing fairness. Though post-processing methods typically underperform in-processing methods, in-processing methods are not applicable in many situations, e.g., the prediction model is a pre-trained black-box regression model, or a flexible and computationally efficient method is required. Whereas, adapting the proposed method to an in-processing setting to obtain better EOC is a possible extension. It would also be interesting to extend our method to the context that sensitive attributes are not available in the test data, possibly through missing data augmentation for the calibration set [55] or prediction-powered inference [56].
In this paper, we introduce a new uncertainty-aware fairness notion, equal opportunity of coverage (EOC), which addresses the limitations of the seminal work of equalized coverage [11]. EOC has several desired properties: It guarantees equal coverage rates for groups with the same labels and marginal coverage rate at a pre-determined level. It also ensures a small prediction interval. The theoretical analyses and empirical findings indicate that EOC and equalized coverage are generally incompatible. We suggest using EOC as an alternative to equalized coverage when equal coverage rates and assessment of uncertainty are needed for more fine-grained demographic groups. To improve EOC, we propose a distribution-free post-processing method, BFQR, based on discretization. Experimental results on synthetic and real-world datasets show that BFQR achieves competitive EOC and ensures guaranteed marginal coverage rates with small mean prediction interval widths compared to the state-of-the-art. Moreover, BFQR is adaptable to various settings, such as classification and other decision-making tasks.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).