Introduction

Machine learning models are being used in numerous applications such as healthcare (De Fauw et al. 2018), online advertising (Perlich et al. 2014), and finance (Malekipirbazari and Aksakalli 2015). Due to its increased proliferation, there is a rising concern in the machine learning community to deploy fair machine learning models (Barocas, Hardt, and Narayanan 2017;Mehrabi et al. 2021). Since decision-making in machine learning comprises of various stages such as the data stage, modeling stage, and prediction stage (Suresh and Guttag 2019), it becomes imperative to look at the fairness problem across stages, rather than limiting the discussion to a single stage. For instance, the data stage could be biased due to members of a subgroup being systematically selected with a higher or a lower probability than others (Medical-Dictionary 2016), also known as sample selection bias. Such biases could propagate to the prediction stage, and the resulting biases in prediction could be compounded by other sources such as model misspecification (Gajane and Pechenizkiy 2017). However, it is unclear precisely how and to what extent the data bias would affect the predictions, and when the resulting prediction biases would be detectable by some auditing approach. Such biases, once detected and precisely characterized, could then be corrected, e.g., by resampling to de-bias the data.

In this paper, we analyze the propagation of differential sampling bias from the data stage to the prediction stage. Differential sampling bias is a form of sample selection bias in which some subpopulation S is sampled non-uniformly, such that the distribution of an outcome variable Y given predictor variables X in the sampled data for S differs from the true (population) distribution of Y given X for S. 1This bias can arise in many different circumstances. For example, in criminal justice, both the organizational biases of police departments (e.g., a policy of conducting large numbers of pedestrian stops in predominantly minority neighborhoods) and the perceptual biases of individual police officers (e.g., higher likelihood of stopping and frisking Black individuals) led to much higher proportions of Black individuals being arrested for marijuana possession, despite similar rates of use in the population as a whole (Edwards et al. 2020). In our analysis of NYPD stop and frisk data, we consider the race of the stopped individual as our outcome variable, and observe that Pr(race = "Black") is significantly increased as compared to a "less biased" alternative policing strategy.

Differential sampling bias can also result from concept shift: a model meant for prediction of outcome variable Y in one setting is learned using data from a different setting where the relationship between Y and the predictor variables X differs. For example, if criminal justice data from one jurisdiction is used to predict a defendant's risk of reoffending in a different jurisdiction, or if historical data is used and reoffending patterns have changed over time, the training data will exhibit differential sampling bias: the proportion of reoffenders for certain demographics may be higher or lower in the training data as compared to the true probabilities for the jurisdiction and time period of interest. In our experimental analysis of the COMPAS dataset, we inject simulated differential sampling bias (assuming concept shift) by weighted resampling of the training data. Step 4: Bias Scan finds the most biased subgroup S * and its log-likelihood ratio score F * based on the predictions.

Here we introduce the first formal analysis of how differential sampling bias induced in the data stage (i.e., biased training data) propagates through the modeling and prediction stages, leading to significant biases in prediction. These propagated biases can then be detected by an auditor that compares the model predictions with the observed outcomes.

Our problem setup is shown in Figure 1: first, in the data stage, we assume initially unbiased training and test data records drawn i.i.d. from some joint probability distribution f X,Y (x, y) of the predictor variables X and binary outcome variable Y . Then differential sampling bias ∆ is injected into the "true" subgroup S T for the training data only. Without loss of generality, we define Y such that the differential sampling bias increases the probability P(Y = 1|X), thus over-sampling records with Y = 1 in subgroup S T . We parameterize the multiplicative increase in the odds of Y = 1 by ∆ > 1. Second, in the modeling stage, a classification model is trained using the biased data. Third, in the prediction stage, the classifier makes predictions p i (the estimated probability that Y = 1 for each data record) for the test data. Fourth, Bias Scan (Zhang and Neill 2016) is used to assess whether the predictions p i are systematically biased as compared to the test outcomes y i for any intersectional subgroup.

Given this problem setup, we present theoretical and empirical results showing (a) the amount of bias that propagates from the data stage to the prediction stage, as measured by the log-likelihood ratio (LLR) score found by Bias Scan; and (b) when the bias will exceed a threshold for significance, assuming a fixed false positive rate α, thus enabling detection by Bias Scan. Our specific contributions are as follows:

1. We define and quantify the differential sampling bias ∆ induced into subgroup S T in the binary outcome Y . 2. We derive a new closed-form expression for the LLR score of Bias Scan, used to audit a consistent classifier trained on large data with differential sampling bias. 3. We present a new asymptotic result for the null distribution of the Bias Scan score, which leads to a threshold score h(α) for detection at a fixed false positive rate α. 4. We demonstrate detection with full asymptotic power, P H1 (Reject H 0 ) → 1, as the data size becomes large. 5. Using the threshold h(α), we find the minimum amount of bias ∆ that needs to be induced in subgroup S T for it to be provably detectable in the finite sample case. 6. We evaluate our theoretical results empirically on two different criminal justice datasets. On the well-known COMPAS dataset, we compare the empirical and theoretical relationships between the Bias Scan score F * and the amount of injected bias ∆, across two different classification models and two types of bias injection (marginal and intersectional). We also analyze historical data from the NYPD's "stop-question-frisk" (SQF) policy, estimating the amount of differential sampling bias ∆ in the data as compared to a "less biased" alternative policing strategy. 7. For both datasets, we observe that the empirical relationship between the propagated bias in predictions (as measured by the Bias Scan score F * ) and the differential sampling bias in data (as measured by ∆) corresponds well to the theoretical values. We also confirm that, if enough bias is present in the data stage, then the affected subgroup is detectable by the auditor in the prediction stage with high accuracy. These two conclusions demonstrate the validity of the theoretical assumptions and provide reasoning when theoretical and empirical results differ.

Related Work

Stage-specific notions of fairness and bias: The machine learning community has typically centered the fairness problem in either the data stage or the prediction stage (Barocas, Hardt, and Narayanan 2017) and Goel (2018) discuss the limitations of these fairness definitions; Kleinberg, Mullainathan, and Raghavan (2016) and Chouldechova (2017) prove that, except in special cases, these definitions are incompatible; Zadrozny (2004) proposes a framework to correct bias in model predictions; and Pedreschi, Ruggieri, and Turini (2009) propose novel measures of discrimination to correct discriminatory patterns. None of the aforementioned works have analyzed how bias propagates downstream, across different stages of the pipeline.

Bias propagation pipelines: Suresh and Guttag (2019) discuss the bias problem holistically, rather than centering it to a particular stage, by laying out a framework comprising of biases originating at different stages of the pipeline. Similarly, an opinion article by Hooker (2021) proposes that bias should be viewed and analyzed as an aggregation of the biases arising in different stages. However, neither of these works provide any formal, quantitative analysis of how bias propagates between stages. Rambachan and Roth (2019) quantitatively analyze how selection bias propagates from the data stage to the prediction stage. However, the study makes a strong assumption about the form of the selection process, and does not discuss whether the propagated bias is detectable or how it can be detected in the prediction stage.

Frameworks for detection of intersectional biases: Several recent approaches have been proposed to detect biases affecting a subpopulation defined along multiple data dimensions (Zhang and Neill 2016;Kearns et al. 2018). Here we apply Bias Scan (Zhang and Neill 2016) to assess models learned from biased data, detecting intersectional subgroups where the model predictions p i most significantly overestimate P(Y = 1 | X = x i ). Bias Scan builds on previous univariate and multivariate subset scan approaches (Neill 2012;Neill, McFowland III, and Zheng 2013). Additionally, McFowland III, Somanchi, and Neill (2018) use a similar multidimensional scan framework to discover the subgroups that are most significantly affected by a treatment in a randomized experiment, and provide statistical guarantees on detection. However, all of these approaches focus on a single pipeline stage (predictions or outcomes), while our work examines the propagation of data biases into model predictions.

Preliminaries Notations

Here Y is a binary outcome variable, and thus we can write f Y |X (y|x) and f Y |X (y|x) as the probabilities P(Y = y | X = x) and

, for test and training data respectively. Let p i = P(Y = 1 | X = x i ) be the true probability that Y = 1 for test record s i = (x i , y i ), and let p i and pi be the estimated probabilities that Y = 1 for test record s i from classification models learned from training data with and without differential sampling bias. Note that p i = pi when ∆ = 1 (under the null hypothesis of no bias).

We assume that X consists of a set of discrete-valued 2 predictor variables {X 1 , . . . , X Q } and that each variable X i takes on a set of values V i . An intersectional subgroup S is defined as a subset of the Cartesian product

Race, and V 2 = {Black, White, Other}, then {Male, Female} × {Black, White} = {(Male, Black), (Female, Black), (Male, White), (Female, White)} is a rectangular subgroup, while {(Male, Black), (Female, White)} is non-rectangular. Let rect(X) denote the set of all rectangular subgroups of X. Finally, for test dataset D, we associate with any given subgroup S the subset of matching data records

Bias Scan

Bias Scan (Zhang and Neill 2016) is a multi-dimensional subset scanning algorithm used to detect intersectional subgroups for which a classifier's probabilistic predictions p i of a 2 Sensitive covariates (e.g. race, ethnicity, and gender) are usually discrete. Continuous covariates can be discretized as a preprocessing step, using the observed covariate distribution or domain knowledge.

binary outcome y i are significantly biased as compared to the observed outcomes y i . More precisely, Bias Scan searches for the rectangular subgroup S * which maximizes a Bernoulli log-likelihood ratio (LLR) scan statistic,foot_1

To obtain the score function for a given subgroup S, Bias Scan computes the generalized log-likelihood ratio F (S) = max q log P (D | H1(S, q)) P (D | H0) , assuming the following hypotheses:

Here we detect biases where the probabilities p i are overestimated, and thus 0 < q < 1. As derived in the Technical Appendix, the resulting log-likelihood ratio score F (S) is

(1) The Bias Scan algorithm for optimizing F (S) over rectangular subgroups is provided in the Technical Appendix.

Differential Sampling Bias

In this section, we quantify differential sampling bias for a subgroup S, as follows: Definition 1. A subgroup S exhibits differential sampling bias ∆ > 1 towards the outcome Y = 1 if, for all x ∈ S,

For example, differential sampling bias could be injected into unbiased training data by re-drawing data elements {( x i , y i )}, for x i ∈ S, with replacement, with sampling weights w i = ∆ for y i = 1 and w i = 1 for y i = 0. We use this approach to inject bias into the COMPAS dataset in our experiments below.

Theoretical Results

In this section, we derive theoretical results to understand the propagated effects of differential sampling bias and to provide statistical guarantees for detectability.

More precisely, we prove four main theorems. Given the problem setup described above and the assumptions listed below, Theorem 1 provides an asymptotic closed-form formulation of the Bias Scan log-likelihood ratio (LLR) score F (S T ) of the injected subgroup S T as a function of the amount of differential sampling bias ∆. If S T is a rectangular subgroup, S T ∈ rect(X), this score is a lower bound on the overall Bias Scan score F * = max S∈rect(X) F (S). Theorem 2 provides an upper bound for the null distribution of F * (i.e., assuming no bias is present), enabling us to compute a threshold score for detection. Finally, Theorems 3 and 4 combine these results to show asymptotic detection with full power for any ∆ > 1 as the sizes of the training and test data go to infinity, as well as computing the minimum amount of bias ∆ needed for detection in finite test data.

These Theorems rely on three key assumptions: (A1) Consistency of the classifier used in the prediction stage, for learning the conditional distribution f Y |X .

(A2) Full support of the biased training data:

Given these assumptions, we first derive the relationship between the amount of differential sampling bias ∆ injected into subgroup S, and the Bias Scan score F (S):

Theorem 1. Assume that a classifier is trained on data D with differential sampling bias ∆ > 1 for subgroup S and makes predictions p i for unbiased test data D = {(x i , y i )}. If Bias Scan is used to assess bias in p i as compared to y i , then under assumptions (A1)-(A3), as the number of training data records | D| → ∞, the Bias Scan score F (S) of subgroup S converges to:

if ∆ > qMLE , and F (S) → 0 otherwise, where qMLE is the maximum likelihood estimate of q for Bias Scan assuming no differential sampling bias (∆ = 1), satisfying

and

is the Bias Scan score of subgroup S assuming no differential sampling bias (∆ = 1).

The proof of Theorem 1 is provided in the Technical Appendix. Critically, under assumptions (A1)-(A3), as | D| → ∞, we have

and the corresponding predicted probabilities with no differential sampling bias, pi

We then show that the maximum likelihood estimate (MLE) of q for Bias Scan is qMLE /∆, where qMLE is the corresponding MLE with no differential sampling bias. Finally, we plug in the expressions for p i , pi , and q M LE , and simplify. Corollary 1. Under the conditions of Theorem 1, as the number of test data records |D| → ∞, the normalized Bias Scan score F (S)/|D| of subgroup S converges to:

an increasing function of ∆.

Next, we provide statistical guarantees for the detection of bias. To do so, we first consider the distribution of the Bias Scan score F * = max S∈rect(X) F (S) under the null hypothesis of no bias, H 0 . For a given false positive rate α, we find a score threshold h(α) such that P H0 (F * > h(α)) ≤ α.

To do so, we make the additional assumption:

(A4) The number of unique covariate profiles in the test data, M , is large enough so that Gaussian approximations hold (e.g., M > 30) but finite (i.e., M remains constant as the number of test data records |D| → ∞).

Then we can show the following: Theorem 2. Assume that a classifier is trained on unbiased training data D and makes predictions pi for unbiased test data D = {(x i , y i )}, and Bias Scan is used to assess bias in pi as compared to y i . Let F * = max S∈rect(X) F (S) be the Bias Scan score, maximized over all rectangular subgroups S. Then under assumptions (A1)-(A4), as the number of training data records | D| → ∞ and the number of test data records |D| → ∞, for a given Type-I error rate α > 0, there exists a critical value h(α) and constants k 1 ≈ 0.202, k 2 ≈ 0.523 such that P(F * > h(α)) ≤ α, where

and Φ is the Gaussian cdf.

Critically, h(α) does not depend on the number of test data records |D|, but only on the number of unique covariate profiles in the test data M . Now, we prove that under the presence of bias ∆, h(α) serves as a threshold for rejecting the null hypothesis of no bias with full asymptotic power.

Theorem 3. Assume that a classifier is trained on data D with differential sampling bias ∆ > 1 for rectangular subgroup S T and makes predictions p i for unbiased test data D = {(x i , y i )}, and Bias Scan is used to assess bias in p i as compared to y i . Let F * = max S∈rect(X) F (S) be the Bias Scan score, and let h(α) be the score threshold for detection at a fixed Type-I error rate of α, as given in Equation (3). Then for any α > 0 and ∆ > 1, under assumptions (A1)-(A4), as the number of training data records | D| → ∞ and the number of test data records |D| → ∞, P(F * > h(α)) → 1.

We now find the minimum bias that needs to be induced into subgroup S to be detectable for a given Type-I error rate.

Theorem 4. Assume that a classifier is trained on data D with differential sampling bias ∆ > 1 for rectangular subgroup S T and makes predictions p i for unbiased test data D = {(x i , y i )}, and Bias Scan is used to assess bias in p i as compared to y i . Let F * = max S∈rect(X) F (S) be the Bias Scan score, and let h(α) be the score threshold for detection at a fixed Type-I error rate of α, as given in Equation (3). Further, assume D S T is fixed, with finite size |D S T | and si∈D S T y i < |D S T |. Then for any α > 0, under assumptions (A1)-(A4), as the number of training data records | D| → ∞, there exists ∆ thresh ≥ 1 such that, if ∆ > ∆ thresh , then P(F * > h(α)) → 1, where

and F old (S T ) is the Bias Scan score of subgroup S T assuming no differential sampling bias (∆ = 1). Proofs of Theorems 1-4 are provided in the Appendix.

Experiments

We perform experiments on two criminal justice datasets to validate our theoretical results: semi-synthetic predictions of recidivism risk derived from the well-known COMPAS dataset, and real-world "stop, question and frisk" (SQF) data from the New York Police Department (NYPD).

Experiments on COMPAS/ProPublica Data

COMPAS is a commercial decision-support algorithm which has been applied in many jurisdictions to estimate a defendant's probability of reoffending, with impacts on criminal justice outcomes such as bail, sentencing, and parole. COM-PAS gained notoriety when investigative journalists from ProPublica published a study arguing that COMPAS was racially biased against Black defendants (Angwin et al. 2016).

The public dataset compiled by ProPublica 4 , including COM-PAS risk predictions for 7,214 defendants in Broward County, Florida, from 2013-2014, and a two-year follow-up to record which defendants were rearrested, has been studied by numerous algorithmic bias researchers (Barenstein 2019). While most of these analyses focus on assessing biases in the COMPAS risk predictions (Chouldechova 2017;Kleinberg et al. 2018), we instead utilize this dataset to learn predictive models for the binary outcome (rearrest within two years) as a function of five categorical predictor variables 5 , and use these models to study how differential sampling bias in the data propagates to the model predictions.

To do so, we consider differential sampling biases ∆ ∈ {1, 1.25, 1.5, . . . , 10} injected into one of two rectangular subgroups. Letting X 1 = Gender, X 2 = Race, and V j = the set of all possible values for attribute X j , we consider the subgroups S T = {Female} × V 2 × . . . × V 5 and S T = {Female} × {Caucasian} × V 3 × . . . × V 5 . The first subgroup represents a marginal bias against females (since we are oversampling females who reoffended, as compared to females who did not reoffend, by a factor of ∆ in the training data, thus leading to an overestimate of their reoffending risk), while the second subgroup represents an intersectional bias against white females. We also consider two different classifiers, random forest and logistic regression, and average results over 100 trials for each combination of classifier, injected subgroup S T , and amount of bias ∆.

4 https://github.com/propublica/compas-analysis/compas-scores-two-years.csv 5 Predictors include gender, race, charge degree, age < 25, and number of prior offenses ("none", "1 to 5", or "more than 5").

For each trial, we randomly partition the data into 80% training and 20% testing data. If ∆ > 1, then differential sampling bias ∆ is injected into subset S T for the training data D, resampling data records ( x i , y i ) ∈ D S T with replacement (where records with y i = 1 have weight ∆ and records with y i = 0 have weight 1), and leaving the test data D and the rest of the training data unchanged. The classifier is trained on the biased training data, and used to make predictions p i on the unbiased test data. Then Bias Scan is used to assess whether these predictions are biased, reporting the highest scoring subgroup S * = arg max S∈rect(X) F (S) and its score F * = F (S * ). We then compare the values of the Bias Scan score F * , the score of the injected subgroup F (S T ) (calculated by equation ( 1)), and the theoretical score of S T , which we denote as F theo (S T ). The value of F theo (S T ) is computed using only the unbiased training and test data, as defined in Theorem 1:

if ∆ > qMLE , and F theo (S T ) = 0 otherwise. We also compute the overlap (Jaccard coefficient) between the injected subset of test data records D S T and the detected subset D S * :

Finally, we use Theorems 2 and 4 to estimate the critical value h(α) and the corresponding threshold value ∆ thresh , for which we expect

Given these values for each amount of bias ∆ (averaged over the 100 trials, for a given classifier and a given injected subgroup S T ), we form two plots: one comparing F * , F (S T ), and F theo (S T ) as a function of ∆, and one showing overlap between D S T and D S * as a function of ∆, as compared to ∆ thresh .

If assumptions (A1)-(A4) hold, as the size of the training data grows to infinity, we expect perfect overlap between the curves for F theo (S T ) and F (S T ) by Thm. 1. As ∆ becomes large compared to ∆ thresh , we expect S * ≈ S T , and thus F * ≈ F (S T ) and overlap ≈ 1, while for small ∆, we expect F * > F (S T ) and overlap ≪ 1. We now examine whether these expectations are met for the finite, real-world COMPAS dataset, for each classifier and each injected subgroup S T .

Experimental results

For the logistic regression classifier learned from training data injected with marginal differential sampling bias (Figure 2), we observe near-perfect overlap between the observed score F (S T ) and theoretical score F theo (S T ) for the injected subgroup S T , suggesting the validity of our theoretical results above. As expected, the Bias Scan score F * ≈ F (S T ) and overlap ≈ 1 for ∆ > ∆ thresh , while F * > F (S T ) and overlap ≪ 1 for small ∆. For the random forest classifier learned from training data injected with marginal differential sampling bias (Figure 3), we see similar results, but with F (S T ) slightly greater than F theo (S T ) for large ∆. This is likely due to data sparsity: the combination of finite training data and high bias may lead to few or no training data records with y i = 0 for some covariate profiles in the injected subgroup, leading to inaccurate estimation of P(Y = 1 | X). This pattern is repeated for the random forest classifier learned from training data injected with intersectional differential sampling bias (Figure 4), with a larger gap between F (S T ) and F theo (S T ), most likely due to the smaller amount of training data in S T . Similarly, the smaller amount of test data in S T leads to some noise in the detected subgroup, resulting in overlap ≈ 0.9 rather than 1, and thus F * = max S∈rect(X) F (S) > F (S T ). Nevertheless, these results suggest that the theoretical values of F theo (S T ) and Δ thresh are good approximations even for finite data.

For the logistic regression classifier learned from training data injected with intersectional differential sampling bias (Figure 5), however, we see a very different picture: as Δ increases, the Bias Scan score F * and the score of the injected subgroup F (S T ) are both much smaller than the theoretical score F theo (S T ), and the overlap between S * and S T plateaus around 0.4 even for large Δ. This is because assumption (A1) is violated: the logistic regression model is misspecified and cannot learn the intersectional bias against white females, instead learning separate (and much smaller) marginal biases against all females and all white individuals via the learned model coefficients on these terms. When an interaction term for white females is manually added to the logistic regression model specification (Figure 6), we observe that this additional term resolves the problem, and we again have a near-perfect match between the theoretical and observed scores for the injected subgroup S T .

Experiments on NYPD Stop and Frisk Data

The New York Police Department (NYPD) has long been plagued with accusations of racially discriminatory policing practices related to its "stop, question, and frisk" (SQF) policies. Gelman, Fagan, and Kiss (2007) found that persons of color "were stopped more frequently than whites, even after controlling for precinct variability and race-specific estimates of crime participation". Goel, Rao, and Shroff (2016) concluded that Black and Hispanic individuals were disproportionately impacted by "low hit rate" stops, where the officer suspected the stopped individual of criminal possession of a weapon (CPW) but the ex ante probability of recovering a weapon was low. Here we assess racial bias in NYPD policing practices by analyzing five years of SQF data during the peak of the stop and frisk policy, prior to a 2013 court ruling (Floyd v. City of New York) that NYPD stop-and-frisk tactics were unconstitutionally targeting New Yorkers of color. Thus our dataset consists of 760,489 pedestrian stops (made by NYPD officers for suspected CPW) from 2008-2012, downloaded from the city's web sitefoot_2 . Following Goel, Rao, and Shroff (2016), we first fit a logistic regression model to predict the probability that each stopped individual was found to have a weapon, using location ("housing", "transit", or "neither"), precinct, and 18 binary variables describing the circumstances of the stopfoot_3 as predictors. Stops with ex ante probability of recovering a weapon at least 0.1 were marked as "high probability". If only high probability stops were conducted, 4.8% of stops would have been made, 46% of weapons would have been recovered, and the proportion of stopped individuals who were neither Black nor Hispanic would have more than doubled, from 9% to 23%.

Next we create a new dataset with the demographics of each stopped individual (borough, sex, race, and age decile, all of which were excluded from the predictive model above), and whether each was a high or low probability stop. We then assess racial bias by considering the race of the stopped individual as the outcome variable, and comparing the original, biased policing data to an alternative, "less biased"foot_4 policing practice in which only high probability stops were made.

More precisely, we perform the following steps, for each value of k ∈ {0, 10, . . . , 100}: Thus, for k > 0, this process can be thought of as injecting differential sampling bias, increasing the odds that Race = Black by some factor Δ > 1, as compared to the alternative policing practice of only making high probability stops. However, this scenario poses several new challenges for our theoretical analysis: we do not know the injected subgroup S T or the amount of bias Δ, and in fact the bias may be heterogeneous (different Δ for different covariate profiles). Thus we make several simplifying assumptions. First, when auditing predictions from the model learned from the most biased training data (k = 100), Bias Scan identifies a large, high-scoring subgroup S * consisting of individuals with Gender ∈ {Male, Female}, Age < 70, and Borough ∈ {Manhattan, Brooklyn, Queens, Staten Island} (excluding the Bronx). We assume that this S * is the true injected subgroup S T . Second, we assume that Δ is constant over subgroup S T , and thus compute the odds ratio Δ = p k (1-p0)

(1-p k )p0 , where p k is the proportion of Black individuals in subgroup S T of the training dataset for a given value of k. Thus we have amounts of differential sampling bias ranging from Δ = 1 for k = 0 to Δ = 2.675 for k = 100. We then use these values of Δ along with the "less biased" training and test data (k = 0) to plot F theo (S T ) as a function of k, and compare these theoretical values to the Bias Scan score F * and the subgroup score F (S T ). In Figure 7, we observe that F * = F (S T ) except when k = 0, i.e., the same subgroup S * is detected for all k > 0. Additionally, we see that F theo (S T ) is a relatively good approximation for F (S T ), with F (S T ) consistently about 16% lower than F theo (S T ) across all values of k. This difference can be explained by our approximation of the heterogeneous bias Δ x , for covariate profiles x ∈ S T , by estimating a single, constant Δ value.

Conclusion

It is critical both to analyze the downstream impacts of biases as they propagate through the learning pipeline, and to create new analytical tools to detect and mitigate propagating biases. With this work, we take a step toward these goals by quantifying how a particular data bias, differential sampling bias, propagates into biased model predictions, and providing theoretical guarantees for detection of the propagated biases. We validate our theoretical results through experiments on real-world criminal justice data where our assumptions are relaxed. In future work, we plan to extend our theoretical analysis of propagating biases to other types of data bias (e.g., measurement bias) as well as biases in other pipeline stages. We are particularly interested in analyzing when model predictions are impacted by multiple, interacting biases, which we believe is often the case in complex, real-world settings.