Because the average treatment effect (ATE) measures the change in social welfare, even if positive, there is a risk of negative effect on, say, some 10% of the population. Assessing such risk is difficult, however, because any one individual treatment effect (ITE) is never observed, so the 10% worst-affected cannot be identified, whereas distributional treatment effects only compare the first deciles within each treatment group, which does not correspond to any 10% subpopulation. In this paper, we consider how to nonetheless assess this important risk measure, formalized as the conditional value at risk (CVaR) of the ITE distribution. We leverage the availability of pretreatment covariates and characterize the tightest possible upper and lower bounds on ITE-CVaR given by the covariate-conditional average treatment effect (CATE) function. We then proceed to study how to estimate these bounds efficiently from data and construct confidence intervals. This is challenging even in randomized experiments as it requires understanding the distribution of the unknown CATE function, which can be very complex if we use rich covariates to best control for heterogeneity. We develop a debiasing method that overcomes this and prove it enjoys favorable statistical properties even when CATE and other nuisances are estimated by black box machine learning or even inconsistently. Studying a hypothetical change to French job search counseling services, our bounds and inference demonstrate a small social benefit entails a negative impact on a substantial subpopulation. This paper was accepted by J. George Shanthikumar, data science. Funding: This work was supported by the Division of Information and Intelligent Systems [Grant 1939704]. Supplemental Material: The data files and online appendices are available at https://doi.org/10.1287/mnsc.2023.4819 .
more »
« less
Reliability Assessment of Scenarios for CVaR Minimization in Two-Stage Stochastic Programs
The mass transportation distance rank histogram (MTDRh) was developed to assess the reliability of any given scenario generation process for a two-stage, risk-neutral stochastic program. Reliability is defined loosely as goodness of fit between the generated scenario sets and corresponding observed values over a collection of historical instances. This graphical tool can diagnose over- or under-dispersion and/or bias in the scenario sets and support hypothesis testing of scenario reliability. If the risk-averse objective is instead to minimize CVaR of cost, the only important, or effective, scenarios are those that produce cost in the upper tail of the distribution at the optimal solution. We describe a procedure to adapt the MTDRh for use in assessing the reliability of scenarios relative to the upper tail of the cost distribution. This adaptation relies on a conditional probability distribution derived in the context of assessing the effectiveness of scenarios. For a risk-averse newsvendor formulation, we conduct simulation studies to systematically explore the ability of the CVaR-adapted MTDRh to diagnose different ways that scenario sets may fail to capture the upper tail of the cost distribution near optimality. We conjecture that, as with the MTDRh and its predecessor minimum spanning tree rank histogram, the nature of the mismatch between scenarios and observations can be observed according to the non-flat shape of the rank histogram. On the other hand, scenario generation methods can be calibrated according to uniform distribution goodness of fit to the distribution of ranks.
more »
« less
- Award ID(s):
- 2132200
- NSF-PAR ID:
- 10352209
- Editor(s):
- Ellis, K.; Ferrell, W.; Knapp, J.
- Date Published:
- Journal Name:
- Proceedings of the IISE Annual Conference & Expo 2022
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We study the chance-constrained bin packing problem, with an application to hospital operating room planning. The bin packing problem allocates items of random sizes that follow a discrete distribution to a set of bins with limited capacity, while minimizing the total cost. The bin capacity constraints are satisfied with a given probability. We investigate a big-M and a 0-1 bilinear formulation of this problem. We analyze the bilinear structure of the formulation and use the lifting techniques to identify cover, clique, and projection inequalities to strengthen the formulation. We show that in certain cases these inequalities are facet-defining for a bilinear knapsack constraint that arises in the reformulation. An extensive computational study is conducted for the operating room planning problem that minimizes the number of open operating rooms. The computational tests are performed using problems generated based on real data from a hospital. A lower-bound improvement heuristic is combined with the cuts proposed in this paper in a branch-and-cut framework. The computations illustrate that the techniques developed in this paper can significantly improve the performance of the branch-and-cut method. Problems with up to 1,000 scenarios are solved to optimality in less than an hour. A safe approximation based on conditional value at risk (CVaR) is also solved. The computations show that the CVaR approximation typically leaves a gap of one operating room (e.g., six instead of five) to satisfy the chance constraint. Summary of Contribution: This paper investigates a branch-and-cut algorithm for a chance-constrained bin packing problem with multiple bins. The chance-constrained bin packing provides a modeling framework for applied operations research problems, such as health care, scheduling, and so on. This paper studies alternative computational approaches to solve this problem. Moreover, this paper uses real data from a hospital operating room planning setting as an application to test the algorithmic ideas. This work, therefore, is at the intersection of computing and operations research. Several interesting ideas are developed and studied. These include a strengthened big-M reformulation, analysis of a bilinear reformulation, and identifying certain facet-defining inequalities for this formulation. This paper also gives a lower-bound generation heuristic for a model that minimizes the number of bins. Computational experiments for an operating room planning model that uses data from a hospital demonstrate the computational improvement and importance of the proposed approaches. The techniques proposed in this paper and computational experiments further enhance the interface of computing and operations research.more » « less
-
Many modern machine learning tasks require models with high tail performance, i.e. high performance over the worst-off samples in the dataset. This problem has been widely studied in fields such as algorithmic fairness, class imbalance, and risk-sensitive decision making. A popular approach to maximize the model’s tail performance is to minimize the CVaR (Conditional Value at Risk) loss, which computes the average risk over the tails of the loss. However, for classification tasks where models are evaluated by the 0/1 loss, we show that if the classifiers are deterministic, then the minimizer of the average 0/1 loss also minimizes the CVaR 0/1 loss, suggesting that CVaR loss minimization is not helpful without additional assumptions. We circumvent this negative result by minimizing the CVaR loss over randomized classifiers, for which the minimizers of the average 0/1 loss and the CVaR 0/1 loss are no longer the same, so minimizing the latter can lead to better tail performance. To learn such randomized classifiers, we propose the Boosted CVaR Classification framework which is motivated by a direct relationship between CVaR and a classical boosting algorithm called LPBoost. Based on this framework, we design an algorithm called alpha-AdaLPBoost. We empirically evaluate our proposed algorithm on four benchmark datasets and show that it achieves higher tail performance than deterministic model training methods.more » « less
-
more » « less
Abstract We present a stochastic optimization model for allocating and sharing a critical resource in the case of a pandemic. The demand for different entities peaks at different times, and an initial inventory for a central agency are to be allocated. The entities (states) may share the critical resource with a different state under a risk‐averse condition. The model is applied to study the allocation of ventilator inventory in the COVID‐19 pandemic by FEMA to different U.S. states. Findings suggest that if less than 60% of the ventilator inventory is available for non‐COVID‐19 patients, FEMA's stockpile of 20 000 ventilators (as of March 23, 2020) would be nearly adequate to meet the projected needs in slightly above average demand scenarios. However, when more than 75% of the available ventilator inventory must be reserved for non‐COVID‐19 patients, various degrees of shortfall are expected. In a severe case, where the demand is concentrated in the top‐most quartile of the forecast confidence interval and states are not willing to share their stockpile of ventilators, the total shortfall over the planning horizon (until May 31, 2020) is about 232 000 ventilator days, with a peak shortfall of 17 200 ventilators on April 19, 2020. Results are also reported for a worst‐case where the demand is at the upper limit of the 95% confidence interval. An important finding of this study is that a central agency (FEMA) can act as a coordinator for sharing critical resources that are in short supply over time to add efficiency in the system. Moreover, through properly managing risk‐aversion of different entities (states) additional efficiency can be gained. An additional implication is that ramping up production early in the planning cycle allows to reduce shortfall significantly. An optimal timing of this production ramp‐up consideration can be based on a cost‐benefit analysis.
-
Bardanis, M. (Ed.)It is unlikely to predict the distribution of soil suction in the field deterministically. It is well established that there are various sources of uncertainty in the measurement of matric suction, and the suction measurements in the field are even more critical because of the heterogeneities in the field conditions. Hence it becomes necessary to probabilistically characterize the suction in the field for enhanced reliability. The objective of this study was to conduct a probabilistic analysis of measured soil suction of two different test landfill covers, compacted clay cover (CC) and engineered turf cover (ETC), under similar meteorological events. The size of the two test landfill covers was 3 m × 3 m (10 ft. × 10 ft.) and 1.2 m (4ft.) in depth. The covers were constructed by excavating the existing subgrade, placing 6-mil plastic sheets, and backfilling the excavated soil, followed by layered compaction. Then the covers were instrumented identically with soil water potential sensors up to specified depths. One of the covers acted as the CC, and the other cover was ETC. In ETC, engineered turf was laid over the compacted soil. The engineered turf consisted of a structured LLDPE geomembrane overlain by synthetic turf (polyethylene fibers tufted through a double layer of woven polypropylene geotextiles). The sensors were connected to an automated data logging system and the collected data were probabilistically analyzed using the R program. There were significant inconsistencies in the descriptive statistical parameters of the measured soil suction at both covers under the same climatic conditions. Soil suction measured in the field ranged between almost 12 to 44 kPa in ETC, while it was in the range of almost 1 to 2020 kPa in the CC. The histogram and quantile-quantile (Q-Q) plot showed the data to be non-normally distributed in the field. A heavy-tailed leptokurtic (Kurtosis=13) distribution of suction was observed in the ETC with substantial outliers. In contrast, the suction distribution in CC was observed skewed to the right containing a thinner tail indicating an almost platykurtic distribution. The distribution of suction in the field under engineered turf was observed to be reasonably consistent with time compared to bare soil under the same meteorological events. The results obtained from this study revealed the engineered turf system to be an effective barrier to inducing changes in soil suction against climatic events.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
