More Like this

1. Response to comment on “Typical physics Ph.D. admissions criteria limit access to underrepresented groups but fail to predict doctoral completion”

   https://doi.org/10.1126/sciadv.aba4647  

   Miller, Casey W.; Zwickl, Benjamin M.; Posselt, Julie R.; Silvestrini, Rachel T.; Hodapp, Theodore (June 2020, Science Advances)

   We provide statistical measures and additional analyses showing that our original analyses were sound. We use a generalized linear mixed model to account for program-to-program differences with program as a random effect without stratifying with tier and found the GRE-P (Graduate Record Examination physics test) effect is not different from our previous findings, thereby alleviating concern of collider bias. Variance inflation factors for each variable were low, showing that multicollinearity was not a concern. We show that range restriction is not an issue for GRE-P or GRE-V (GRE verbal), and only a minor issue for GRE-Q (GRE quantitative). Last, we use statistical measures of model quality to show that our published models are better than or equivalent to several alternates. 

   more » « less
2. Safe Learning of Lifted Action Models

   https://doi.org/10.24963/kr.2021/36  

   Juba, Brendan; Le, Hai S.; Stern, Roni (September 2021, Proceedings of the 18th International Conference on Principles of Knowledge Representation and Reasoning)

   Creating a domain model, even for classical, domain-independent planning, is a notoriously hard knowledge-engineering task. A natural approach to solve this problem is to learn a domain model from observations. However, model learning approaches frequently do not provide safety guarantees: the learned model may assume actions are applicable when they are not, and may incorrectly capture actions' effects. This may result in generating plans that will fail when executed. In some domains such failures are not acceptable, due to the cost of failure or inability to replan online after failure. In such settings, all learning must be done offline, based on some observations collected, e.g., by some other agents or a human. Through this learning, the task is to generate a plan that is guaranteed to be successful. This is called the model-free planning problem. Prior work proposed an algorithm for solving the model-free planning problem in classical planning. However, they were limited to learning grounded domains, and thus they could not scale. We generalize this prior work and propose the first safe model-free planning algorithm for lifted domains. We prove the correctness of our approach, and provide a statistical analysis showing that the number of trajectories needed to solve future problems with high probability is linear in the potential size of the domain model. We also present experiments on twelve IPC domains showing that our approach is able to learn the real action model in all cases with at most two trajectories. 

   more » « less
3. Reducing Simulation Input-Model Risk via Input Model Averaging

   https://doi.org/https://doi.org/10.1287/ijoc.2020.0994  

   Nelson, Barry L.; Wan, A.T.K.; Zou, G; Xinyu Zhang, X; Jiang, X (October 2020, INFORMS journal on computing)

   null (Ed.)

   Input uncertainty is an aspect of simulation model risk that arises when the driving input distributions are derived or “fit” to real-world, historical data. Although there has been significant progress on quantifying and hedging against input uncertainty, there has been no direct attempt to reduce it via better input modeling. The meaning of “better” depends on the context and the objective: Our context is when (a) there are one or more families of parametric distributions that are plausible choices; (b) the real-world historical data are not expected to perfectly conform to any of them; and (c) our primary goal is to obtain higher-fidelity simulation output rather than to discover the “true” distribution. In this paper, we show that frequentist model averaging can be an effective way to create input models that better represent the true, unknown input distribution, thereby reducing model risk. Input model averaging builds from standard input modeling practice, is not computationally burdensome, requires no change in how the simulation is executed nor any follow-up experiments, and is available on the Comprehensive R Archive Network (CRAN). We provide theoretical and empirical support for our approach. 

   more » « less
4. Arsm Gradient Estimator for Supervised Learning to Rank

   https://doi.org/10.1109/ICASSP40776.2020.9053127  

   Dadaneh, Siamak Zamani; Boluki, Shahin; Zhou, Mingyuan; Qian, Xiaoning (May 2020, The 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020))

   We propose a new model for supervised learning to rank. In our model, the relevance labels are assumed to follow a categorical distribution whose probabilities are constructed based on a scoring function. We optimize the training objective with respect to the multivariate categorical variables with an unbiased and low-variance gradient estimator. Learning-to-rank methods can generally be categorized into pointwise, pairwise, and listwise approaches. Although our scoring function is pointwise, the proposed framework permits flexibility over the choice of the loss function. In our new model, the loss function need not be differentiable and can either be pointwise or listwise. Our proposed method achieves better or comparable results on two datasets compared with existing pairwise and listwise methods. 

   more » « less
5. Transfer Learning with Uncertainty Quantification: Random Effect Calibration of Source to Target (RECaST)

   Hickey, Jimmy; Williams, Jonathan P; Hector, Emily C (November 2024, Journal of machine learning research)

   Transfer learning uses a data model, trained to make predictions or inferences on data from one population, to make reliable predictions or inferences on data from another population. Most existing transfer learning approaches are based on fine-tuning pre-trained neural network models, and fail to provide crucial uncertainty quantification. We develop a statistical framework for model predictions based on transfer learning, called RECaST. The primary mechanism is a Cauchy random effect that recalibrates a source model to a target population; we mathematically and empirically demonstrate the validity of our RECaST approach for transfer learning between linear models, in the sense that prediction sets will achieve their nominal stated coverage, and we numerically illustrate the method's robustness to asymptotic approximations for nonlinear models. Whereas many existing techniques are built on particular source models, RECaST is agnostic to the choice of source model, and does not require access to source data. For example, our RECaST transfer learning approach can be applied to a continuous or discrete data model with linear or logistic regression, deep neural network architectures, etc. Furthermore, RECaST provides uncertainty quantification for predictions, which is mostly absent in the literature. We examine our method's performance in a simulation study and in an application to real hospital data. 

   more » « less