More Like this

1. Better null models for assessing predictive accuracy of disease models

   https://doi.org/10.1371/journal.pone.0285215  

   Keyel, Alexander C.; Kilpatrick, A. Marm (May 2023, PLOS ONE)

   Ansari, Ali R. (Ed.)

   Null models provide a critical baseline for the evaluation of predictive disease models. Many studies consider only the grand mean null model (i.e. R 2 ) when evaluating the predictive ability of a model, which is insufficient to convey the predictive power of a model. We evaluated ten null models for human cases of West Nile virus (WNV), a zoonotic mosquito-borne disease introduced to the United States in 1999. The Negative Binomial, Historical (i.e. using previous cases to predict future cases) and Always Absent null models were the strongest overall, and the majority of null models significantly outperformed the grand mean. The length of the training timeseries increased the performance of most null models in US counties where WNV cases were frequent, but improvements were similar for most null models, so relative scores remained unchanged. We argue that a combination of null models is needed to evaluate the forecasting performance of predictive models for infectious diseases and the grand mean is the lowest bar. 

   more » « less
2. Post Hoc Explanations of Language Models Can Improve Language Models

   Krishna, Satyapriya; Ma, Jiaqi; Slack, Dylan; Ghandeharioun, Asma; Singh, Sameer; Lakkaraju, Himabindu (December 2023, Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

   Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks. Moreover, recent research has shown that incorporating human-annotated rationales (e.g., Chain-of-Thought prompting) during in-context learning can significantly enhance the performance of these models, particularly on tasks that require reasoning capabilities. However, incorporating such rationales poses challenges in terms of scalability as this requires a high degree of human involvement. In this work, we present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges by automating the process of rationale generation. To this end, we leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. More specifically, we construct automated natural language rationales that embed insights from post hoc explanations to provide corrective signals to LLMs. Extensive experimentation with real-world datasets demonstrates that our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks, including those where prior approaches which rely on human-annotated rationales such as Chain-of-Thought prompting fall short. Our work makes one of the first attempts at highlighting the potential of post hoc explanations as valuable tools for enhancing the effectiveness of LLMs. Furthermore, we conduct additional empirical analyses and ablation studies to demonstrate the impact of each of the components of AMPLIFY, which, in turn, lead to critical insights for refining in context learning. 

   more » « less
3. Post Hoc Explanations of Language Models Can Improve Language Models

   Krishna, Satyapriya; Ma, Jiaqi; Slack, Dylan; Ghandeharioun, Asma; Singh, Sameer; Lakkaraju, Himabindu (December 2023, Advances in Neural Information Processing Systems)
4. Identifiability of Bifactor Models

   https://doi.org/10.5705/ss.202020.0386  

   Fang, Guanhua; Guo, Jinxin; Xu, Xin; Ying, Zhiliang; Zhang, Susu (January 2020, Statistica Sinica)

   Chen, Rong; Huang, Su-Yun; Shen, Xiaotong (Ed.)

   The bifactor model and its extensions are multidimensional latent vari- able models, under which each item measures up to one subdimension on top of the primary dimension(s). Despite their wide applications to educational and psycho- logical assessments, these multidimensional latent variable models may suffer from nonidentifiability, which can further lead to inconsistent parameter estimation and invalid inference. The current work provides a relatively complete characterization of identifiability for linear and dichotomous bifactor models and the linear extended bifactor model with correlated subdimensions. In addition, similar results for the two-tier models are developed. Illustrative examples on checking model identifia- bility by inspecting the factor loading structure are provided. Simulation studies examine the estimation consistency when the identifiability conditions are/are not satisfied. 

   more » « less
5. Models of random knots

   https://doi.org/10.1007/s41468-017-0007-8  

   Even-Zohar, Chaim (December 2017, Journal of Applied and Computational Topology)

   null (Ed.)