More Like this

1. Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4

   Chang, Kent K.; Cramer, Mackenzie; Soni, Sandeep; Bamman, David (April 2023, arXivorg)

   In this work, we carry out a data archaeology to infer books that are known to ChatGPT and GPT-4 using a name cloze membership inference query. We find that OpenAI models have memorized a wide collection of copyrighted materials, and that the degree of memorization is tied to the frequency with which passages of those books appear on the web. The ability of these models to memorize an unknown set of books complicates assessments of measurement validity for cultural analytics by contaminating test data; we show that models perform much better on memorized books than on non-memorized books for downstream tasks. We argue that this supports a case for open models whose training data is known. 

   more » « less
2. Proceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation

   https://doi.org/10.1145/3574318  

   Kashyapi, Sumanta; Dietz, Laura (December 2022, ACM)

   Ganguly, Debasis; Gangopadhyay, Surupendu; Mitra, Mandar; Majumder, Prasenjit (Ed.)

   For most queries, the set of relevant documents spans multiple subtopics. Inspired by the neural ranking models and query-specific neural clustering models, we develop Topic-Mono-BERT which performs both tasks jointly. Based on text embeddings of BERT, our model learns a shared embedding that is optimized for both tasks. The clustering hypothesis would suggest that embeddings which place topically similar text in close proximity will also perform better on ranking tasks. Our model is trained with the Wikimarks approach to obtain training signals for relevance and subtopics on the same queries. Our task is to identify overview passages that can be used to construct a succinct answer to the query. Our empirical evaluation on two publicly available passage retrieval datasets suggests that including the clustering supervision in the ranking model leads to about 16% improvement in identifying text passages that summarize different subtopics within a query. 

   more » « less
3. A Feasibility Study of Answer-Unaware Question Generation for Education

   https://doi.org/10.18653/v1/2022.findings-acl.151  

   Dugan, Liam; Miltsakaki, Eleni; Upadhyay, Shriyash; Ginsberg, Etan; Gonzalez, Hannah; Choi, DaHyeon; Yuan, Chuning; Callison-Burch, Chris (January 2022, Findings of the Association for Computational Linguistics: ACL 2022)

   We conduct a feasibility study into the applicability of answer-agnostic question generation models to textbook passages. We show that a significant portion of errors in such systems arise from asking irrelevant or un-interpretable questions and that such errors can be ameliorated by providing summarized input. We find that giving these models human-written summaries instead of the original text results in a significant increase in acceptability of generated questions (33% → 83%) as determined by expert annotators. We also find that, in the absence of human-written summaries, automatic summarization can serve as a good middle ground. 

   more » « less
4. Scaling hermeneutics: a guide to qualitative coding with LLMs for reflexive content analysis

   https://doi.org/10.1140/epjds/s13688-025-00548-8  

   Dunivin, Zackary Okun (December 2025, EPJ Data Science)

   Abstract Qualitative coding, or content analysis, is more than just labeling text: it is a reflexive interpretive practice that shapes research questions, refines theoretical insights, and illuminates subtle social dynamics. As large language models (LLMs) become increasingly adept at nuanced language tasks, questions arise about whether—and how—they can assist in large-scale coding without eroding the interpretive depth that distinguishes qualitative analysis from traditional machine learning and other quantitative approaches to natural language processing. In this paper, we present a hybrid approach that preserves hermeneutic value while incorporating LLMs to scale the application of codes to large data sets that are impractical for manual coding. Our workflow retains the traditional cycle of codebook development and refinement, adding an iterative step to adapt definitions for machine comprehension, before ultimately replacing manual with automated text categorization. We demonstrate how to rewrite code descriptions for LLM-interpretation, as well as how structured prompts and prompting the model to explain its coding decisions (chain-of-thought) can substantially improve fidelity. Empirically, our case study of socio-historical codes highlights the promise of frontier AI language models to reliably interpret paragraph-long passages representative of a humanistic study. Throughout, we emphasize ethical and practical considerations, preserving space for critical reflection, and the ongoing need for human researchers’ interpretive leadership. These strategies can guide both traditional and computational scholars aiming to harness automation effectively and responsibly—maintaining the creative, reflexive rigor of qualitative coding while capitalizing on the efficiency afforded by LLMs. 

   more » « less
5. Sample-Driven Connectivity Learning for Motion Planning in Narrow Passages

   Li, Sihui; Dantam, Neil T (May 2023, International Conference on Robotics and Automation)

   Sampling-based motion planning works well in many cases but is less effective if the configuration space has narrow passages. In this paper, we propose a learning-based strategy to sample in these narrow passages, which improves overall planning time. Our algorithm first learns from the configuration space planning graphs and then uses the learned information to effectively generate narrow passage samples. We perform experiments in various 6D and 7D scenes. The algorithm offers one order of magnitude speed-up compared to baseline planners in some of these scenes. 

   more » « less