More Like this

1. Generating Multiple Choice Questions with a Multi-Angle Question Answering Model

   Olney, A. ( July 2022 , Proceedings of The Third Workshop of the Learner Data Institute , The 15th International Conference on Educational Data Mining (EDM 2022))

   Multi-angle question answering models have recently been proposed that promise to perform related tasks like question generation. However, performance on related tasks has not been thoroughly studied. We investigate a leading model called Macaw on the task of multiple choice question generation and evaluate its performance on three angles that systematically reduce the complexity of the task. Our results indicate that despite the promise of generalization, Macaw performs poorly on untrained angles. Even on a trained angle, Macaw fails to generate four distinct multiple-choice options on 17% of inputs. We propose augmenting multiple choice options by paraphrasing angle input and show this increases overall success to 97.5%. A human evaluation comparing the augmented multiple-choice questions with textbook questions on the same topic reveals that Macaw questions broadly score highly but below human questions. 

   more » « less
2. Generating multiple choice questions from a textbook: LLMs match human performance on most metrics

   Olney, A. ( June 2023 , Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation at the AIED’23 Conference)

   Multiple choice questions are traditionally expensive to produce. Recent advances in large language models (LLMs) have led to fine-tuned LLMs that generate questions competitive with human-authored questions. However, the relative capabilities of ChatGPT-family models have not yet been established for this task. We present a carefully-controlled human evaluation of three conditions: a fine-tuned, augmented version of Macaw, instruction-tuned Bing Chat with zero-shot prompting, and humanauthored questions from a college science textbook. Our results indicate that on six of seven measures tested, both LLM’s performance was not significantly different from human performance. Analysis of LLM errors further suggests that Macaw and Bing Chat have different failure modes for this task: Macaw tends to repeat answer options whereas Bing Chat tends to not include the specified answer in the answer options. For Macaw, removing error items from analysis results in performance on par with humans for all metrics; for Bing Chat, removing error items improves performance but does not reach human-level performance. 

   more » « less
3. Generating multiple choice questions from a textbook: LLMs match human performance on most metrics

   Olney, Andrew M ( September 2023 , CEUR workshop proceedings)

   Moore, S ; Stamper, J ; Cao, T ; Liu, Z ; Hu, X ; Lu, Y ; Liang, J ; Khosravi, H ; Denny, P ; Singh, A (Ed.)

   Multiple choice questions are traditionally expensive to produce. Recent advances in large language models (LLMs) have led to fine-tuned LLMs that generate questions competitive with human-authored questions. However, the relative capabilities of ChatGPT-family models have not yet been established for this task. We present a carefully-controlled human evaluation of three conditions: a fine-tuned, augmented version of Macaw, instruction-tuned Bing Chat with zero-shot prompting, and humanauthored questions from a college science textbook. Our results indicate that on six of seven measures tested, both LLM’s performance was not significantly different from human performance. Analysis of LLM errors further suggests that Macaw and Bing Chat have different failure modes for this task: Macaw tends to repeat answer options whereas Bing Chat tends to not include the specified answer in the answer options. For Macaw, removing error items from analysis results in performance on par with humans for all metrics; for Bing Chat, removing error items improves performance but does not reach human-level performance. 

   more » « less
4. Leveraging Large Language Models for Multiple Choice Question Answering

   Joshua Robinson ; David Wingate ( January 2023 , International Conference on Learning Representations)

   While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they generally lag behind the MCQA state of the art (SOTA). MCQA tasks have traditionally been presented to LLMs like cloze tasks. An LLM is conditioned on a question (without the associated answer options) and its chosen option is the one assigned the highest probability after normalization (for length, etc.). A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.g., “A”) associated with its chosen answer option. This approach allows the model to explicitly compare answer options, reduces computational costs, and mitigates the effects of tokenization scheme and answer option representations on answer selection. For the natural approach to be effective, the LLM it is used with must be able to associate answer options with the symbols that represent them. The LLM needs what we term multiple choice symbol binding (MCSB) ability. This ability varies greatly by model. We show that a model with high MCSB ability performs much better with the natural approach than with the traditional approach across 20 diverse datasets and largely closes the gap with the SOTA, suggesting that the MCQA ability of LLMs has been previously underestimated. 

   more » « less
5. Improving student understanding of quantum mechanics underlying the Stern–Gerlach experiment using a research-validated multiple-choice question sequence

   https://doi.org/10.1088/1361-6404/ab2135  

   Justice, Paul ; Marshman, Emily ; Singh, Chandralekha ( July 2019 , European Journal of Physics)

   Engaging students with well-designed multiple-choice questions during class and asking them to discuss their answers with their peers after each student has contemplated the response individually can be an effective evidence-based active-engagement pedagogy in physics courses. Moreover, validated sequences of multiple-choice questions are more likely to help students build a good knowledge structure of physics than individual multiple-choice questions on various topics. Here we discuss a framework to develop robust sequences of multiple-choice questions and then use the framework for the development, validation and implementation of a sequence of multiple-choice questions focusing on helping students learn quantum mechanics via the Stern–Gerlach experiment (SGE) that takes advantage of the guided inquiry-based learning sequences in an interactive tutorial on the same topic. The extensive research in developing and validating the multiple-choice question sequence (MQS) strives to make it effective for students with diverse prior preparation in upper-level undergraduate quantum physics courses. We discuss student performance on assessment task focusing on the SGE after traditional lecture-based instruction versus after engaging with the research-validated MQS administered as clicker questions in which students had the opportunity to discuss their responses with their peers.

    

   more » « less