Multiple choice questions are widely used in education. In contrast to free response questions, multiple choice questions are scored objectively and at great speed, simply by checking the selected answer code, e.g. A-D, against an answer key. However, the ease of grading multiple choice questions (MCQs) comes at a nontrivial cost of creating them, with the greatest effort spent on creating distractor answer options [1,2].
Automatic generation of MCQs has received increasing research interest over the past two decades. Early approaches had little training data and so approached the MCQ generation task as four subtasks in a pipeline architecture: sentence selection, answer selection from selected sentences, question generation using the sentence and answer, and distractor generation [3]. Each of these subtasks can be addressed using NLP approaches developed outside the MCQ literature, like summarization techniques for sentence selection, keyword extraction techniques for answer selection, general question generation [4], and semantic similarity approaches for was popularized by GPT-3, which was shown to successfully complete various tasks without explicit training as long as it was given task instructions [17] (so-called zero-shot learning), with performance typically improving if additional demonstrations were provided (so-called few-shot learning). Remarkably, GPT-3 was able to exceed fine-tuned state-of-the-art performance on some benchmark tasks using this approach. Succeeding work has investigated instructiontuning, which fine-tunes an LLM to follow instructions written in natural language for various tasks, and has found improvements over GPT-3 on benchmark tasks using models with fewer parameters [18,19], but of course, even greater gains have been found using instruction tuning on the largest LLMs (>100B parameters) [20,21]. The success of instruction-tuned models has led to commercialization successes like ChatGPT and Microsoft's Bing Chat, which present an instruction-following LLM that can follow directions across conversational turns (so-called chatbot interaction). While these recent instruction-tuned LLMs are impressive in casual interactions, it is not clear how well they can generate MCQs compared to fine-tuned LLMs.
The present paper attempts to clarify the state of the art in MCQ generation by conducting a carefully-controlled human evaluation of three conditions: the fine-tuned augmented version of Macaw [16], instruction-tuned Bing Chat with zero-shot prompting, and human-authored questions from a college anatomy and physiology textbook [22]. In order to control for concept coverages, conditions are content-aligned, i.e. AI conditions generate MCQs based on the same input. Our primary research questions are (1) how well do the MCQs produced by the LLMs compare to textbook MCQs in a human evaluation study, (2) what errors do the LLMs make in the task, and (3) can LLM errors explain the human performance gap.
A human evaluation study was conducted to compare the fine-tuned augmented version of Macaw (Macaw+) [16], instruction-tuned Bing Chat with zero-shot prompting (Bing Chat), and human-authored questions from a college anatomy and physiology textbook (Textbook).
The evaluation study used a within-subjects design with Macaw+, Bing Chat, and Textbook as conditions. Conditions were presented using a 6 x 3 balanced Latin square (i.e., 3! = 3 × 2 × 1) to counterbalance condition order and prevent carryover effects between conditions. However, the underlying context of each MCQ (i.e., the input source sentence and correct answer for the LLMs) was not counterbalanced. This design decision means that in a fully-used Latin square, a context in a particular location would be paired with each condition, making fatigue effects equivalent across conditions. The human ratings were analyzed using mixed-effects beta regression with random intercepts for rater and rating question using the glmmTMB R package [23]. Beta regression is appropriate for continuous bounded outcome variables, unlike linear regression, which isn't suitable for bounded outcomes, and logistic regression, which can be used for proportions, but only when the proportion is a ratio of two counts [24]. Because beta regression is defined on the open interval (0,1), we use a standard transformation to squeeze our closed interval outcome variables to the open interval [25]. We conducted statistical tests Write a multiple choice question using the following sentence and answer. Convert the sentence into a question that matches the answer. Use JSON format. Sentence: <sentence> Answer: <answer> at α = .05 to address our research questions. If beta regression revealed a significant effect of condition, post hoc tests were conducted to determine differences between conditions.
Raters (N = 16) were recruited through the Amazon Mechanical Turk (AMT) marketplace from April to May of 2023 using the CloudResearch platform [26]. Raters were recruited using an occupation screener that paid 1 cent for their reporting of occupation. Raters were required to reside in the U.S., Canada, New Zealand, United Kingdom, Ireland, or Australia, and be employed as a nurse (n = 9), doctor (n = 5), or allied health provider with anatomy and physiology expertise (n = 2). The educational and occupational constraints we designed to ensure raters were experts in the evaluation subject domain: they had passed anatomy and physiology in their studies and used this knowledge on a daily basis. Demographic constraints are enforced by CloudResearch based on rater responses to previous demographic surveys. Raters were further required to have completed at least 100 previous AMT tasks with at least a 95% approval rating. Raters were paid $12 regardless of reliability, based on an estimated 120 minutes to complete the task. In addition, raters were paid up to $50 in bonuses for passing quality checks determined by intra-rater reliability: a $5 bonus for passing each check, and an additional $20 bonus for passing a comprehensive check.
A textbook on anatomy and physiology [22] from OpenStax was used as the source of 120 textbook questions. Questions were web scraped from the OpenStax websitefoot_0 , manually checked, and aligned with the answer key accessible by registering as an instructor. The 120 questions for this evaluation were drawn from the first 4-5 questions from each of the textbook's 28 chapters.
MCQs for the three conditions were produced as follows. The Textbook condition used the MCQ as it appeared in the textbook. The LLM conditions both required sentence and answer as input, so the textbook questions were manually transformed into sentences, and these sentences and original answers were used as inputs to the LLM models. For example, the textbook question "Which of the following specialties might focus on studying all of the structures of the ankle and foot?" with associated answer "regional anatomy" was transformed into the sentence "Regional anatomy might, for example, focus on studying all of the structures of the ankle and foot. " Each sentence/answer was input to the fine-tuned, augmented Macaw model described in [16] to create MCQs for the Macaw+ condition (see Section 1 for key details).
Because Bing Chat uses prompt-based input and there is no known best prompt for generating MCQs, several different prompts were designed based on templates from existing datasets [18,19] and informally evaluated using a handful of the above sentence/answer pairs. The best-performing prompt given in Figure 1 was used to generate all Bing Chat MCQs using the EdgeGPT API [27]. Note the prompt includes the same sentence/answer used in the Macaw+ condition. Thus all three conditions are aligned on each MCQ context.
Six surveys were created with Qualtrics, an online survey tool, using a balanced Latin square to define the order of conditions. Because each row of the Latin square only contains 3 orderings, each ordering was repeated 20 times in a survey for a total of 60 MCQs. The same ordering of 60 contexts was used in each survey; only the condition applied to each position of the ordering varied across surveys. Each question, correct answer, and answer options were formatted vertically in that order on a single survey page using the direct assessment methodology [28,29]. These three elements each had two associated ratings, followed by an overall quality rating, for a total of seven ratings per question, as shown in Table 1. All ratings were in horizontal slider format and arranged in descending order. The 0-100 sliders had no numeric indicators and were initialized at the midpoint. The remaining sliders had numeric indicators and snapped to integer positions. Each survey had instructions at the beginning to explain the task and the seven ratings. Survey instructions and an example survey page are shown in Appendix A.
Following the direct assessment methodology, degraded items were created to evaluate the internal reliability of each rater [29,28,30]. Degraded items were created by copying the question, answer, and options on an existing survey page and then applying the following transformations. Questions were degraded by deleting a span of words [28], where the length of the span was determined by the equation span length = 0.21696 * word count + 0.78698 [31]. Degraded answers were created by replacing the correct answer with one of the other answer options selected at random. Degraded answer options were created by randomly selecting a remaining incorrect answer option and then duplicating it while removing another incorrect option at random. Thus each survey of 80 pages contained 60 distinct pages and 20 degraded versions of distinct pages. An example degraded item is shown in Appendix A.
We refer to a distinct page and its degraded version as a control pair. A sample size of 20 control pairs is sufficient to detect a large (.8 SD) effect using a Wilcoxon signed-ranks test for matched pairs at α = .05 and .95 power on a one-tailed test. Thus if we do not detect a Table 1 Ratings used in human evaluation study
The question contains correct information 0-100 The question is grammatical and fluent 0-100 The given correct answer is correct 0-100 The given correct answer is present in the answer options 0-100 Number of answer options that give a correct answer 0-4 Number of answer options that are distinct (no duplicates)
1-4 Quality of the question, given answer, and answer options combined 0-100 large effect between ratings of distinct pages and their degraded versions, we infer the rater is not reliable (i.e., poor intra-rater reliability). The degraded pages were in randomly assigned positions in each survey and were evenly distanced from their matched distinct pages, modulo 50. This ensured that pages in control pairs had 50 other items between them, making it less likely that raters would remember their rating on a previous item.
We additionally developed an occupation survey to help us find more qualified raters. The occupation survey consisted of two questions, a generic occupation question from the standard Qualtrics demographics library with 20 answer options, and a conditional branch question that only appeared if a respondent selected healthcare on the first question. The conditional branch question asked for a more specific healthcare occupation, with nine total options including six matching our recruiting criteria. This indirect approach to asking about specific healthcare occupations was designed to avoid demand characteristics (i.e., false responses) from asking such questions directly.
Six surveys were sequentially with a default quota of four raters. If a survey had sufficiently good reliability with less than four raters, it was terminated early; this only happened with survey 3. Likewise, if a survey had insufficient reliability with four raters, it was extended with a quota of an additional four raters; this only happened for survey 2. Raters were allowed to participate in more than one survey if they passed the comprehensive quality check.
Raters accessed the surveys through AMT and completed the surveys using Qualtrics. Because the study is a system evaluation and not human subjects research, informed consent was not obtained. Raters saw the instructions for the survey twice, once as a preview on AMT before undertaking the survey, and again once they clicked on the survey link. On each following page, raters read the question, the correct answer, and the answer options, and then completed the ratings described in Table 1. Raters were paid upon completion of the survey and received bonuses based on the quality checks passed, i.e. based on their intra-rater reliability for each rating, with the final rating in Table 1 serving as the comprehensive check.
Median completion time across surveys was 133 minutes, giving approximately 100 seconds to read the question, answer, and options and make 7 judgments. Control checks were considered to be passed if p < .05 on the aforementioned Wilcoxon signed-ranks test. Seven raters failed to pass the comprehensive check on a given survey and were excluded from future surveys. Every check on each survey was passed by at least two raters with the exception of survey 2, where all but question informative and question fluent were passed by two raters.
Initially, the Cronbach's alpha inter-rater reliability statistic was calculated for raters passing control checks in each survey. However, inspection of the raw ratings revealed that while some raters used 0-100 sliders as a measure of confidence (as intended), other raters used it in a binary fashion, leading to nonlinearity and lack of variability among raters. Because Cronbach's alpha was invalid for these data, a new agreement metric was constructed based on the contrast between distinct and degraded items. If we consider that the distinct items are likely good and so should have ratings above 50 on most scales and that degraded items are likely bad and so should have ratings below 50 on the same scales, then we can threshold all ratings to be either 1 (good) or 0 (bad) and calculate balanced accuracy for true positives and true negatives. The same approach works for correct options if we assume that there is 1 correct option by default (good) and for distinct options if we assume that there are 4 distinct options by default (good) and that degraded items have all other values on these metrics. We will refer to these assumptions as pseudo-truth because we are assuming that the generated MCQs are generally good and that their degraded variants are generally bad. Inter-rater reliability was calculated for each rating within a survey using the following method. First, all ratings were converted to 1/0 as described above, and their balanced accuracy based on pseudo-truth was calculated. The top two most accurate ratings were kept for interrater reliability (regardless of absolute accuracy) and any additional ratings with accuracy greater than .8 were also included. Inter-rater reliability was then calculated between included ratings using Fleiss's kappa, adjusted for unbalanced classes [32]. The net effect of this approach is that comparing to pseudo-truth was a stricter criterion of intra-rater reliability than using the Wilcoxon signed-ranks test (e.g. 5 raters passed the Wilcoxon signed-ranks test on survey 1 for question informative, but only 2 raters passed the pseudo-truth procedure), and using only these raters ensures high inter-rater reliability without sacrificing validity. Intra-rater reliabilities are shown in Table 2 in the same order as Table 1 but using abbreviated labels. Final kappas showed substantial agreement (κ > .60) on 39 of 42 ratings. Ratings shown in Table 2 were used in all further analyses.
To answer our research question of how MCQs produced by the LLMs compare to textbook MCQs, we ran separate mixed-effects beta regressions with random intercepts for rater and question, using the source of the question as the fixed effect (Bing Chat, Macaw+, or Textbook). Descriptive statistics and regression results are shown in Table 3, with associated p values from one-way ANOVA. Significant differences between conditions were found only for answer in options and combined quality. However, pairwise contrasts for answer in options revealed no significant differences between Bing Chat (M = .83, SE = .02) and Textbook (M = .86, SE = .02), p = .068, and no significant differences between Macaw+ (M = .86, SE = .02) and Textbook, p = .083. Pairwise contrasts for combined quality, however, revealed significant differences between Bing Chat (M = .85, SE = .02) and Textbook (M = .88, SE = .02), p = .009, and significant differences between Macaw+ (M = .85, SE = .02) and Textbook, p = .038. Altogether, the LLM conditions were not significantly different from Textbook on six of seven measures. However, answer in options and relatively low p-values of answer correct and distinct options warrant further investigation.
To answer our research question of what errors the LLMs make in MCQ generation, we conducted an error analysis for distinct options and answer in options. MCQs generated by Macaw+ and Bing Chat were automatically scored using exact string match to determine if the given answer appeared in the answer options and if the four answer options were distinct from each other. Errors detected by exact string match were then manually reviewed to determine if they were actual errors. For example, a failed string match where the only difference was punctuation or an article like "a" would not be considered an actual error. Macaw+ had 111 questions with distinct options or 92.5%. This is notably lower than the previously reported success rate of 97.5% [16]. Six of the nine failures occurred when response options contained lists, e.g. "carbon, hydrogen, oxygen, and nitrogen" which may explain the difference in the previous result if such lists are resistant to paraphrasing. Macaw+ had 120 questions where the given correct answer was one of the answer options, i.e. 100%. So the primary failure mode of Macaw+ in the evaluation was a failure to generate distinct answer options in 7.5% of cases, with the majority of these caused by answer options that are lists.
Bing Chat had 120 questions with distinct options, i.e. 100%. Bing Chat had 109 questions where the given correct answer was one of the answer options or 91%. Of the remaining 11 mismatches, 8 could be considered to be valid questions overall, in the sense that one of the answer options was the correct answer, but that answer differed from the given correct answer in a nontrivial way. For example, 2 failures used a letter (A-D) to indicate the correct answer rather than using the given correct answer in the prompt, and another 3 failures used either a wider or narrower scoping of the answer than was presented in the options, e.g. answer option "Axons from the retinal ganglion cells in the retina" is a narrower scoping of given correct answer "retinal ganglion cells. " If such errors are judged leniently, then 8 of 11 of the errors can be viewed as a failure to precisely follow the prompt's instructions. So the primary failure mode of Bing Chat in the evaluation was a failure to include the given correct answer among the answer options in 9% of cases, with the majority of these cases being otherwise valid questions. Example errors for Macaw+ and BingChat are shown in Appendix B.
The error analysis in Section 3 potentially explains the pattern of results in the human evaluation in Section 2.5. Recall answer in options was significant with Bing Chat having the lowest mean score, tracking Bing Chat's failure to include the given correct answer among the answer options in 9% of cases. The same error could potentially explain the relatively low p-values of answer correct, since the mismatch between the given correct answer and the answer options could lower the confidence of human raters that the given answer is correct. Similarly for Macaw+, distinct options had a relatively low p-value and Macaw+ had the lowest mean score, which tracks Macaw+'s failure to generate distinct answer options in 7.5% of cases.
To answer our research question of how LLM errors explain the human performance gap, we reanalyzed the human evaluation data to determine the effect of the above errors on combined quality, which was the only measure for which pairwise significant differences were found between Bing Chat, Macaw+, and Textbook. Two analyses were conducted for Bing Chat and Macaw+. First, we used the rating of their primary error type to predict their combined quality. Second, we tested the difference between LLM and Textbook combined quality when MCQs marked as bad were excluded. For example, if a rater marked a Macaw+ MCQ as having 4 distinct options (good), then their combined quality rating for that MCQ would be included in the analysis, otherwise it would be excluded. Both analyses used separate mixed-effects beta regressions with random intercepts for rater and question.
Mixed-effects beta regression for Bing Chat ratings using answer in options to predict combined quality revealed a significant effect, p < .001. When answer in options is 0, estimated combined quality is low (M = .39, SE = .07), and when answer in options is 100, estimated combined quality is high (M = .89, SE = .02). An additional mixedeffects beta regression was conducted comparing Bing Chat to Textbook using only MCQs where answer in options was scored highly (above 50). This filtering procedure removed 34 ratings from the Bing Chat condition and 5 ratings from the Textbook condition out of 360 total ratings. Mixed-effects beta regression revealed that with these errors removed, combined quality for Bing Chat (M = .92, SE = .02) was still rated lower than Textbook (M = .93, SE = .01), p < .012. A follow-up simulation analysis on the choice of 50 as a threshold revealed that the significant difference between conditions remained up to a threshold of 98. These results suggest that while answer in options errors strongly influence combined quality for Bing Chat, they do not fully explain the human performance gap.
Mixed-effects beta regression for Macaw+ ratings using distinct options to predict combined quality revealed a significant effect, p < .001. When distinct options is not 4 (bad), estimated combined quality is low (M = .59, SE = .06), and when distinct options is 4 (good), estimated combined quality is high (M = .86, SE = .02). An additional mixed-effects beta regression was conducted comparing Macaw+ to Textbook using only MCQs where distinct options was scored highly (equal to 4). This filtering procedure removed 45 ratings from the Macaw+ condition and 7 ratings from the Textbook condition out of 360 total ratings. Mixed-effects beta regression revealed no significant difference between Macaw+ (M = .91, SE = .02) and Textbook (M = .92, SE = .01), p = .149. These results suggest that distinct options errors may explain the human performance gap for Macaw+.
The goal of the present work was to clarify the state of the art in MCQ generation by comparing two LLMs, the fine-tuned augmented version of Macaw [16] and instruction-tuned Bing Chat with zero-shot prompting, to human-authored questions in a carefully-controlled human evaluation. Our results indicate that on six of seven measures tested, both LLM's performance was not significantly different from human performance. These six measures relate to individual components of the MCQ, specifically the question stem, the answer, and the answer options, and are very fine-grained, so the lack of significant difference is particularly notable. Only on the overall measure of combined quality was a significant difference found in favor of the human-authored questions.
Analysis of LLM errors indicates that Macaw and Bing Chat have different failure modes for this task: Macaw tends to repeat answer options whereas Bing Chat tends to not include the specified answer in the answer options. Each of these error types is strongly predictive of combined quality ratings. For Macaw, removing error items from analysis results in combined quality ratings that are not significantly different from human-authored questions on combined quality. For Bing Chat, removing error items improves combined quality, but resulting ratings remain significantly different from human-authored questions. Altogether, these results suggest that the LLMs are both remarkably capable of creating MCQs, and the error analyses suggest future research directions for each LLM on this task.
These results are based on a high-quality human evaluation. It is widely agreed that human evaluations provide the best evidence of system performance, yet as few as 20% of research papers on natural language generation include them [33]. We were careful to recruit raters whose profession required them to be highly knowledgeable in the MCQ content domain. Our evaluation includes both intra-rater reliability (can raters distinguish between actual items and intentionally degraded items) as well as inter-rater reliability (do raters agree with each other). Only raters with high intra-and inter-rater reliability were included in our analysis. Additionally, the evaluation was designed to minimize confounding effects of fatigue as well as individual rater characteristics like extreme responses.
Our study has two primary limitations. First, we only evaluated questions on the topic of anatomy and physiology. It is possible that the LLMs would perform differently on other topics, though neither model was trained specifically for this topic. Second, the task given to the LLMs simplifies the canonical task of generating MCQs from text by providing sentences and answers instead of selecting them [3]. Therefore, our results should not be taken as representative of MCQ generation from freeform text.
The LLMs differ in the types of errors they make during MCQ generation. The primary error mode of Macaw+ is to fail to make enough distinct options as distractors, especially when the distractors are lists, as shown in Figure 5. The primary error of Bing Chat is failing to follow the instructions of the prompt precisely. As shown in Figure 6, Bing Chat has generated a valid question if the given correct answer had been "fauces, " but the given correct answer was "oral cavity. " A typical Bing Chat generation error where the prompt was not followed precisely. The input sentence was "The fauces connects the oral cavity to the oropharynx. " and the input answer was "oral cavity. " The respective prompt using these elements is given in Figure 1.
https://openstax.org/details/books/anatomy-and-physiology