More Like this

1. Best 2019 PIC III Paper: Do They Understand Your Language? Access Their Fluency with Vector Representation

   https://doi.org/10.18260/1-2--34205  

   Davishahl, Eric ; Haskell, Todd ; Davishahl, Jill ; Singleton, Lee ; Goodridge, Wade ( June 2020 , 2020 ASEE Virtual Annual Conference Content Access)

   In teaching mechanics, we use multiple representations of vectors to develop concepts and analysis techniques. These representations include pictorials, diagrams, symbols, numbers and narrative language. Through years of study as students, researchers, and teachers, we develop a fluency rooted in a deep conceptual understanding of what each representation communicates. Many novice learners, however, struggle to gain such understanding and rely on superficial mimicry of the problem solving procedures we demonstrate in examples. The term representational competence refers to the ability to interpret, switch between, and use multiple representations of a concept as appropriate for learning, communication and analysis. In engineering statics, an understanding of what each vector representation communicates and how to use different representations in problem solving is important to the development of both conceptual and procedural knowledge. Science education literature identifies representational competence as a marker of true conceptual understanding. This paper presents development work for a new assessment instrument designed to measure representational competence with vectors in an engineering mechanics context. We developed the assessment over two successive terms in statics courses at a community college, a medium-sized regional university, and a large state university. We started with twelve multiple-choice questions that survey the vector representations commonly employed in statics. Each question requires the student to interpret and/or use two or more different representations of vectors and requires no calculation beyond single digit integer arithmetic. Distractor answer choices include common student mistakes and misconceptions drawn from the literature and from our teaching experience. We piloted these twelve questions as a timed section of the first exam in fall 2018 statics courses at both Whatcom Community College (WCC) and Western Washington University. Analysis of students’ unprompted use of vector representations on the open-ended problem-solving section of the same exam provides evidence of the assessment’s validity as a measurement instrument for representational competence. We found a positive correlation between students’ accurate and effective use of representations and their score on the multiple choice test. We gathered additional validity evidence by reviewing student responses on an exam wrapper reflection. We used item difficulty and item discrimination scores (point-biserial correlation) to eliminate two questions and revised the remaining questions to improve clarity and discriminatory power. We administered the revised version in two contexts: (1) again as part of the first exam in the winter 2019 Statics course at WCC, and (2) as an extra credit opportunity for statics students at Utah State University. This paper includes sample questions from the assessment to illustrate the approach. The full assessment is available to interested instructors and researchers through an online tool. 

   more » « less
2. Do They Understand Your Language? Assess Their Fluency with Vector Representations

   Davishahl, E. ; Haskell, T. R. ; Davishahl, J. ; Singleton, L. ; Goodridge, W. H. ( June 2019 , ASEE Annual Conference proceedings)

   In teaching mechanics, we use multiple representations of vectors to develop concepts and analysis techniques. These representations include pictorials, diagrams, symbols, numbers and narrative language. Through years of study as students, researchers, and teachers, we develop a fluency rooted in a deep conceptual understanding of what each representation communicates. Many novice learners, however, struggle to gain such understanding and rely on superficial mimicry of the problem solving procedures we demonstrate in examples. The term representational competence refers to the ability to interpret, switch between, and use multiple representations of a concept as appropriate for learning, communication and analysis. In engineering statics, an understanding of what each vector representation communicates and how to use different representations in problem solving is important to the development of both conceptual and procedural knowledge. Science education literature identifies representational competence as a marker of true conceptual understanding. This paper presents development work for a new assessment instrument designed to measure representational competence with vectors in an engineering mechanics context. We developed the assessment over two successive terms in statics courses at a community college, a medium-sized regional university, and a large state university. We started with twelve multiple-choice questions that survey the vector representations commonly employed in statics. Each question requires the student to interpret and/or use two or more different representations of vectors and requires no calculation beyond single digit integer arithmetic. Distractor answer choices include common student mistakes and misconceptions drawn from the literature and from our teaching experience. We piloted these twelve questions as a timed section of the first exam in fall 2018 statics courses at both Whatcom Community College (WCC) and Western Washington University. Analysis of students’ unprompted use of vector representations on the open-ended problem-solving section of the same exam provides evidence of the assessment’s validity as a measurement instrument for representational competence. We found a positive correlation between students’ accurate and effective use of representations and their score on the multiple choice test. We gathered additional validity evidence by reviewing student responses on an exam wrapper reflection. We used item difficulty and item discrimination scores (point-biserial correlation) to eliminate two questions and revised the remaining questions to improve clarity and discriminatory power. We administered the revised version in two contexts: (1) again as part of the first exam in the winter 2019 Statics course at WCC, and (2) as an extra credit opportunity for statics students at Utah State University. This paper includes sample questions from the assessment to illustrate the approach. The full assessment is available to interested instructors and researchers through an online tool. 

   more » « less
3. Dynamic Assessment of English Learners in the Content Areas: An Exploratory Study in Fifth‐Grade Science

   https://doi.org/10.1002/tesq.3059  

   Grapin, Scott E. ; Llosa, Lorena ( August 2021 , TESOL Quarterly)

   Dynamic assessment (DA) seeks to understand learners’ abilities that are still developing by embedding mediation (i.e., contingent forms of assistance) into the assessment process. To date, DA studies have been carried out primarily in contexts focused on language learning. However, in content classrooms (e.g., science, math), English learners (ELs) are developing content and English language proficiency simultaneously and may therefore be able to demonstrate their learning more fully with mediation than in independent performance alone. This study applied DA to a novel context—the science classroom—and examined whether embedding mediation (in the form of contingent questions/probes) into two science assessment tasks yielded more complete and accurate information about what fifth‐grade ELs and their peers knew and could do. Findings indicated that mediation supported ELs in clarifying their science ideas that were initially communicated inexplicitly or imprecisely. Mediation also revealed that some non‐ELs’ seemingly accurate initial responses were not underpinned by science understanding. By offering a theoretically grounded approach to assessment that is dually responsive to students’ content and language learning needs, DA could help refute deficit views of ELs in the content areas and create more equitable opportunities for these students to demonstrate their content learning.

    

   more » « less
4. Investigating crowdsourcing to generate distractors for multiple-choice assessments

   Scheponik, T. ; Golaszewski, E. ; Herman, G. ; Offenberger, S. ; Oliva, L. ; Peterson, P. A. ; Sherman, A. T. ( June 2019 , National Cyber Summit)

   We present and analyze results from a pilot study that explores how crowdsourcing can be used in the process of generating distractors (incorrect an-swer choices) in multiple-choice concept inventories (conceptual tests of under-standing). To our knowledge, we are the first to propose and study this approach. Using Amazon Mechanical Turk, we collected approximately 180 open-ended responses to several question stems from the Cybersecurity Concept Inventory of the Cybersecurity Assessment Tools Project and from the Digital Logic Concept Inventory. We generated preliminary distractors by filtering responses, grouping similar responses, selecting the four most frequent groups, and refining a repre-sentative distractor for each of these groups. We analyzed our data in two ways. First, we compared the responses and resulting distractors with those from the aforementioned inventories. Second, we obtained feedback from Amazon Mechanical Turk on the resulting new draft test items (including distractors) from additional subjects. Challenges in using crowdsourcing include controlling the selection of subjects and filtering out re-sponses that do not reflect genuine effort. Despite these challenges, our results suggest that crowdsourcing can be a very useful tool in generating effective dis-tractors (attractive to subjects who do not understand the targeted concept). Our results also suggest that this method is faster, easier, and cheaper than is the tra-ditional method of having one or more experts draft distractors, building on talk-aloud interviews with subjects to uncover their misconceptions. Our results are significant because generating effective distractors is one of the most difficult steps in creating multiple-choice assessments. 

   more » « less
5. Investigating crowdsourcing to generate distractors for multiple-choice assessments

   Scheponik, T. ; Golaszewski, E. ; Herman, G. ; Offenberger, S. ; Oliva, L. ; Peterson, P. A. ; Sherman, A. T. ( June 2019 , Proceedings of the National Cyber Summit)

   We present and analyze results from a pilot study that explores how crowdsourcing can be used in the process of generating distractors (incorrect answer choices) in multiple-choice concept inventories (conceptual tests of under-standing). To our knowledge, we are the first to propose and study this approach. Using Amazon Mechanical Turk, we collected approximately 180 open-ended responses to several question stems from the Cybersecurity Concept Inventory of the Cybersecurity Assessment Tools Project and from the Digital Logic Concept Inventory. We generated preliminary distractors by filtering responses, grouping similar responses, selecting the four most frequent groups, and refining a repre-sentative distractor for each of these groups. We analyzed our data in two ways. First, we compared the responses and resulting distractors with those from the aforementioned inventories. Second, we obtained feedback from Amazon Mechanical Turk on the resulting new draft test items (including distractors) from additional subjects. Challenges in using crowdsourcing include controlling the selection of subjects and filtering out responses that do not reflect genuine effort. Despite these challenges, our results suggest that crowdsourcing can be a very useful tool in generating effective dis-tractors (attractive to subjects who do not understand the targeted concept). Our results also suggest that this method is faster, easier, and cheaper than is the traditional method of having one or more experts draft distractors, building on talk-aloud interviews with subjects to uncover their misconceptions. Our results are significant because generating effective distractors is one of the most difficult steps in creating multiple-choice assessments. 

   more » « less