More Like this

1. Paper or Silicon: Assessing Student Understanding in a Computer-based Testing Environment Using PrairieLearn

   Ardister, J; Recktenwald, G; Roccabianca, S (June 2024, ASEE)

   Computer-based testing is a powerful tool for scaling exams in large lecture classes. The decision to adopt computer-based testing is typically framed as a tradeoff in terms of time; time saved by auto-grading is reallocated as time spent developing problem pools, but with significant time savings. This paper seeks to examine the tradeoff in terms of accuracy in measuring student understanding. While some exams (e.g., multiple choice) are readily portable to a computer-based format, adequately porting other exam types (e.g., drawings like FBDs or worked problems) can be challenging. A key component of this challenge is to ask “What is the exam actually able to measure?” In this paper the authors will provide a quantitative and qualitative analysis of student understanding measurements via computer-based testing in a sophomore level Solid Mechanics course. At Michigan State University, Solid Mechanics is taught using the SMART methodology. SMART stands for Supported Mastery Assessment through Repeated Testing. In a typical semester, students are given 5 exams that test their understanding of the material. Each exam is graded using the SMART rubric which awards full points for the correct answer, some percentage for non-conceptual errors, and zero points for a solution that has a conceptual error. Every exam is divided into four sections; concept, simple, average, and challenge. Each exam has at least one retake opportunity, for a total of 10 written tests. In the current study, students representing 10% of the class took half of each exam in Prairie Learn, a computer-based auto-grading platform. During this exam, students were given instant feedback on submitted answers (correct or incorrect) and given an opportunity to identify their mistakes and resubmit their work. Students were provided with scratch paper to set up the problem and work out solutions. After the exam, the paper-based work was compared with the computer submitted answers. This paper examines what types of mistakes (conceptual and non-conceptual) students were able to correct when feedback was provided. The answer is dependent on the type and difficulty of the problem. The analysis also examines whether students taking the computer-based test performed at the same level as their peers who took the paper-based exams. Additionally, student feedback is provided and discussed. 

   more » « less
2. Work in Progress: The Development of a Tactile Spatial Ability Instrument for Assessing Spatial Ability in Blind and Low-vision Populations

   Goodridge, W. H.; & Shaheen, N. L.; & Hunt, A. T.; & Kane, D. (January 2021, 2021 ASEE Virtual Annual Conference Content Access)

   There is significant work indicating that spatial ability has correlations to student success in STEM programs. Work also shows that spatial ability correlates to professional success in respective STEM fields. Spatial ability has thus been a focus of research in engineering education for some time. Spatial interventions have been developed to improve student’s spatial ability that range from physical manipulatives to the implementation of entire courses. These interventions have had positive impact upon student success and retention. Currently, researchers rely on a variety of different spatial ability instruments to quantify participants spatial ability. Researchers classify an individual’s spatial ability as the performance indicated by their results on such an instrument. It is recognized that this measured performance is constrained by the spatial construct targeted with that spatial instrument. As such, many instruments are available for the researchers use to assess the variety of constructs of spatial ability. Examples include the Purdue Spatial Visualization Test of Rotations (PSVTR), the Mental Cutting Test (MCT), and the Minnesota Paper Foam Board Test. However, at this time, there are no readily accessible spatial ability instruments that can be used to assess spatial ability in a blind or low vision population (BLV). Such an instrument would not only create an instrument capable of quantifying the impacts of spatially focused interventions upon BLV populations but also gives us a quantitative method to assess the effectiveness of spatial curriculum for BLV students. Additionally, it provides a method of assessing spatial ability development from tactile perspective, a new avenue for lines of research that expand beyond the visual methods typically used. This paper discusses the development of the Tactile Mental Cutting Test (TMCT), a non-visually accessible spatial ability instrument, developed and used with a BLV population. Data was acquired from individuals participating in National Federation of the Blind (NFB) Conventions across the United States as well as NFB sponsored summer engineering programs. The paper reports on a National Science Foundation funded effort to garner initial research findings on the application of the TMCT. It reports on initial findings of the instrument’s validity and reliability, as well as the development of the instrument over the first three years of this project. 

   more » « less
3. Differences Between Science and Engineering Undergraduate Students’ Perceived Support: Exploring the Potential of College Profiles

   https://doi.org/10.1109/FIE43999.2019.9028454  

   Hall, Janice L.; Verdin, Dina; Lee, Walter C.; Godwin, Allison; Knight, David (October 2019, Frontiers in Education Conference)

   This work-in-progress research paper stems from a larger project where we are developing and gathering validity evidence for an instrument to measure undergraduate students' perceptions of support in science, technology, engineering, and mathematics (STEM). The refinement of our instrument functions to extend, operationalize, and empirically test the model of co-curricular support (MCCS). The MCCS is a conceptual framework of student support that explains how a student's interactions with the professional, academic and social systems within a college could influence their success more broadly in an undergraduate STEM degree program. Our goal is to create an instrument that functions diagnostically to help colleges effectively allocate resources for the various financial, physical, and human capital support provided to undergraduate students in STEM. While testing the validity of our newly developed instrument, an analysis of the data revealed differences in perceived support among College of Engineering (COE) and College of Science (COS) students. In this work-in-progress paper, we examine these differences at one institution using descriptive statistics and Welch's t-tests to identify trends and patterns of support among different student groups. 

   more » « less
4. Broadening Participation and Success in AP CSA: Predictive Modeling from Three Years of Data

   https://doi.org/10.1145/3408877.3432421  

   Boda, Phillip A.; McGee, Steven (March 2021, ACM Tech. Sym. Comp. Sci. Ed. (SIGCSE’21))

   null (Ed.)

   The AP Computer Science A course and exam continually exhibit inequity among over- and under-represented populations. This paper explored three years of AP CS A data in the Chicago Public School district (CPS) from 2016-2019 (N = 561). We analyzed the impact of teacher and student-level variables to determine the extent AP CS A course taking and exam passing differences existed between over- and under-represented populations. Our analyses suggest four prominent findings: (1) CPS, in collaboration with their Research-Practice Partnership (Chicago Alliance for Equity in Computer Science; CAFÉCS), is broadening participation for students taking the AP CS A course; (2) Over- and under- represented students took the AP CS A exam at statistically comparable rates, suggesting differential encouragement to take or not take the AP CS A exam was not prevalent among these demographics; (3) After adjusting for teacher and student-level prior experience, there were no significant differences among over- and under-represented racial categorizations in their likelihoods to pass the AP CS A exam, albeit Female students were 3.3 times less likely to pass the exam than Males overall; (4) Taking the Exploring Computer Science course before AP CS A predicted students being 3.5 times more likely to pass the AP CS A exam than students that did not take ECS before AP CS A. Implications are discussed around secondary computer science course sequencing and lines of inquiry to encourage even greater broadening of participation in the AP CS A course and passing of the AP CS A exam. 

   more » « less
5. An Item Response Theory Evaluation of a Language-Independent CS1 Knowledge Assessment

   https://doi.org/10.1145/3287324.3287370  

   Xie, Benjamin; Davidson, Matthew J.; Li, Min; Ko, Andrew J. (January 2019, ACM Technical Symposium on Computer Science Education)

   Tests serve an important role in computing education, measuring achievement and differentiating between learners with varying knowledge. But tests may have flaws that confuse learners or may be too difficult or easy, making test scores less valid and reliable. We analyzed the Second Computer Science 1 (SCS1) concept inventory, a widely used assessment of introductory computer science (CS1) knowledge, for such flaws. The prior validation study of the SCS1 used Classical Test Theory and was unable to determine whether differences in scores were a result of question properties or learner knowledge. We extended this validation by modeling question difficulty and learner knowledge separately with Item Response Theory (IRT) and performing expert review on problematic questions. We found that three questions measured knowledge that was unrelated to the rest of the SCS1, and four questions were too difficult for our sample of 489 undergrads from two universities. 

   more » « less