skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments
Title: Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments
This Complete Research paper investigates the holistic assessment of creativity in design solutions in engineering education. Design is a key element in contemporary engineering education, given the emphasis on its development through the ABET criteria. As such, design projects play a central role in many first-year engineering courses. Creativity is a vital component of design capability which can influence design performance; however, it is difficult to measure through traditional assessment rubrics and holistic assessment approaches may be more suitable to assess creativity of design solutions. One such holistic assessment approach is Adaptive Comparative Judgement (ACJ). In this system, student designs are presented to judges in pairs, and they are asked to select the item of work that they deem to have demonstrated the greatest level of a specific criterion or set of criteria. Each judge is asked to make multiple judgements where the work they are presented with is adaptively paired in order to create a ranked order of all items in the sample. The use of this assessment approach in technology education has demonstrated high levels of reliability among judges (~0.9) irrespective of whether the judges are students or faculty. This research aimed to investigate the use of ACJ to holistically assess the creativity of first-year engineering students design solutions. The research also sought to explore the differences, if any, that would exist between the rank order produced by first-year engineering students and the faculty who regularly teach first-year students. Forty-six first-year engineering students and 23 faculty participated in this research. A separate ACJ session was carried out with each of these groups; however, both groups were asked to assess the same items of work. Participants were instructed to assess the creativity of 101 solutions to a design task, a “Ping Pong problem,” where undergraduate engineering students had been asked to design a ping pong ball launcher to meet specific criteria. In both ACJ sessions each item of work was included in at least 11 pairwise comparisons, with the maximum number of comparisons for a single item being 29 in the faculty ACJ session and 50 in the student ACJ session. The data from the ACJ sessions were analyzed to determine the reliability of using ACJ to assess creativity of design solutions in first-year engineering education, and to explore whether the rankings produced from the first-year engineering students ACJ session differed significantly from those of the faculty. The results indicate a reasonably high level of reliability in both sessions as measured by the Scale Separation Reliability (SSR) coefficient, SSRfaculty = 0.65 ± 0.02, SSRstudents = 0.71 ± 0.02. Further a strong correlation was observed between the ACJ ranks produced by the students and faculty both when considered in terms of the relative differences between items of work, r = .533, p < .001, and their absolute rank position, σ = .553, p < .001. These findings indicate that ACJ is a promising tool for holistically assessing design solutions in engineering education. Additionally, given the strong correlation between ranks of students and faculty, ACJ could be used to include students in their own assessment to reduce the faculty grading burden or to develop a shared construct of capability which could increase the alignment of teaching and learning.  more » « less
Award ID(s):
2020785
PAR ID:
10480526
Author(s) / Creator(s):
; ; ; ;
Publisher / Repository:
ASEE Peer
Date Published:
Journal Name:
Review directory American Society for Engineering Education
ISSN:
0092-4326
Format(s):
Medium: X
Location:
Baltimore, MD
Sponsoring Org:
National Science Foundation
More Like this
  1. ; (, SEFI Conference Proceedings)
    This study aims to investigate the development of creativity in engineering education and how spatial skills relate to creativity of design solutions. Undergraduate students in the first (n=86) and fourth/fifth year (n=48) of their engineering programme were invited to participate. Students completed four spatial tests to precisely measure visualisation skills. In a separate session, students were invited back to solve two engineering design tasks: a ping pong problem where they designed a ping pong ball launcher game to meet specified criteria and a rain catcher problem where they were tasked with developing as many ideas for capturing rainwater as a water source for a remote location as they could. Students were asked not to consider feasibility, cost, etc. and to come up multiple radical solutions to the rainwater capture problem. The creativity of design solutions was assessed using Adaptive Comparative Judgement. Statistical analysis indicated significant relationships between spatial skills, students’ year of study and gender. A statistically significant relationship was also found between students’ creativity scores on both design challenges. No statistical differences were determined in the creativity of first and fourth/fifth year students’ solutions. These findings will be discussed relative to existing research, future work, and potential implications for education practice. 
    more » « less
  2. ; ; ; (, Review directory American Society for Engineering Education)
    Design projects are an important part of many first-year engineering programs. The desire to employ holistic assessment strategies to student work with open-ended and divergent responses has been widely noted in the literature. Holistic strategies can provide insight into the role of qualities (e.g., professional constructs) that are not typically conducive to standard assessment rubrics. Adaptive Comparative Judgement (ACJ) is an assessment approach that is used to assess design projects holistically. The assessment of projects using ACJ can be carried out by experts or students to scaffold their learning experience. This Work-in-Progress paper explores the use and benefits of ACJ for assessing design projects specifically focusing on first-year engineering students and educators. Further, conference attendees will be provided the opportunity throughout the conference to engage with the ACJ software to experience how this system can work in practice for assessing student design projects. 
    more » « less
  3. (, ICERI Intenational Conference on Educational Research and Innovation Conference Proceedings)
    Design is a core attribute of engineering practice; in fact, the etymology of the word engineer is traced to the Latin “ingeniare,” which translates as inventor or designer. In order to prepare our students for success in an engineering career, they must be proficient at design and able to think creatively and flexibly about optimal solutions to problems. Additionally, numerous studies have demonstrated the need for well-developed spatial skills for success in engineering, especially in engineering problem solving. Studies also show a link between spatial thinking and technical creativity. The focus of this research is on understanding the relationship between spatial visualization skills and engineering design. As a first phase of testing, 127 undergraduate engineering students completed four tests of spatial ability. In a second phase, 102 students returned and were asked to complete three tasks. The first was designing a ping pong ball launcher to hit a target at a specific height from a given distance. They were then asked to list as many factors as possible that should be considered when designing a retaining wall for mitigating flood damage along the Mississippi River in the Midwest in the United States. The third task was to sketch as many ideas as possible in a given timeframe for a rainwater collection system in a remote location. This research paper will present preliminary analysis of the Midwest Flood problem and spatial ability data. The initial insights will be discussed relative to the overall study and how this work could inform undergraduate engineering education, and specifically the provision of design education. 
    more » « less
  4. ; ; (, Review directory American Society for Engineering Education)
    This research paper details a study investigating spatial visualization skills relation to design problem-solving for undergraduate engineering students. Design is outlined as one of the seven attributes that engineering students must demonstrate prior to their graduation as set out through the ABET guidelines. It is important to understand the factors that contribute to design capability to achieve this learning goal. Design problems by their nature are cognitive tasks and as such require problem solvers to draw both on learned knowledge and pertinent cognitive abilities for their solution. In the context of engineering design problem solving, spatial visualization is one such cognitive ability that likely plays a role. Previous research has demonstrated a link between spatial visualization and design. This work aims to advance on that research by exploring how spatial visualization relates to the design process enacted by undergraduate engineering students. There were two phases to data collection for this research. In the first phase, 127 undergraduate engineering students completed four spatial tests. In the second phase, 17 students returned to complete three design tasks. This paper will focus on one of these design tasks, the Ping Pong problem where individuals are asked to design a ping pong launcher to hit a target from a given distance at a specific height. A purposive sample of 9 first-year and 8 senior students were selected to engage in a think aloud protocol during the problemsolving task based on their spatial visualization skill levels (high vs. low). The think aloud protocol was used to assign pre-defined codes for design activity for each of the 17 participants. Through analysis of these codes, results indicated that there is an association between the spatial skills of students and the design processes/actions that they employ. These insights will be discussed relative to their potential influence on engineering education, specifically in developing design capability. 
    more » « less
  5. ; ; (, Frontiers in Education)
    Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students ( n = 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email