skip to main content


Title: WIP: Evaluating the Impact on a User’s Motivation To Improve Their Sketching Ability Due to the Gamification of a User Interface
Sketching skills are developed over time through practice, requiring students to stay motivated to continue improving. Gamification has been shown to be helpful in keeping users motivated, so this work seeks to investigate the impact of gamification on the user’s motivation to practice sketching skills in the intelligent tutoring system, SketchTivity. Specifically, this work will evaluate the impact of gamified elements including achievement banners, star ratings, and performance statistics to give users feedback about their level of success after a sketching lesson. This concept will be explored through within subjects focus group testing where participants will interact with each version of the interface, describe their experiences in a think-aloud fashion, and discuss their preferences in a post-interview. The motivational impact of the gamified elements will be synthesized through thematic analysis of the think-aloud comments and interview data as well as statistical analysis of performance differences in terms of SketchTivity’s sketch quality metrics.  more » « less
Award ID(s):
2013612
NSF-PAR ID:
10418342
Author(s) / Creator(s):
Date Published:
Journal Name:
Joint Proceedings of the ACM IUI Workshops 2023
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Engineers are called to play an important role in addressing the complex problems of our global society, such as climate change and global health care. In order to adequately address these complex problems, engineers must be able to identify and incorporate into their decision making relevant aspects of systems in which their work is contextualized, a skill often referred to as systems thinking. However, within engineering, research on systems thinking tends to emphasize the ability to recognize potentially relevant constituent elements and parts of an engineering problem, rather than how these constituent elements and parts are embedded in broader economic, sociocultural, and temporal contexts and how all of these must inform decision making about problems and solutions. Additionally, some elements of systems thinking, such as an awareness of a particular sociocultural context or the coordination of work among members of a cross-disciplinary team, are not always recognized as core engineering skills, which alienates those whose strengths and passions are related to, for example, engineering systems that consider and impact social change. Studies show that women and minorities, groups underrepresented within engineering, are drawn to engineering in part for its potential to address important social issues. Emphasizing the importance of systems thinking and developing a more comprehensive definition of systems thinking that includes both constituent parts and contextual elements of a system will help students recognize the relevance and value of these other elements of engineering work and support full participation in engineering by a diverse group of students. We provide an overview of our study, in which we are examining systems thinking across a range of expertise to develop a scenario-based assessment tool that educators and researchers can use to evaluate engineering students’ systems thinking competence. Consistent with the aforementioned need to define and study systems thinking in a comprehensive, inclusive manner, we begin with a definition of systems thinking as a holistic approach to problem solving in which linkages and interactions of the immediate work with constituent parts, the larger sociocultural context, and potential impacts over time are identified and incorporated into decision making. In our study, we seek to address two key questions: 1) How do engineers of different levels of education and experience approach problems that require systems thinking? and 2) How do different types of life, educational, and work experiences relate to individuals’ demonstrated level of expertise in solving systems thinking problems? Our study is comprised of three phases. The first two phases include a semi-structured interview with engineering students and professionals about their experiences solving a problem requiring systems thinking and a think-aloud interview in which participants are asked to talk through how they would approach a given engineering scenario and later reflect on the experiences that inform their thinking. Data from these two phases will be used to develop a written assessment tool, which we will test by administering the written instrument to undergraduate and graduate engineering students in our third study phase. Our paper describes our study design and framing and includes preliminary findings from the first phase of our study. 
    more » « less
  2. Previous studies have convincingly shown that traditional, content-centered, and didactic teaching methods are not effective for developing a deep understanding and knowledge transfer. Nor does it adequately address the development of critical problem-solving skills. Active and collaborative instruction, coupled with effective means to encourage student engagement, invariably leads to better student learning outcomes irrespective of academic discipline. Despite these findings, the existing construction engineering programs, for the most part, consist of a series of fragmented courses that mainly focus on procedural skills rather than on the fundamental and conceptual knowledge that helps students become innovative problem-solvers. In addition, these courses are heavily dependent on traditional lecture-based teaching methods focused on well-structured and closed-ended problems that prepare students to plug variables into equations to get the answer. Existing programs rarely offer a systematic approach to allow students to develop a deep understanding of the engineering core concepts and discover systematic solutions for fundamental problems. Without properly understanding these core concepts, contextualized in domain-specific settings, students are not able to develop a holistic view that will help them to recognize the big picture and think outside the box to come up with creative solutions for arising problems. The long history of empirical learning in the field of construction engineering shows the significant potential of cognitive development through direct experience and reflection on what works in particular situations. Of course, the complex nature of the construction industry in the twenty-first century cannot afford an education through trial and error in the real environment. However, recent advances in computer science can help educators develop virtual environments and gamification platforms that allow students to explore various scenarios and learn from their experiences. This study aims to address this need by assessing the effectiveness of guided active exploration in a digital game environment on students’ ability to discover systematic solutions for fundamental problems in construction engineering. To address this objective, through a research project funded by the NSF Division of Engineering Education and Centers (EEC), we designed and developed a scenario-based interactive digital game, called Zebel, to guide students solve fundamental problems in construction scheduling. The proposed gamified pedagogical approach was designed based on the Constructivism learning theory and a framework that consists of six essential elements: (1) modeling; (2) reflection; (3) strategy formation; (4) scaffolded exploration; (5) debriefing; and (6) articulation. We also designed a series of pre- and post-assessment instruments for empirical data collection to assess the effectiveness of the proposed approach. The proposed gamified method was implemented in a graduate-level construction planning and scheduling course. The outcomes indicated that students with no prior knowledge of construction scheduling methods were able to discover systematic solutions for fundamental scheduling problems through their experience with the proposed gamified learning method. 
    more » « less
  3. We propose and evaluate a lightweight strategy for tracing code that can be efficiently taught to novice programmers, building off of recent findings on "sketching" when tracing. This strategy helps novices apply the syntactic and semantic knowledge they are learning by encouraging line-by-line tracing and providing an external representation of memory for them to update. To evaluate the effect of teaching this strategy, we conducted a block-randomized experiment with 24 novices enrolled in a university-level CS1 course. We spent only 5-10 minutes introducing the strategy to the experimental condition. We then asked both conditions to think-aloud as they predicted the output of short programs. Students using this strategy scored on average 15% higher than students in the control group for the tracing problems used the study (p<0.05). Qualitative analysis of think-aloud and interview data showed that tracing systematically (line-by-line and "sketching" intermediate values) led to better performance and that the strategy scaffolded and encouraged systematic tracing. Students who learned the strategy also scored on average 7% higher on the course midterm. These findings suggest that in <1 hour and without computer-based tools, we can improve CS1 students' tracing abilities by explicitly teaching a strategy. 
    more » « less
  4. This work-in-progress paper shares findings of the early stage of a 3-year research funded by the National Science Foundation. The major aim of the project is to advance engineering and mathematics (EM) education theory and practice related to students’ self-regulation of cognition and motivation skills during problem-solving activities. The self-regulation includes students’ metacognitive knowledge about task (MKT) and self-regulation of cognition (SRC). The motivational component of self-regulation (SRM) includes self-control of the motivation needed to maintain the level of engagement and deliberate practice necessary for scientific thinking and reasoning. To be effective problem-solvers, students must understand the relationship between the MKT, SRC and SRM throughout the problem-solving activities. Four research questions will guide the research: (1) How do students perceive their self-regulation of cognition (SRC) and motivation (SRM) skills for generic problem-solving activities in EM courses; (2) How does students’ metacognitive knowledge about problem-solving tasks (MKT) inform their Task interpretation?; (3) How do students’ SRC and SRM dynamically evolve?; and (4) How do students’ SRC and SRM reflect their perceptions of self-regulation of cognition and motivation for generic EM problemsolving activities? A sequential mixed-methods research design involving quantitative and qualitative methods are used to develop complementary coarse- and fine-grained understandings of undergraduate students’ SRC and SRM during academic problem-solving activities. Two 2nd year EM courses: Engineering Statics, and Ordinary Differential Equations were purposefully selected for the contexts of the study. One hundred forty two students from both courses were invited and participated in quantitative data collection using two validated surveys during spring 2022 semester. Later in the semester, qualitative data will be generated with twenty students in both courses through one-on-one interviews with students and course instructors, think-aloud protocols with students, and classroom observations. Coarse-grained understandings of students’ SRC and SRM are currently developed through analysis of quantitative data collected using self-report surveys (i.e., BRoMS and PMI). Fine-grained understandings of students’ SRC and SRM will be developed through analysis of qualitative data gathered via one-on-one interviews, think-aloud protocols, classroom observations, and course artifacts gathered as students engage in EM problem-solving activities. 
    more » « less
  5. As the need for interdisciplinary collaboration increases, industry needs engineers who are not only affluent in technical engineering skills but also efficient in skills such as communication, problem-solving, engineering ethics, and business management. As a result, engineering programs are tasked with providing students with sufficient opportunities to develop non-technical professional skills to better prepare them for the workforce. Previous research has focused on exploring how and where students tend to develop profession skills and assessments have been established to measure the level of professional skills. However, without a means to measure whether students are getting sufficient opportunities for development, it is hard for educators and engineering programs to determine whether or where scaffolding are needed. We developed an instrument to assess undergraduate engineering students’ opportunities for professional skill development. To increase content validity, we conducted 20 think-aloud interviews with students from a large Midwestern university. The aim of this WIP is two-fold. We present the preliminary results of the think-aloud interview to determine what changes need to be made to existing items and what emerging themes appear regarding to participants’ professional skill development opportunities. After thematic analysis of the interview transcripts, we revised 10 items by simplifying the grammar or altering certain words that tend to confuse participants or carry negative connotations. We found that, compared to students who have only been involved in class projects, those with co-curricular experiences tend to report more opportunities in skills related to business management principles and problem-solving skills. Co-curricular activities were also the most referenced in building communication skills. Our next step will be piloting the instrument across multiple institutions and conducting validation analysis. 
    more » « less