<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>An insight into user experiences of an intelligent tutoring system to learn sketching skills</title></titleStmt>
			<publicationStmt>
				<publisher>ASEE</publisher>
				<date>06/27/2023</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10515612</idno>
					<idno type="doi"></idno>
					<title level='j'>ASEE Annual Conference proceedings</title>
<idno>1524-4644</idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>D Jaison</author><author>H E Merzdorf</author><author>L White</author><author>K A Douglas</author><author>K Watson</author><author>T Hammond</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Engineers who learn to sketch develop many essential skills, such as spatial visualization, design idea representation and fluency, and communication. However, most engineering programs focus on digital design tools and no longer teach freehand sketching. In addition, large engineering classrooms make it challenging for instructors to provide personalized teaching and immediate feedback on sketching assignments. We developed an intelligent tutoring system for teaching 2D and 3D sketching fundamentals in perspective. The system has been deployed at three universities for 4 years in undergraduate and graduate mechanical engineering and design graphics courses. It has also been used by undergraduate instructors outside of engineering. While our research has demonstrated the impact of classroom instruction with the software on student learning, self-efficacy, sketching skills, and design ideation, there has been little research into instructors' experiences teaching freehand sketching with the system.This study evaluates how instructors implemented an intelligent tutoring system for sketching in their classrooms. We hypothesize that course level, subject area, and class size can be sources of variety in instructors' teaching approaches. Our research is guided by the following question: In what ways do engineering and design instructors teach freehand perspective sketching with an intelligent tutoring software?This study follows a qualitative research methodology that included interviews to capture details on instructors' use of the intelligent sketching tutoring software. Three instructors who implemented the system in their mechanical engineering and design visualization courses at both undergraduate and graduate levels, one instructor of first-year engineering, and one industrial design instructor were purposefully recruited. Three instructors taught with the software for multiple semesters. We follow a semi-structured interview protocol asking how instructors introduce the software into their course, how student work with the software is incorporated into the course learning objectives, what benefits instructors saw with using the software to their own instruction and to students' learning, and what difficulties and areas for improvement they identified at the end of each semester. Thematic analysis was performed on the interview responses by two researchers using a qualitative data analysis tool namely MAXQDA. Our results will examine each instructor's practices in detail. We will report the degree to which the intelligent tutoring software was integrated with lessons and assignments and the ways that instructors scaffold student use of the software. We will also identify key design strengths and weaknesses of the system which helped or hindered its use. We will discuss instructors' opinions on the software's support in reaching each course's learning objectives and compare experiences by level (undergraduate or graduate), subject area (engineering, design, or integrated), and class size. These findings will pinpoint future design features and functions for the software, and]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><p>generate wider recommended best practices for sketching instruction in engineering and design courses when using intelligent systems.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.">Background &amp; Literature Review</head><p>Engineers who learn to sketch develop many essential skills, such as spatial visualization, design idea representation and fluency, and communication. Spatial visualization has been linked positively to the retention of engineering students and also improved performance. Research has been ongoing for the past four decades, indicating the importance of spatial visualization in STEM education. Spatial visualization skills in students can be improved with training, and training students in sketching is a significant contributor to improving spatial visualization skills <ref type="bibr">[1]</ref>. Engineering design is an integral part of all engineering disciplines. There are various steps in engineering design, and sketching can contribute in various ways at the various stages of engineering design. At the early stages of engineering design, where ideas are being brainstormed, sketching significantly helps in representing ideas and sharing those ideas visually among the team members. Since the nature of sketches is rough, it supports changeability and encourages the generation of new ideas leading to increased creativity. Graphical communication is yet another skill that is critical to engineering students; by learning to sketch, students gain a tool to communicate.</p><p>Sketching benefits engineering students in diverse ways that also contribute to enhancing the creativity of engineering students <ref type="bibr">[2,</ref><ref type="bibr">3]</ref>. Creativity is critical to engineering design; creative engineers are a necessity for solving the complex challenges of the 21st century <ref type="bibr">[4]</ref>. However, most engineering programs focus on digital design tools and no longer teach freehand sketching <ref type="bibr">[5]</ref>. In addition, large engineering classrooms make it challenging for instructors to provide personalized teaching and immediate feedback on sketching assignments. Sketching instruction is very rarely included as part of the engineering curriculum, while CAD instruction has been integrated into the engineering curriculum. Many engineering instructors have called for sketching to be reintegrated into the classroom so that students may develop essential visualization, communication, and creative skills.</p><p>SketchTivity is an intelligent tutoring system for teaching 2D and 3D sketching fundamentals in perspective to engineering students that was developed at Texas A&amp;M University <ref type="bibr">[6]</ref><ref type="bibr">[7]</ref><ref type="bibr">[8]</ref><ref type="bibr">[9]</ref><ref type="bibr">[10]</ref><ref type="bibr">[11]</ref>. The system has been deployed at three universities for 4 years in undergraduate and graduate mechanical engineering and design graphics courses. It has also been used by undergraduate instructors outside of engineering. Students receive real-time feedback on their speed, precision and smoothness and also an additional tip to help them improve. SketchTivity has repeatedly improved the sketching self-efficacy of students along with sketching skill development. A survey instrument that measures the self-efficacy of students was developed and validated as part of the project that helped us <ref type="bibr">[12]</ref>.</p><p>A few years ago, an instructor who used SketchTivity as part of his instruction collected feedback from students about their experiences with the tool <ref type="bibr">[13]</ref>. Instructors have been willing to continue using the tool for instructional purposes in their classes for teaching sketching. Recently, as part of our commitment to improving the tool, we examined the experience of undergraduate and graduate engineering students from three institutions regarding their use of SketchTivity to learn sketching and their perspectives on improving the intelligent tutoring system <ref type="bibr">[14]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.1.">Research Motivation</head><p>While our research has demonstrated the impact of classroom instruction with the software on student learning, self-efficacy, sketching skills, and design ideation, there has been little research into instructors' experiences teaching freehand sketching with the system. This study evaluates how instructors implemented an intelligent tutoring system for sketching in their classrooms.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.2.">Research Questions</head><p>We hypothesize that course level, subject area, and class size can be sources of variety in instructors' teaching approaches. Our research is guided by the following question: In what ways do engineering and design instructors teach freehand perspective sketching with an intelligent tutoring software?</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.">Methods</head><p>This study follows a qualitative research methodology that included interviews to capture details on instructors' use of the intelligent sketching tutoring software. Thematic analysis was conducted using an inductive approach to find patterns across data <ref type="bibr">[15]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1.">Interview Protocol</head><p>We followed a semi-structured interview protocol with 10 initial questions about instructors' use of SketchTivity in their teaching. Nearly all main questions had at least one probing question to follow up with participants and gain more detail into their experiences. Interviews were conducted and transcribed via Zoom and lasted around 15 minutes. The researchers cleaned Zoom transcripts for clarity and compiled all responses by question. Not all probing questions were asked to all instructors; however, all instructors provided responses to Questions 1-8 and provided general reflections for Questions 9-10.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.">Participants</head><p>Three instructors who implemented the system in their mechanical engineering and design visualization courses at both undergraduate and graduate levels, one instructor of first-year engineering, and one industrial design instructor were purposefully recruited. The first-year engineering instructor had previous experience teaching with the software at the high school level (see Table <ref type="table">2</ref>). As experienced instructors who were experts in their fields and had taught multiple semesters with SketchTivity, we were interested in discovering their long-term perspective on teaching with the software. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.">Results</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1.">Courses where SketchTivity was implemented</head><p>The first instructor was a mechanical engineering professor who taught a freshman-level computer-aided design-based class. This course was one of the first courses taken by primarily freshman students in the mechanical engineering program, more than 80%; but also by civil and aerospace engineering majors, including those who plan to change majors in the future. The second instructor taught a three-course sequence in first-year engineering mechanics, which included a lecture and a lab. The third instructor taught in a first-year industrial design twocourse sequence of labs in the fall and spring semesters, which met for two hours per week and were primarily for industrial design majors.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2.">Significant changes in implementation and instruction over time</head><p>For the mechanical engineering instructor, who taught the course with a graduate teaching assistant, course lecture content became more specific towards the application of the software over time. They reported becoming more confident over time in answering questions about software functionality and adapting lectures from other team members towards the needs of their students.</p><p>"With respect to the implementation I think as we went from one semester to the other, we were a bit more confident in how the software works, and we were able to answer the questions ourselves. And also, we tailored the lectures a little bit more towards the application of the software.</p><p>Throughout the implementation, the mechanical engineering instructor focused on teaching areas which could be easily sketched by students with SketchTivity.</p><p>The first-year engineering instructor reported the benefit of prior experience teaching with SketchTivity at the high school level in structuring their teaching at the university level. They affirmed that this had helped them understand how students engage with the software, which informed their design of a program structured around their interest. The first-year instructor described starting with all seven sketching items and focusing on those which they observed to have the strongest impact on students' skill development.</p><p>According to the industrial design instructor, SketchTivity was not a standalone learning platform, and therefore was not developed to take the role of an instructor. Instead, the software was meant to monitor student learning and provide feedback:</p><p>"... [SketchTivity's] intent was to help monitor or assess students' ability to do certain things in perspective drawing. It was meant as a supplement, at least the way we used it in our class, as a supplement to a physical in person instruction. If we take that as a basis for that answer, it served that purpose quite well."</p><p>Specific features supported the industrial design professor's teaching over time, such as dashboards which indicated number of attempts and logins, achievement metric scores, and progress over time. These data helped the instructor with grading and judging student effort, as well as showing improvement throughout the semester. The instructor cited this as providing insight into student motivation, with one expected motivator being the line drawing practice game.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3.">Introduction of SketchTivity with course homework and grades</head><p>The mechanical engineering instructor taught perspective sketching lessons during the same week as implementing SketchTivity in the lab section of the course. The instructor-reported changing one computer-aided design assignment to a freehand sketching assignment, paired with a "more detailed lecture on perspective sketching and how the software can help students," which the instructor felt was a natural connection between instruction and software.</p><p>Approximately halfway through the semester, the first-year engineering instructor integrated SketchTivity into lab weeks where students were learning with hands-on measurement activities as well as online simulations. In these weeks, SketchTivity was presented as an intelligent tutoring system that could help develop sketching capabilities. As most students in these courses did not have any sketching experience and were primarily using CAD design, the instructor emphasized the value of sketching:</p><p>"I just wanted to introduce them to say, 'you're going to have time in your career where you may not have technology available, or you may be more effective just to sketch out [a] design, so we're going to make sure you have at least know-how to do that effectively."</p><p>Students in the first-year engineering courses were free to use SketchTivity as available on laptops or tablets during this time as they continued into the labs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.4.">Length of time for student practice with SketchTivity</head><p>Most instructors reported using SketchTivity for anywhere from 15-45 minutes in class. The mechanical engineering instructor assigned homework in addition to the free time during labs where students could practice with the software. The first-year engineering instructor reported that many students wanted to try more activities than were assigned, especially when ZenSketch was a feature of the software that motivated them to play. At the same time, the first-year engineering instructor reported that some students were limited by tactile issues when sketching with a tablet and digital pen compared with paper. The industrial design instructor advised their students to practice with SketchTivity for a minimum of 15-20 minutes per day for 5 days, with one hour spent on the system being sufficient. However, the instructor realized that logins were not the only indicator of engagement, as students could log in but not complete the lessons, which caused them to use "a mixture of how long they were on it, and how many attempts they had."</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.5.">Support of course learning objectives by SketchTivity</head><p>Instructors had mixed responses regarding SketchTivity's support of their course learning objectives, which originated from the larger role of sketching in the curriculum. The mechanical engineering professor noted that teaching with SketchTivity allowed them to bring sketching into focus alongside computer-aided design:</p><p>"The way the course is organized, it's more focused towards computer-aided design. And that's something we are struggling with, students are more comfortable with CAD models than handson sketches. So hands-on sketches are much easier to make and communicate with somebody else, but students are not comfortable with sketching, right? So that's one of the reasons why I wanted to incorporate SketchTivity so the students could practice sketching a little bit more."</p><p>The mechanical engineering instructor observed that students became more confident freehand sketchers with practice, which they believed was an essential skill for engineers. Similarly, the first-year engineering instructor did not have specific course objectives for sketching and used SketchTivity as a way to incorporate the sketching skills which they felt were missing from the course:</p><p>"I was able to carve out a week to focus on some things that weren't necessarily elicited as learning objectives for the course, but [I] knew to be underlying things that are good for engineers to know."</p><p>The industrial design instructor listed three objectives for introducing SketchTivity into classroom instruction: documenting and monitoring students' effort and participation in the course, measuring learning progression for sketch accuracy or fluidity, and finding ways to support students' extrinsic motivations "to spur them to try to continue to work by introducing the level of competition. There was something there about having a leaderboard or gamifying it that turned it into something there that helps spur the students to work on their own."</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.6.">Benefits provided by SketchTivity to teaching</head><p>In the mechanical engineering instructor's courses, the syllabus did not include sketching in regular instruction and left them little time to teach perspective sketching principles. Therefore, the mechanical engineering instructor reported that SketchTivity was helpful for giving students the opportunity to practice outside the course: " It's not really changing anything with respect to the course learning objectives. But I do see personally, the benefits of SketchTivity as a tool that can help students to get that experience in sketching."</p><p>The first-year engineering instructor referenced their prior experience implementing the product of a teacher professional development training in their own course, and how SketchTivity benefited students in different learning teams:</p><p>"I could see when I put them in teams to work through the EDP [engineering design process]. and I don't have a concurrent control group to compare that to, but thinking through when I've done some EDP work prior, I noticed that there was, within the groups, a lot more sketching of ideas as they're coming, especially the brainstorming element."</p><p>In this context, the first-year engineering instructor observed that students who were familiar with SketchTivity were able to provide visual drawings for their quick brainstorms to help other students understand their ideas. Compared to the prior semesters before the first-year engineering instructor taught with SketchTivity, they observed students involved in engineering design projects sketched more frequently.</p><p>For the industrial design professor, SketchTivity provided value to their teaching by using realtime assessment about student sketching activity to streamline many of the one-on-one interactions between students and teachers. Compared to traditional art or trade schools where instructors monitor students' work individually, having intelligent tutoring system provided the instructor with a resource to alleviate the repetition in large classes:</p><p>"In essence, that is quite good. Because then you could just log into the dashboard, and at a very quick glance see your entire classroom, and see the kind of progression and kind of ability at that point in time of students doing certain exercises. That's kind of the best impact from a tedium teaching kind of standpoint. It's real time assessment, and it's instant, and constantly updates. So that says it's a great read on to your students."</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.7.">Benefits provided by SketchTivity to student learning</head><p>Overall, the mechanical engineering instructor reported that students were increasingly comfortable with relying on sketching to communicate their ideas, rather than immediately relying on CAD models:</p><p>"In general, I notice students are getting more comfortable with the art of sketching. So instead of like when I ask them a question. Show me a system instead of just creating a CAD model immediately. The students are sketching more, especially some of these students who came to my follow up classes like dynamics."</p><p>Similarly, the first-year engineering instructor noticed that students who previously had low selfefficacy in their beliefs that they could draw improved in their confidence:</p><p>"Some students would go into the SketchTivity saying, 'I really don't draw well at all.' And actually saw how you can learn how to draw better. And by the end of that they were much more comfortable with their sketching and their ability to sit down and just kind of sketch out something pretty quickly, where I don't think they would have tried that before."</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.8.">Areas for improvement with SketchTivity instruction</head><p>To maintain student interest in using the software outside of lectures, the mechanical engineering professor preferred to have more gamification features in SketchTivity. Features such as puzzles or games would keep students interested in playing, as students "are very interested in gamebased platforms, so maybe that's a way to go, so they will continue using that, and they develop their hands-on sketching abilities." The first-year engineering instructor felt that animated tutorials were helpful and agreed that additional gamification features would help reinforce standalone principles of primitive shapes, to "actually see it in a particular environment that works on improving the skill sets."</p><p>The industrial design instructor described two limitations with SketchTivity that could be improved. First, they wished to see software support for more complex primitive combinations to help students advance beyond drawing basics. Second and conversely, the industrial design instructor recognized that students need motivation to perform the repetition necessary to become skilled in the basics, which SketchTivity could not always provide:</p><p>"Drawing, like art and creativity, or acting, or painting, or music, it is something that requires practice, something that requires 10,000 hours of doing rote things, or doing the thing same things over and over again in order to get good enough so that it unlocks your creativity, because you're no longer thinking consciously about doing something. So in that sense, right?</p><p>You know, the software has a limitation on its motivation."</p><p>The industrial design instructor noted that gamified features were motivating to a certain extent, but that small changes such as digital avatars and star ratings did not fully tap into students' deeper intrinsic motivations to sketch. Compared to experts who were intrinsically motivated, the instructor felt that the software being tailored towards beginners was more based on extrinsic motivation of competition and rewards. Because of this, the instructor concluded that "We never really got into what would encourage someone to do this type of drawing exercise. But I think that's one that we could probably look into, or at least talk about."</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Discussion</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Instructor Scaffolding and Classroom Integration</head><p>Instructors of all courses were intentional in how they incorporated sketching into their classes.</p><p>In particular, they considered gaps in the curriculum where sketching could be used to teach essential skills which would benefit students in their future practice. Instructors provided students with both structured and unstructured opportunities to practice sketching with SketchTivity, including in-class free use and out-of-class homework assignment hours requirements. Instructors had learning objectives which they aimed for students to accomplish with SketchTivity. Outcomes of sketching included improving motivation to sketch, increased confidence to use sketching during engineering design, and willingness to sketch for informal communication.While each instructor taught with SketchTivity for a different length of time and in different courses, each leveraged their professional experience and teaching goals for helping students use SketchTivity effectively.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>System Design Strengths and Weaknesses</head><p>The software had strengths and weaknesses identified by the instructors. While its design supported repetition and practice of sketching fundamentals, it was not able to provide feedback on advanced sketching technique, which made it more suitable for beginners. One feature which all instructors agreed was beneficial for engagement and motivation was the game for line drawing practice. However, they acknowledged that gamification also has limitations, and further research is needed to know what can motivate students to continue sketching long-term.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Software Support of Learning Objectives</head><p>In large courses, SketchTivity software supported instructors in scaling instruction to provide individualized feedback, while in smaller courses instructors created open-ended opportunities for students to practice independently. It also supported formative assessment through real-time updates on student performance and a dashboard of metrics for an entire class. While some instructors felt that the metrics were limited, they were useful for teaching fundamentals and provided a starting point for with little experience or confidence in sketching.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Conclusions and Future Directions</head><p>As this software continues to be improved and implemented in other engineering design contexts, we plan to support more advanced learning objectives and explore features that can support motivation and engagement. These findings can help pinpoint future design features and functions for the software, and generate wider recommended best practices for sketching instruction in engineering and design courses when using intelligent systems. In addition to improving gamification features for student engagement, there is potential for greater alignment between curriculum objectives and sketching practice. Further, we hope to continue developing the pedagogy around the software to systematically teach sketching fundamentals in engineering design, using it to build on basics and support more advanced technique.</p></div></body>
		</text>
</TEI>
