skip to main content


Title: Eye-Track Modeling of Problem-Solving in Virtual Manufacturing Environments
Problem-solving focuses on defining and analyzing problems, then finding viable solutions through an iterative process that requires brainstorming and understanding of what is known and what is unknown in the problem space. With rapid changes of economic landscape in the United States, new types of jobs emerge when new industries are created. Employers report that problem-solving is the most important skill they are looking for in job applicants. However, there are major concerns about the lack of problem-solving skills in engineering students. This lack of problem-solving skills calls for an approach to measure and enhance these skills. In this research, we propose to understand and improve problem-solving skills in engineering education by integrating eye-tracking sensing with virtual reality (VR) manufacturing. First, we simulate a manufacturing system in a VR game environment that we call a VR learning factory. The VR learning factory is built in the Unity game engine with the HTC Vive VR system for navigation and motion tracking. The headset is custom-fitted with Tobii eye-tracking technology, allowing the system to identify the coordinates and objects that a user is looking at, at any given time during the simulation. In the environment, engineering students can see through the headset a virtual manufacturing environment composed of a series of workstations and are able to interact with workpieces in the virtual environment. For example, a student can pick up virtual plastic bricks and assemble them together using the wireless controller in hand. Second, engineering students are asked to design and assemble car toys that satisfy predefined customer requirements while minimizing the total cost of production. Third, data-driven models are developed to analyze eye-movement patterns of engineering students. For instance, problem-solving skills are measured by the extent to which the eye-movement patterns of engineering students are similar to the pattern of a subject matter expert (SME), an ideal person who sets the expert criterion for the car toy assembly process. Benchmark experiments are conducted with a comprehensive measure of performance metrics such as cycle time, the number of station switches, weight, price, and quality of car toys. Experimental results show that eye-tracking modeling is efficient and effective to measure problem-solving skills of engineering students. The proposed VR learning factory was integrated into undergraduate manufacturing courses to enhance student learning and problem-solving skills.  more » « less
Award ID(s):
1830741
NSF-PAR ID:
10285346
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
ASEE annual conference proceedings
ISSN:
1524-4857
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Virtual reality (VR) technology allows for the creation of fully immersive environments that enable personalized manufacturing learning. This case study discusses the development of a virtual learning factory that integrates manual and automated manufacturing processes such as welding, fastening, 3D printing, painting, and automated assembly. Two versions of the virtual factory are developed: (1) a multiplayer VR environment for the design and assembly of car toys; which allows for the collaboration of multiple users in the same VR environment, and (2) a virtual plant that utilizes heavy machinery and automated assembly lines for car manufacturing. The virtual factory also includes an intelligent avatar that can interact with the users and guide them to the different sections of the plant. The virtual factory enhances the learning of advanced manufacturing concepts by combining virtual objects with hands-on activities and providing students with an engaging learning experience. 
    more » « less
  2. null (Ed.)
    Virtual reality (VR) technology allows for the creation of fully immersive environments that enable personalized manufacturing learning. This case study discusses the development of a virtual learning factory that integrates manual and automated manufacturing processes such as welding, fastening, 3D printing, painting, and automated assembly. Two versions of the virtual factory are developed: (1) a multiplayer VR environment for the design and assembly of car toys; which allows for the collaboration of multiple users in the same VR environment, and (2) a virtual plant that utilizes heavy machinery and automated assembly lines for car manufacturing. The virtual factory also includes an intelligent avatar that can interact with the users and guide them to the different sections of the plant. The virtual factory enhances the learning of advanced manufacturing concepts by combining virtual objects with hands-on activities and providing students with an engaging learning experience. 
    more » « less
  3. Problem-solving is an iterative process that requires brainstorming, analysis of the problem, development and testing of solutions. It relies on under-standing what is known and what is unknown about the problem. That knowledge of the knowns and unknowns is called metacognition. Today’s engineers must understand their own metacognition and that of other team members to derive the best solutions for engineering problems given the different constraints. Engineers working in design and manufacturing fields confront challenges due to a lack of important metacognitive understanding of their own and their team’s problem-solving skills. This research suggests measuring metacognition within teams by using manufacturing simulations with virtual reality and eye tracking 
    more » « less
  4. Engineering education aims to create a learning environment capable of developing vital engineering skill sets, preparing students to enter the workforce and succeed as future leaders. With all the rapid technological advancements, new engineering challenges continuously emerge, impeding the development of engineering skills. This insufficiency in developing the required skills resulted in high regression rates in students’ GPAs, resulting in industries reporting graduates’ unsatisfactory performance. From a pedagogical perspective, this problem is highly correlated with traditional learning methods that are inadequate for engaging students and improving their learning experience when adopted alone. Accordingly, educators have incorporated new learning methodologies to address the pre-defined problem and enhance the students’ learning experience. However, many of the currently adopted teaching methods still lack the potential to expose students to practical examples, and they are inefficient among engineering students, who tend to be active learners and prefer to use a variety of senses. To address this, our research team proposes integrating the technology of virtual reality (VR) into the laboratory work of engineering technology courses to improve the students’ learning experience and engagement. VR technology, an immersive high-tech media, was adopted to develop an interactive teaching module on hydraulic gripper designs in a VR construction-like environment. The module aims to expose engineering technology students to real-life applications by providing a more visceral experience than screen-based media through the generation of fully computer-simulated environments in which everything is digitized. This work presents the development and implementation of the VR construction lab module and the corresponding gripper designs. The virtual gripper models are developed using Oculus Virtual Reality (OVR) Metrics Tool for Unity, a Steam VR Overlay utility created to make visualizing the desktop in a VR setting simple and intuitive. The execution of the module comprises building the VR environment, designing and importing the gripper models, and creating a user-interface VR environment to visualize and interact with the model (gripper assembly/mechanism testing). Besides the visualization, manipulation, and interaction, the developed VR system allows for additional features like displaying technical information, guiding students throughout the assembly process, and other specialized options. Thus, the developed interactive VR module will serve as a perpetual mutable platform that can be readily adjusted to allow future add-ons to address future educational opportunities. 
    more » « less
  5. Familiarity with manufacturing environments is an essential aspect for many engineering students. However, such environments in real world often contain expensive equipment making them difficult to recreate in an educational setting. For this reason, simulated physical environments where the process is approximated using scaled-down representations are usually used in education. However, such physical simulations alone may not capture all the details of a real environment. Virtual reality (VR) technology nowadays allows for the creation of fully immersive environments, bringing simulations to the next level. Using rapidly advancing gaming technology, this research paper explores the applicability of creating multiplayer serious games for manufacturing simulation. First, we create and validate a hands-on activity that engages groups of students in the design and assembly of toy cars. Then, a corresponding multiplayer VR game is developed, which allows for the collaboration of multiple VR users in the same virtual environment. With a VR headset and proper infrastructure, a user can participate in a simulation game from any location. This paper explores whether multiplayer VR simulations could be used as an alternative to physical simulations. 
    more » « less