The building industry has a major impact on the US economy and accounts for $1 trillion in annual spending [1] and 9 million jobs [2]. Despite its massive impact, the industry has been criticized for poor productivity compared with other industries and also billions of dollars in annual waste due to the lack of interoperability [3]. Furthermore, the industry has been approaching a "labor cliff," meaning there are not enough new individuals entering the industry to offset the vacancies left by an aging, retiring workforce [4]. To remain effective, this critical industry will need to do more with less.
In order to encourage and prepare students for a career in this industry, educators have often aimed to replicate real-world project processes through physical design/build educational activities in events like the Department of Energy's (DOE) Solar Decathlon, Sacramento Municipal Utility District's (SMUD) Tiny House Competition, and DOE's Challenge Home Competition, among others. These experiences help situate learning concepts within a real-world context and provide an authentic environment for learning. Unfortunately, not all universities have the financial resources necessary to fund these types of hands-on projects. Thankfully, technology-like mixed reality-has the potential to help mitigate this inequity. To prepare students with the skills to shift the building industry to do more with less, educators need a better way to prepare more students with fewer resources.
The larger project surrounding the research described in this paper takes a critical step toward this ambitious challenge and explores a fundamental research question: To what extent can mixed reality (MR) technology enable engineering educators to simulate physical design and construction activities? Doing so will enable engineering educators to create low-cost experiences that enable students at all institutions to gain exposure to authentic, hands-on learning experiences. This paper will provide highlights of the most recent progress in the second year of the three-year study. It will also present preliminary findings associated with the assessment that will be used in the final year of the study.
To address the overarching research question, this study uses an iterative development approach according to two guiding frameworks: Human Centered Design (HCD) principles [5] and Shulman's Three Apprenticeships Model (i.e. learning related to "Head", "Hand", and "Heart") [6], [7]. In short, HCD principles facilitate an approach for designing computing tools for human users whose needs may not be fully understood by the developers. On the other hand, the Three Apprenticeships model is a theoretical framework for designing learning experiences that integrate three different types of competencies, including the head (knowledge), hand (skills), and heart (values and attitudes) [8]. Each of these three competencies will be integrated into the study at various stages.
The research involves several phases of work aimed at specifically exploring MR-enabled learning experiences related to the: Heart (Year 1); Head and Hand (Year 2); and Head, Heart, and Hand (Year 3). This paper will present highlights from Year 2, focusing specifically on the development of an instrument designed to measure the extent to which engineers value the heart-related constructs presented in the Three Apprenticeships framework.
The research team achieved several objectives during Year 2. The primary milestone was the creation of a mixed reality prototype where participants are asked to construct a wood frame wall using HoloLens technology and a physical tape measure. The experience is designed to test the 'head' and 'hand' skills of participants, requiring them to demonstrate their knowledge in understanding the construction plans and assembling the wall. Participants are tasked with measuring components, orienting them, and installing them in the correct location. Fig. 1 presents a sample of the plans provided to each participant, and Fig 2 . shows a section of the wall during construction. Year 3 will focus on the development of MR experiences for learning related Head, Heart, and Hand apprenticeships.
Based on the findings from years 1 and 2, the team intends to challenge students to complete a MR design and construction activity and consider attributes related to all three apprenticeships. The objective is to have students assess design and construction considerations for a space intended for small children. They will be asked to consider both intellectual and practical considerations required for this space. In short, this task invokes an emphasis on the heartdimension of the framework by requiring them to design a space that meets the requirements of a set of users whose needs are very different from their own.
The objective of Year 3 requires the research team to develop and pilot test an instrument with the ability to assess the heart component of the Three Apprenticeships framework. While there are many assessments focused on the cognitive (head) and practical (hand) skills associated with engaging in the engineering design process, there is an absence of instruments focused on the role of affective dimensions (heart) of engineering design. The instrument developed in this study will measure the engineering students' intentions to infuse heart-related constructs during the engineering design process. More specifically, it will measure the value engineering students place on heart-related constructs throughout the design experience. The constructs of highest interest in this study include empathy, safety, and humanitarian considerations when making decisions surrounding engineered solutions. These types of professional, human-centered skills are often underemphasized in the engineering curriculum [9], creating a need for new metrics for their assessment. The instrument will be developed using Messick's validation theory as a guiding lens, which focuses on collecting evidence for six different types of validity, including content, substantive, structural, generalizability, external, and consequential validity [10]. Together, these types of evidence provide a complete picture of how validity is assessed for the construct, making it an appropriate framework for this instrument. It must be noted, however, that the pilot study discussed in this paper is focused on evidence of content and substantive validity. The following sections summarize the pilot study results, including an item analysis of the survey responses.
The instrument created in this study was developed based on Messick's theory of instrument development, using content experts, existing literature, and the research team to develop the items. Twenty-two (22) Likert-style questions were created with a 4-point scale including response choices ranging from "Strongly Agree" to "Strongly Disagree." The instrument was pilot tested via Qualtrics in several sections of a first-year engineering course at a large university on the east coast of the U.S. The course is required for all engineering students and has a typical class size of roughly 70 students. Throughout the semester, students work in small groups on an open-ended design project. This course was chosen due to its ability to expose students to the engineering design process and provide them with critical opportunities to demonstrate their understanding of heart-related constructs during engineering design. Though the students in the pilot study had not yet chosen their major, this instrument will be primarily focused on civil engineering students as further field tests are conducted. Participant data was exported from Qualtrics into Excel and converted as follows: Strongly Disagree = 1, Disagree = 2, Agree = 3, Strongly Agree = 4. Though "Not Applicable" was also an option on the instrument, these responses were not considered. The Excel file was then imported into JMetrik, an open-source program designed for psychometric analysis.
The results of the pilot test are presented in Table 1. The table lists the difficulty, standard deviation, and discrimination values for each item in the instrument. On typical (non-Likert scale) instruments, the difficulty indicates how challenging the item is. On instruments that use Likert scales (as is the case in this study), the difficulty value simply indicates the mean response value. Discrimination, on the other hand, indicates the strength of the relationship between the item and the instrument as a whole. Said differently, it indicates how effective the item is at differentiating between respondents who value heart-related constructs and those who do not. In this instrument, difficulty values range from 1.0 to 4.0 and discrimination values range from -1.0 to 1.0. For most normreferenced instruments, ideal results include moderate difficulty values and high discrimination values [11]. As such, satisfactory results for this instrument are indicated by difficulty values around 2.5 and positive discrimination values near 1.0.
Results indicate that the items have good discrimination values, but high difficulties. All discrimination values are positive and above 0.5, indicating that they adequately discriminate between those who place a high value on heartrelated constructs and those who do not. The difficulties, however, have an average over 3.0, even though values around 2.5 are ideal. Despite this, the difficulty values are not high enough to eliminate any items (values between 3.5-4.0 would be cause for concern), so the results of the item analysis were deemed satisfactory. In addition, the instrument received a reliability value (Cronbach's alpha) of 0.9703, which is very high and indicative of excellent reliability.
The results of this pilot study indicate that the instrument satisfactorily measures the value that engineering students place on heart-related constructs during the engineering design process. Given that all difficulty and discrimination values were acceptable, no items will be edited or deleted. It must be noted, however, that these results only represent a small sample and thus are limited in terms of generalizability. As such, next steps for this instrument include a larger pilot study which may include additional items that were not tested during this pilot study. Afterwards, a large field test will be conducted to further validate and refine the instrument.
This project is designed to understand the extent to which mixed reality environments can replicate physical design and construction learning environments, specifically focusing on their ability to allow students to demonstrate their knowledge and judgement. This work-in-progress paper has outlined the major accomplishments of Year 2, providing an overview of the creation of a mixed reality prototype designed to help students demonstrate their 'head' and 'hand' knowledge through the construction of a virtual wood frame wall. It also provided a detailed discussion on the development and pilot study of a 'heart-centered' instrument that measures the extent to which students value heart-centered constructs during the engineering design process. This instrument directly addresses a gap in current literature, drawing attention to professional, human-centered skills in a field that is often criticized for placing too much emphasis on technical knowledge and skills [9]. This instrument will begin to fill that gap and will provide a quantifiable method for assessing the heart component of the Three Apprenticeships Model. Moving forward, the instrument will be pilot tested again before moving to a larger-scale field test. This instrument will also be used in Year 3 of this project assess the heartrelated constructs play in the final MR design and construction activity. The research team looks forward to the final year of the project and hopes to contribute to a better understanding of how mixed reality experiences can be used to facilitate authentic learning experiences for engineering students.