In recent years, it has increasingly been recognized that spatial visualization skills are important in supporting student success in Science, Technology, Engineering, and Math (STEM) education and retention of these students in STEM careers ( [1]- [5]). Many first-year college engineering programs and high schools with pre-engineering curriculum have incorporated spatial visualization training into their curriculum and have seen large learning gains in spatial visualization skills ([6]- [11]).
Several researchers have attempted to provide a clear definition of the underlying factors of spatial ability such as Spatial Visualization, Spatial Orientation, and Speeded Rotations [12]. Other studies have broken down speeded rotations into sub-topics such as Closure Speed, Flexibility of Closure and Perceptual Speed [13]. Tests have been developed to ascertain spatial reasoning skills. Lohman [12] said that the tests that define spatial visualization are complex. Some require rotation, reflection, or folding of complex figures, while others require combining different figures or multiple transformations. Yilmaz, who conducted a literature review of spatial reasoning issues, summarized different ways of measuring spatial ability and said that some spatial visualization tests have been shown to be more advantageous for males, as well as some biological and socio-economic factors [14].
At the high school and college level, the Purdue Spatial Visualization Test: Rotations (PSVT:R), which is recognized as a gold standard assessment tool, is used to measure students' learning gains in spatial skills [15]. The PSVT:R is a 20-minute timed test consisting of 30 multiplechoice three-dimensional rotations problems.
There is no reason why spatial visualization training could not occur at a much younger age, like elementary school. Cognitive psychologists have indicated that children's spatial ability does not reach an adult level before age twelve [16] with varying levels of abilities ranging from infants being able to track the direction of a moving item and its distance from them, to being able to understand simple routes, to the ability to use maps by age nine or ten [17]. While the PSVT:R test has been well validated, the tools used to teach and assess elementary aged children need to be different than those of fully developed adults. The debate on the nature and types of the spatial ability still continues, as well as the development of tests to adequately assess spatial reasoning skills.
Researchers at the University of California, San Diego developed an assembly pre-and post-test based upon a timed Lego™ exercise. Students are timed to see how long it would take them to build small Lego™ shapes using only a picture of the final assembly, but no step-by-step instructions. The assessment is given once at the beginning of the term and then again at the end of the term using a different Lego™ shape. While the PSVT:R test has been well validated, there are benefits to developing alternative methods of assessing spatial visualization skills:
Lego™ Assembly can be used in elementary school where the 20-minute timed PSVT:R is too challenging, not as engaging, and therefore inappropriate. The Lego™ Assembly test relates to applying a broad set of spatial visualization skills rather than just answering multiple choice questions related to rotations only. The PSVT:R results are capped at 30, which can skew analytical results. However, the Lego™ Assembly time is not limited (except for the speed of one's hands).
Another benefit of this new assessment is that it is more engaging and can illustrate to students a skill that could be perceived as more relevant to their future careers, thus possibly increasing their motivation for spatial visualization training. This paper describes the development of the alternative spatial visualization assessment tool using Lego™ assemblies, its correlation to the PSVT:R assessment after the Spatial Vis™ mobile sketching app was used for training in several large college classrooms, and its suitability as a spatial visualization assessment for elementary school aged students.
The motive for developing the Lego™ Assembly test is to identify an assessment tool that could apply to younger grades. Rather than just focusing on rotations about different axes, which is the emphasis of the PSVT:R, the new assessment will evaluate students' ability to take simple shapes and pieces (individual Lego™ bricks) and create a more complex assembly, thereby emphasizing a number of spatial reasoning skills.
In order to validate the new instrument, the Lego™ Assembly test was implemented in a pilot study in a college freshman engineering graphics course in spring 2018 using students who could do both the PSVT:R and the Lego™ Assembly assessments. At the beginning of the course, all students took the PSVT:R test. Half of the students then performed a Lego™ Assembly of one shape while the other half did the assembly test with another shape. The two initial shapes chosen for this pilot, a whale and a lion, are shown in Figure 1. These parts were taken from the Lego™ Classic Blue Creativity Box 10706 and Classic Orange Creativity Box 10709, respectively. During the term, the students completed spatial visualization training which taught them how to sketch orthographic and isometric assignments using the Spatial Vis™ mobile sketching app (https://egrove.education). The app provided automatic grading and hint feedback to help students when they were stuck. At the end of the course, the PSVT:R test was repeated for all students. The Lego™ Assembly test was also repeated, but the students switched which shape they were building. This approach allowed the difficulty of the assembly tasks to be normalized based upon the average time it took to build the shapes in the pre-test. Also, by switching shapes, the chance of students remembering the specific assembly steps from the pre-test, which would skew results of the post-test, could be eliminated.
Students were informed that they would be given the Lego™ assessment at the start and end of the term and were told to do their best but that the time would not impact their grades. Students were given a manila envelope with the Lego™ pieces in a plastic bag and one image showing the 3D isometric of the completed shape (refer to Figure 1). The students first checked that all the Lego™ pieces were present. Then they timed themselves on how long it took them to properly build the given shape. The teaching assistants checked the final assembly, and for the few cases that were incorrect, the students were told to restart their timers and continue working on the task. Appendix A shows the instructions provided to the students for the whale assembly. A similar set of instructions were provided for the Lion kit.
There was some concern that college students would consider a Lego™ test as "beneath" them. However, the college students where highly engaged in the test, leading to the belief that elementary school students would be even more engaged. Initial analyses of the assembly times indicate that the difficulty of the two Lego™ shapes were significantly different. The average time to build the Lion was 7.3 minutes compared to 3.55 minutes for the whale. This represents a 35% difference between each shape's average time and the overall mean of 5.4 minutes. One issue with the Lion shape was that the isometric view did not clearly show the tail, which caused some students a fair amount of trial and error. Therefore, for comparison purposes, the completion times on the Lego™ Assembly tests were normalized. These normalized times were correlated against the PSVT:R scores in order to see how the new assessment compared to the "gold standard". The PSVT:R scores are capped at 100% and it can be seen that a small of number of students achieved 100% in the pre-test, but a much larger number of students achieved 100% in the post-test. Despite the non-linear effect of the capping at 100%, the data showed a R-Squared correlation of 0.11 between the Lego™ Assembly times and the PSVT:R scores for the pre-test and 0.14 for the post-test (see Figure 2). Overall, the average Lego™ build times decreased significantly after using the Spatial Vis™ app, indicating an improvement in students' spatial reasoning skills. However, in an attempt to better validate the new assessment, it was decided to repeat the study with Lego™ shapes of more similar difficulty.
During two courses in spring 2019, the Lego™ Assembly assessment was repeated but the Lion assembly was replaced by an Airplane, which was also from the Lego™ Classic Orange Creativity Box 10709 (see Figure 1). This was to create more equivalence in difficulty and time with the Whale set. Instructions were the same, and the results were also compared against the PSVT:R tests. Results show that the selection of the Airplane shape may have been an overcompensation because the average build time for the Airplane was 2.63 min as opposed to the average build time for the Whale which was 4.58 min. Albeit, the difference between the two times dropped to 27% difference from the mean. The average assembly times are shown in Table 1. The Lego™ Assembly times were normalized for each shape based on the average pre-test "difficulty". Accordingly, all the pre-and post-Whale assembly times were divided by the average pre-test assembly time, and the same was done for the Airplane shape. A comparison of the Lego™ test to the PSVT:R scores is shown in Figures 4A and4B. In both cases, the assembly time tends to decrease as the PSVT:R scores increase as anticipated. In can be seen in Figure 4B that for the post-test, the PSVT:R scores are compressed at the high end since the perfect score of 30 is the maximum that can be achieved regardless of the spatial visualization ability. However, there is a more continual decrease in Lego™ assembly times, which allows for a measurement of spatial visualization ability above the 30-point threshold in the PSVT:R test.
There are statistically significant correlations between the assembly times and the PSVT:R scores.
For the pre-test, the data showed a correlation coefficient of R = -0.31 with a P = 0.0031. For the post-test, the correlation was R = -0.25 with a P = 0.0176. The correlation coefficients are negative since increased PSVT:R scores corresponds to reduced assembly times. In both cases, the P value is less than 0.05, which is a typical threshold for statistically significant correlations. The higher level of correlation and significance for the pre-test can be due to the fact that the PSVT:R scores are capped at 30, which has a higher impact on post-tests.
4A: Pre-Tests 4B: Post-Tests Figure 4: Normalized Lego Assembly Times vs PSVT:R Scores Another comparison between each assembly test and the PSVT:R test is shown in Figure 5, which compares the change between the pre-and post-tests. A larger improvement in the PSVT:R score resulted in an increased reduction in assembly times. This trend indicates that the more students increased their spatial visualization ability, the more their assembly times decreased.
A Lego™ based assessment tool was developed that would be suitable for elementary school students. To validate this test, a trial was conducted at University of California, San Diego with college students who could complete both the PSVT:R and Lego™ tests. The first time the Lego™ assessment was implemented, the Lion Lego™ assembly took on average substantially more time than the Whale to build. The assembly times were normalized to account for differences in shape difficulty. However, in order to re-validate the assessment tool, the Lego™ test was repeated in another trial by replacing the Lion shape with an Airplane shape. This change ended up overcompensating the results with the Airplane now taking on average substantially less time than the Whale. Nevertheless, the original and repeat trials yielded similar results. The repeat trial showed a statistically significant correlation between the Lego™ based assessment test and the PSVT:R (p values 0.0031 on the pre-test and 0.0176 on the post-test). Overall, students with initially weaker spatial visualization skills took longer to build their Lego™ models. However, spatial visualization training using the Spatial Vis™ app led to improvements in their post-test Lego™ building times.
While the PSVT:R was designed for older students, the Lego™ assessment provides a new metric for evaluating improvement in spatial visualization that is suitable for use in elementary schools. Furthermore, the Lego™ Assembly test applies to a broader set of spatial visualization skills rather than just rotations, which is the focus of the PSVT:R assessment. From a quantitative perspective, the PSVT:R cannot measure very high visualization skills because the multiple-choice test is capped at 100%. However, the assembly test can see improvements in assembly time even for students at the higher spatial visualization levels. Perhaps the most important aspect of the Lego™ assessment for elementary school implementation is that it is likely to be much more engaging for students than a multiple-choice test.
Future work includes identifying two Lego™ Assembly shapes that are much closer in average assembly times. Another area for future study may involve using the same Lego™ shape in the pre-and post-tests to see if there is a memorization effect of how to build a specific shape that is retained over a period of a few weeks. Furthermore, we are interested in evaluating whether students' prior familiarization with Legos affects performance on the Lego™ Assembly assessments.