VR training simulations offer a safe and cost-effective way to teach complex skills in an embodied manner, which enhances learning compared to traditional click-through methods [11,28]. However, designing effective VR training simulations presents several challenges. The content must engage learners while preventing them from experiencing excessive confusion or getting stuck [23]. VR environments are often less familiar for users than traditional interfaces, making basic tasks like navigation and object interaction difficult. This can lead to errors in encoding input, decoding output, and discovering mechanics [3]. There is also increasing interest in self-directed VR training systems capable of teaching complex skills without human instructors [27]. This requires clear and reliable instructions describing tasks that users can consistently perform in order to avoid getting stuck [7]. The embodied nature of VR also introduces accessibility challenges, such as objects being placed out of reach for shorter users or text being too small for some to read. These complexities necessitate usability testing and iterative design to ensure VR training simulations are both effective and accessible.
Improving the usability of VR experiences can involve watching playtests or videos of playtests, which is labor-intensive. Participants asked to review videos of user playtests in MacCormick et al. [22] found the process to be "tedious." Game User Research (GUR) methods have introduced tools for reviewing data collected during gameplay. These tools commonly log and visualize events, the position of players or objects, player gaze, or other biometrics. These multiple data streams can be difficult for researchers to interpret [10].
To address these challenges, we developed an algorithm and associated visualization tool called DEPTH (Design Evaluation using Position Tracking and Highlighting). DEPTH uses only the position of a player's head and hands in 3D space, and the amount of time it took the user to complete each step in a procedure-based training simulation. DEPTH aggregates position data for each tracked object into voxels, or "volume elements", which are the three-dimensional counterpart to pixels [33]. Each point of position data is associated with the amount of time required by the player who generated that data point to complete that step. For each voxel, we determine the average of all completion times from all players associated with the voxel. We call this value a DEPTH score.
We also developed a tool for quickly and intuitively visualizing DEPTH scores. It integrates with the Unity game engine, allowing designers to see DEPTH scores in the relevant context. Our visualization places a cube at the center of each voxel, the color of which represents the DEPTH score for that voxel. We highlight areas where users who took longer to complete the simulation spent more time. We assume that players who took more time to complete the simulation may have experienced more difficulty. Multiple players who took more time to complete the simulation with similar spatiotemporal data likely indicate a design flaw rather than user error [25]. Our algorithm visualizes these potential design issues, helping designers identify where users experienced unintended difficulty.
In sum, our contributions are:
• An algorithm that aggregates player position data in 3D space and associates it with task completion times.
• A Unity-integrated visualization tool that displays the algorithm's output in context.
• A case study that demonstrates the use of this algorithm and visualization to investigate design flaws in an industrial VR training scenario.
Traditional GUR methods such as playtesting with think-aloud protocols or user interviews are often labor-intensive and timeconsuming [12]. To address this, various analysis tools have been developed to streamline the process. Most of these tools are designed for non-VR games. While the types of data used in GUR vary, many previous GUR applications have used spatiotemporal data to allow researchers, game designers, or players to review how players moved through the environment during gameplay [9,16,34]. A common method of doing this is to record the position of the player and the actions they take over the course of gameplay, and allow GUR researchers to replay and analyze recorded 2D games as an animation or animated map [1,16,22,30]. Several tools represent player position over time by drawing movement traces over the actual map, or a simplified representation of the map [9,32]. These traces are generally represented as Catmull-Rom splines [6]. Other tools utilize 2D heatmaps to represent aggregate player positions over the course of the game [5,8,18].
Compared to games, much less work has been conducted on VR, or in XR more broadly, regarding spatiotemporal data analysis. Tools for visualizing spatiotemporal data in VR often use the same techniques as those used in GUR for 2D games. These include 3D traces [4,13,20,29,31], heat maps [4,29], or animated playback of user movement and actions [13,20,21,29]. For example, PLUME records player positions and enables playback in a standalone application [13]. PEARL, an application for the Microsoft HoloLens 2, records and replays user positions by placing 3D models of heads and hands in the appropriate positions [20]. AvatAR represents fullbody motion capture data through animated 3D models of generic avatars, also on the HoloLens 2 [29]. GAMR (Game Analytics -Mixed Reality) records the positions of both players and tagged objects during mixed reality gameplay and allows for playback in mixed reality [31]. FRVRIT (Full Room Virtual Reality Investigation Tool) is a background application that records a user's position data while they play a VR game and displays it as point clouds and voxel heatmaps [21]. These tools are mostly designed for exploring spatiotemporal data from one player at a time.
Analyzing data from individual players sequentially is inefficient, particularly when evaluating large datasets. Looking at spatiotemporal data from multiple users simultaneously is more efficient. However, including spatiotemporal data from multiple users in the same visualization can lead to occlusion issues, making it difficult to interpret.
Visualizations that represent player movement with trajectories become difficult to understand when the trajectories of several players are displayed simultaneously. For example, Kloiber et al. [17] used trajectories to visualize VR movement data and found that "the current prototype does not scale well enough for a larger number of participants," to a degree which made it impractical for some intended use cases in data analysis for psychology studies. Kepplinger et al. [15] also found that traces of player movement in a short game were difficult to interpret. They reduced the complexity of their visualization by drawing a simplified trajectory through centroids, rather than visualizing every individual path.
Heatmaps are a common technique for aggregating spatiotemporal data which visually highlights where activity is concentrated. This information is made easier to see by compressing 3D data into a 2D plane [13,20,21,31]. In VR applications, collapsing 3D position data to a 2D projection discards important spatial information from 3D environments. One notable exception to this is AvatAR, which generates heatmaps that only show where players' feet make contact with the ground, which is fundamentally 2D data [29]. FRVRIT is unique in its use of a voxel heatmap to display values within 3D space, rather than collapsing this information in a 2D visualization [21]. Values for each voxel were represented with a cube which was the size of the voxel itself. The resulting occlusion was addressed by making the voxels transparent. The usability of this visualization, and the extent to which occlusion made it difficult to interpret, was not evaluated by a usability study. FRVRIT also allowed the user to see all recorded positions of the player over time as a pointcloud, which, while similar to a voxel heatmap, suffers from significant visual occlusion due to the density of points. Like the aforementioned examples, DEPTH can display spatiotemporal data from many users at once. Like FRVRIT, it displays 3D data using a 3D, voxel-based heatmap, rather than collapsing information into a 2D plane. Unlike FRVRIT, voxels are represented by small cubes at the center of each voxel, rather than as cubes which are the same size as the space they represent, thereby reducing occlusion through visible gaps between cubes. Some researchers have explored other algorithmic methods for simplifying and visualizing large spatiotemporal datasets. For example, Afonso et al. [1] displayed animated maps of multiple League of Legends games on the same map by using DBSCAN algorithm for data clustering. Aung et al. [2] used the DEDICOM decomposition model and k-means clustering to simplify spatiotemporal data of more than 5000 players of Just Cause 2 creating interpretable graphs from a very large dataset. Like Afonso et al. [1] and Aung et al. [2], DEPTH uses an algorithm to aggregate player data and use it to describe properties of space. While straightforward, our algorithm, described in the next section, appears to be unique in its approach. We did not find any instances of tools visualizing an algorithm that displays average completion times for all data within a voxel.
Highlighting) is designed for procedure-based training simulations, or simulations with discrete tasks that happen in a specific order [19]. In these simulations, time is commonly used as a proxy for difficulty [39]. DEPTH identifies areas associated with slow completion times, which suggest potential design flaws.
We record the position of the player's head and hands, along with the time taken to complete each step. Unlike eye tracking or other biometrics, this simple data can be tracked by most head-mounted displays (HMDs). We logged position data three times per second, though this frequency can be adjusted.
For each step of the training and each tracked object, all position data is aggregated into voxels. We used a voxel size of 10 cm because our case study used HTC Vive controllers, which are bulky and do not support fine motor movements. For use cases in which greater precision is required, such as those involving hand tracking, a smaller voxel size would be more appropriate. Each data point is associated with the amount of time in seconds that the player took to complete that step. The average completion time for all data points within a voxel is the DEPTH score for that voxel. The DEPTH score ď ğ for each voxel is calculated as:
where:
• ď ğ : DEPTH score for the voxel • Ĥ ğ : Number of data points in the voxel • Đ ğ Ġ : Completion time for the Ġ-th data point in the voxel This generates three matrices of DEPTH scores per step-one each for the head, left hand, and right hand. No DEPTH score is calculated for voxels for which there is no data.
To visualize DEPTH scores within the VR scene, we developed a Unity package that creates a voxel-based visualization. Small cubes are placed at the center of each voxel. The color of these cubes indicates the task completion time associated with each voxel. We expose a "threshold" variable which can be modified interactively using the UI shown in Figure 2, allowing designers to fine-tune the visualization to highlight relevant issues. Cubes with scores above the threshold are red, and cubes below the threshold are blue. We chose to use a threshold rather than a gradient to give users a single, adjustable parameter, aligning with DEPTH's focus on quick and intuitive design insights. By examining the concentration and distribution of red cubes, designers can pinpoint where users may be struggling, and direct their attention to these areas.
Using a small visual representation at the center of each voxel, rather than visual representations which are the same size as the voxel, as is done in MacCormick et al [21], reduces occlusion issues, as users can see around the cubes. Users can also rotate the scene in the Unity editor to adjust their perspective, making occlusion less of an issue than it is in the static, 2D images in this paper.
We demonstrated the utility of DEPTH by applying it to a VR training simulation (described in Nasri et al. [24]) for cold spray, an advanced manufacturing technique.
Cold spray, also called supersonic particle deposition, is a method for applying metals, metal alloys, and metal blends. Cold spray uses an electrically heated high-pressure carrier gas, like nitrogen or helium, to accelerate metal powders through a supersonic de Laval nozzle above a critical velocity for particle adhesion [26].
We created a training simulation by collaborating with experts in the process, and breaking the process into steps following Van Merriënboer and Kirschner's task decomposition model [36]. In the simulation, trainees engage with 3D models of the cold spray equipment and tools, such as various sizes of wrenches, to perform tasks. For each step, participants are presented with written instructions and asked to physically execute the task using the tools in a manner akin to the real-world process. Successful task completion is reinforced through animations demonstrating the correct execution. The training structure is designed to be highly embodied, requiring participants to physically interact with the VR environment by performing tasks such as turning bolts with wrenches. This approach was chosen to prevent passive engagement, which can occur in click-through training modules, and to leverage embodied learning, suggesting associating physical actions with tasks enhances memory retention.
The DEPTH algorithm was applied to data collected from a previous user study, in which 19 participants were asked to play through two of six total modules, in order to identify and visualize potential problem areas. We sequentially reviewed each of the three DEPTH maps (representing the left hand, right hand, and head) for each of the 33 steps of the training within the Unity editor. When we observed notable patterns in the data, we referred to the text of the instructions for the step and notes taken during the study to understand the potential cause. The following section outlines our key findings based on this analysis.
4.2.1 Incorrect Tools. During the study from which our data was obtained, participants interacted with the powder feeder-a metal cylinder with a lid that holds the metal powder used in the cold spray process-and a table of tools. One of the more straightforward uses of DEPTH was to identify instances where participants used the incorrect tool. For example, in step 3 of the training, participants were instructed to disconnect the gas hose, a small hose at the back of the powder feeder connecting it to the wall. The specific task instructions were: "Check the pressure gauge for zero pressure. Disconnect the gas hose. " To complete this step, participants needed to grab the gas hose with their hand and detach it from the powder feeder.
Figure 1 illustrates the VR scene with the DEPTH visualization for the right hand during this step. The threshold is set at 173 seconds. Red cubes represent voxels where the average completion time exceeded 173 seconds, while blue cubes indicate average times below this threshold. Voxels without cubes represent areas where no participant positioned their right hand during the step.
The red cubes near the small wrenches on the table suggest that participants who took longer to complete the step reached for these wrenches, even though no tool was needed for this task in the training simulation. This indicates a potential flaw in the design of the training, as it was unclear to participants that no tool was necessary. It might be beneficial to require players to use wrenches for this step, as this would be more realistic and true to the actual process of interacting with physical powder feeder (removing the real gas hose from the real powder feeder is done using wrenches), and some participants seemed to expect the use of a wrench. 4.2.2 Behavior Detection. Figure 3 illustrates the head positions of players during step 1 of the training. The threshold is set to 60 seconds. The lower part of the visualization contains more red cubes, indicating slower completion times, while the upper part is mostly blue, indicating faster task completion. This is a particularly visually clear instance of a phenomenon that we saw in several stepsplayers who took longer to complete steps seem to have crouched or bent over. This could be an expression of fatigue, or bending over to look more closely at the powder feeder when confused. While DEPTH does not clearly indicate the cause of this behavior, it can tell us that bending over or crouching seems to be associated with slow completion times. In future implementations, detecting this behavior could be a cue to offer a hint to the player, though doing so without inadvertently discriminating against shorter players would require careful consideration. 4, participants are asked to insert eight bolts into the top of the powder feeder. In Figure 4a, there are red cubes near the lid of the powder feeder. As the threshold was set to 210 seconds, these are areas associated with completion times above three and a half minutes. In Figure 4b, making cubes associated with faster completion times transparent makes the red cubes on the lid of the powder feeder more visible. In Figure 4c, we similarly focus on just the faster completion times. The fact that the area around the lid of the powder feeder is associated with completion times slower than 210 seconds, rather than faster than 210 seconds, suggests that participants understood where the bolts should go, but struggled to screw them into place with the virtual wrench. Our design goal is to teach participants where to put the bolts, not how to use a wrench. It may therefore be advisable to adjust the balance between realism and training duration by simplifying the wrench interaction.
Our case study demonstrates how DEPTH identifies potential design flaws in VR training simulations by highlighting areas where users who struggled spent more time. Unlike traditional visualizations such as density heatmaps or individual time-based heatmaps, DEPTH provides unique insights by aggregating spatiotemporal data from multiple players into 3D voxels and associating them with task completion times. This approach highlights correlations between user behavior and task performance, helping designers identify problem areas that might otherwise go unnoticed.
Density heatmaps indicate the amount of data in each voxel, while time-based heatmaps show how long an individual player spent in specific areas. In contrast, DEPTH visualizations display the average task completion time for all data points within a voxel (as described in Section 3). For instance, in Step 1 of our case study (Figure 3), a density heatmap could show that players crouched in a specific location, but only DEPTH reveals that this behavior
(a) (b) (c) correlates with slower task completion times. By summarizing differences between fast and slow-performing players, DEPTH maps highlight areas where design improvements may be needed.
DEPTH is a simple and highly generalizable approach. While our case study is a VR industrial training, our approach can be extended to a wide range of applications. It requires spatiotemporal data (the position of any object over time), and the time it took for a cohort of players to complete steps, and displays the relationship between these values. DEPTH can therefore be generalized to any VR application that tracks the position of the user or other relevant objects and in which there are steps that players complete in a measurable amount of time. Given that many VR industrial training applications focus on procedural skills, DEPTH's approach should be applicable across many VR training simulations [14,35,37,38]. We would expect DEPTH to also be applicable to other VR applications, beyond the domain of training simulations. For example, in VR games without fine-grained sequential tasks, DEPTH could be adapted to correlate player positions with the time taken to complete levels, or specific in-game tasks or challenges. We would expect certain parameters of how DEPTH is used, such as the frequency of data sampling or size of voxel used, to from tuning for the specific task. For example, tasks focused on fine motor skills should use smaller voxel sizes.
DEPTH operates on the assumption that (1) more difficult tasks take longer to complete on average and (2) completion times may be indicative of a design flaw. While DEPTH can show where players who were slow to finish a task spent their time, it cannot explain why they took a long time to finish the task. While players may have taken a long time to complete the task because they experienced difficulty, they may have been taking their time to appreciate the experience, or distracted by something external to the application. Instances of slow completion times caused by players experiencing difficulty also do not always indicate a design flaw. Players may be experiencing difficulty which is in-loop (difficulty that designers intended for players to experience), or may be experiencing difficulty which is out of loop (does not support the design intent of the experience) [3]. DEPTH maps can show where players who took a long time were, helping designers to determine whether users are getting stuck in the intended areas or if there are unintended problem spots. The interpretation of DEPTH maps and subsequent design decisions benefit from integration with other metrics and the expertise of a skilled designer familiar with the application. Rather than replacing existing tools, DEPTH complements a designer's toolkit by providing a temporal perspective that integrates with other forms of analysis. Future work could explore how different designers interpret DEPTH visualizations and use them to make actionable changes, validating its utility across diverse design contexts.
This paper presents DEPTH, an algorithm and visualization tool designed to evaluate VR training simulations by highlighting areas where players may struggle based on position tracking data and task completion times. By aggregating player movements into 3D voxels and associating them with completion times, DEPTH provides different insights than existing methods. We demonstrate how DEPTH can be used by evaluating a VR industrial training simulation. DEPTH's flexibility and generalizability make it applicable beyond VR training, offering potential benefits for other VR applications.