INTRODUCTION

Virtual Reality (VR) is a transformative technology which has fundamentally changed how we interact with digital spaces. By utilizing specialized hardware such as VR headsets and input controllers, users can immerse themselves in three-dimensional, computergenerated realistic virtual environments. This level of immersion has propelled VR from a specialized niche to a mainstream platform for a range of applications including gaming [16], social interaction [10], remote conferencing [9] and virtual tourism [2]. As VR technologies continue to mature, their adoption rates have skyrocketed. As of February 2023, over 171 million people worldwide, with 65.9 million in the U.S. alone, were using VR applications [1].

While the VR technology enables exciting new applications, it also raises pressing security and privacy concerns. These concerns are not merely hypothetical and could be a significant barrier to VR technology adoption [23]. These issues are further exacerbated by the fact that VR platforms often interface with other smart devices, such as mobile phones and wearables, creating a complex ecosystem ripe for security and privacy vulnerabilities. Existing research efforts in the literature have exposed a variety of security/privacy vulnerabilities in VR platforms, ranging from motion sensor-based inference attacks [82] and eye-tracking exploits that harvest sensitive personal data [41] to exploring how gait and movement data from VR headsets can be used to create deepfake videos [78].

In this paper, we focus on an unexplored, yet highly relevant, privacy risk that arises when a user is simultaneously engaged with a VR platform and a non-VR mobile/wearable device equipped with motion sensors. Individually, an adversary's access to data from either system is usually considered non-threatening. However, within the broader ecosystem, is it possible that a user's privacy may suddenly become vulnerable if an adversary gains access to both data streams? Our work attempts to answer this question by studying if a combination of these disparate data sources can be potentially used to de-anonymize users and compromise their privacy within the VR application ecosystem. Despite recent research efforts focusing on uncovering and overcoming privacy challenges in VR applications, the potential threat (and associated risks) of correlating real-world motion sensor data with in-VR visual data has been largely overlooked. This paper aims to address this specific, yet critical, gap.

This work is the first to systematically investigate the user anonymity and privacy implications stemming from a combination of data from VR and non-VR (mobile/wearable) platforms, with the following specific contributions: • Correlation Framework: We develop a framework for correlating motion sensor data from users' mobile/wearable devices with the visual movement of their virtual avatars in VR applications. • Empirical evaluation: To validate the performance and efficacy of our correlation framework, we collect test data from human subject participants and perform a comprehensive empirical evaluation under various practical settings. • Optimization: We propose improvements and optimizations to our correlation framework, tailored for large-scale attacks. These enhancements provide a more efficient and scalable solution to the identified user-privacy issue. • Mitigation strategies: We discuss potential mitigation policies and recommendations that provide actionable insights for both developers and policy-makers.

RELATED WORK

We categorize related research efforts into those that attempt to infer private information from mobile device motion sensors and those that employ VR systems and applications.

Information leakage from mobile device motion sensors: Mobile and wearable device motion sensors such as accelerometers and gyroscopes have been heavily scrutinized in the research literature for their potential to be employed as a side-channel for leaking users' private information. For instance, motion sensor data have been utilized to infer keystrokes and passwords [27,46,47,64], identify lock screen patterns [85], deduce travel routes and location [36,58,60], infer speeches [35,37,52], infer handwritten text [79], reconstruct 3D models from printer vibrations [67], and estimate demographic information [31,66]. Application of such on-body motion sensors for user authentication [44,80,84] has also received significant attention. However, such biometric authentication systems require training data from individual users. Information leakage in VR systems and applications: Albeit relatively new as a consumer technology, VR has garnered a host of security and privacy concerns. Attacks such as password inference from finger movements (using motion sensors) when typing a password in the virtual world can become a security problem if the same password is reused by the user in real world [30]. Some VR headsets include eye-tracking, which can reveal valuable personal information [41]. VR, when used in conjunction with Deepfakes [78], can also become a serious threat as an adversary can potentially utilize personal gait and movement data collected from a VR headset to create a very authentic-looking fake video. These type of attacks can be used to damage personal reputation [12], conduct social engineering attacks [81], and spread misinformation [34,40]. Authentication using authorized sensors on the VR headset or paired on-body controllers [68] and synchronization among multiple (on-body) VR sensors [28,39] for utility focused applications are another closely related research topics. Unlike these prior research efforts, in our attack we focus on out-of-band motion sensor data, which are not natively paired with the VR system. Deanonymization solely using movements observed in the virtual world is difficult, especially when the confusion set size is large. In this work, we attempt to de-anonymize VR users by correlating visually observed movements of their virtual world avatars with available out-of-band motion data.

One of the most desirable features in a VR experience is the ability to use anonymous avatars and identity transformation. Without appropriate identity protection VR users can be hesitant to participate in the ecosystem [23]. Previous works on de-anonymization of VR users utilized in-band data, such as sensors on the VR systems and/or movement characteristics of virtual avatars, to infer users' identity [53,54], anthropometrics [59], environment [59], device information [59,72], and demographics [59]. To the best of our knowledge, the proposed de-anonymization attack using out-of-band motion data has never been studied before.

SYSTEM AND ADVERSARY MODEL

The target in our proposed attacks are users participating in a VR application. For this, users employ a primary VR device/hardware (from manufacturers such as Microsoft, Meta, Apple and HTC), which typically comprises of a headset running a manufacturerprovided platform or OS. Users are able to execute a variety of manufacturer-provided or third-party VR applications [7, 9-11, 18, 19, 21] on the headset. Users may optionally also employ additional manufacturer-provided or third-party hardware such as hand controllers and headphones/earphones. In addition to the application that the user is directly interacting with, the VR headset may also be running other support applications (in parallel and/or in the background), for example, for live streaming the VR experience.

We refer to all these apps running on the VR headset as VR-apps. VR-apps often generate visual/video streams comprising of the users' virtual embodiment (in the form of an avatar) interacting with other virtual users (or avatars) or the virtual environment. Depending on the VR-app, these visual or video stream data may be publicly available to all users of the VR-app or to everyone (in case it is live streamed on a public platform).

While interacting with VR-apps, users may also have in their possession other smart devices such as smartphones and smartwatches. There may be applications running on these smart devices, typically distinct from the VR-apps (e.g., operated by different manufacturers or providers), and are referred by us as non-VR-apps. These non-VR-apps may be able to access local sensors on these devices (smartphones and smartwatches) either with or without the explicit permissions of the user. For instance, accessing camera and microphone may require explicit user permissions, while accessing motion sensors such as accelerometers and gyroscopes may not.

We consider an adversary whose goal is to de-anonymize a target VR user (or users) by correlating the visual movements of his/her anonymous virtual world avatar from the publicly-accessible visual/video stream data generated by the VR-app with out-of-band, but identifiable, mobile/wearable motion sensor data from a set of potential target users collected through non-VR-apps. The size of the labeled motion dataset of users (collected from non-VR-apps) in the possession of the adversary, representing the confusion set of the target VR user or avatar, may vary between a large-scale where the cardinality (of the dataset) may be very high, to a small-scale. Similarly, the video recordings of VR users or avatars will result

Correlate Avatars' Visual Movements to Users' Motion Sensor Data User A Wears VR Headset Anonymous Virtual Avatar Virtual World Adversary Adversary Observes a Set of Virtual Avatars and Composes a Visual Movement Dataset (V) Phone In Pocket Adversary's Malicious App Motion Sensor Data (Accelerometer + Gyroscope) Motion Sensors Dataset (M) User A Identified ? User J User K v i m j Figure 1: Threat model and Attack Summary in a visual movement dataset, which can also range between a large-scale where its cardinality may be very high, to a significantly smaller small-scale such as avatars present within a (targeted) virtual room or playing a (targeted) virtual game. The goal of the adversary, as depicted in Figure 1, is to de-anonymize a target user (i.e., its avatar) in the VR space by matching an element in the labeled motion dataset to the element (corresponding to the target user or avatar) in the visual movement dataset. This adversarial goal can also be extended to include de-anonymization of multiple VR users or avatars. In order to compile the visual movement dataset (denoted by 𝑉 = {𝑣 1 , 𝑣 2 , . . . , 𝑣 𝑝 }, with cardinality 𝑝), the adversary has to join the virtual world, observe and record each avatar for a baseline duration of time within which a series of movements are likely observed. In case of the VR service provider being the adversary, this process can scale easily. In order to compile the labeled motion dataset (denoted by 𝑀 = {𝑚 1 , 𝑚 2 , . . . , 𝑚 𝑞 }, with cardinality 𝑞), the adversary promiscuously records zero-permission motion (accelerometer and gyroscope) sensor data from a targeted set of users. Typically, this can be achieved by means of a malicious SDK and/or a trojan app that offers some utility to the users on the front-end (e.g., a game or a social networking service), while surreptitiously recording the motion data on the back-end. User may also be compelled by a higher authority, for example an employer, school, or government, to download such a (malicious) app onto their smartphone or smartwatch [3,4,13,17,24]. Several research efforts in the literature have studied/uncovered other significant privacy issues under a similar assumption [27, 45, 48-50, 55, 57, 63, 75-77, 87, 89]. We also assume that both datasets (𝑉 and 𝑀) contains timestamps which are fairly in sync with the standard global time.

CORRELATION FRAMEWORK

Our proposed correlation framework (Figure 2) is composed of two key components. The first component converts both the (out-ofband) motion sensor data and the visual movement data into a comparable format, referred as activity-vector series. The activityvector series enables us to directly compare and match elements from the two datasets (𝑉 and 𝑀) using a matching heuristic. The second component ranks the closest matches across the elements of both the datasets, such that the highly ranked matches are most likely associated with the target user (identifiable from 𝑀).

Activity-Vector Series

Our motivation behind defining an activity-vector series stems from the fact that the two data sources (𝑉 and 𝑀) are not directly comparable to each other. The motion sensor data 𝑀 comprises of samples measuring linear acceleration and orientation changes of a user's body, whereas the visual movement data 𝑉 consists of video frames recording an anonymous avatar's movements. Consequently, we define an activity-vector series as a sequence of activities observed (classified by a ML model), combined with a pairwise sequence of "magnitudes" for each observed activity from each of the data sources. The magnitude quantification (Section 4.5) associated with an observed activity is approximate, but serves as a critical attribute in our correlation framework.

More precisely, our activity-vector series is composed of the following commonly observed activities: idle, body rotation, head rotation, hand movements, walking, bending, jumping, and "other". These were the common movements observed in over 2000 hours of activity data collected by us inside VRChat [21] (more details in Section 5). These activity classifications combined with magnitude calculations form a vector-like representation where each observed activity has a corresponding magnitude information. An activityvector series from either data sources can be depicted as:

walking walking idle bending walking walking jumping idle walking jumping 𝑎 4 𝑎 3 𝑎 1 𝑎 7 𝑎 6 𝑎 10 𝑎 8 𝑎 2 𝑎 5 𝑎 9 Activity Magnitude Left-front Hip Pocket (Motion Sensor)

where 𝑎 𝑖 ∈ R + is the positive real magnitude of an activity time window, such that 𝑎 1 > 𝑎 2 > . . . > 𝑎 10 . In order to generate this activity-vector series, we next detail the steps taken to preprocess and utilize supervised machine learning models to classify the activities observed in individual sequences.

Pre-Processing

We first segment both the physical motion data (obtained from the mobile device motion sensors) and the visual movement data (obtained from the VR apps) into small time windows (of 𝑤 seconds each) and classify each window as one of the eight aforementioned actions. We empirically evaluate the effect of the size of 𝑤 on correlation accuracy in Section 6.1 and use the optimal value for rest of the evaluation. For the visual movement data, we further separate individual user's avatar from the background, so as to better classify the movements of the avatar without any background noise. Pad-dleSeg [29], an open-source toolkit that applies image segmentation using different techniques, was used to segment out the individual avatars. More specifically, we used a pre-trained ORCNet model with HRNet backbone that was trained using the Cityscapes dataset [15]. For the motion sensor data, we apply a Savitzky-Golay filter [61] to smooth the signals for noise reduction before classification.

Training Data Generation

To generalize and scale our activity classification for a large-scale attack, we generate a training dataset by adding synthetically generated variations that capture a wide range of bodily variances and anomalies (often caused by imperfections in the VR systems), otherwise imfeasible to collect from real human subjects, to a wellknown visual movement dataset in the literature. Specifically, we generate the training data of our visual movement classifier using the 3D game engine Unity [20] (Figure 3), utilizing the CMU MoCap [5] dataset and synthetically generated variations of motions captured in the CMU MoCap dataset. The CMU MoCap dataset was created using a motion capture system where the subjects wore 41 markers and performed various activities. It is a well-known dataset for evaluation of activity recognition frameworks [26,56,65], and can be applied to reproduce avatar movements inside Unity using corresponding body keypoints.

Our synthetically generated movement variations randomized the speed between 0.25× and 2× of CMU MoCap speeds, and rotation angle between -10°and +10°of CMU MoCap rotation angles. In addition to the CMU MoCap model avatar, we also train using another freely available avatar, namely the Futuristic soldier -Scifi character 1 . As the video movement data is dependent on the viewpoint of the adversary, we also capture varying camera positions around the virtual avatar in Unity (Figure 3). Specifically, the camera position was randomized around the avatar (across all angles for which the avatar is visible), enabling different visual perspectives and thus improving classifier training. The visual movements of avatars were recorded using OBS Studio [14].

Additionally, in Unity we attached a custom-made virtual motion sensor to the avatar (Figure 3), which is able to capture acceleration and orientation changes of the avatar. This virtual motion sensor closely captures the kinematic forces experienced by the avatar in the same way a smartphone or smartwatch motion sensor on a real person would experience, and it allows us to collectively train a classifier for the motion sensor data alongside the visual movement classifier. Such a strategy of using virtual sensors enables training data collection without requiring real human subject participants and also eliminates related synchronization errors. Even though our activity classifiers were trained with publicly-available avatar datasets (and synthetically generated variants of it), for the experimental evaluation and testing of our framework (from an adversarial standpoint) we compose a test dataset with the help of real/actual human subject participants and also address synchronization errors between the motion sensor and visual movement data (Section 6). Such a strategy is not only easily generalizable but also realistic, as in practice an adversary may be unable to train activity classifiers using the target users' data.

Activity Classification

We collectively utilize Apple's Core ML 2 and Create MLfoot_2 libraries to generate two classification models (each trained separately), one using the video movement training data and another using the motion sensor training data as described above. Core ML is a state-of-the-art model pre-trained by Apple for generic action and activity classification, and we further fine-tuned it with the help of transfer learning [51] using the training data described in Section 4.3. Prior research has already demonstrated the feasibility of such activity recognition using Core ML [42]. Moreover, Apple's Vision framework 4 is already pre-trained for keypoint detection on humans, which can also be utilized with Core ML on humanoid avatars. Applying these trained models on test visual movement and motion sensor data (as described in Section 5.4), split into 𝑤 second windows, will result in a sequence of activities observed on the two data sources, which is one of the two sequences in the activity-vector series described earlier.

Activity Magnitude

Intuitively, when the same classified activity is observed in both data sources (in a given time window), we can improve our identity correlation by ranking smaller magnitude differences above larger magnitude difference. For example, if an anonymous avatar is observed to be jumping fast in the virtual world (high magnitude), it is likely that their activity magnitude will also be high on the motion sensor data. As mentioned earlier, our magnitude quantification of an observed activity is approximate. For the motion sensor, we calculate magnitude of each 𝑤 second activity window as the average magnitude of acceleration vectors in the motion sensor data. For the visual movement data, we utilize optical flow to compute the average acceleration of areas on the avatar's body where the motion sensor may be attached. Optical flow estimates the motion of objects between consecutive frames in a video, caused by the relative movement between the object and camera [33,38].

However, as some activities tend to generate disproportionate levels of motion in various parts of the body, it may result in different magnitudes of movements for the same activity. Furthermore, as the adversary may not have knowledge of the motion sensor's positioning for each user's data, the visually observed magnitude of movement experienced by an avatar's different body keypoints is another attribute that should be factored in to improve our correlation model. We consider six usual body positions where the motion sensor is likely to be attached, such as a smartphone in pant pocket or a smartwatch on the wrist: left-front hip pocket, right-front hip pocket, left-back hip pocket, right-back hip pocket, left wrist, and right wrist. As a result, the activity-vector series calculated from the visual movement dataset will consists of six different magnitude sequences (for the same activity sequence) as follows:

walking idle bending walking walking jumping idle idle jumping 𝑎 4 𝑎 3 𝑎 1 𝑎 7 𝑎 6 -𝑎 8 𝑎 2 𝑎 5 𝑎 9 Activity Magnitude walking walking idle bending walking walking jumping idle idle jumping 𝑎 4 𝑎 3 𝑎 2 𝑎 7 𝑎 6 𝑎 10 𝑎 8 𝑎 1 𝑎 5 𝑎 9 Activity Magnitude walking walking idle bending walking walking jumping idle idle jumping 𝑎 4 𝑎 2 𝑎 3 𝑎 7 𝑎 6 𝑎 10 𝑎 8 𝑎 1 𝑎 5 𝑎 9 Activity Magnitude walking walking idle bending walking walking jumping idle idle jumping 𝑎 4 𝑎 3 𝑎 1 𝑎 7 𝑎 6 𝑎 9 𝑎 8 𝑎 2 𝑎 5 -Activity Magnitude walking walking idle bending walking walking jumping idle idle jumping 𝑎 3 𝑎 4 𝑎 1 𝑎 6 𝑎 7 𝑎 8 -𝑎 2 𝑎 5 𝑎 9 Activity Magnitude walking walking idle bending walking walking jumping idle idle jumping 𝑎 1 𝑎 3 𝑎 4 𝑎 6 𝑎 7 𝑎 9 𝑎 8 𝑎 2 𝑎 5 𝑎 10 Activity Magnitude Left-front Hip (Visual) Right-front Hip (Visual) Left-back Hip (Visual) Right-back Hip (Visual) Left Wrist (Visual) Right Wrist (Visual)

where "-" implies unobservable position for optical flow calculations, all 𝑎 𝑖 in red depict mismatched magnitude rank with the left-front hip pocket motion sensor activity-vector series shown in Section 4.1, and all green 𝑎 𝑖 imply matching magnitude rank. Moreover, there is an activity misclassification in this example at the ninth window, highlighted as 𝑖𝑑𝑙𝑒. All of these seven magnitude sequences (one from motion sensor data and six from visual movement data) are utilized in the correlation and identity ranking processes described next.

Correlation and Identity Ranking

The first intuitive assumption in our correlation framework is that the order of activities by a user (and his/her avatar) will be unique when observed for a long enough duration. Intuitively, this observation duration can be shorter in a small-scale attack where the confusion set is smaller. In a large-scale attack, the observation duration has to be longer because with a large confusion set the occurrence of more than one anonymous user conducting the same sequence of activities within a short observation duration is more probable, thus creating confusion between them. We use this assumption to filter out unlikely matches from our identity ranking calculations, using the activity sequences in the activity-vector series.

Our second intuitive assumption is that varying activity magnitudes caused by disproportional levels of motion in various parts of the body can be utilized to identify closely correlated visual movement and motion sensor sequences. Accordingly, we utilize magnitude correlation rankings to rank known identities (from dataset 𝑀) such that users with motion sensor magnitude sequence closely matching to a visual movement magnitude sequence (best of the six visual positions) are ranked closer to 1.

4.6.1 Activity-based Filtering. As the activity classification is not perfect, we cannot reliably use the sequence of activities for correlation. Instead, we use a high degree of mismatch between sequences of activities (across visual movement and motion sensor data) to filter out identities whose motion sensor data are objectively different from an anonymous avatar being observed. More specifically, we calculate the Hamming distance between the motion sensor activity sequence and the visual movement activity sequence (which is the same for all six activity-vector series generated from the visual movement data). Thereafter, we eliminate pairs with distance threshold > 𝑡 from further magnitude-based identity rankings. We empirically evaluate threshold 𝑡 in Section 6.1 as part of our framework parameter optimization. For example, between the pair of activity-vector series illustrated in Section 4.1 and Section 4.5, this Hamming distance is 1 (or 10%) due to the activity mismatch in the ninth time window. 4.6.2 Magnitude-based Ranking. After filtering, we are left with identities whose motion sensor activity sequences closely matched at least one of the six visual movement activity sequences. We utilize Spearman's rank correlation coefficient [88] to correlate and rank potential identities based on magnitude sequences, which is computed as follows:

where 𝑛 is the number of observations (of 𝑤 second windows) in the activity-vector series and 𝑑 𝑖 is the difference in the paired ranks of the two magnitudes (across the visual movement and motion sensor data sequences) at the 𝑖 𝑡ℎ time window. The higher the Spearman's coefficient, the more likely the two sequences correlate to each other, and thus the corresponding identity from 𝑀 would be ranked closer to 1 out of the 𝑞 (minus the identities that did not pass the activity-based filtering). As the adversary does not have positioning information of the motion sensor on the users' body, we compute Spearman's correlation coefficient for the six likely positioning of the motion sensors (Section 4.5), and consider only the maximum for identity ranking. Between the examples shown in Section 4.1 and Section 4.5, magnitude from the visual data sequence of the left-front hip will have the highest Spearman's correlation coefficient with the left-front hip pocket motion sensor magnitudes. When activity-based filtering threshold 𝑡 is set very low (i.e., only tolerance for very minor or no mismatches in the activity sequences), it is also possible that all identities are eliminated from this magnitude-based raking, thus resulting in no identity ranking. The entire correlation procedure is described in Algorithm 1.

Algorithm 1 Correlation Algorithm.

1: Input: 2: 𝑣𝑖𝑑𝑒𝑜[] ⊲ Video's activity-vectors series 3: 𝑚𝑜𝑡𝑖𝑜𝑛[] ⊲ Motion's activity-vectors series 4: 𝑡 ⊲ Filtering threshold 5: Output: 6: 𝑟𝑎𝑛𝑘𝑒𝑑[] ⊲ Ranked list of correlated motion-video indexes with maximum Spearman's RCC 7: procedure Correlate 8: 𝑐𝑜𝑟𝑟𝑒𝑙𝑎𝑡𝑒𝑑[] ⊲ Maps motion indexes to correlated video indexes 9: 𝑢𝑛𝑟𝑎𝑛𝑘𝑒𝑑[] ⊲ Unranked list of correlated motion-video indexes with maximum Spearman's RCC 10: for 𝑖 in range(𝑣𝑖𝑑𝑒𝑜.𝑠𝑖𝑧𝑒() -1) do 11: for 𝑗 in range(𝑚𝑜𝑡𝑖𝑜𝑛.𝑠𝑖𝑧𝑒() -1) do 12: if 𝐻𝑎𝑚𝑚𝑖𝑛𝑔𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒(𝑣𝑖𝑑𝑒𝑜[𝑖], 𝑚𝑜𝑡𝑖𝑜𝑛[𝑗]) < 𝑡 then 13: 𝑐𝑜𝑟𝑟𝑒𝑙𝑎𝑡𝑒𝑑[𝑖].𝑎𝑝𝑝𝑒𝑛𝑑(𝑗) 14: end if 15: end for 16: end for 17: for 𝑖 in range(𝑣𝑖𝑑𝑒𝑜.𝑠𝑖𝑧𝑒() -1) do 18: for 𝑗 in range(𝑐𝑜𝑟𝑟𝑒𝑙𝑎𝑡𝑒𝑑[𝑖].𝑠𝑖𝑧𝑒() -1) do 19: 𝑚 𝑖𝑑𝑥 = 𝑐𝑜𝑟𝑟𝑒𝑙𝑎𝑡𝑒𝑑[𝑖][𝑗] ⊲ Motion index 20: 𝑚𝑎𝑥𝑆𝑝𝑒𝑎𝑟𝑚𝑎𝑛 = 𝑚𝑎𝑥(𝑆𝑝𝑒𝑎𝑟𝑚𝑎𝑛(𝑣𝑖𝑑𝑒𝑜[𝑖], 𝑚𝑜𝑡𝑖𝑜𝑛[𝑚 𝑖𝑑𝑥 ])) 21: 𝑢𝑛𝑟𝑎𝑛𝑘𝑒𝑑[𝑖].𝑎𝑝𝑝𝑒𝑛𝑑({𝑚𝑎𝑥𝑆𝑝𝑒𝑎𝑟𝑚𝑎𝑛, 𝑚 𝑖𝑑𝑥 }) 22: end for 23: 𝑟𝑎𝑛𝑘𝑒𝑑[𝑖] = 𝑢𝑛𝑟𝑎𝑛𝑘𝑒𝑑[𝑖].𝑠𝑜𝑟𝑡 () ⊲ Sorted based on Spearman's RCC 24: end for 25: end procedure

EXPERIMENTAL SETUP

To evaluate our proposed correlation framework and training methodology, we collect test data (visual and motion sensor) from human subject participants using a real VR application. Here we outline details of our test data collection procedure.

Participants' Task

Our participants (details in Section 5.3) carry out a set of representative activities in a virtual reality app while carrying a smartphone and smartwatch on their body. The controlled activities include movement of the head, arms, palms, legs, and also actions that require combinations of them. These different actions were chosen to generate a wide variety of different movements within the limited time we had with the participants. During the uncontrolled activity phases, participants were free to interact with the VR app on their own volition, not limited by the aforementioned activities.

The average time each of our participants spent on the VR app to provide us data for our study was 1 hours and 8 minutes.

Adversarial Viewpoint

We continuously observe and record the participants' avatar in the virtual world by means of five different virtual camera positions, where each camera position represents a different adversarial viewpoint. Four of these positions are static and positioned at different corners of the virtual room, each of which represents the fixed (or static) position of an adversarial avatar observing the target participant from that position. The fifth camera is mobile, and represents the view of an adversarial avatar moving and navigating in the proximity of the (target) participant's avatar. We carried out our experiments in two different virtual worlds -one in a public world (called Black Cat) where other users' avatars may be present, and second in a private world (called Home) where access is restricted to a select group of users. We refer to these five adversarial viewpoints in these two worlds by means of a legend outlined in Table 1. In our evaluation (Section 6), we will also analyze the effect of combining these five viewpoints on the accuracy of activity classification (where the viewpoints are referred to as HCC and BCC for Home and Black Cat, respectively).

Participants

We recruited 64 participants for test data collection, however, due to various personal, technical, and medical challenges, only 35 of them completed the study and whose data is included in our evaluation. The participants' ages were between 18 and 48, with a median age of 19. Additional demographic and other details about our participants are listed in Table 2. Participants were appropriately compensated for their time and our study was approved by our institution's Institutional Review Board (IRB).

Data Collection Apparatus

VR Device and App. We utilize the Meta Quest 2 VR device 5 and the popular VRChat [21] app (installed on the Quest 2) for generating and collecting test data from the participants in our study. As of July 2022, VRChat had more than 200,000 daily active users and more than 7 million registered users [22]. Although other popular apps also have full-body avatars [8], the fundamental nature of data generation (and collection) does not significantly differ across a majority of the VR apps. Motion Sensors. Participants' body motion was captured at 20 𝑚𝑠 sampling interval on a smartwatch (TicWatch 2) worn by the participants on their wrist and on a smartphone (Moto G7 Play) placed in their pocket. 10 participants chose to wear the smartwatch on their right wrist, while the rest chose to wear it on their left wrist. 23 participants placed the smartphone in one of their front pockets, while the rest place it in one of their back pockets. Data Logging. The VRChat app was installed on five different desktops to record the viewpoints/perspective of an adversary as described in Section 3, and OBS Studio [14] was used to record the each adversarial perspective into individual video files with timestamps. The motion sensors were logged in respective devices with timestamps, and later transferred to another desktop for analysis. Analysis Computer. A MacBook Pro, equipped with 10-Core M1 CPU, 16-Core GPU, 16GB memory, 1TB SSD storage and 16-core Neural Engine, was used to train and classify activities, and for the activity-based filtering and magnitude-based ranking tasks. For the large-scale analysis (Section 7), we used a desktop with Ryzen 5 3600 6-Core 3.6GHz CPU, RTX 3060 12GB GPU, 1TB SSD storage, and 16GB memory, to train and generate large datasets using CTGAN [6,83].

EVALUATION

We evaluate the proposed correlation framework utilizing the test data collected from participants, which represents a small-scale attack with confusion set size of 271 (accumulating different motion sensor locations from individual participants). After comprehensively evaluating the framework in the small-scale setting, we generate and evaluate a representative dataset for a large-scale correlation in Section 7.

Framework Parameters

Our correlation framework has two key parameters: (i) activity window size (𝑤), which is the time duration used to classify an action, and (ii) Hamming distance used as the activity-based filtering threshold (𝑡), which is the minimum requirement for an activity-vector to be considered in the identity ranking. As the total observation time, and thus the number of observed activity windows, will vary between different target users, the activity-based filtering threshold (𝑡) is normalized with respect to the number of observed activity windows. No filtering occurs when the filtering threshold is set at 100%, whereas at 0% even one mismatch in the activity sequence will result in that activity-vector being filtered out. Figures 4 and 5 show the correlation accuracy, where "None Correlated" occurs when the activity-based filtering filters all candidate activity-vectors, Correlated" occurs when the top ranked identity is incorrect, and "Correctly Correlated" occurs when the top ranked identity is correct. From these figures, we can see that as we increase 𝑤, the percentage of identities that passes the activity-based filtering and then used for identity ranking also grows. Conversely, the percentage of "None Correlated" is diminished as 𝑤 is increased. This can primarily be attributed to: (i) the size of activity sequence in the activity-vector is inversely proportional to 𝑤 for a constant observation time period thereby reducing the number of probable mismatches, and (ii) the activity inference tends to perform more accurately for larger 𝑤.

While the above observation should compel us to select a larger 𝑤, in Figures 4 and 5 we also observe that there exists a trade-off between 𝑤 and correctly correlated identities for different activitybased filtering thresholds. For instance, when 𝑤 = 5𝑠 we observe that the percentage of correctly correlated identities starts to decrease beyond the filtering threshold of 70% in Figure 4e. This is most likely because as the size of activity-vector is reduced with increasing 𝑤, the probability of confusion with another person's activity magnitudes is increased. This trend was consistent across other experimental variables, such as different adversarial viewpoints, motion sensors, and motion sensor positions on the body.

Based on empirical observations across different experimental variables, we set 𝑤 = 1𝑠 and 𝑡 = 30% for the rest of our analyses. On average, these selected values are best suited for maximizing the percentage of correctly correlated identities. The average correctly correlated identities using these parameter values within top-1 of the ranking was 16.3%, and 17.0% of the identities were within top-3. These values are significant as training did not consider participant data at all! In an alternate adversarial model where the motion sensor positions on the body is known to the adversary, more specific (i.e., per target user) 𝑤 and 𝑡 values can be selected to further improve the percentage of correctly correlated identities.

Activity Confusions

The accuracy of the activity classification models play an important role in the correlation framework's overall success rate. Activity classification between visual and motion sensor data differs significantly due to the modality of input signal, and is subject to different types of noises and interference signals. Different adversarial viewpoint angles, distances, and occlusion levels affect the visual data classification. For instance, if only half of the avatar is visible due to being behind a coach or another avatar is in front of the target avatar, the chance of a misclassification is significantly increased. The positioning and orientation of the device used to collect motion sensor data also imposes certain limitations on the activity classification accuracy, especially as we assume that the adversary is unaware of the exact position of the motion sensor. For instance, if the motion sensor data is from a smartwatch worn on the right hand, it is very useful to classify activities involving the right hand, but may result in high misclassification of activities not involving the right arm.

Due to these apparent limitations, we analyze the direct consequence of misclassifications, i. e., the confusion of activities between the visual and motion sensor data. In Figure 6, we observe that the idle activity has noticeably low accuracy (36% and 22% for right wrist smartwatch and front right pocket smartphone, respectively), and is often confused with other activities. An unexpected, yet clearly discernible, confusion exists between motion sensor idle and visual walking. One possible factor behind this observation is that VR users may be using the VR joystick to walk in the virtual world. As a result, the target user appears idle in the motion sensor data, while their virtual avatar is visually walking. Another noteworthy observation is that head movements had high confusion due to the fact that placement of motion sensors around hip and wrist areas is not suitable for capturing the target user's head movements, whereas a head-mounted VR device is accurately able to capture head movements and apply them to the avatar in the virtual world.

In light of these insights, we further optimize our framework as follows. Rather than considering all the classified actions, we only utilize activities with less than 60% of confusion -body, hand, walk, bend, jump, and others -for our activity-based filtering. Remaining activities in the activity-vector are ignored from the Hamming distance calculations. The average correctly correlated identities after this optimization within top-1 of the ranking was 37.3%, while 38.7% of the identities were within top-3. The correlation accuracy plateaued beyond top-3 due instances of the real identity's activityvector series being eliminated from the rankings by the activitybased filter. This suggests that as activity classification models improve in the future, our attack's correlation accuracy will also improve. (a) 𝑤 = 0.5𝑠 (b) 𝑤 = 1𝑠 (c) 𝑤 = 2𝑠 (d) 𝑤 = 3𝑠 (e) 𝑤 = 5𝑠 Idle Body Head Hand Walk Bend Jump Other Video Action Other Jump Bend Walk Hand Head Body Idle Motion Sensor Action 0.01 0.18 0.05 0.03 0.02 0.09 0.00 0.62 0.00 0.02 0.00 0.01 0.00 0.00 0.91 0.06 0.01 0.13 0.00 0.01 0.00 0.72 0.00 0.12 0.00 0.08 0.01 0.01 0.81 0.02 0.00 0.07 0.03 0.09 0.00 0.75 0.03 0.01 0.00 0.09 0.26 0.27 0.09 0.16 0.06 0.00 0.03 0.14 0.01 0.68 0.03 0.11 0.04 0.06 0.00 0.07 0.36 0.00 0.04 0.01 0.58 0.00 0.00 0.00 0.0 0.2 0.4 0.6 0.8 (a) Using right wrist smartwatch. Idle Body Head Hand Walk Bend Jump Other Video Action Other Jump Bend Walk Hand Head Body Idle Motion Sensor Action 0.04 0.15 0.11 0.07 0.04 0.06 0.01 0.51 0.00 0.00 0.00 0.03 0.00 0.00 0.95 0.02 0.00 0.04 0.01 0.00 0.02 0.87 0.02 0.03 0.00 0.10 0.06 0.02 0.79 0.02 0.00 0.01 0.15 0.27 0.01 0.34 0.13 0.01 0.04 0.05 0.72 0.02 0.10 0.01 0.00 0.00 0.07 0.07 0.06 0.53 0.08 0.17 0.10 0.00 0.00 0.07 0.22 0.08 0.14 0.14 0.38 0.00 0.00 0.03 0.0 0.2 0.4 0.6 0.8 (b) Using front right pocket smartphone.

Different Sensor Locations

We next detail how different positions of the motion sensor on the (human) body and different adversarial viewpoints affect the correct correlation of our proposed framework. Overall, smartwatch (motion sensor) on left or right wrist performed better than the smartphone in the hip pockets (Figure 8). For example, for the Home world the smartwatch yielded about 41% and 68% correct correlations (top-1 rank), for left and right wrists, respectively. In contrast, the front left-front pocket smartphone data resulted in about 9.1% correct correlations, while other smartphone locations are in a similar range. Intuitively, one of the main factors behind this observation is the inability of smartphone motion sensors to pick up hand movements when they are located in the hip area pockets. This causes higher confusion between activities (Figure 6), resulting in the activity-vector of the target user being filtered out with high likelihood. As far as the impact of different adversarial viewpoints on the correlation accuracy of our framework is concerned, we can see from Figure 8 that, except for BC1, all other camera locations (or adversarial viewpoints) yielded comparable results within each of the motion sensor locations. The reason behind BC1 performing particularly poorly is that its location was near the entrance point of the Black Cat world and most participants eventually moved away from the field-of-view of this camera during the data collection experiments. In summary, combining multiple viewpoints and the availability of wrist-based motion sensor data are the most favorable conditions for the adversary.

Similar Activity Sequences

There can be situations where multiple users perform a similar or even an identical sequence of activities. In such cases, the magnitudebased ranking should ideally still rank the real identity (of the target user) higher than others. In this part of our analysis, we study the extent to which our magnitude-based ranking is able to do so, by comparing correlation accuracy when participants (and their avatars) performed the same sequence of activities. In Figure 9b, we observe 16.5% correct correlation for motion data from the right wrist in top-1 of identity rankings and 50.1% correct correlation within the top-3 ranks. This demonstrates that magnitude-based ranking is able to, to an extent, discern the difference between identities based on the magnitude of movements. activity matching and filtering. However, as even a single mismatch between two activity sequences will result in completely different hash values (i.e., the keys in a hash table), we design a larger hash table that allows for some degree of mismatch. Specifically, we populate a hash table with keys based on permutations of the 𝑞 activity sequences in 𝑀 (each of length 𝑘) from the motion sensors data, accounting for possible errors allowable within the Hamming distance threshold (𝑡). Let us assume that the numbers 0 to 7 denotes each of the eight activities we classify. If 𝑘 = 5, an example of the activity string would be ⟨47634⟩. If our hamming distance threshold is 𝑡 = 2, then any two activities can be mismatched and still pass the threshold. Now, assume the character * as a wildcard activity that may or may not be a match. To populate the hash table exhaustively, we compute every possible permutation of each activity sequence in 𝑀 including up to two * . For our previous example, ⟨47634⟩, some of the permutations generated would be ⟨ * * 634⟩, ⟨4 * 6 * 4⟩, and ⟨47 * 3 * ⟩. All these permutations are then used as the key in our hash table, while the corresponding value is the identity of users from the motion sensor data (𝑀). Thereafter, during the correlation process, each activity sequence from the video dataset also undergoes permutations with up to two * , and then queried against the above hash table for a match. If a matching key exists, the corresponding identity and activity-vector has satisfied the activity-based filtering and is included in the identity ranking.

Optimized Performance Analysis. The number of permutations per activity-vector does not scale with the size of datasets and thus can be treated as 𝑂(1) time complexity. Similarly, hash table search and insertion is 𝑂(1) time complexity. Therefore, with the use of our hash table, the new time complexity becomes 𝑂(𝑝 + 𝑞), where 𝑂(𝑞) time is required to create the hash table, and 𝑂(𝑝) time is require to iterate through 𝑉 for filtering. Our empirical results show that with this optimization, the activity-filtering is significantly faster. For instance with 𝑝 = 𝑞 = 100000, 𝑘 = 10, and 𝑡 = 3, using the optimization technique was 575 times faster than the default activity-based filter. Another important aspect we also evaluate in the empirical results is the number of collisions, which occur when multiple unique data sources satisfies the activitymatching threshold. For 𝑘 = 5, we observe 1594.87 average number of collisions, and for 𝑘 = 10, we observe only 4.26 average collisions (with 𝑝 = 𝑞 = 100000). This implies that the adversary should increase the value of 𝑘 if it observes a very high number of collisions.

CONCLUSION

We proposed a novel framework to correlate anonymous avatars in virtual worlds with identified out-of-band motion sensor data.

Our work highlights a newfound privacy risk to users of the growing VR ecosystem. Specifically, VR users can be vulnerable to deanonymization attack if they carry a smartphone or wear a smartwatch while using a VR system. Our evaluation of the proposed framework is a step towards demonstrating the feasibility of such an attack, utilizing real-world data from human participants. Through our empirical analyses, we were able to optimize framework parameters, improve scalability, and identified current limitations and potential for further improvements.