Physical environments are being digitized at an accelerating speed and ever-increasing fidelity as a result of the advancement in cameras and depth sensors. Recent years have seen the success of "virtual tours", captured by 360º cameras or smartphone cameras, these digital replicas of the physical environment have been used to provide a semi-immersive experience, allowing users to remotely "tour" the environment. These virtual tours are used by national parks 1and museums 2 for accommodating remote experiences. Another popular use of these virtual tours is in the real estate domain in which property owners often offer virtual tours to accommodate potential buyers who cannot attend open-house tours physically. The hype in this trend of digitization fuels an emerging concept of "digital twins" as new thrusts in many industries such as manufacture [36], construction [1,18], and AI [2].
In this research, we first investigated the benefits and caveats of using virtual tours, in general -3D digital replicas of physical environments in an important application, which is people with limited mobility remotely assessing the accessibility of unfamiliar environments, to improve independence in living and quality of life. Through a background survey, we found the remote assessment of accessibility is much in need by people with limited mobility as they often need to visit unfamiliar environments full of obstacles such as narrow aisles, low tabletops, and tall shelves. As it has been estimated that 30% of the global population have access requirements [11,14] and most people will have limited mobility at some stage in their life, we expect the knowledge acquired from this research to have a transformative and wide impact across society.
Existing approaches to assess environment accessibility mostly rely on 1) standardized accessibility assessment (ADA inspection) which labels accessibility of environments in a coarsely grained manner, or 2) manual labor (e.g., family members, friends, crowd workers) which undermines independence and increases cost. In response, we identify the great potential in using virtual tour technology a powerful tool for remote accessibility assessment which could adapt to a finer-grained and personalized assessment. Additionally, in virtual tour technology, 3D digital replicas of environments are easy to acquire and share, which significantly lowers the barrier for accessibility inspection. Finally, these digital replicas in concert with virtual tour platforms uniquely allow user to annotate at nearly zero cost. These annotations which are extremely useful in improving accessibility would not be easy to gather in physical environments.
Prior work has looked into using digital replicas of physical environments for accessibility assessment. For example, Project Sidewalk [37] is a seminal crowd-source platform that leverages satellite images to assess the accessibility of the urban environments, labelling critical information such as existence, number, and type of ramps, obstacles, and surface problems. Compared with this pioneering effort, our research focuses on indoor environments that could be more complex and come with a drastically different set of accessibility challenges, user needs, and interaction preferences. Additionally, by letting users with disability to be the inspectors, we enable personalized and finer-grained accessibility assessments, which were widely appreciated by our participants in the user study.
Americans with Disabilities Act (ADA) has regulated physical environments so that they can accommodate people with disabilities, including those with limited mobilityfoot_2 . For example, public spaces and urban streets must be constructed wide enough for wheelchairs and other accessibility mobility tools to traverse unabated. However, ADA is failing to meet drastically different requirements of people with limited mobility due to their very different types and levels of disabilities. This observation has been reported in prior research which unveiled that people with limited mobility often need different accommodations in terms of uses of everyday objects and computing devices [23,27,34,41]. To accommodate for this difference, researchers have taken a user-centric approach, serving people with limited mobility with customizable technology on their wheelchairs to meet individual needs [6][7][8].
Researchers have also investigated the use cases and experience of a broader spectrum of sensing systems including ones that are deployed in environments for people with physical disabilities [20]. To compensate for a user's limited capabilities in actuating everyday objects, prior work has studied home automation systems to mitigate challenges and obstacles in the daily lives of people with limited mobility [19,24,30,32]. Uses of special-purpose electronic devices could be accommodated through dense deployments in the environment [33] or embodied mobility via drones [28]. Additionally, Represe [9] and Roman [25] improve the usability of everyday objects with 3D printed enhancements that make their host objects easier to grasp and manipulate. MiniKers [42] are self-powered home automation systems leveraging kinetic energy harvested from user interactions. Despite these efforts, physical environments are still full of challenges and obstacles to people with limited mobility. For this reason, effective assessment approaches are much needed to facilitate people with disabilities to plan ahead of their visits, prepare assistance/tools needed, and provide feedback to improve the accessibility of environments.
Remote environment accessibility assessment has been mostly conducted visually through digital approaches such as real-time camera feeds [13,26,28]. This approach could be further assisted by another person moving the camera as instructed by the remote users [33]).
Studies have also been conducted for remote home-modification evaluation using a video-conference system [38] and Virtual Tours for the elderly through immersive and non-immersive VR to assess the living environment before moving in [40]. Several protocols for assessment that could be implemented by video conference techniques have been developed. In this paper, we explore the use of contemporary virtual tour technologies enabling people with limited mobility to interactively explore previously digitized environments on their own. Closer to our work, there has been prior research investigating remote environment assessment approaches for people with physical disabilities. Kim et al. [22] developed a seminal remote environment accessibility assessment system consisting of 3D models of environments. These works used commercial software to construct high-fidelity 3-D environments from photographs and developed custom screening algorithms and instruments for analyzing accessibility. This system was further used in a study investigating the effectiveness of accessibility assessment in the form of immersive virtual reality presented to the users on a projected screen [21]. Only parts of the environments (e.g., entrance) were provided in the study and a checklist was provided to participants to guide the assessment. No investigation on user needs or feedback was conducted. In comparison, our work takes advantage of the latest development in environment digitization to provide comprehensive and high-fidelity 3D replicas of the environments including details like objects, tools, and utilities.
Finally, Project Sidewalk [15,37] demonstrated a crowd-source approach to improving urban accessibility by reporting crowdidentified issues with city street ramps. This pioneering work has also demonstrated a high scalability with a massive deployment. In comparison, our work also targets remote accessibility assessment with a high scalability but focuses on a different type of environments -indoor spaces which feature different sets of configurations, objects, and functionalities, constituting innately different challenges and opportunities.
We conducted a commercial survey on existing environment 3D scanning technologies including only those that are commercially available to facilitate the conversion of results from this work. This survey identified six products including Matterport [29], Metareals [31], Canvas [5], 3D Scanner App [3], Polycam [35], and CamToPlan -3D Scanner & LiDAR [3]. We compared the quality of the scanned models, ease of implementation, supported interactivity, number of users, and accommodation for annotations/edits, and eventually selected Matterport as the platform in our study for its overall superior performance and popularity in virtual tour applications.
We used Matterport to scan four common indoor environments each featuring a unique combination of size (i.e., small vs. large) and functionality (i.e., residential vs. commercial). These environments can be found in Figure 1. On average, there are 17 (SD=11) scanning positions in each environment. We followed the recommendations and guidelines from Matterport to conduct the scanning. Insta360 ONE and Matterport PRO 3D scanners were used in concert with the Matterport reconstruction software to create the virtual tours. Of note that Matterport supports measurements in virtual tours and we made this feature available to participants in the study as prior work [21] has shown that measurements in virtual environments could be of help to the assessment of accessibility (e.g., clearance of space is sufficient for a wheelchair to maneuver around). Additionally, we used the annotation feature to collect participants' feedback to improve the accessibility in the virtual environments but did not specify what should / should not participants focus on, allowing for maximum space for feedback hoping to draw richer insights given this flexibility. The study involved 14 participants (7 males, average age = 34.7; 7 females, average age = 30.3) with different levels of mobility. All participants rely on assistive mobility technologies, specifically wheelchairs, on a daily basis. We conducted a demographic survey before the study, the result of which is shown in Table 1. The main study began after the demographic survey and our brief introduction of the study. The studies with all participants were conducted remotely over zoom. Figure 2 shows examples of study configurations from two participants during the study.
Our study consisted of three phases each featuring one or a few major tasks for participants: 1) exploring the four environments using virtual tours and making annotations on the accessibility aspect of the environments, 2) reviewing results together with the experimenter and completing a questionnaire on the usability of the virtual tour techniques for remote environment assessment, and 3) participating in a semi-structured interview with in-depth discussions. The study took around 1.5 hours to finish and each participant was reimbursed $40.
Before phase 1 started, we taught participants how to perform navigation, make annotations, and take measurements using the platform with a short introductory video followed by Q&A. Photos and videos of the four environments were shown to the participants, and were later referred as baselines by participants when evaluating the usability of Virtual Tours. In phase 1, all participants were able to use virtual tours smoothly except for P6 for their limited muscle capability in the upper extremity and P11 for insufficient experience with computing devices. For these two participants, navigation and annotations were performed by the experimenter under participants' verbal instructions. In phase 2, we developed the hypothesis and questionnaire items according to the Technology Acceptance Model (TAM) framework [12] with a focus on the usability of the Virtual Tours as an approach for remote accessibility assessment. We compared the usability of Virtual Tours, with two conventional approaches commonly used in accessibility assessment (i.e., inperson and photo/video assessment) on the fronts of feasibility, operability, accuracy, and learnability. Participants were asked to complete Likert-scale questionnaires consisting of seven points, where 1 indicated the most negative perception and 7 indicated the most positive perception. Finally, in phase 3, we conducted semistructured interviews with participants, guided by responses from them on these usability metrics to draw further insights. Thematic analysis using affinity diagramming was conducted. We identified the pros and cons of Virtual Tours in assessing the accessibility of remote environments and propose future improvements that could make this promising technology in accessibility even more useful.
Participants had a remote spatial exploration experience via the Matterport platform, with which they left 489 annotations in four spaces (Figure 3). These annotations cover discussions on the overall structure and layout of the entire space. They contain measurements of the wall as well as the essential furniture such as tables and beds. We group annotations from participants into two categories, which are discussed below.
This type of annotations discusses configurations that are structurally static and cannot be easily changed in the space, such as the height of walls, the width of corridors, the height of bed frames, etc. Common words covered in the relevant annotations are "narrow", "high", "inaccessible to wheelchairs", etc. Areas marked by users are generally focused on clearance that are too narrow and on surfaces that are positioned too high, such as worktops and bed frames, etc. Several scenarios involve nuanced configurations that wheelchair users would find inconvenient, for example, a TV console at a corner of a roomeven though the space does not appear to be narrow, operations of cable connections require wheelchair users to reach the back of devices which are inaccessible from both sides. Potential solutions to these inaccessible configurations are also mentioned in the annotations, which can be divided into two main categories. The first category is about use of products with actuation mechanisms which could provide adjustable height, such as lift tables, lift bed frames, etc. Another category is about reconfiguration of environment to allow users to get closer to the table by lifting up its top or clearing out the area under it so that wheelchairs could easily fit underneath. For corridors with narrow spaces, participants could not propose a viable solution. However, by measuring the environment remotely, participants felt that they could clearly understand whether their wheelchair can fit to make better decisions.
The second type of annotations concerns the dynamic configurations of environments, such as the way doors open and the direction in which various appliances open. We noted that some objects (e.g., tables, doors) often received more attention from participants than others. Participants commented on these objecdt's mobility, radius, and range of motion, which can significantly affect the accessibility of environments to wheelchair users. For example, if a door opens toward a wheelchair user, the user will have to move back a certain distance to make space for the door. Depending on the static configuration of the environment, this clearance might not be easily achieved. Most participants recommended uses of sliding doors in their annotations. We found three common directions of gross movements on household objects either these objects: up-down, forward-backward, and left-right. We conclude from the annotations that participants prefer the leftright movement over the other two directions, between which the up-down movement is the most inconvenient for requiring larger ranges of limb movements. For example, when it comes to the way the washing machine is opened, they prefer a door that is pulled open on the front side of the machine, rather than a door that is on the top and has to be lifted. This is because that doors on the top require up-down movements that often exceed what wheelchair users could perform. Additionally doors on the top make it difficult for wheelchair users who often have low vantage points to see interiors of washing machines.
Figure 4 shows p-value statistics on all responses across all questions (i.e., 4 metrics x 3 modes) in our survey, the result of which accepts the hypothesis of normality (p>0.05). These Likert-Scale responses are shown in Figure 4 with significance in pair-wise differences shown in asterisks (* p<0.05, ** p<0.01, and *** p<0.005). Paired-T tests indicate superior preferences in most usability metrics of Virtual Tour (VT) assessment than the two baselines -In-Person (IP), and Video & Photo (VP) assessment, especially in Feasibility. Interestingly, participants perceived VP as less feasible for the need for an extensive number of photos taken to be able to match the amount of information provided by VT, lowering its feasibility as participants would rely on people who have access to the remote environments to take these photos and videos, which defeats the purpose of remote assessment. However, the perception on all four metrics of VP varied quite a lot across users, as seen in its large ranges between minimums and maximums, highlighting the fact that this conventional approach might not be able to meet various individual needs. For example, photos and videos of environments might suffice in delivering sufficient information for participants with higher level of mobility, while being insufficient for those that are more limited in mobility, who will rely on more comprehensive information of environments including specific measurements of widths and heights for effective assessment.
We conducted a thematic analysis [4] with affinity diagramming on quotes and notes collected from our interviews with participants. Below we categorized our findings into 5 themes.
Supporting personalization in accessibility assessment is acknowledging various individual disabilities and needs, which have yet to be fully accommodated by the state-of-the-art accessibility assessment approaches. Take the ADA inspection for example -only routine inspections are performed on a few functional objects (i.e., bathtubs) and coarse configurations of the environment (e.g., the width of corridors), which is far from being enough to constitute the wide spectrum of aspects in the daily living of people with limited mobility. The issue was mostly due to the lack of information needed for answering individually different queries rather than the lack of personalized interaction techniques. Participants commented that personalized accessibility assessment is much in need because there is simply too much variance in what people with disability would need. In the case of people with limited mobility, they have different levels of mobility (e.g., different types and levels of spinal cord injury) and the physical disability could be at different parts of body. Additionally, this difference is amplified when the user resides on mobility technologies of different configurations (front-drive vs. rear-drive). What makes it even more complicated is that residents of different environments might have different expectations regarding the tasks they need to perform depending on their length of stay (e.g., long-term vs. shortterm residence), and people with disabiity might have different living situations (e.g., alone vs. with friends/family), both of which result in drastically different accessibility needs and therefore aspects to pay attention to in the assessment of the environment. For even the same environment, assessment results could differ because of the change in tasks and needs of a resident. For example, P14 commented that "if I am staying in the room by myself, I would need more assistive features", and P5 "I don't mind if there is a short stair which I can overcome with some effort; if I am only staying at that place for a few days". Finally, P11 mentioned that an effective assessment approach could help them get better prepared mentally for challenges in the environments to visit. This mental preparation is equally important as physical preparation to her.
We noted a negative correlation between feasibility and accuracy -conventional approaches that yield high accuracy often demand more efforts to achieve and thus have lower feasibility. Participants uniformly referred to in-person assessment as the most accurate approach and yet this approach was recognized as the most challenging one to achieve in practice. As P10 commented, "in-person assessment is accurate. The sense of real objects present near me is important for me to check if the environment is accessible. But it is just very hard for me to do that independently". In addition to independence, participants also mentioned time and monetary commitments when explaining their lower scores for the feasibility of the in-person approach. These findings around feasibility ground a strong need for accurate tools for remote accessibility assessment. One immediate requirement for this tool is to provide comprehensive information to accommodate differences in user needs.
Our result revealed that Virtual Tours have the benefits of bothbeing a digital approach, VT allows users to assess environments remotely, while the comprehensive information presented by VT can facilitate personalized assessment beyond the previous possible. All participants praised VT for leading to more accurate assessments. Specifically, P2 and P3 gave credit to the impact of VT as a potential platform on which people with limited mobility could easily express assistance needed with the space owner. P5 commented, "compared with conventional modalities, assessing the environment with this system is easier to achieve and more accurate than phone calls". They further commented that in some aspects VT facilitates the assessment of hard-to-reach locations of environments (e.g., measuring the height of a tall shelf), for which this approach is even better than in-person assessment. P7 noted that "online assessment with VT is comprehensive and meets all the information I need in my assessment. Additionally, P8 and P10 expressed their favor for VT for it being a convenient approach that eliminates the need for in-person assessments.
From participants' feedback, we found Accuracy to have much overlap with Personalization in addition to the negative correlation with Feasibility. 5 participants mentioned in their interviews that they assessed the accessibility of unfamiliar environments remotely using photos, videos, word of mouth, and phone calls with their owners. Though these approaches were feasible to conduct, they did not provide sufficient information to accommodate individual differences in informational needs and thus lowered the accuracy of assessments. P7, P10, and P14 mentioned that VT is comparatively more feasible to get to locations in environments that are difficult to measure (e.g., underneath the table, height of an overhead cabinet). Estimations at distance result in errors that hurt accuracy and therefore accuracy and feasibility used to contradict one another before VT for all participants in our study.
Additionally, participants mentioned that the devils are in the details (P1, P5, and P7) as they noted that most photos and videos they found online are missing critical details because these media files are often engineered for aesthetics that rarely expose problems in accessibility. For example, online photos and videos often highlight major appliances and objects in the room while excluding nuanced information such as the clearance of a corner where wheelchair users need to turn. Exposing these details requires photogrpahers to have the needs of people with disability in mind knowing what they need to pay attention to, while VT mitigates the problem by conducting comprehensive scans of environments. We found that the lack of information with commonly used remote accessibility assessment approaches hindered participants' comprehension of the environment leading to inaccurate assessments. For example, P2, P5, and P7 relied on owners to describe the accessibility of environments but the information offered was often inaccurate and misleading because of the lack of awareness that there is much variety in disability. A common example from our participants was that owners of environments tend to be overly confident in claiming that their environments are accessible simply because they have hosted wheelchair users before.
3.5.4 Operability. We suspected that operability could affect our participants' perception of the usability of VT compared with baseline approaches. Interestingly this was not the case with the current set of participants. We did not find evidence that shows demands in personalized device interactions in our study. We are cautious that the variety of disabilities in our participant group might not be wide enough to draw further insights and therefore future studies of Virtual Tours considering a wider spectrum of physical disabilities merit further efforts. All participants except P6 used computer mice in our study and did not report physical challenges in doing so. No improvements in operability for VT were mentioned once participants learned how to nevigate around and create annotations using the Matterport platform. One suggestion from participants was on the support of speech-to-text in annotating, however we noted that such features could be enabled by system-level supports on mainstream operating systems (e.g., macOS dictate feature, Azure Speech services). None the less, we expect that an seamless integration of speech-to-text into annotations could improve the usability of Virtual Tours.
3.5.5 Learnability. Most participants were positive towards VT in their comments regarding its learnability. All participants except P11 effective learned operations in VT after receiving a training session that lasted around 5 minutes by watching a tutorial video the research team prepared that demonstrated how to 1) navigate around, 2) make annotations, and 3) take measurements. P11 had limited experience with computers but managed to use VT with help of a remote experimenter. We suspect that similar situations with people with limtied upper extremety mobility and with insufficient experience of computers could be resolved by help from volunteers, friends and family members. Though this approach adds a layer of cost but it should still strike a better ballance between feasibility and accuracy than conventional approaches. Finally, participants noted that the similarity between interaction techniques in VT and those of Google Street helped their learning.
3.6.1 Missing information on physical properties. Though Virtual Tours compared favorably against baseline techniques on multiple aforementioned metrics, participants pointed out much information that is not currently supported by state-of-the-art Virtual Tour systems. Since the comprehension of environments largely lies in the comprehensiveness of provided information in remote accessibility assessment, the missing information should be paid attention to for future Virtual Tour systems, especially for those potentially geared towards people with disabilities. Specifically, participants noted the missed information on the physical layer -many non-visual properties such as weight of objects (e.g., a cooking pot), textures of surfaces (e.g., carpet), abd torque and force of object uses (e.g., door) cannot be conveyed by Virtual Tours. For example, P10 commented that "it is unclear how much force the door needs to open". Additionally, P9, P10, and P12 recommended custom vantage points in VT navigation for different wheelchairs often having different sitting heights. P12 referred to texture of ground surfaces as a piece of critical information needed to understand the accessibility of the environment. A common example from several participants was that navigating on a carpet took much more effort than on hard floors. As for now, only in-person assessment could offer information on these aspects, which constitute both limitations and opportunities for future research.
3.6.2 Improving accuracy using visual references. Even for visual information, there are improvements needed to allow users to make better use of it with higher accuracy, efficiency, and confidence. For example, P2 pointed out the fact that it remained challenging to them on how to interpret measurements in the environments. It would be even more challenging for users who do not recall their wheelchair dimensions. In this regard, P7 and P9 suggested setting the height of the scanning camera to the eye level of the seated wheelchair user. However, since different users have different seeting heights, this camera height adjustment could only be done in an approximate manner. A more precise solution to this problem is to have a scanning instrument with multiple cameras or a time-multiplex camera platform that can yield scans from vantage points of various heights. Additionally, P2, P7, and P14 suggested adding references to VT to improve accuracy. Specifically, they suggested two types of references -1:1 ratio dummy wheelchair or at least its footprint shown in the virtual environment to give users a better sense of the spatial relationships -e.g., whether the space has enough clearance for their wheelchairs to make turns, to pass through or fit under certain furniture.
3.6.3 Convey information on dynamic configurations of the environments. Current Virtual Tours do not support visualization that shows dynamic configurations of the environment. However, this information was much needed, as pointed out by P10 with in a comment that it was unclear how the cabinet door would be opened since its handle is at the center of the door pane. This information is important because if the cabinet door opens to the right, the door will be positioned in between the participant and the cabinet making it difficult to access objects inside. To address this issue, future remote accessibility assessment platforms could leverage modeling techniques to add semantic labels on the dynamic configurations of objects (e.g., how parts of objects move when they are in use). Additionally, rich rendering of these models could be offered in visualization platforms geared towards virtual environments in 3D such as virtual reality.
3.6.4 Rich explorations through VR. Fortunately, the aforementioned challenges could be mitigated by Virtual Reality (VR), an emerging platform that has been increasingly adopted by wheelchair users [16,17,39]. For example, physical properties of objects could be rendered with VR haptic technologies (e.g., Grabity [10]). We also found haptic techniques that have been commercialized to render textures of remote environments (e.g., carpet, work surfaces). The visual reference needed could be achieved through assigning wheelchair users virtual avatars modeled after their physical capabilities. Additionally, dynamic configurations of environment could be rendered in VR through animations or virtual manipulations allowing users to conduct rich explorations of environments in a more immersive manner. Finally, rich explorations through VR could contribute to rich experience which is important in environmental assessment as the participants noted, and P7 recommended using VR to make Virtual Tours closer to real-world tour experience.
3.6.5 Reduce informational load with semantic labels. We also received feedback from participants to remove clutters on the floor that is irrelavent to the core functionality of the environment. In this regard, P2 recommended having at least two scans of each environment -with, and without clutters and unnecessary objects. The "clean" scan of environments could give participants a quick understanding of their fundamental configurations. As for now, they had to first imagine these environments with all clutters removed and then perform evaluation. Additionally, removing clutters is often what participants would do in practice to improve the accessibility of environments (e.g., move objects out of moving paths to make clearance). This feature requires semantic labels of objects to add information such as mobility and necessity. Such labeling processes could be done using the latest advances in artificial intelligence (e.g., object recognition and image semantic analysis).
3.6.6 Synergy with Google Street. Almost all participants had experience with Google Street and three of them (P6, P9, and P14) recommended future VT accessibility assessment technology could work in concert with Google Street having a seamless transition between indoor and outdoor environments, which is increasingly feasible for Google's recent developments on Indoor Mapsfoot_3 . Collaborations between research and user communities, and industry to add assistive features in Indoor Maps to facilitate people with disabilities in their remote accessibility assessment could further amplify the impact of this technology.
3.6.7 Support richer input modalities. Finally, three participants recommended richer input modalities beyond touch interactions, computer mice, and keyboards. P7 recommended allowing power wheelchair users to use the wheelchair controllers to navigate in virtual environments. Additionally, P4 and P7 wanted to see voice input (e.g., speech-to-text) being more utilized in future Virtual Tours to accommodate for users with limited upper extremity mobility. Several other relevent ideas were mentioned such as using voice commands to navigate and take measurements and leaving voice messages to provide feedback to owners of the environments.
Digitization approaches in our physical environments have become increasingly mature and given birth to numerous revolutionary applications that improve the life quality for all. In this work, we first conducted a user study of Virtual Tours in remote accessibility assessment applications for people with limited mobility. The study consisted of 14 participants who used Virtual Tours in four representative indoor environments and left 489 annotations on the accessibility challenges based on their various individual needs. We drew insights into personalization and usability and discuss potential modalities for Virtual Tours and assistive features that could make this technology more useful in remote accessibility assessment.
Virtual Yosemite: https://www.virtualyosemite.org
These 12 Famous Museums Offer Virtual Tours You Can Take on Your Couch: https://www.travelandleisure.com/attractions/museums-galleries/museumswith-virtual-tours
Americans with Disabilities Act of 1990: https://en.wikipedia.org/wiki/Americans_ with_Disabilities_Act_of_1990
Google Indoor Maps: https://www.google.com/maps/about/partners/indoormaps/