skip to main content

This content will become publicly available on March 13, 2023

Title: Environmental, User, and Social Context-Aware Augmented Reality for Supporting Personal Development and Change
Robust pervasive context-aware augmented reality (AR) has the potential to enable a range of applications that support users in reaching their personal and professional goals. In such applications, AR can be used to deliver richer, more immersive, and more timely just in time adaptive interventions (JITAI) than conventional mobile solutions, leading to more effective support of the user. This position paper defines a research agenda centered on improving AR applications' environmental, user, and social context awareness. Specifically, we argue for two key architectural approaches that will allow pushing AR context awareness to the next level: use of wearable and Internet of Things (IoT) devices as additional data streams that complement the data captured by the AR devices, and the development of edge computing-based mechanisms for enriching existing scene understanding and simultaneous localization and mapping (SLAM) algorithms. The paper outlines a collection of specific research directions in the development of such architectures and in the design of next-generation environmental, user, and social context awareness algorithms.
; ; ; ;
Award ID(s):
2046072 1908051 1903136
Publication Date:
Journal Name:
IEEE Workshop for Building the Foundations of the Metaverse
Sponsoring Org:
National Science Foundation
More Like this
  1. In a social context where two or more interlocutors interact with each other in the same space, one’s sense of copresence with the others is an important factor for the quality of communication and engagement in the interaction. Although augmented reality (AR) technology enables the superposition of virtual humans (VHs) as interlocutors in the real world, the resulting sense of copresence is usually far lower than with a real human interlocutor. In this paper, we describe a human-subject study in which we explored and investigated the effects that subtle multi-modal interaction between the virtual environment and the real world, where a VH and human participants were co-located, can have on copresence. We compared two levels of gradually increased multi-modal interaction: (i) virtual objects being affected by real airflow as commonly experienced with fans in summer, and (ii) a VH showing awareness of this airflow. We chose airflow as one example of an environmental factor that can noticeably affect both the real and virtual worlds, and also cause subtle responses in interlocutors.We hypothesized that our two levels of treatment would increase the sense of being together with the VH gradually, i.e., participants would report higher copresence with airflow influence than withoutmore »it, and the copresence would be even higher when the VH shows awareness of the airflow. The statistical analysis with the participant-reported copresence scores showed that there was an improvement of the perceived copresence with the VH when both the physical–virtual interactivity via airflow and the VH’s awareness behaviors were present together. As the considered environmental factors are directed at the VH, i.e., they are not part of the direct interaction with the real human, they can provide a reasonably generalizable approach to support copresence in AR beyond the particular use case in the present experiment.« less
  2. Many have predicted the future of the Web to be the integration of Web content with the real-world through technologies such as Augmented Reality (AR). This has led to the rise of Extended Reality (XR) Web Browsers used to shorten the long AR application development and deployment cycle of native applications especially across different platforms. As XR Browsers mature, we face new challenges related to collaborative and multi-user applications that span users, devices, and machines. These collaborative XR applications require: (1) networking support for scaling to many users, (2) mechanisms for content access control and application isolation, and (3) the ability to host application logic near clients or data sources to reduce application latency. In this paper, we present the design and evaluation of the AR Edge Networking Architecture (ARENA) which is a platform that simplifies building and hosting collaborative XR applications on WebXR capable browsers. ARENA provides a number of critical components including: a hierarchical geospatial directory service that connects users to nearby servers and content, a token-based authentication system for controlling user access to content, and an application/service runtime supervisor that can dispatch programs across any network connected device. All of the content within ARENA exists as endpointsmore »in a PubSub scene graph model that is synchronized across all users. We evaluate ARENA in terms of client performance as well as benchmark end-to-end response-time as load on the system scales. We show the ability to horizontally scale the system to Internet-scale with scenes containing hundreds of users and latencies on the order of tens of milliseconds. Finally, we highlight projects built using ARENA and showcase how our approach dramatically simplifies collaborative multi-user XR development compared to monolithic approaches.« less
  3. Mobile devices supporting the "Internet of Things" (IoT), often have limited capabilities in computation, battery energy, and storage space, especially to support resource-intensive applications involving virtual reality (VR), augmented reality (AR), multimedia delivery and artificial intelligence (AI), which could require broad bandwidth, low response latency and large computational power. Edge cloud or edge computing is an emerging topic and technology that can tackle the deficiency of the currently centralized-only cloud computing model and move the computation and storage resource closer to the devices in support of the above-mentioned applications. To make this happen, efficient coordination mechanisms and “offloading” algorithms are needed to allow the mobile devices and the edge cloud to work together smoothly. In this survey paper, we investigate the key issues, methods, and various state-of-the-art efforts related to the offloading problem. We adopt a new characterizing model to study the whole process of offloading from mobile devices to the edge cloud. Through comprehensive discussions, we aim to draw an overall “big picture” on the existing efforts and research directions. Our study also indicates that the offloading algorithms in edge cloud have demonstrated profound potentials for future technology and application development.
  4. As improvements in medicine lower infant mortality rates, more infants with neuromotor challenges survive past birth. The motor, social, and cognitive development of these infants are closely interrelated, and challenges in any of these areas can lead to developmental differences. Thus, analyzing one of these domains - the motion of young infants - can yield insights on developmental progress to help identify individuals who would benefit most from early interventions. In the presented data collection, we gathered day-long inertial motion recordings from N = 12 typically developing (TD) infants and N = 24 infants who were classified as at risk for developmental delays (AR) due to complications at or before birth. As a first research step, we used simple machine learning methods (decision trees, k-nearest neighbors, and support vector machines) to classify infants as TD or AR based on their movement recordings and demographic data. Our next aim was to predict future outcomes for the AR infants using the same simple classifiers trained from the same movement recordings and demographic data. We achieved a 94.4% overall accuracy in classifying infants as TD or AR, and an 89.5% overall accuracy predicting future outcomes for the AR infants. The addition of inertialmore »data was much more important to producing accurate future predictions than identification of current status. This work is an important step toward helping stakeholders to monitor the developmental progress of AR infants and identify infants who may be at the greatest risk for ongoing developmental challenges.« less
  5. Participating in a research experience for undergraduates (REU) site provides opportunities for students to develop their research and technical skills, network with other REU students/professors, raise their awareness of graduate studies, and understand the social context of research. In support of this mission, our REU site at the University of Alabama is exploring research at the intersection of engineering and communicative disorders. Beyond research training though, an REU site provides the opportunity for professional development, social activities, and cultural activities to enrich the student experience. These are important features of an REU, which typically range from 9-10 weeks. Students that participate in summer REUs are recruited from around the country and are brought together at a central research site. Each student brings with them their unique perspectives and lived experiences. To form a cohesive cohort from the individual students, it is important to facilitate shared experiences early in their 9-10 week REU. Supporting the development of a student community through shared experiences has a significant impact on student perspectives of the program. Shared experiences also provide the opportunity to increase the students’ understanding of the new city/state/region that is the setting for the REU. The 2019 iteration of our REUmore »Site, which has a theme of developing technology to support clinical practice in the field of communicative sciences and disorders, aimed to increase the level of social and cultural activities of the cohort in comparison to previous REU sites on campus. This was achieved with multiple professional development, cultural, and social activities. For professional development, students participated in a Practicing Inclusive Engagement workshop to build skills for intercultural engagement that in turn foster a more inclusive REU cohort. Students participated in this workshop within the first three days of arriving on campus. This workshop focused on identity, inclusive language, and creative ways to invite and engage in diverse perspectives. For cultural activities, full-day field trips were taken to the U.S. Space & Rocket Center in Huntsville, AL and The Legacy Museum / The National Memorial for Peace and Justice in Montgomery, AL. These trips engaged students in very different aspects of Alabama's history. One showcasing achievements of the U.S. space and rocket program and the other investigating the racial injustice in American history and its legacy. While many of the students were familiar with these histories, the museums and their compelling visuals and data-rich exhibits provided a far deeper insight into these topics and facilitated further conversation between the REU cohort. The REU cohort spent much of their summer learning with and from graduate students enrolled in the masters of speech-language pathology (SLP) program at the University of Alabama. At the end of the summer experience, a BBQ event was facilitated (food, yard games) to spur on friendly competition between REU and SLP students. This provided both groups an informal opportunity to debrief about the summer experiences. In this work an overview of the REU site will be provided with a focus on the logistical elements to pilot the social, cultural and professional development efforts, a summary of the student feedback from the written reflections and focus groups, experiences of the program coordinators, and future plans to refine and improve these elements will be presented.« less