More Like this

1. VRFromX: From Scanned Reality to Interactive Virtual Experience with Human-in-the-Loop

   https://doi.org/10.1145/3411763.3451747  

   Ipsita, Ananya; Li, Hao; Duan, Runlin; Cao, Yuanzhi; Chidambaram, Subramanian; Liu, Min; Ramani, Karthik (May 2021, Conference on Human Factors in Computing Systems (CHI EA '21))

   null (Ed.)

   There is an increasing trend of Virtual-Reality (VR) applications found in education, entertainment, and industry. Many of them utilize real world tools, environments, and interactions as bases for creation. However, creating such applications is tedious, fragmented, and involves expertise in authoring VR using programming and 3D-modelling softwares. This hinders VR adoption by decoupling subject matter experts from the actual process of authoring while increasing cost and time. We present VRFromX, an in-situ Do-It-Yourself (DIY) platform for content creation in VR that allows users to create interactive virtual experiences. Using our system, users can select region(s) of interest (ROI) in scanned point cloud or sketch in mid-air using a brush tool to retrieve virtual models and then attach behavioral properties to them. We ran an exploratory study to evaluate usability of VRFromX and the results demonstrate feasibility of the framework as an authoring tool. Finally, we implemented three possible use-cases to showcase potential applications. 

   more » « less
2. Develop and evaluate an augmented reality posture training tool to promote work safety

   https://doi.org/10.1177/1071181320641496  

   Chen, Ken; Perera, Gimantha; Li, Li; Xu, Xu; Chen, Karen B (December 2020, Proceedings of the Human Factors and Ergonomics Society Annual Meeting)

   null (Ed.)

   Conventional work posture training tools included pamphlets, one-time training orientation, and/or videos. These tools did not always yield satisfactory training outcomes, and the incident rate of work-related musculoskeletal disorders did not substantially lower. In this research, modern augmented reality (AR) technology was leveraged to deliver interactive, holistic, whole-body visual information to convey safe work postures. The developmental procedure followed DMAIC by first defining specifications of training content, which led to the development of the training tool, including 3D reconstruction of a virtual instructor and building of user interface based on user-centered framework. This AR training tool was measured and analyzed through usability evaluation, and quantitative and qualitative data were obtained for cross-validation and usability issue source identification. Findings revealed the utility of 3D reconstruction of a virtual instructor and practicality of adopting conventional usability evaluation method for AR user interface usability evaluation. Feedback from the usability evaluation via questionnaire, think aloud, and post-task open-ended responses are employed to iteratively design the next version of the AR posture training tool. 

   more » « less
3. DA Wand: Distortion-Aware Selection using Neural Mesh Parameterization

   https://doi.org/10.1109/CVPR52729.2023.01606  

   Liu, Richard; Aigerman, Noam; Kim, Vladimir; Hanocka, Rana (July 2023, IEEE Conference on Computer Vision and Pattern Recognition)

   We present a neural technique for learning to select a local sub-region around a point which can be used for mesh parameterization. The motivation for our framework is driven by interactive workflows used for decaling, texturing, or painting on surfaces. Our key idea is to incorporate segmentation probabilities as weights of a classical parameterization method, implemented as a novel differentiable parameterization layer within a neural network framework. We train a segmentation network to select 3D regions that are parameterized into 2D and penalized by the resulting distortion, giving rise to segmentations which are distortion-aware. Following training, a user can use our system to interactively select a point on the mesh and obtain a large, meaningful region around the selection which induces a low-distortion parameterization. 

   more » « less
4. The Design of a Virtual Prototyping System for Authoring Interactive Virtual Reality Environments From Real-World Scans

   https://doi.org/10.1115/1.4062970  

   Ipsita, Ananya; Duan, Runlin; Li, Hao; Chidambaram, Subramanian; Cao, Yuanzhi; Liu, Min; Quinn, Alex; Ramani, Karthik (March 2024, Journal of Computing and Information Science in Engineering)

   Abstract Domain users (DUs) with a knowledge base in specialized fields are frequently excluded from authoring virtual reality (VR)-based applications in corresponding fields. This is largely due to the requirement of VR programming expertise needed to author these applications. To address this concern, we developed VRFromX, a system workflow design to make the virtual content creation process accessible to DUs irrespective of their programming skills and experience. VRFromX provides an in situ process of content creation in VR that (a) allows users to select regions of interest in scanned point clouds or sketch in mid-air using a brush tool to retrieve virtual models and (b) then attach behavioral properties to those objects. Using a welding use case, we performed a usability evaluation of VRFromX with 20 DUs from which 12 were novices in VR programming. Study results indicated positive user ratings for the system features with no significant differences across users with or without VR programming expertise. Based on the qualitative feedback, we also implemented two other use cases to demonstrate potential applications. We envision that the solution can facilitate the adoption of the immersive technology to create meaningful virtual environments. 

   more » « less
5. Visually-Grounded Library of Behaviors for Manipulating Diverse Objects across Diverse Configurations and Views

   Yang, Jingyun; Tung, Hsiao-Yu; Zhang, Yunchu; Pathak, Gaurav; Pokle, Ashwini; Atkeson, Christopher G; Fragkiadaki, Katerina (June 2021, 5th Annual Conference on Robot Learning)

   We propose a visually-grounded library of behaviors approach for learning to manipulate diverse objects across varying initial and goal configurations and camera placements. Our key innovation is to disentangle the standard image-to-action mapping into two separate modules that use different types of perceptual input:(1) a behavior selector which conditions on intrinsic and semantically-rich object appearance features to select the behaviors that can successfully perform the desired tasks on the object in hand, and (2) a library of behaviors each of which conditions on extrinsic and abstract object properties, such as object location and pose, to predict actions to execute over time. The selector uses a semantically-rich 3D object feature representation extracted from images in a differential end-to-end manner. This representation is trained to be view-invariant and affordance-aware using self-supervision, by predicting varying views and successful object manipulations. We test our framework on pushing and grasping diverse objects in simulation as well as transporting rigid, granular, and liquid food ingredients in a real robot setup. Our model outperforms image-to-action mappings that do not factorize static and dynamic object properties. We further ablate the contribution of the selector's input and show the benefits of the proposed view-predictive, affordance-aware 3D visual object representations. 

   more » « less