Data physicalizations (3D printed terrain models, anatomical scans, or even abstract data) can naturally engage both the visual and haptic senses in ways that are difficult or impossible to do with traditional planar touch screens and even immersive digital displays. Yet, the rigid 3D physicalizations produced with today's most common 3D printers are fundamentally limited for data exploration and querying tasks that require dynamic input (e.g., touch sensing) and output (e.g., animation), functions that are easily handled with digital displays. We introduce a novel style of hybrid virtual + physical visualization designed specifically to support interactive data exploration tasks. Working toward a "best of both worlds" solution, our approach fuses immersive AR, physical 3D data printouts, and touch sensing through the physicalization. We demonstrate that this solution can support three of the most common spatial data querying interactions used in scientific visualization (streamline seeding, dynamic cutting places, and world-in-miniature visualization). Finally, we present quantitative performance data and describe a first application to exploratory visualization of an actively studied supercomputer climate simulation data with feedback from domain scientists.
more »
« less
Depth-assisted calibration on learning-based factorization for a compressive light field display
Due to the widespread applications of high-dimensional representations in many fields, the three-dimension (3D) display technique is increasingly being used for commercial purpose in a holographic-like and immersive demonstration. However, the visual discomfort and fatigue of 3D head mounts demonstrate the limits of usage in the sphere of marketing. The compressive light field (CLF) display is capable of providing binocular and motion parallaxes by stacking multiple liquid crystal screens without any extra accessories. It leverages optical viewpoint fusion to bring an immersive and visual-pleasing experience for viewers. Unfortunately, its practical application has been limited by processing complexity and reconstruction performance. In this paper, we propose a dual-guided learning-based factorization on polarization-based CLF display with depth-assisted calibration (DAC). This substantially improves the visual performance of factorization in real-time processing. In detail, we first take advantage of a dual-guided network structure under the constraints of reconstructed and viewing images. Additionally, by utilizing the proposed DAC, we distribute each pixel on displayed screens following the real depth. Furthermore, the subjective performance is increased by using a Gauss-distribution-based weighting (GDBW) toward the concentration of the observer’s angular position. Experimental results illustrate the improved performance in qualitative and quantitative aspects over other competitive methods. A CLF prototype is assembled to verify the practicality of our factorization.
more » « less- NSF-PAR ID:
- 10394522
- Publisher / Repository:
- Optical Society of America
- Date Published:
- Journal Name:
- Optics Express
- Volume:
- 31
- Issue:
- 4
- ISSN:
- 1094-4087; OPEXFF
- Page Range / eLocation ID:
- Article No. 5399
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)We, as a society, need artists to help us interpret and explain science, but what does an artist's studio look like when today's science is built upon the language of large, increasingly complex data? This paper presents a data visualization design interface that lifts the barriers for artists to engage with actively studied, 3D multivariate datasets. To accomplish this, the interface must weave together the need for creative artistic processes and the challenging constraints of real-time, data-driven 3D computer graphics. The result is an interface for a technical process, but technical in the way artistic printmaking is technical, not in the sense of computer scripting and programming. Using metaphor, computer graphics algorithms and shader program parameters are reimagined as tools in an artist's printmaking studio. These artistic metaphors and language are merged with a puzzle-piece approach to visual programming and matching iconography. Finally, artists access the interface using a web browser, making it possible to design immersive multivariate data visualizations that can be displayed in VR and AR environments using familiar drawing tablets and touch screens. We report on insights from the interdisciplinary design of the interface and early feedback from artists.more » « less
-
In recent years, virtual/augmented reality (VR/AR) technology has received great attention due to its capability of creating various levels of immersive experiences. However, current wireless VR/AR devices are quite expensive, which hinders its large-scale deployment in practice. In this demo, we present a wireless interactive VR/AR teaching system based on popular Android phones. In such a demo, when a teacher explains a 3D model, multiple students can see it from exactly the same perspective as the teacher does through VR/AR glasses. When one student has a concern or question regarding a particular part of the 3D model, he/she can point it out, and a corresponding blue cursor will appear on screens of all users. Moreover, in the absence of 3D models in Android phones, we broadcast 3D models based on their visual priorities.more » « less
-
As augmented and virtual reality (AR/VR) technology matures, a method is desired to represent real-world persons visually and aurally in a virtual scene with high fidelity to craft an immersive and realistic user experience. Current technologies leverage camera and depth sensors to render visual representations of subjects through avatars, and microphone arrays are employed to localize and separate high-quality subject audio through beamforming. However, challenges remain in both realms. In the visual domain, avatars can only map key features (e.g., pose, expression) to a predetermined model, rendering them incapable of capturing the subjects’ full details. Alternatively, high-resolution point clouds can be utilized to represent human subjects. However, such three-dimensional data is computationally expensive to process. In the realm of audio, sound source separation requires prior knowledge of the subjects’ locations. However, it may take unacceptably long for sound source localization algorithms to provide this knowledge, which can still be error-prone, especially with moving objects. These challenges make it difficult for AR systems to produce real-time, high-fidelity representations of human subjects for applications such as AR/VR conferencing that mandate negligible system latency. We present Acuity, a real-time system capable of creating high-fidelity representations of human subjects in a virtual scene both visually and aurally. Acuity isolates subjects from high-resolution input point clouds. It reduces the processing overhead by performing background subtraction at a coarse resolution, then applying the detected bounding boxes to fine-grained point clouds. Meanwhile, Acuity leverages an audiovisual sensor fusion approach to expedite sound source separation. The estimated object location in the visual domain guides the acoustic pipeline to isolate the subjects’ voices without running sound source localization. Our results demonstrate that Acuity can isolate multiple subjects’ high-quality point clouds with a maximum latency of 70 ms and average throughput of over 25 fps, while separating audio in less than 30 ms. We provide the source code of Acuity at: https://github.com/nesl/Acuity.more » « less
-
With design teams becoming more distributed, the sharing and interpreting of complex data about design concepts/prototypes and environments have become increasingly challenging. The size and quality of data that can be captured and shared directly affects the ability of receivers of that data to collaborate and provide meaningful feedback. To mitigate these challenges, the authors of this work propose the real-time translation of physical objects into an immersive virtual reality environment using readily available red, green, blue, and depth (RGB-D) sensing systems and standard networking connections. The emergence of commercial, off-the-shelf RGB-D sensing systems, such as the Microsoft Kinect, has enabled the rapid three-dimensional (3D) reconstruction of physical environments. The authors present a method that employs 3D mesh reconstruction algorithms and real-time rendering techniques to capture physical objects in the real world and represent their 3D reconstruction in an immersive virtual reality environment with which the user can then interact. Providing these features allows distributed design teams to share and interpret complex 3D data in a natural manner. The method reduces the processing requirements of the data capture system while enabling it to be portable. The method also provides an immersive environment in which designers can view and interpret the data remotely. A case study involving a commodity RGB-D sensor and multiple computers connected through standard TCP internet connections is presented to demonstrate the viability of the proposed method.more » « less
