An official website of the United States government
Placing and orienting a camera to compose aesthetically meaningful shots of a scene is not only a key objective in real-world photography and cinematography but also for virtual content creation. The framing of a camera often significantly contributes to the story telling in movies, games, and mixed reality applications. Generating single camera poses or even contiguous trajectories either requires a significant amount of manual labor or requires solving highdimensional optimization problems, which can be computationally demanding and error-prone. In this paper, we introduce GAIT, a Deep Reinforcement Learning (DRL) agent, that learns to automatically control a camera to generate a sequence of aesthetically meaningful views for synthetic 3D indoor scenes. To generate sequences of frames with high aesthetic value, GAIT relies on a neural aesthetics estimator, which is trained on a crowed-sourced dataset. Additionally, we introduce regularization techniques for diversity and smoothness to generate visually interesting trajectories for a 3D environment, and to constrain agent acceleration in the reward function to generate a smooth sequence of camera frames. We validated our method by comparing it to baseline algorithms, based on a perceptual user study, and through ablation studies. The source code of our method will be released with the final version of our paper.
more »
« less
View Selection in Knot Deformation
Extracting good views from a large sequence of visual frames is quite difficult but a very important task across many fields. Fully automatic view selection suffers from high data redundancy and heavy computational cost, thus fails to provide a fast and intuitive visualization. In this paper we address the automatic viewpoint selection problem in the context of 3D knot deformation. After describing viewpoint selection criteria, we detail a brute-force algorithm with a minimal distance alignment method in a way to not only ensure the global best viewpoint but also present a sequence of visually continuous frames. Due to the intensive computation, we implement an efficient extraction method through parallelization. Moreover, we propose a fast and adaptive method to retrieve best viewpoints in real-time. Despite its local searching nature, it is able to generate a set of visually continuous key frames with an interactive rate. All these combine provide insights into 3D knot deformation where the critical changes of the deformation are fully represented.
more »
« less
- Award ID(s):
- 1651581
- PAR ID:
- 10140254
- Date Published:
- Journal Name:
- 2019 IEEE International Conference on Big Data (Big Data)
- Page Range / eLocation ID:
- 3365 to 3372
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Mathematical knots are different from everyday ropes in that they are infinitely stretchy and flexible when being deformed into their ambient isotopic. For this reason, a number of challenges arise when visualizing mathematical knot's static and changing structures during topological deformation. In this paper we focus on computational methods to visually communicate the mathematical knot's dynamics by computationally simulating the topological deformation and capturing the critical changes during the entire simulation. To further improve our visual experience, we propose a fast and adaptive method to extract key moments where only critical changes occur to represent and summarize the long deformation sequence. We conduct evaluation study to showcase the efficacy and efficiency of our methods.more » « less
-
For many applications with limited computation, com- munication, storage and energy resources, there is an im- perative need of computer vision methods that could select an informative subset of the input video for efficient pro- cessing at or near real time. In the literature, there are two relevant groups of approaches: generating a “trailer” for a video or fast-forwarding while watching/processing the video. The first group is supported by video summa- rization techniques, which require processing of the entire video to select an important subset for showing to users. In the second group, current fast-forwarding methods de- pend on either manual control or automatic adaptation of playback speed, which often do not present an accurate rep- resentation and may still require processing of every frame. In this paper, we introduce FastForwardNet (FFNet), a re- inforcement learning agent that gets inspiration from video summarization and does fast-forwarding differently. It is an online framework that automatically fast-forwards a video and presents a representative subset of frames to users on the fly. It does not require processing the entire video, but just the portion that is selected by the fast-forward agent, which makes the process very computationally efficient. The online nature of our proposed method also enables the users to begin fast-forwarding at any point of the video. Experiments on two real-world datasets demonstrate that our method can provide better representation of the input video (about 6%-20% improvement on coverage of impor- tant frames) with much less processing requirement (more than 80% reduction in the number of frames processed).more » « less
-
null (Ed.)There is a growing importance in characterizing 3D shape quality in additive manufacturing (a.k.a. 3D printing). To accurately define the shape deviation between the designed product and actual build, shape registration of scanned point cloud data serves as a prerequisite for a reliable measurement. However, manual registration is currently heavily involved, for example, in obtaining initial matching of the design and the scanned product based on landmark features. The procedure can be inefficient, and more importantly, introduce potentially large operator-to-operator variations for complex geometries and deformation. Finding a sparse shape correspondence before refined registration would be meaningful to address this problem. In that case, automatic landmark selection has been a challenging issue, particularly for complicate geometric shapes like teeth. In this work we present an automatic landmark selection method for complicated 3D shapes. By incorporating subject matter knowledge (e.g., dental biometric information), a 3D shape will be first segmented through a new density-based clustering method. The geodesic distance is proposed as the distance metric in the revised clustering procedure. Geometrically informative features in each segment are automatically selected through the principal component analysis and Hotelling's T2 statistic. The proposed method is demonstrated in dental 3D printing application and could serve as a basis of sparse shape correspondence.more » « less
-
We present a method to reconstruct dynamic scenes from monocular continuous-wave time-of-flight (C-ToF) cameras using raw sensor samples that achieves similar or better accuracy than neural volumetric approaches and is 100×faster. Quickly achieving high-fidelity dynamic 3D reconstruction from a single viewpoint is a significant challenge in computer vision. In C-ToF radiance field reconstruction, the property of interest—depth—is not directly measured, causing an additional challenge. This problem has a large and underappreciated impact upon the optimization when using a fast primitive-based scene representation like 3D Gaussian splatting, which is commonly used with multi-view data to produce satisfactory results and is brittle in its optimization otherwise. We incorporate two heuristics into the optimization to improve the accuracy of scene geometry represented by Gaussians. Experimental results show that our approach produces accurate reconstructions under constrained C-ToF sensing conditions, including for fast motions like swinging baseball bats. https://visual.cs.brown.edu/gftorfmore » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/BigData47090.2019.9005987
