An official website of the United States government
Real-time detection of 3D obstacles and recognition of humans and other objects is essential for blind or low- vision people to travel not only safely and independently but also confidently and interactively, especially in a cluttered indoor environment. Most existing 3D obstacle detection techniques that are widely applied in robotic applications and outdoor environments often require high-end devices to ensure real-time performance. There is a strong need to develop a low-cost and highly efficient technique for 3D obstacle detection and object recognition in indoor environments. This paper proposes an integrated 3D obstacle detection system implemented on a smartphone, by utilizing deep-learning-based pre-trained 2D object detectors and ARKit- based point cloud data acquisition to predict and track the 3D positions of multiple objects (obstacles, humans, and other objects), and then provide alerts to users in real time. The system consists of four modules: 3D obstacle detection, 3D object tracking, 3D object matching, and information filtering. Preliminary tests in a small house setting indicated that this application could reliably detect large obstacles and their 3D positions and sizes in the real world and small obstacles’ positions, without any expensive devices besides an iPhone.
more »
« less
Experimental Autonomous Deep Learning-Based 3D Path Planning for a 7-DOF Robot Manipulator
In this paper, we examine the autonomous operation of a high-DOF robot manipulator. We investigate a pick-and-place task where the position and orientation of an object, an obstacle, and a target pad are initially unknown and need to be autonomously determined. In order to complete this task, we employ a combination of computer vision, deep learning, and control techniques. First, we locate the center of each item in two captured images utilizing HSV-based scanning. Second, we utilize stereo vision techniques to determine the 3D position of each item. Third, we implement a Convolutional Neural Network in order to determine the orientation of the object. Finally, we use the calculated 3D positions of each item to establish an obstacle avoidance trajectory lifting the object over the obstacle and onto the target pad. Through the results of our research, we demonstrate that our combination of techniques has minimal error, is capable of running in real-time, and is able to reliably perform the task. Thus, we demonstrate that through the combination of specialized autonomous techniques, generalization to a complex autonomous task is possible.
more »
« less
- PAR ID:
- 10134879
- Date Published:
- Journal Name:
- ASME 2019 Dynamic Systems and Control Conference
- Volume:
- 2
- Page Range / eLocation ID:
- V002T14A002
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
North-Morris, Michael B.; Creath, Katherine; Porras-Aguilar, Rosario (Ed.)A novel Vision ray metrology technique is reported that estimates the geometric wavefront of a measurement sample using the sample-induced deflection in the vision rays. Vision ray techniques are known in the vision community to provide image formation models even when conventional camera calibration techniques fail. This work extends the use of vision rays to the area of optical metrology. In contrast to phase measuring deflectometry, this work relies on differential measurements, and hence, the absolute position and orientation between target and camera do not need to be known. This optical configuration significantly reduces the complexity of the reconstruction algorithms. The proposed vision ray metrology system does not require mathematical optimization algorithms for calibration and reconstruction – the vision rays are obtained using a simple 3D fitting of a line.more » « less
-
null (Ed.)Many recent video applications |including autonomous driving, traffic monitoring, drone analytics, large-scale surveillance networks, and virtual reality require reasoning about, combining, and operating over many video streams, each with distinct position and orientation. However, modern video data management systems are largely designed to process individual streams of video data as if they were independent and unrelated. In this paper, we present VisualWorldDB, a vision and an initial architecture for a new type of database management system optimized for multi-video applications. VisualWorldDB ingests video data from many perspectives and makes them queryable as a single multidimensional visual object. It incorporates new techniques for optimizing, executing, and storing multi-perspective video data. Our preliminary results suggest that this approach allows for faster queries and lower storage costs, improving the state of the art for applications that operate over this type of video data.more » « less
-
Recent studies on quadruped robots have focused on either locomotion or mobile manipulation using a robotic arm. However, legged robots can manipulate large objects using non-prehensile manipulation primitives, such as planar pushing, to drive the object to the desired location. This paper presents a novel hierarchical model predictive control (MPC) for contact optimization of the manipulation task. Using two cascading MPCs, we split the loco-manipulation problem into two parts: the first to optimize both contact force and contact location between the robot and the object, and the second to regulate the desired interaction force through the robot locomotion. Our method is successfully validated in both simulation and hardware experiments. While the baseline locomotion MPC fails to follow the desired trajectory of the object, our proposed approach can effectively control both object's position and orientation with minimal tracking error. This capability also allows us to perform obstacle avoidance for both the robot and the object during the loco-manipulation task.more » « less
-
The paper discusses a machine learning vision and nonlinear control approach for autonomous ship landing of vertical flight aircraft without utilizing GPS signal. The central idea involves automating the Navy helicopter ship landing procedure where the pilot utilizes the ship as the visual reference for long-range tracking, but refers to a standardized visual cue installed on most Navy ships called the ”horizon bar” for the final approach and landing phases. This idea is implemented using a uniquely designed nonlinear controller integrated with machine vision. The vision system utilizes machine learning based object detection for long-range ship tracking, and classical computer vision for object detection and the estimation of aircraft relative position and orientation during the final approach and landing phases. The nonlinear controller operates based on the information estimated by the vision system and has demonstrated robust tracking performance even in the presence of uncertainties. The developed autonomous ship landing system is implemented on a quad-rotor vertical take-off and landing (VTOL) capable unmanned aerial vehicle (UAV) equipped with an onboard camera and was demonstrated on a moving deck, which imitates realistic ship deck motions using a Stewart platform and a visual cue equivalent to the horizon bar. Extensive simulations and flight tests are conducted to demonstrate vertical landing safety, tracking capability, and landing accuracy while the deck is in motion.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/https://doi.org/10.1115/DSCC2019-8951
