Image sensors with internal computing capability enable in-sensor computing that can significantly reduce the communication latency and power consumption for machine vision in distributed systems and robotics. Two-dimensional semiconductors have many advantages in realizing such intelligent vision sensors because of their tunable electrical and optical properties and amenability for heterogeneous integration. Here, we report a multifunctional infrared image sensor based on an array of black phosphorous programmable phototransistors (bP-PPT). By controlling the stored charges in the gate dielectric layers electrically and optically, the bP-PPT’s electrical conductance and photoresponsivity can be locally or remotely programmed with 5-bit precision to implement an in-sensor convolutional neural network (CNN). The sensor array can receive optical images transmitted over a broad spectral range in the infrared and perform inference computation to process and recognize the images with 92% accuracy. The demonstrated bP image sensor array can be scaled up to build a more complex vision-sensory neural network, which will find many promising applications for distributed and remote multispectral sensing.
Optical lace for synthetic afferent neural networks
Whereas vision dominates sensing in robots, animals with limited vision deftly navigate their environment using other forms of perception, such as touch. Efforts have been made to apply artificial skins with tactile sensing to robots for similarly sophisticated mobile and manipulative skills. The ability to functionally mimic the afferent sensory neural network, required for distributed sensing and communication networks throughout the body, is still missing. This limitation is partially due to the lack of cointegration of the mechanosensors in the body of the robot. Here, lacings of stretchable optical fibers distributed throughout 3D-printed elastomer frameworks created a cointegrated body, sensing, and communication network. This soft, functional structure could localize deformation with submillimeter positional accuracy (error of 0.71 millimeter) and sub-Newton force resolution (~0.3 newton).
more »
« less
- Award ID(s):
- 1719875
- NSF-PAR ID:
- 10149143
- Date Published:
- Journal Name:
- Science Robotics
- Volume:
- 4
- Issue:
- 34
- ISSN:
- 2470-9476
- Page Range / eLocation ID:
- eaaw6304
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract -
Multi-robot cooperative control has been extensively studied using model-based distributed control methods. However, such control methods rely on sensing and perception modules in a sequential pipeline design, and the separation of perception and controls may cause processing latencies and compounding errors that affect control performance. End-to-end learning overcomes this limitation by implementing direct learning from onboard sensing data, with control commands output to the robots. Challenges exist in end-to-end learning for multi-robot cooperative control, and previous results are not scalable. We propose in this article a novel decentralized cooperative control method for multi-robot formations using deep neural networks, in which inter-robot communication is modeled by a graph neural network (GNN). Our method takes LiDAR sensor data as input, and the control policy is learned from demonstrations that are provided by an expert controller for decentralized formation control. Although it is trained with a fixed number of robots, the learned control policy is scalable. Evaluation in a robot simulator demonstrates the triangular formation behavior of multi-robot teams of different sizes under the learned control policy.more » « less
-
Soft robots actuate themselves and their world through induced pressure and strain, and can often sense these quantities as well. We hypothesize that coordination in a tightly coupled collective of soft robots can be achieved with purely proprioceptive sensing and no direct communication. In this paper, we target a platform of soft pneumatic modules capable of sensing strain on their perimeter, with the goal of using only the robots' own soft actuators and sensors as a medium for distributed coordination. However, methods for modelling, sensing, and controlling strain in such soft robot collectives are not well understood. To address this challenge, we introduce and validate a computationally efficient spring-based model for two-dimensional sheets of soft pneumatic robots. We then translate a classical consensus algorithm to use only proprioceptive data, test in simulation, and show that due to the physical coupling between robots we can achieve consensus-like coordination. We discuss the unique challenges of strain sensors and next steps to bringing these findings to hardware. These findings have promising potential for smart materials and large-scale collectives, because they omit the need for additional communication infrastructure to support coordination.more » « less
-
Robotic automation of construction tasks is a growing area of research. For robots to successfully operate in a construction environment, sensing technology must be developed which allows for accurate detection of site geometry in a wide range of conditions. Much of the existing body of research on computer vision systems for construction automation focuses on pick-and-place operations such as stacking blocks or placing masonry elements. Very little research has focused on framing and related tasks. The research presented here aims to address this gap by designing and implementing computer vision algorithms for detection and measurement of building framing elements and testing those algorithms using realistic framing structures. These algorithms allow for a stationary RGB-D camera to accurately detect, identify, and measure the geometry of framing elements in a construction environment and match the detected geometry to provided building information modeling (BIM) data. The algorithms reduce identified framing elements to a simplified 3D geometric model, which allows for robust and accurate measurement and comparison with BIM data. This data can then be used to direct operations of construction robotic systems or other machines/equipment. The proposed algorithms were tested in a laboratory setting using an Intel RealSense D455 RGB-D camera, and initial results indicate that the system is capable of measuring the geometry of timber-frame structures with accuracy on the order of a few centimeters.more » « less
-
We study two multi-robot assignment problems for multi-target tracking. We consider distributed approaches in order to deal with limited sensing and communication ranges. We seek to simultaneously assign trajectories and targets to the robots. Our focus is on \emph{local} algorithms that achieve performance close to the optimal algorithms with limited communication. We show how to use a local algorithm that guarantees a bounded approximate solution within $\mathcal{O}(h\log{1/\epsilon})$ communication rounds. We compare with a greedy approach that achieves a $2$--approximation in as many rounds as the number of robots. Simulation results show that the local algorithm is an effective solution to the assignment problem.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1126/scirobotics.aaw6304
