An official website of the United States government
Abstract Image sensors with internal computing capability enable in-sensor computing that can significantly reduce the communication latency and power consumption for machine vision in distributed systems and robotics. Two-dimensional semiconductors have many advantages in realizing such intelligent vision sensors because of their tunable electrical and optical properties and amenability for heterogeneous integration. Here, we report a multifunctional infrared image sensor based on an array of black phosphorous programmable phototransistors (bP-PPT). By controlling the stored charges in the gate dielectric layers electrically and optically, the bP-PPT’s electrical conductance and photoresponsivity can be locally or remotely programmed with 5-bit precision to implement an in-sensor convolutional neural network (CNN). The sensor array can receive optical images transmitted over a broad spectral range in the infrared and perform inference computation to process and recognize the images with 92% accuracy. The demonstrated bP image sensor array can be scaled up to build a more complex vision-sensory neural network, which will find many promising applications for distributed and remote multispectral sensing.
more »
« less
Optical lace for synthetic afferent neural networks
Whereas vision dominates sensing in robots, animals with limited vision deftly navigate their environment using other forms of perception, such as touch. Efforts have been made to apply artificial skins with tactile sensing to robots for similarly sophisticated mobile and manipulative skills. The ability to functionally mimic the afferent sensory neural network, required for distributed sensing and communication networks throughout the body, is still missing. This limitation is partially due to the lack of cointegration of the mechanosensors in the body of the robot. Here, lacings of stretchable optical fibers distributed throughout 3D-printed elastomer frameworks created a cointegrated body, sensing, and communication network. This soft, functional structure could localize deformation with submillimeter positional accuracy (error of 0.71 millimeter) and sub-Newton force resolution (~0.3 newton).
more »
« less
- Award ID(s):
- 1719875
- PAR ID:
- 10149143
- Date Published:
- Journal Name:
- Science Robotics
- Volume:
- 4
- Issue:
- 34
- ISSN:
- 2470-9476
- Page Range / eLocation ID:
- eaaw6304
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Soft robots actuate themselves and their world through induced pressure and strain, and can often sense these quantities as well. We hypothesize that coordination in a tightly coupled collective of soft robots can be achieved with purely proprioceptive sensing and no direct communication. In this paper, we target a platform of soft pneumatic modules capable of sensing strain on their perimeter, with the goal of using only the robots' own soft actuators and sensors as a medium for distributed coordination. However, methods for modelling, sensing, and controlling strain in such soft robot collectives are not well understood. To address this challenge, we introduce and validate a computationally efficient spring-based model for two-dimensional sheets of soft pneumatic robots. We then translate a classical consensus algorithm to use only proprioceptive data, test in simulation, and show that due to the physical coupling between robots we can achieve consensus-like coordination. We discuss the unique challenges of strain sensors and next steps to bringing these findings to hardware. These findings have promising potential for smart materials and large-scale collectives, because they omit the need for additional communication infrastructure to support coordination.more » « less
-
null (Ed.)Due to their ability to move without sliding relative to their environment, soft growing robots are attractive for deploying distributed sensor networks in confined spaces. Sensing of the state of such robots would add to their capabilities as human-safe, adaptable manipulators. However, incorporation of distributed sensors onto soft growing robots is challenging because it requires an interface between stiff and soft materials, and the sensor network needs to undergo significant strain. In this work, we present a method for adding sensors to soft growing robots that uses flexible printed circuit boards with self-contained units of microcontrollers and sensors encased in a laminate armor that protects them from unsafe curvatures. We demonstrate the ability of this system to relay directional temperature and humidity information in hard-to-access spaces. We also demonstrate and characterize a method for sensing the growing robot shape using inertial measurement units deployed along its length, and develop a mathematical model to predict its accuracy. This work advances the capabilities of soft growing robots, as well as the field of soft robot sensing.more » « less
-
We study two multi-robot assignment problems for multi-target tracking. We consider distributed approaches in order to deal with limited sensing and communication ranges. We seek to simultaneously assign trajectories and targets to the robots. Our focus is on \emph{local} algorithms that achieve performance close to the optimal algorithms with limited communication. We show how to use a local algorithm that guarantees a bounded approximate solution within $$\mathcal{O}(h\log{1/\epsilon})$$ communication rounds. We compare with a greedy approach that achieves a $$2$$--approximation in as many rounds as the number of robots. Simulation results show that the local algorithm is an effective solution to the assignment problem.more » « less
-
Multi-robot cooperative control has been extensively studied using model-based distributed control methods. However, such control methods rely on sensing and perception modules in a sequential pipeline design, and the separation of perception and controls may cause processing latencies and compounding errors that affect control performance. End-to-end learning overcomes this limitation by implementing direct learning from onboard sensing data, with control commands output to the robots. Challenges exist in end-to-end learning for multi-robot cooperative control, and previous results are not scalable. We propose in this article a novel decentralized cooperative control method for multi-robot formations using deep neural networks, in which inter-robot communication is modeled by a graph neural network (GNN). Our method takes LiDAR sensor data as input, and the control policy is learned from demonstrations that are provided by an expert controller for decentralized formation control. Although it is trained with a fixed number of robots, the learned control policy is scalable. Evaluation in a robot simulator demonstrates the triangular formation behavior of multi-robot teams of different sizes under the learned control policy.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1126/scirobotics.aaw6304
