During in-hand manipulation, robots must be able to continuously estimate the pose of the object in order to generate appropriate control actions. The performance of algorithms for pose estimation hinges on the robot's sensors being able to detect discriminative geometric object features, but previous sensing modalities are unable to make such measurements robustly. The robot's fingers can occlude the view of environment- or robot-mounted image sensors, and tactile sensors can only measure at the local areas of contact. Motivated by fingertip-embedded proximity sensors' robustness to occlusion and ability to measure beyond the local areas of contact, we present the first evaluation of proximity sensor based pose estimation for in-hand manipulation. We develop a novel two-fingered hand with fingertip-embedded optical time-of-flight proximity sensors as a testbed for pose estimation during planar in-hand manipulation. Here, the in-hand manipulation task consists of the robot moving a cylindrical object from one end of its workspace to the other. We demonstrate, with statistical significance, that proximity-sensor based pose estimation via particle filtering during in-hand manipulation: a) exhibits 50% lower average pose error than a tactile-sensor based baseline; b) empowers a model predictive controller to achieve 30% lower final positioning error compared to when using tactile-sensor based pose estimates.
more »
« less
A Simple, Inexpensive, Wearable Glove with Hybrid Resistive‐Pressure Sensors for Computational Sensing, Proprioception, and Task Identification
Wearable devices have many applications ranging from health analytics to virtual and mixed reality interaction, to industrial training. For wearable devices to be practical, they must be responsive, deformable to fit the wearer, and robust to the user's range of motion. Signals produced by the wearable must also be informative enough to infer the precise physical state or activity of the user. Herein, a fully soft, wearable glove is developed, which is capable of real‐time hand pose reconstruction, environment sensing, and task classification. The design is easy to fabricate using low cost, commercial off‐the‐shelf items in a manner that is amenable to automated manufacturing. To realize such capabilities, resisitive and fluidic sensing technologies with machine learning neural architectures are merged. The glove is formed from a conductive knit which is strain sensitive, providing information through a network of resistance measurements. Fluidic sensing captured via pressure changes in fibrous sewn‐in flexible tubes, measuring interactions with the environment. The system can reconstruct user hand pose and identify sensory inputs such as holding force, object temperature, conductability, material stiffness, and user heart rate, all with high accuracy. The ability to identify complex environmentally dependent tasks, including held object identification and handwriting recognition is demonstrated.
- NSF-PAR ID:
- 10158466
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Advanced Intelligent Systems
- Volume:
- 2
- Issue:
- 6
- ISSN:
- 2640-4567
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
In soft devices, complex actuation sequences and precise force control typically require hard electronic valves and microcontrollers. Existing designs for entirely soft pneumatic control systems are capable of either digital or analog operation, but not both, and are limited by speed of actuation, range of pressure, time required for fabrication, or loss of power through pull-down resistors. Using the nonlinear mechanics intrinsic to structures composed of soft materials—in this case, by leveraging membrane inversion and tube kinking—two modular soft components are developed: a piston actuator and a bistable pneumatic switch. These two components combine to create valves capable of analog pressure regulation, simplified digital logic, controlled oscillation, nonvolatile memory storage, linear actuation, and interfacing with human users in both digital and analog formats. Three demonstrations showcase the capabilities of systems constructed from these valves: 1) a wearable glove capable of analog control of a soft artificial robotic hand based on input from a human user’s fingers, 2) a human-controlled cushion matrix designed for use in medical care, and 3) an untethered robot which travels a distance dynamically programmed at the time of operation to retrieve an object. This work illustrates pathways for complementary digital and analog control of soft robots using a unified valve design.more » « less
-
Locus is a NIME designed specifically for an interactive, immersive high density loudspeaker array environment. The system is based on a pointing mechanism to interact with a sound scene comprising 128 speakers. Users can point anywhere to interact with the system, and the spatial interaction utilizes motion capture, so it does not require a screen. Instead it is completely controlled via hand gestures using a glove that is populated with motion-tracking markers. The main purpose of this system is to offer intuitive physical interaction with the perimeter based spatial sound sources. Further, its goal is to minimize user-worn technology and thereby enhance freedom of motion by utilizing environmental sensing devices, such as motion capture cameras or infrared sensors. The ensuing creativity enabling technology is applicable to a broad array of possible scenarios, from researching limits of human spatial hearing perception to facilitating learning and artistic performances, including dance. Below we describe our NIME design and implementation, its preliminary assessment, and offer a Unity-based toolkit to facilitate its broader deployment and adoption.more » « less
-
more » « less
Abstract Highly stretchable fiber sensors have attracted significant interest recently due to their applications in wearable electronics, human–machine interfaces, and biomedical implantable devices. Here, a scalable approach for fabricating stretchable multifunctional electrical and optical fiber sensors using a thermal drawing process is reported. The fiber sensors can sustain at least 580% strain and up to 750% strain with a helix structure. The electrical fiber sensor simultaneously exhibits ultrahigh stretchability (400%), high gauge factors (≈1960), and excellent durability during 1000 stretching and bending cycles. It is also shown that the stretchable step‐index optical fibers facilitate detection of bending and stretching deformation through changes in the light transmission. By combining both electrical and optical detection schemes, multifunctional fibers can be used for quantifying and distinguishing multimodal deformations such as bending and stretching. The fibers’ utility and functionality in sensing and control applications are demonstrated in a smart glove for controlling a virtual hand model, a wrist brace for wrist motion tracking, fiber meshes for strain mapping, and real‐time monitoring of multiaxial expansion and shrinkage of porcine bladders. These results demonstrate that the fiber sensors can be promising candidates for smart textiles, robotics, prosthetics, and biomedical implantable devices.
-
more » « less
Haptic devices are in general more adept at mimicking the bulk properties of materials than they are at mimicking the surface properties. Herein, a haptic glove is described which is capable of producing sensations reminiscent of three types of near‐surface properties: hardness, temperature, and roughness. To accomplish this mixed mode of stimulation, three types of haptic actuators are combined: vibrotactile motors, thermoelectric devices, and electrotactile electrodes made from a stretchable conductive polymer synthesized in the laboratory. This polymer consists of a stretchable polyanion which serves as a scaffold for the polymerization of poly(3,4‐ethylenedioxythiophene). The scaffold is synthesized using controlled radical polymerization to afford material of low dispersity, relatively high conductivity, and low impedance relative to metals. The glove is equipped with flex sensors to make it possible to control a robotic hand and a hand in virtual reality (VR). In psychophysical experiments, human participants are able to discern combinations of electrotactile, vibrotactile, and thermal stimulation in VR. Participants trained to associate these sensations with roughness, hardness, and temperature have an overall accuracy of 98%, whereas untrained participants have an accuracy of 85%. Sensations can similarly be conveyed using a robotic hand equipped with sensors for pressure and temperature.
