More Like this

1. Neuromorphic vision and tactile fusion for upper limb prosthesis control

   https://doi.org/10.1109/NER.2019.8716890  

   Hays, Mark; Osborn, Luke; Ghosh, Rohan; Iskarous, Mark; Hunt, Christopher; Thakor, Nitish V. (March 2019, IEEE Transactions on Biomedical Circuits and Systems)

   null (Ed.)

   A major issue with upper limb prostheses is the disconnect between sensory information perceived by the user and the information perceived by the prosthesis. Advances in prosthetic technology introduced tactile information for monitoring grasping activity, but visual information, a vital component in the human sensory system, is still not fully utilized as a form of feedback to the prosthesis. For able-bodied individuals, many of the decisions for grasping or manipulating an object, such as hand orientation and aperture, are made based on visual information before contact with the object. We show that inclusion of neuromorphic visual information, combined with tactile feedback, improves the ability and efficiency of both able-bodied and amputee subjects to pick up and manipulate everyday objects.We discovered that combining both visual and tactile information in a real-time closed loop feedback strategy generally decreased the completion time of a task involving picking up and manipulating objects compared to using a single modality for feedback. While the full benefit of the combined feedback was partially obscured by experimental inaccuracies of the visual classification system, we demonstrate that this fusion of neuromorphic signals from visual and tactile sensors can provide valuable feedback to a prosthetic arm for enhancing real-time function and usability. 

   more » « less
2. First Demonstration of Functional Task Performance Using a Sonomyographic Prosthesis: A Case Study

   https://doi.org/10.3389/fbioe.2022.876836  

   Engdahl, Susannah M.; Acuña, Samuel A.; King, Erica L.; Bashatah, Ahmed; Sikdar, Siddhartha (May 2022, Frontiers in Bioengineering and Biotechnology)

   Ultrasound-based sensing of muscle deformation, known as sonomyography, has shown promise for accurately classifying the intended hand grasps of individuals with upper limb loss in offline settings. Building upon this previous work, we present the first demonstration of real-time prosthetic hand control using sonomyography to perform functional tasks. An individual with congenital bilateral limb absence was fitted with sockets containing a low-profile ultrasound transducer placed over forearm muscle tissue in the residual limbs. A classifier was trained using linear discriminant analysis to recognize ultrasound images of muscle contractions for three discrete hand configurations (rest, tripod grasp, index finger point) under a variety of arm positions designed to cover the reachable workspace. A prosthetic hand mounted to the socket was then controlled using this classifier. Using this real-time sonomyographic control, the participant was able to complete three functional tasks that required selecting different hand grasps in order to grasp and move one-inch wooden blocks over a broad range of arm positions. Additionally, these tests were successfully repeated without retraining the classifier across 3 hours of prosthesis use and following simulated donning and doffing of the socket. This study supports the feasibility of using sonomyography to control upper limb prostheses in real-world applications. 

   more » « less
3. Texture Discrimination with a Soft Biomimetic Finger Using a Flexible Neuromorphic Tactile Sensor Array That Provides Sensory Feedback

   https://doi.org/10.1089/soro.2020.0016  

   Sankar, Sriramana; Balamurugan, Darshini; Brown, Alisa; Ding, Keqin; Xu, Xingyuan; Low, Jin Huat; Yeow, Chen Hua; Thakor, Nitish (September 2020, Soft Robotics)

   null (Ed.)

   The compliant nature of soft fingers allows for safe and dexterous manipulation of objects by humans in an unstructured environment. A soft prosthetic finger design with tactile sensing capabilities for texture discrimination and subsequent sensory stimulation has the potential to create a more natural experience for an amputee. In this work, a pneumatically actuated soft biomimetic finger is integrated with a textile neuromorphic tactile sensor array for a texture discrimination task. The tactile sensor outputs were converted into neuromorphic spike trains, which emulate the firing pattern of biological mechanoreceptors. Spike-based features from each taxel compressed the information and were then used as inputs for the support vector machine classifier to differentiate the textures. Our soft biomimetic finger with neuromorphic encoding was able to achieve an average overall classification accuracy of 99.57% over 16 independent parameters when tested on 13 standardized textured surfaces. The 16 parameters were the combination of 4 angles of flexion of the soft finger and 4 speeds of palpation. To aid in the perception of more natural objects and their manipulation, subjects were provided with transcutaneous electrical nerve stimulation to convey a subset of four textures with varied textural information. Three able-bodied subjects successfully distinguished two or three textures with the applied stimuli. This work paves the way for a more human-like prosthesis through a soft biomimetic finger with texture discrimination capabilities using neuromorphic techniques that provide sensory feedback; furthermore, texture feedback has the potential to enhance user experience when interacting with their surroundings. 

   more » « less
4. Stiffness Perception using Transcutaneous Electrical Stimulation during Active and Passive Prosthetic Control

   https://doi.org/10.1109/EMBC44109.2020.9176078  

   Vargas, Luis; Huang, Helen; Zhu, Yong; Hu, Xiaogang (July 2020, International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC))

   null (Ed.)

   Haptic feedback allows an individual to identify various object properties. In this preliminary study, we determined the performance of stiffness recognition using transcutaneous nerve stimulation when a prosthetic hand was moved passively or was controlled actively by the subjects. Using a 2×8 electrode grid placed along the subject's upper arm, electrical stimulation was delivered to evoke somatotopic sensation along their index finger. Stimulation intensity, i.e. sensation strength, was modulated using the fingertip forces from a sensorized prosthetic hand. Object stiffness was encoded based on the rate of change of the evoked sensation as the prosthesis grasped one of three objects of different stiffness levels. During active control, sensation was modulated in real time as recorded forces were converted to stimulation amplitudes. During passive control, prerecorded force traces were randomly selected from a pool. Our results showed that the accuracy of object stiffness recognition was similar in both active and passive conditions. A slightly lower accuracy was observed during active control in one subject, which indicated that the sensorimotor integration processes could affect haptic perception for some users. 

   more » « less
5. Multichannel Sensorimotor Integration with a Dexterous Artificial Hand

   https://doi.org/10.3390/robotics13070097  

   Abd, Moaed A; Engeberg, Erik D (July 2024, Robotics)

   People use their hands for intricate tasks like playing musical instruments, employing myriad touch sensations to inform motor control. In contrast, current prosthetic hands lack comprehensive haptic feedback and exhibit rudimentary multitasking functionality. Limited research has explored the potential of upper limb amputees to feel, perceive, and respond to multiple channels of simultaneously activated haptic feedback to concurrently control the individual fingers of dexterous prosthetic hands. This study introduces a novel control architecture for three amputees and nine additional subjects to concurrently control individual fingers of an artificial hand using two channels of context-specific haptic feedback. Artificial neural networks (ANNs) recognize subjects’ electromyogram (EMG) patterns governing the artificial hand controller. ANNs also classify the directions objects slip across tactile sensors on the robotic fingertips, which are encoded via the vibration frequency of wearable vibrotactile actuators. Subjects implement control strategies with each finger simultaneously to prevent or permit slip as desired, achieving a 94.49% ± 8.79% overall success rate. Although no statistically significant difference exists between amputees’ and non-amputees’ success rates, amputees require more time to respond to simultaneous haptic feedback signals, suggesting a higher cognitive load. Nevertheless, amputees can accurately interpret multiple channels of nuanced haptic feedback to concurrently control individual robotic fingers, addressing the challenge of multitasking with dexterous prosthetic hands. 

   more » « less