Upper-limb amputees abandon prostheses at a high rate [1], [2], in part due to unintuitive and poor control [3]. One solution to improving prosthetic control is to automate the prosthesis using embedded sensors to aid in conforming the grasp to an object [4]- [7]. However, to date, autonomous prostheses have been designed to accomplish a specific task and are not necessarily adaptable to general use. Semiautonomous hands have been demonstrated to outperform human control when handling fragile objects [4] and can increase prosthesis contact area to a given object [6]. Although the benefits of shared control have been evaluated with respect to task performance, the impact of shared control on a user's physiological state (i.e., cognitive and physical effort) has not yet been evaluated.
In this study, we introduce a novel autonomous controller that predicts an object's distance from the fingers using embedded proximity and pressure sensors in the fingertips. We also introduce a novel shared-control algorithm that attenuates a user's EMG output based on the prediction of the autonomous controller. We validate the shared-control algorithm using a holding task, in which participants attempt to continuously hold an object while completing a secondary detection response task (DRT). We show that shared control improves grasp security and decreases the physical and cognitive burden on a user compared to that of EMG control alone. Making a hand more dexterous, while increasing its ease of use, may ultimately decrease prosthetic hand abandonment and increase patient quality of life.
A left-handed TASKA hand (TASKA, Christchurch, New Zealand) was retrofitted with fingers containing pressure and infrared proximity sensors (Point Designs LLC, Lafayette, CO, USA) [4], [8]. A multilayer perceptron (MLP) was designed with 10 hidden layers to predict object distance from the infrared and pressure sensors. Training data was collected by oscillating the thumb towards and away from white 3Dprinted objects in the shape of a cylinder, cube, and cone. The ground-truth distance from the object for the training data was computed by measuring the kinematic position of the finger upon initial contact with the object (i.e., when pressure was first recorded), and then retroactively using the difference between that kinematic position and the current position to determine the current distance to the object. Example training data can be seen in Fig. 1.
For training the MLP, 70% of the total data was randomly selected as the training data, 15% was selected for validation data, and the remaining 15% was used for testing. During run-time, the kinematic prediction from the autonomous controller was computed as the sum of the current position of the thumb and the predicted distance to the object.
Three healthy intact participants (21.67±0.58 years old; 33% female) were recruited for this study. All the participants were right-hand dominant. None of the participants had prior experience with myoelectric prostheses. Informed consent and experiment protocols were carried out in accordance with the University of Utah Institutional Review Board.
Surface EMG from the participants was collected using a custom EMG sleeve [9]. EMG was sampled at 1 kHz and filtered using the Summit Neural Interface Processor (Ripple Neuro Med LLC) as described in [10]. EMG features used for estimating motor intent consisted of the 300-ms smoothed mean absolute value (MAV) on 528 channels (32 single-ended channels and 496 calculated differential pairs) calculated at 30 Hz, as described in [10]. The embedded fingertip sensor readings were sampled at 30 Hz and passed through a median filter with a time window of 10 samples. Sensor drift was removed from the pressure sensors using a high-pass filter that was toggled on and off by the sensor crossing a threshold.
The human position goal, 𝑢 , was computed using a modified Kalman filter (MKF) [11]. The MKF was fit to surface EMG (sEMG) data collected during five preprogrammed trials of full-flexion pinch between the thumb and index fingers and five trials of full extension of the same digits. Both control techniques only affected one degree of freedom in the form of a pinch grip between the thumb and index finger. To form the pinch grip, the index finger was set to a constant position while the thumb was flexed from 0 (felly extended) to 1 (fully flexed). Control of the hand was shared between the human position goal from the MKF, denoted 𝑢 , and the machine position goal as computed by the MLP, 𝑢 . Both human and machine control are normalized to a range of zero to one. The shared goal, 𝑢 , was then computed as the following:
This effectively attenuates the human's control of the hand in proportion to the remaining range of the digit. A minimum threshold was set such that the thumb would only move if the infrared sensor rose above a given value. Once contact was detected, the machine predictions were frozen until the pinch was released by the human controller. An sEMG toggle was used to trigger the shared-control algorithm [4]. The sEMG toggle switched the user between shared control and human control based on the output of the MKF. For example, if the MKF predicted that the human was attempting to extend to a position The thumb was programmed to approach each object 20 times. Each approach consisted of incrementally moving the thumb forward, then waiting approximately 3 s before moving the thumb forward again. After contact, the thumb retracted from the object in a similar fashion. The autonomous controller is more accurate when the object is closer. That is, the predicted distance (red) more closely aligns with the ground-truth measured distance (black) as the distance becomes smaller.
greater than 50% of the extension range, the output would be switched to the human-only control scheme. To return to the machine-control state the user had to flex 1% of the flexion range. This aided the participant in releasing the object without altering the evaluation of the shared controller's performance during object grasping.
The participants donned the prosthesis using a custom bypass socket [11]. The participants then completed a holding task in which they were instructed to use the prosthesis to pick up a white 3Dprinted cube and hold it for two minutes without dropping it. If the object was dropped the participant was instructed to pick it up as quickly as possible and continue the task. The number of times the cube was dropped during a trial was recorded as a measure of grip security. The participant completed the holding task for four trials using human control and shared control in a pseudo-randomized counter-balanced format. While completing the holding task, the participant also simultaneously completed a tactile detection-response task (DRT) to measure their cognitive load [12]. The DRT requires the participant to push a button in response to a small vibrating motor on their collar bone. Both the response rate (i.e., how often they respond to the vibratory stimuli) and response time (i.e., how long it takes to press the button after a vibratory stimuli) are used as direct measures of cognitive load. The EMG MAV on the 32 electrodes was also recorded during the hold task as an indicator of the physical effort (muscle activity) needed to complete the task. Data was aggregated across all participants and screened for normality. Pairwise comparisons between the human control and shared control were then performed using a paired t-test for parametric data or Wilcoxon rank-sum test for non-parametric data. All values are reported and shown as mean ± standard deviation.
Combining human and machine control allowed the participants to hold the object more reliably than when only human control was used. Only one drop was recorded under shared control, whereas up to eight drops were recorded under human control, with a median of three drops per trial for human control (Fig. 2A). Shared control resulted in no significant difference in the response rate of the DRT compared to human control (63.19 ± 13.87% vs 63.58 ± 11.43%, respectively). However, participants had significantly faster response times to the DRT when using shared control, indicating less cognitive demand (0.719 ± 0.056 s vs 0.444 ± 0.022 s, respectively; p < 0.001, Wilcoxon rank-sum test; Fig. 2B). Furthermore, participants had significantly lower EMG activity with shared control than with human control (24.10 ± 10.48 vs 47.13 ± 17.53; p < 0.001, paired t-test; Fig. 2C).
Error in human-control is dampened in sharedcontrol Using the shared-control algorithm, the participants were able to hold an object more reliably while using less physical and cognitive effort. Example traces of the human control, machine control, shared control, and contact position from a single participant are shown in Fig. 3. The thumb made contact with the cube at a position of 0.3, and, although the machine did not give a perfect prediction, the thumb moves to a position near this goal. By starting closer to the object, and adding the two control techniques together, the shared algorithm is able to decrease the physical effort of the user. The human control also has a high degree of variability, and variations in the human goal below the point of object contact, would have resulted in multiple drops of the object. By scaling the human control based on the machine prediction, we effectively decreased the variability in the kinematic position. This in turn results in greater performance and less physical effort, which then allow the user to focus on other tasks.
In this study we demonstrated that sharing control between a human EMG decoder and an autonomous controller can benefit the user by decreasing their physical and mental effort while increasing their grip security on a given object. Decreasing the complexity and effort of use of upper-limb prostheses may ultimately lead to a reduction in prosthesis abandonment. Being able to hold an object securely without considerable mental or physical strain comes naturally to someone with an intact hand, but is still a challenge for users of upper-limb prostheses.
In future iterations of this design, we intend to scale the human control logarithmically rather than linearly. Scaling the EMG decoder output in a logarithmic fashion follows a biomimetic paradigm and would grant users finer levels of control with smaller objects. We will also train the MLP predictions of object distance using continuous data as opposed to the discrete steps used for this work. The MLP estimates were found to overestimate the object distance when the digit was in motion. Providing a more accurate machine estimate would lead to a shared-control algorithm capable of handling fragile objects. We also intend to expand the shared-control approach presented here to multiple DOFs. We anticipate that shared control implemented across all the digits will further increase the benefits seen here when working with complex objects by reducing the variation in grasping force between digits.