We propose a novel framework for inferring human skill-level and dynamically adjusting the assistance based on the inferred skill-level in complex dynamic control tasks. Along with the development of state-of-the-art artiicial intelligence technology, questions have been raised about the role, ability, and necessity of humans. The irst question about the motivation of human training might be "Why do we have to train humans in the era of artiicial intelligence?" We can irst consider the technical aspects of this question. For instance, the ever-increasing demand for well-trained human pilots in the ield of urban air mobility (UAM) has received a lot of attention recently. To promote the safety and eiciency of UAM operations in urban areas, the role of human pilots is actively being discussed [41,46]. Some tasks such as detection and avoidance, emergency procedures, communication, takeof and landings, and planning and decision-making still require well-trained human pilots in speciic context [5]. Regulatory and social aspects also provide further context. For example, the Federal Aviation Administration (FAA) pointed out that a human pilot is ultimately responsible for the operation and safety during unmanned aerial vehicle (UAV) light [18]. A human supervisor with responsibility and authority must have the ability to ensure safe operation [19]. Therefore, well-trained human operators are required, and assisting human training is a signiicant beneit.
Since human training is important for safe and eicient system operations, the next question to ask is "How can we train humans eiciently?" In daily life, humans learn motor skills for vehicle control through repetitive practice which induces a physical change in human brains [28]. To shorten the repetitive learning process, humans have sought assistance from experts. This can be analogous to the parent-child metaphor [40]. When children learn how to ride a bike, an adult assists to balance that bike. In general, the adult lets the children take control of the bike if it is in a safe operational range so that the children can practice balancing themselves. One important point is that assistance should only be provided as needed. The goal of assistance is to help novices perform their task independently, so the assistant should not have full control of the target system. As the novice's skill-level improves, the assistant needs to reduce intervention so that the novice can practice the task.
The primary motivation of this paper is that if human-human (parent-child or expert-novice) interaction can expedite human learning for dynamic control tasks, then autonomy can serve as a tutor to build an eicient human-machine interaction (HMI). The expert plays an important role in human-human interaction, but the number of experts is limited. In our proposed approach, autonomy models the performance of experts to build an expert model. The expert models enable autonomy to provide appropriate assistance to novices, as experts do. The past and current performance of novices can be utilized by autonomy to infer their skill-levels online. Then, autonomy can personalize the level of assistance based on the inferred skill-level. This approach reduces the demand for experts for human training while giving novices rapid access to quality training. In the following section, we provide related work. Various HMI approaches have been developed to assist human training such as modulating practice conditions, robot-assisted motor skill rehabilitation, and shared control applications. The skill-level inference methods have been proposed in many applications since they may not be straightforward in complex scenarios.
1.1.1 Modulating Practice Conditions for Human Training. Efective practice conditions have been widely explored, since practice is generally considered the most important factor for motor learning and human training. The optimal challenge point (OCP) framework formulates the learning rate as a function of two variables, skill-level and task diiculty [23]. Hence, the learning rate can be optimized by modulating the task diiculty based on the skill-level of a learner. Note that skill refers to sensory-motor performance during tasks based on an intention [51]. The OCP framework suggests that as skill-level improves, the task should be more diicult to obtain further learning beneits [60]. In human-computer interaction, the dynamic diiculty adjustment (DDA) utilizes the OCP framework. The DDA has been adopted in computer games, healthcare, and education to ofer appropriate diiculty to learners based on their dexterity, adapting ability, and personal characteristics [64]. The intelligent tutoring system is another approach to utilize the OCP framework [29]. The learners can engage in educational tasks with the intelligent tutor, which provides appropriate interventions to expedite the learning process based on the learner's previous performance [62].
However, inferring skill-level and adjusting task diiculty are task-dependent and may not be straightforward for complex dynamic control tasks. For instance, education games can assess the skill-level using a testing score and provide easy or diicult problems based on the skill-level. On the other hand, the testing score and pre-deined easy or diicult practice conditions may not be available in complex scenarios such as operating UAM vehicles.
1.1.2 Robot-Assisted Motor Skill Rehabilitation. Robotic exoskeleton systems have been used for motor learning and rehabilitation. Recent experimental studies have revealed that the motor learning rate can be improved or hindered by adjusting the assistance, feedback, and training tasks based on the current status of trainees [3,45,48,52,53]. In various rehabilitation studies, skill-level-based haptic guidance and error augmentation techniques have been demonstrated [3,54]. Haptic guidance refers to providing appropriate torque to assist the learners. The error augmentation imposes artiicial error to intentionally ofer variations in training environments [42,58]. Their studies showed that providing haptic assistance is generally beneicial for more impaired learners and error augmentation is efective for less impaired learners. Their ideas and some of their results coincide with the OCP framework [23]. One study addressed the most similar approach to this paper, called the minimal assist-as-needed controller [48]. As its name implies, the minimal assist-as-needed controller modulates the assistance from a robotic algorithm which is afected by the skill-level of the learner and a pre-deined error bound.
However, there are some concerns over the existing work. The existing algorithms for rehabilitation devices only have considered limited degrees of freedom, and they may not be applicable to high-dimensional tasks. Furthermore, the existing algorithms usually employ simple reference trajectories for human training, for instance, straight lines. In complex high-dimensional scenarios, reference trajectories should consider the dynamics of the target systems (e.g., vehicle dynamics in UAM scenarios). Also, reference trajectories might be situation dependent (e.g., the UAM vehicle should slow down near the landing area). In this regard, data-driven methods could be better options in complex cases (e.g., learning reference trajectories from expert demonstrations).
1.1.3 Shared Control. Shared control has no global deinition, but the method of controlling a system by a convex combination of independent control inputs from the human and autonomy has been widely used [40]. In general, autonomy assists humans to achieve goals by making tasks easier [16]. Popular applications include automated vehicles [1], UAVs [33], robot-assisted surgery [57], and manufacturing robots [44]. For enhancing the human learning rate, some applications utilized shared control [1]. Haptic shared control schemes use force or torque inputs from humans to infer their intentions [31]. Interestingly, haptic shared control with ixed gain (i.e., non-adaptive assistance) can decrease the human learning rate [36]. The researchers claimed that ill-designed shared controls may cause humans to adapt to a new task transformed by the shared controller rather than to the actual task [47]. To realize a well-designed shared control scheme for human training, a progressive shared control was proposed [35]. The progressive shared controller adjusts its assistance based on the skill-level of the learners. Physically decoupled shared control schemes (i.e., non-haptic shared controllers [1]) have also been studied for human training. Visual and auditory feedback can be provided [45,52] or a shared controller can be used to provide assistant inputs obtained from expert demonstrations based on inferred skill-level of the learners [7,9].
Studies on adaptive shared control provide a solid foundation for human training, but their target systems are relatively simple, i.e., controlling mass points in two-dimensional space [31,35,49]. Thus, their skill-level inference approaches need to be extended in complex dynamic control tasks. More complex scenarios, such as aerial vehicle landing scenarios, can be found in [7,9], but only deterministic skill-level inference approaches are available. Those approaches cannot address uncertainties in human behaviors [56]. More details of the skill-level inference will be followed in Section 1.1.4.
Inference. Skill-level inference techniques aim to ind quantiiable indicators of humans' skilllevel by observing human motion patterns [17]. If a given task is simple, an intuitive indicator such as a trajectory tracking error can be used. On the other hand, for complex tasks such as surgery using robotic arms, human expert demonstrations can be utilized to identify the performance metric. In [20], the authors extracted comparison criteria for novice and expert performance in robot-controlled surgery. However, their comparison criteria depend on tasks, and they cannot perform the skill-level inference online (i.e., the skill-level is inferred after each trial). In [57], a convolution neural network technique was used to divide the skill-level into two categories (adept or rusty). This approach requires labeled human demonstrations for each skill-level. In robotics, human skill-levels were successfully quantiied in a complex task and human-exoskeleton interactive system but pre-deined performance features are required [21,22]. Some machine learning techniques have been investigated to infer the skill-level [38]. They utilize human demonstrations to model human behavioral patterns. For instance, inverse optimal control and inverse reinforcement learning were used to compare the given demonstrations from novices and experts [8,9,38]. These methods can infer and quantify the skill-level as a continuous variable. Nevertheless, these techniques cannot account for the variability of human behaviors. The variability refers to the intrinsic uncertainties that can be observed in human behaviors [56]. For instance, when an expert performs a task multiple times, resultant demonstrations might not be perfectly consistent due to variability. In our pilot study, experts tend to control the multi-rotor vehicle very accurately near an obstacle or waypoint, but relatively larger variations are observed in the open air. Similar observations can be found in civil aircraft pilots who are highly qualiied [15]. Even if a novice deviates from the expert nominal trajectory, if there is a contextual situation in which the expert frequently deviates from the nominal trajectory, we do not want to underestimate the skill-level of the novice. These complexities may induce additional problems in assessing skill-level.
Our main contributions are three folds: 1) a novel stochastic-skill-level (SSL) inference method is developed to estimate the SSL of each human in complex dynamic control tasks. The proposed inference method can account for not only the nominal human behavior but also the variability in human behaviors. The SSL can be assessed online; 2) the SSL-based shared control scheme is proposed to adapt the assistance based on the inferred SSL in complex dynamic control tasks. It can personalize the assistance according to the inferred SSL; and 3) a human subject experiment is conducted to demonstrate the proposed framework.
More details are provided to contextualize the contributions. The proposed shared control approach enables us to apply the OCP framework in complex dynamic control tasks. To improve the human learning rate in the dynamic control task, the diiculty of the task is modulated to prevent frustration from tasks being too hard or boredom from tasks being too easy [64]. Consequently, two methods are required to facilitate task diiculty modulation: skill-level inference and skill-level-based task modulation. Some related applications have been investigated in robot-assisted motor skill rehabilitation [3,48,53], robot-assisted surgery [20,57], and shared control [35,49,61]. However, their target systems are relatively simple in terms of the system dynamics (e.g., mass points in the two-dimensional space with spring-damper systems) or the degree of freedom (DOF) (e.g., single-axis control of rehabilitation devices). Also, their tasks are simple (e.g., tracking a pre-deined path), and applied skill-level inference methods are only suitable for certain simple tasks (details in Section 1.1.4). Several concerns could arise if the target system and tasks are complex. In our case, the target system is a multi-rotor system with the nonlinear 6-DOF dynamics. The task is lying through the urban area by safely passing through multiple waypoints, which requires more sophisticated path planning rather than simply connecting waypoints by curved lines since the vehicle dynamics adds constraints (e.g., obstacle avoidance maneuvers and acceleration constraints). In the worst case, the waypoints may not be available, but they need to be derived from expert demonstrations. Even if the optimal trajectory is available, it might not be preferred by humans due to the perceived risk [32] (e.g., human pilots do not want to ly close to buildings although autonomy says it is optimal). Thus, a data-driven method can be applied to eiciently generate the nominal trajectory from the expert demonstrations. Furthermore, multiple expert demonstrations also include the variability [56] which represents the inherent uncertainties in human behaviors. For instance, pilots tend to control the vehicle very accurately near obstacles and waypoints, but positional variations in the open air are much larger [15]. The proposed SSL inference method can explicitly account for the variability to identify acceptable or unacceptable deviations from the nominal trajectory. Then, the proposed shared control framework can adjust the assistant scheme based on the inferred SSL. Our human subject study shows how the proposed method works in an illustrative scenario.
The rest of the paper is organized as follows. In Section 2, a human training scenario using a UAM simulator is introduced. Section 3 addresses the structure of the proposed framework. The proposed framework is demonstrated via a human subject experiment in Section 4. Discussion and conclusions are given in Section 5 and 6, respectively. We introduce our human training scenario that helps readers understand how the proposed framework works. It is used for the human subject experiment in Section 4. In this scenario, a human user is requested to control and move a multi-rotor vehicle from an initial position to a destination for UAM operation training.
A 3-D virtual training environment is constructed in a constrained rectangular region with a take-of pad, a landing pad, two large buildings, and detailed surroundings that imitate an urban environment as shown in Fig. 1. This environment is powered by AirSim, an open-source simulator for physically and visually realistic simulations [55]. Inside the environment, the two buildings lie between the starting and ending points, creating a path shaped like the letter "S" between them (Fig. 1a). Along the path, six virtual waypoints, which have no collision efect, are strategically placed in the air (Fig. 1b). The human users use a physical light controller (Xbox controller) to control the vehicle's roll, pitch, yaw rate, and thrust. A graphical representation of the vehicle and the current system state information (e.g., current position and velocity) are displayed on a monitor (Fig. 1c). Vibration can be given to the human user through the light controller to notify changes in the assistant scheme. The vehicle takes of automatically from its initial position to its designated altitude before human users start each trial. When a release signal is triggered by human users via the light controller, they are requested to trace the path by lying through the waypoints successively to their best ability whether passing through each waypoint or not, i.e., they do not need to retry passing through each waypoint even if they missed it. The last waypoint is three times wider to give lexibility on the inal approach to the landing pad. Note that we only consider the skill training in this scenario, but path re-planning (e.g., choosing alternative paths) is not considered. Above the landing pad, there is a target area marked by a semi-transparent sphere, which triggers an automatic landing sequence only if the vehicle approaches the sphere with a speed of less than 7 m/s (Fig. 1d). During the entire training, human users control the vehicle in the irst-person view to replicate a remote pilot control situation. A sub-window, which shows beneath the vehicle, is provided to the human user to enhance the user's depth perception. Along with each trial, the vehicle states such as position, velocity, acceleration, attitude, angular velocity, angular acceleration, and human behavior data such as roll, pitch, yaw rate, and thrust inputs are recorded in 20 Hz. The human user has to complete the given task within three minutes time limit, otherwise, the simulation is terminated. Fig. 2 shows recorded trajectories from novice and expert demonstrations using the training environment.
In the SSL-based shared control framework, we utilize autonomy as a catalyst to expedite skill transfer from experts to novices. The proposed framework assists novices to emulate experts by permitting manual control (no assistance) to the novice unless the predicted behavior is deviating from a certain conidence interval. The probability of conducting the given task properly can be mapped into the SSL and the assistant scheme would be personalized since each novice has a diferent SSL in various situations. We formally present all necessary technical elements of the SSL inference and SSL-based shared control. We consider a 6-DOF nonlinear dynamical system operated by humans (like the multi-rotor system in Section 2):
where x k ∈ R 3 is the position, ẋk ∈ R 3 is the velocity, ϕ k ∈ R 3 is the Euler angle, and ω k ∈ R 3 is the angular velocity of the system, respectively. k denotes the time step. u k ∈ U ⊂ R m denotes the m-dimensional control input and f : R 12+m → R 12 denotes the nonlinear system dynamics. Throughout this paper, our main interest is to model human behaviors in controlling the trajectory of the system. The system model can be reformulated as: where U : R 12+m → R 3 is the nonlinear mapping function from the position, velocity, Euler angle, angular velocity, and control input into the acceleration input: ẍk = U(x k , ẋk , ϕ k , ω k , u k ). I and ∆t denote the identity matrix and the time step, respectively. The acceleration mapping function ẍk = U(u k |ϕ k ) = U k and its inverse function u k = U -1 (U k |ϕ k ) for a multi-rotor system are feasible with high precision [34]. See Appendix A.1 and A.2 for the details.
The structure of the SSL-based shared control is given in Fig. 3. For the SSL inference, the proposed framework requires two trajectory distributions: the expert trajectory distribution and the predicted novice trajectory distribution. Even if experts perform the very same control task multiple times, their input sequences may not be the same multiple times because of unintended variations in behavior, known as the variability of human behavior in motor skills [56]. For novices, the variability is usually greater and their behaviors are unintentionally inconsistent [9]. Thus, we model and estimate the probability distribution of multiple trajectories instead of only considering the single mean trajectory. The expert trajectory distribution can be obtained from expert demonstrations using a data-driven method. Since the expert trajectory distribution is only used as a reference, oline computation is acceptable. The predicted novice trajectory distribution can be computed from novice demonstrations and the current state of the system. By combining the novice's past data and the current state, it is possible to predict how the novice will control the system in the future. This process should be conducted in real-time to account for the time-varying SSL of the novice. Then, the SSL can be inferred by comparing the expert trajectory distribution and the predicted novice trajectory distribution. Note that the novice model can also be updated after each trial to examine SSL improvement trial by trial as shown in Fig. 3.
A data-driven method is used to reproduce the trajectory distribution from human demonstrations. The proposed framework models the trajectory distribution using the hidden semi-Markov model (HSMM). The HSMM has advantages over the existing trajectory distribution encoding algorithms [25]. The Gaussian mixture model (GMM) can represent the structure of the demonstrations but cannot provide the state transition information (i.e., a transition from one Gaussian mixture to another). The hidden Markov model (HMM) provides the state transition probabilities, but its self-state transitions are known to be inaccurate. The HSMM can encode the state duration probabilities which provide further advantages in accurate learning. See [10] for the details and tutorial source codes.
We aim to model the trajectory distribution using HSMM with a given set of human demonstrations {x k , ẋk , ẍk } N k =1
where N = M m=1 T m denotes the total number of data points from M demonstrations with T m data points for each demonstration. Let ζ ∈ R 9N and x ∈ R 3N be the concatenated states as:
The given training data points ζ can be encoded into a GMM using the expectation maximization (EM) algorithm [10,43], i.e., P (ζ ) ∼ K i=1 w i N (µ i , Σ i ) where K denotes the number of Gaussian mixture component, w i is the weight of each Gaussian mixture component with K i=1 w i = 1, N (•) is the normal distribution, and µ i and Σ i denote the mean and covariance of each Gaussian mixture component. In the next step, the HSMM parameters
are trained and the mixture duration is post-estimated from the training data points [12], where π i is the initial probability of being the Gaussian mixture component i and α i, j is the transition probability from the Gaussian mixture component i to that of j. µ D i and Σ D i are the mean and variance of the mixture duration time.
Based on the trained HSMM parameters, the expert trajectory distribution can be reconstructed as a sequence of Gaussian states The likelihood of the mixture duration for the Gaussian mixture i is given as:
where
denotes the evaluated Gaussian function with input x, mean µ, and covariance Σ. Then, the sequence of Gaussian mixture can be found by:
where D = min(d max , t -1) with initialization γ (i, 1) = π i . Then, (5) can be recursively computed. We pick the maximum probability Gaussian component to reconstruct the trajectory distribution [12]:
The likelihood of trajectory ζ given the Gaussian mixture sequence s is described as:
where
is the square diagonal matrix with the elements on the diagonal. The log-likelihood of P (ζ | s) can be obtained with a large sparse matrix
Then, P (ζ |s) = P (Φx|s) and the trajectory mean μx and covariance Σx are given as a least-square solution [10]:
where σ > 0 denotes the scale factor. Then, the reproduced trajectory distribution is given by xk ∼ N ( μx k , Σx k ). Fig. 4 shows exemplar trajectory distributions from human demonstrations. Note that 10 Gaussian components are used in our training environment (K = 10). This number can be determined based on the Bayesian information criterion (BIC). One can ind the best itting number of Gaussian components by selecting K that yields the lowest information criterion [59]. In the novice case, the trajectory distribution is relatively wide since the novice demonstrations are inconsistent due to the lack of skill. The expert can perform the task relatively accurately and consistently, and thus the trajectory distribution is relatively narrow. At the inal stage of the trajectory (marked by black squares in Fig. 4b), the last waypoint is three times wider than the others. The wider waypoint is designed to induce more variability in trajectories.
The proposed framework needs an assistant policy to provide expert-like assistant input. The assistant policy can be represented as a state feedback controller which can reproduce the learned expert nominal trajectory. Let xe k ∼ N ( μe,x k , Σe,x k ) be the expert trajectory distribution. Then, a trajectory tracking cost can be formulated as a quadratic cost function c k for all k ∈ {0, • • • ,T } [10]:
where R = ρI is the control input cost with the scale factor ρ > 0. We consider a constrained optimization problem with the constrained acceleration input A u U k ≤ b u to conduct realistic and safe operations. This optimization problem can be solved using the inite-time horizon model predictive control (MPC) [30]. Let the current time step k = 1, without loss of generality, and the inite-time horizon l. The cost function C is given by:
where
Then, the MPC problem can be solved using the quadratic programming (QP) [6].
where
can be computed using the Cholesky factorization [24]. This QP problem can be solved very eiciently using the standard QP algorithm, and the assistant acceleration input U a k computation is tractable in real-time. Then, the assistant policy Π a is given by solving the QP problem (17):
which is a deterministic feedback policy with respect to the current state ζ k and the data-driven expert model { μe,x k , Σe,x k }. Finally, the assistant input u a k for a multi-rotor system can be obtained using the mapping function from acceleration to control input (see Appendix A.2).
Next step is to obtain the novice control policy using the novice demonstrations (past data) and system state (current data). This step aims to obtain the predicted novice trajectory distribution to assess the novice's SSL based on the observed variability. Assuming that the current state
be a set of the inferred trajectory distribution using the novice demonstrations and ( 9)- (10). Then, the current phase j ∈ {1, • • • ,T } of the system needs to be identiied to align time series data:
where ∥•∥ denotes the L 2 norm and i ∈ {1, • • • ,T } denotes the time step. Then, we pick the Gaussian component that has the maximum probability as similar to (6) with the inite-time horizon l:
where k ∈ {j, • • • , j + l -1}. The trajectory distribution can be obtained using ( 7)-( 10) and the initial condition
where Σ j denotes the initial state covariance. Note that if the state ζ j is perfectly known without uncertainty, the initial state covariance can be Σ j = ϵI where ϵ < 1 denotes the small positive scale factor. The novice model {s n , µ s n , Σ s n } is deined as a set of sequence, mean, and covariance of each Gaussian component, as shown in (7). The prediction result { μnp,x k , Σnp,x k
} is given by the mean and covariance of the novice trajectories with respect to the current time step k. In other words, the predicted novice trajectory distribution is given as
), as shown in ( 9)- (10). The novice control policy Π n represents the predicted novice trajectory distribution as a stochastic policy:
Fig. 5 shows the predicted novice trajectory distribution
) from the current state ζ k and the novice model {s n , µ s n , Σ s n }, using an example from the human training environment in Section 2.
We present a formal representation of the SSL-based shared control in this section. The primary role of the shared control is to determine the control authority α k ∈ [0, 1] at each time step k. The control authority is deined as a weight value of the human input on the shared input space [2]. The shared control input u k is then a convex combination of the human input u h k ∈ U and assistant (autonomy) input u a k ∈ U as
Note that α k = 1 indicates the full manual control and α k = 0 denotes the full autonomous control, respectively.
In the proposed shared control framework, we only use a set of inite parameters such as α k ∈ A = {0.1, 0.5, 1.0}, since it is observed in our previous study [9] that continuous control authority change may provoke mode confusion to the human. The control authority is allocated in a discretized manner and the corresponding control authority is mapped into the discrete mode of the hybrid system [37]. The hybrid systems refers to the dynamic systems that involve the interaction of continuous states and discrete states. Accordingly, the SSL can be inferred in a discretized manner as well, and the SSL and the control authority are one-to-one mapped. Note that α k = 0.1, α k = 0.5 and α k = 1.0 correspond to the assistant schemes for low-skill-level, intermediate-skill-level, and high-skill-level human users, respectively. Also, the mapping can be modiied depending on the speciic task and context (e.g., more inely discretized skill-level). The hybrid shared controller is formalized as a tuple of the following elements [37], H = (Q, I, O, r, G) where each element is deined as follows.
• Q = {1, 2, • • • , N q } denotes the set of discrete modes. q k ∈ Q denotes the discrete mode at time step k. Note that N q = 3 for the application in this paper. • I = Z × Π a × Π n denotes the input of the hybrid system, which consists of the state space Z, the assistant policy Π a , and the novice control policy Π n . • O = U is the output of the hybrid system, i.e., the shared input u k .
• r : Q × I → O denotes the shared control law:
where the control authority α k is an one-to-one mapping from the discrete mode q k , i.e., α k = α (q k ), where α : Q → [0, 1] is the one-to-one mapping function. • G : Q × Q → 2 Z×Π a ×Π n denotes the guard condition which assigns the previous discrete mode q k -1 to the current discrete mode q k if the state Z, the assistant policy Π a , and the novice control policy Π n satisfy the guard condition. The guard condition is covered in detail in Section 3.3.
We deine the SSL to quantify the guard condition which determines the mode change of the hybrid system or the control authority allocation in the proposed framework. The SSL represents the probability that a novice emulates an expert's behavior while performing a given task. With the control authority being allocated based on the SSL, the proposed framework is named the SSL-based shared control. The proposed framework uses the Mahalanobis distance [50] as a performance measure to determine the assistant scheme by considering the variability. Let the mean of expert trajectory distribution μe,x k be the reference trajectory and the covariance Σe,x k be the weighting matrix for the Mahalanobis distance.
Let the current time step k = 1 without loss of generality. If the discrete mode is ixed for a inite-time horizon L, i.e., q k = q for all k ∈ {1, • • • , L}, the predicted Mahalanobis distance DL (q) of the system, controlled by the shared control law (27), is given by:
where the future predicted state x L (q) with the constant discrete mode q is given as:
subject to the state prediction under the assistant acceleration input
for all k ∈ {1, • • • , L} and the initial condition {x a 1 , ẋa 1 } = {x 1 , ẋ1 } is known. The predicted Mahalanobis distance DL (q) for the ixed discrete mode q is a stochastic value that represents the skill-level. Indeed, if a novice performs a given task similar to an expert, then DL (q) would be a small value which indicates that the skill-level of that novice is high, and vice versa.
The covariance matrix of the expert trajectory distribution Σe,x k is used as a weighting matrix of the performance measure DL (q) to model the uncertainty of the expert demonstrations. If the expert demonstrations show a smaller covariance at a region, it implies that the region is highly constrained. Thus, any small deviation (x L (q) -μe,x L ) may cause higher DL (q) which would induce a smaller control authority to the novice, and vice versa. This design ensures that the shared controller relects the variability in the expert demonstrations.
The core part of designing the hybrid shared control is to determine the guard condition, which represents the mode transition condition in the hybrid system. The discrete mode q k is one-to-one mapped into the control authority α k and the mode transition from q k -1 to q k is governed by the guard condition. Note that if α k = 0 for every time step k, then the system is fully controlled by the autonomy, and the performance measure is optimized. However, this is not a desirable case for human training. The proposed framework is designed to minimize the intervention from the autonomy during the human training so that novices can learn through their own control experience. Thus, the control authority α k needs to be maintained high enough when the novice can perform a given task similar enough to the expert performance. In this context, the guard condition is designed to guarantee a certain conidence level of similarity between the expert performance and the novice performance while maximizing the control authority of the novice:
where D denotes the largest Mahalanobis distance of the expert demonstrations. P ∈ [0, 1] is the conidence level threshold and it is a tunable design parameter. Note that P can be tuned in diferent training situations: a higher P will impose more assistance and vice versa. Then, the discrete mode q k at time step k is determined by an inverse mapping
) is an one-to-one mapping. Note that the control authority can be updated in real-time. The probability P ( DL (q) < D) in ( 32) can be computed using the generalized Chi-square (GCS) method which provides a numerical probability density function (PDF) [14]. To compute the PDF, the Mahalanobis distance can be reformulated as:
) and the standard GCS method can provide the PDF of the Mahalanobis distance which represents the SSL in the proposed framework. Then, the computed PDF can be used to determine the control authority α k by solving (32). Note that ( DL (q) < D) ⇔ (( DL (q)) 2 < D2 ) since DL (q) and D are positive.
The purpose of the human subject experiment is to demonstrate the proposed framework in the complex UAM vehicle control scenario with the environment described in Section 2. Participants are randomly divided into three groups. The irst group is a control group that is not assisted by any method during their practice. The second group is assisted by a baseline shared control approach called Maxwell's Demon Algorithm (MDA), which is known to aid human learning [7]. The baseline approach is a deterministic and non-personalized hybrid shared control method. The MDA has only two modes: MDA fully accepts human input if it is in the same half-hyperplane (in R m ) with assistant input (i.e., α k = 1), or MDA discards human input otherwise (i.e., α k = 0 and set u k = 0). The MDA only blocks particularly bad input from the human but does not provide any assistant input. The MDA is formally presented in Appendix B. The third group is assisted by the proposed framework. A hypothesis to be investigated is given as follows.
• Hypothesis: the proposed framework can improve the human learning rate, in terms of the immediate skill retention, in the complex UAM vehicle control scenario in Section 2 compared to the practice-only approach and the baseline approach.
A total of 40 human subjects were selected as novices for the human subject study, which is approved by the Institutional Review Board at Purdue University (IRB-2020-755). Participants who can pass through more than 50% of waypoints at their initial trial were excluded. Novice participants were randomly divided into three groups.
• Group 1 (practice-only group): 10 novices who are not assisted during training.
• Group 2 (baseline group): 15 novices who are assisted by the baseline approach during training.
• Group 3 (proposed group): 15 novices who are assisted by the proposed framework during training.
Two human subjects who can pass through all waypoints without failure for more than 10 consecutive trials and touch the target area in Fig. 1d with stable speed and attitude were selected as the experts to provide expert demonstrations.
An experiment instruction was explained in advance to all participants so that they had the same information regarding the task, basic control technique, and expectations. All participants were allowed three minutes to familiarize themselves with the light controller and the environment. We did not request the participants to ly fast but ly accurately. The only constraint for speed was to complete each trial within three minutes, otherwise, the trial was terminated in the middle and data was stored up to that point. There were only three trials out of 1,000 trials of terminating in the middle due to timeout, and all of them were included in the experimental data.
The experimental procedure for each individual was separated into the following three phases:
• Phase 1 (Initial performance): All groups conduct the task 10 times by manual control. Their initial performance measures before any training are recorded. The proposed framework exploits the participant demonstrations to estimate parameters for the novice model. A ive minutes break follows Phase 1. • Phase 2 (Training): All groups conduct the task 10 times. Group 1 performs manual control. Group 2 is assisted by the baseline approach. Group 3 is assisted by the proposed framework during their trials. For Group 3, their novice models are automatically updated after each trial by incorporating the latest trial and excluding the oldest trial. A ive minutes break follows Phase 2. • Phase 3 (Final performance): All groups conduct the task ive times by manual control. It is expected that their performance measures are improved compared to their initial performance measures.
The control authority was allocated from inite sets for Group 2 and Group 3. For Group 2, the control authority was determined in a inite set A g2 = {0.0, 1.0}: full autonomous control when α k = 0.0 or full manual control when α k = 1.0. Participants can recognize the current control authority via vibration cues: when the baseline approach intervenes a participant (α k = 0.0), the light controller vibrates, and there is no vibration for the manual control mode (α k = 1.0). For Group 3, the control authority was determined in a inite set A g3 = {0.1, 0.5, 1.0}. Note that the minimum control authority α k = 0.1 was set to prevent abuse of the autonomy. Vibration cues were given with respect to the control authority: a high-frequency vibration for the irst mode (α k = 0.1), a low-frequency vibration for the second mode (α k = 0.5), and no vibration for the manual control mode (α k = 1.0). All participants in Group 3 reported that they can distinguish the vibration cues during the instruction. The threshold for the Mahalanobis distance D = 9.65 in (32) was obtained from the worst Mahalanobis distance of expert demonstrations. The conidence level threshold P = 0.5 was selected by our pilot study. These thresholds can be interpreted as the assistance will be given only if the conidence level that the novice is worse than the expert's worst case is more than 1 -P. The inite-time horizon for the shared control scheme was set to l = 40 which is equivalent to two seconds in real-time. A dwell-time of three seconds was imposed on the mode transitions in Group 2 and Group 3 to prevent frequent mode changes.
After each trial, the participants were instructed to answer the subjective survey form that evaluates their self-conidence and workload for each trial on an integer scale of 1-10 [26, 63]. During the experiment, physical variables such as position, velocity, and acceleration were recorded, as well as human input variables such as roll, pitch, yaw rate, and thrust input. The SSL-based shared control parameters (e.g., α k and D k ) were also recorded.
The following performance measures are used to quantify the results.
• The Mahalanobis distance D k in (28), for all k, quantiies the weighted tracking error to the expert nominal trajectory. This value is computed after applying the dynamic time wrapping (DTW) [11] so that the expert's and novice's trajectories are temporally aligned with each other. • The minimum distance to each waypoint, d w i for all i ∈ {1, 2, • • • , 6}, measures how well novices perform the high-level task of passing through the waypoints.
• The inal stage performance denotes the Mahalanobis distance D k after passing the last waypoint. It represents the ability of the novice to stabilize and slow down the vehicle for safe landing near the inal target.
• The learning rate is used to it the performance improvement over the training trials. The power function (y = ax -b ) is used for the regression analysis [4]. A larger b value indicates a faster learning rate. • The averaged control authority α avg ≜ 1/T T k=0 α k represents control authority of the novices for their training in Phase 2. A higher averaged control authority means that more human control is engaged in Phase 2.
• Two self-reported cognitive states, self-conidence and workload, are used to measure the user experience [63].
is constrained in roll, pitch, yaw rate, and throttle commands.
• Mean and standard deviation of the mission completion time T in each phase are used to examine execution time and consistency.
The Mahalanobis distance D k is compared in three groups to answer the research question of whether the proposed framework can enhance the human learning rate. In Fig. 6, the box plots are used to visualize the means of the Mahalanobis distance of each participant. The analysis of variance (ANOVA) is used for statistical testing [27]. The two-way mixed ANOVA is used to determine how performance is afected by two variables, phase and group. One-way repeated measures ANOVAs are used to examine the training results between phases within each group. Note that the one-way repeated measures ANOVAs are presented in each igure. Conventional symbols are used to represent p-values in igures: * p < 0.05, * * p < 0.01, * * * p < 0.001, and * * * * p < 0.0001. The ANOVA result reveals that there is no signiicant two-way interaction between group and phase on the Mahalanobis distance, F (2.63, 48.71) = 0.94, p = 0.42, η 2 = 0.005. There is a signiicant main efect of phase, F (1.32, 48.71) = 55.755, p < 0.0001, η 2 = 0.135. All pairwise comparisons are analyzed between each group. Only Group 1 and Group 3 show a signiicant diference of the Mahalanobis distance (p = 0.0162). Group 2 is not signiicantly diferent from Group 1 (p = 0.374) and Group 3 (p = 0.0863). The one-way repeated measures ANOVAs in Fig. 6 show that Group 3 has better training results compared to the others. The minimum distance to each waypoint d w i is used to compare three groups in terms of the performance of the high-level task, passing through the waypoints. The expert performance was 0.689 m on average. As participants are trained through Phase 2, their mean, median, and variance are all reduced in Fig. 7. There is no signiicant two-way interaction between group and phase on the minimum distance to waypoints, F (3.25, 60.2) = 0.93, p = 0.44, η 2 = 0.006. There is a signiicant main efect of phase, F (1.63, 60.2) = 41.948, p < 0.0001, η 2 = 0.117. The pairwise comparison results between each group reveal that Group 3 is signiicantly diferent from Group 1 (p = 0.018) and Group 2 (p = 0.0194). Group 1 and Group 2 are not signiicantly diferent (p = 0.78). The one-way repeated measures ANOVAs results in Fig. 7 indicate that Group 3 outperforms other groups.
The inal stage performance is a subset of the Mahalanobis distance result. The inal stage is diferent from the rest of the trajectory since it requires slowing down the vehicle to gently touch the target. In Fig. 8, all groups show improvement from Phase 1 to Phase 3, but only Group 3 shows a signiicant diference. There is no signiicant twoway interaction between group and phase on the inal stage performance, F (3.03, 55.98) = 1.25, p = 0.3, η 2 = 0.02. There is a signiicant main efect of phase, F (1.51, 55.98) = 14.167, p < 0.0001, η 2 = 0.093. The pairwise comparison results show that Group 1 and Group 3 are signiicantly diferent (p = 0.0485). Group 2 is not signiicantly diferent from Group 1 (p = 0.44) and Group 3 (p = 0.176). The one-way repeated ANOVAs in Fig. 8 show that Group 3 performs better than Group 1 and Group 2.
A regression analysis using the power function itting technique is used to quantify the human learning rate. The power function y = ax -b is widely adopted to measure the learning rate [4], and it shows the highest R 2 value among the linear, power, and exponential function itting for the presented experiment results. A bigger b indicates a higher learning rate. The learning rate is estimated using each raw data over a total of 25 trials. The one-way ANOVA method is used for comparing each group in Fig. 9. The results show that the learning rate diferences (i.e., itted b values) are not signiicant except for the inal stage, between Group 1 and Group 3 (p = 0.0293). The results show that the itted power functions are not signiicantly diferent, even if the performance improvements show signiicant diferences in Fig. 678. Nevertheless, Group 3 still shows higher mean and median values of b for all the cases. Fig. 10a represents the averaged control authority α avg over Phase 2 of each group. Larger values indicate that the humans take more control authority over the system. Group 1 has full control authority (no assistance), i.e., α k = 1.0 for all k. Group 2 takes signiicantly less control authority during the training (lower α avg ) compared to Group 3. However, its performance improvement is still worse than that of Group 3. This fact indicates that the proposed framework can infer the SSL and adjust the level of assistance appropriately such that the human learning rate is increased while allowing more control authority to the humans. Fig. 10b shows the time portion of each control authority (discrete mode) in Group 2 and Group 3. Each trial is divided into three segments to get more detailed information, from the start to the third waypoint (initial stage), from the third waypoint to the last waypoint (mid stage), and from the waypoint to landing area (inal stage). Some participants reported that a curved trajectory in the mid stage is diicult compared to the initial and inal stage with the straight trajectories. The proposed framework can relect the task diiculties since it provides more assistance (i.e., less control authority to the humans) in the mid stage to Group 3 as shown in Fig. 10b. The cognitive states are investigated using a subjective survey form. The surveyed values are normalized using the z-score method [63]. Fig. 11 shows similar trends in all groups: self-conidence is increasing, and workload is decreasing over phases, respectively. In the self-conidence survey, there is no statistically signiicant two-way interaction between group and phase, F (2.97, 54.95) = 0.21, p = 0.89, η 2 = 0.003. There is a signiicant main efect of phase, F (1.49, 54.95) = 77.682, p < 0.0001, η 2 = 0.385. One observed point is that Group 3 is the only group that shows a signiicant diference in post-training, from Phase 2 to Phase 3 (p = 0.018). In the workload survey, there is no signiicant two-way interaction between group and phase, F (4, 74) = 0.98, p = 0.42, η 2 = 0.02. However, there is a signiicant main efect of phase, F (2, 74) = 29.172, p < 0.0001, η 2 = 0.249. Group 3 reported a signiicantly lower workload in Phase 2 compared to Group 1 (p = 0.0238) and Group 2 (p = 0.0277). Some Phase 1 Phase 2 Phase 3 -2 0 2 Self-confidence Group 1: Practice-only F(2,18)=18.48, p<0.0001, 2 =0.28 ** ** Phase 1 Phase 2 Phase 3 -2 0 2 Self-confidence Group 2: Baseline F(2,28)=25.88, p<0.0001, 2 =0.36 *** *** Phase 1 Phase 2 Phase 3 -2 0 2 Self-confidence Group 3: Proposed F(1.17,16.34)=42.76, p<0.0001, 2 =0.62 **** * **** Phase 1 Phase 2 Phase 3 -2 0 2 Workload Group 1: Practice-only F(1.18,10.6)=19.05, p=0.00077, 2 =0.28 ** ** ** Phase 1 Phase 2 Phase 3 -2 0 2 Workload Baseline p=0.013, =0.14 * Phase 1 Phase 2 Phase 3 -2 0 2 Workload Group 3: Proposed F(1.45,20.37)=15.09, p=0.00029, =0.42 **** ** Fig. 11. Self-reported cognitive states: self-confidence and workload.
participants in Group 2 described that the baseline shared controller intervenes too frequently, and they tried to explore the right control input to avoid the intervention. This may cause a higher workload in Phase 2 for Group 2. This observation coincides with the average control authority result in Fig. 10a since Group 2 shows the lower averaged control authority (more intervention from autonomy) in Phase 2. The results indicate that the proposed framework could reduce workload during the training. Another interesting observation is that Group 3 shows lower variability in self-conidence and workload. This observation can be interpreted as a result of the personalized nature of the proposed method, but further investigation is required to conclude. We plan to examine the correlation between workload and training with further experiments. The control efort to monitor the behavioral changes of the participants over the phases. There is signiicant two-way interaction between group and phase, F (2.78, 51.49) = 6.06, p = 0.002, η 2 = 0.06. All groups show signiicant main efects of the phase as shown in Fig. 12. Group 2 demonstrates signiicantly higher control eforts in Phase 2 compared to Group 1 (p = 0.0195) and Group 3 (p = 0.0102). Some Group 2 reported that the assistant scheme was too restrictive for them, and they tried to avoid vibration which was regarded as punishment instead of assistance (details in Section 5). They also pointed out that they had explored the control space to stop the assistance and vibration in Phase 2. The exploring behaviors can be interpreted as adaptive behaviors to the changed nature of the task due to the baseline approach. This aspect is also relected in the workload survey in Fig. 11 since Group 2 reported a higher workload in Phase 2 compared to Group 3. On the other hand, Group 3 made signiicantly lower control eforts in Phase 2 compared to Phase 1. This fact shows that the proposed framework does not change the nature of the training task, and thus, it did not cause the exploring behaviors in Phase 2.
The mission completion time should be carefully analyzed since the participants were not requested to inish the task fast but accurately. In our pilot study, some participants tended to ind shortcuts while ignoring the waypoints. Those strategies are not appropriate for UAM operations especially near the landing area due to safety concerns. Thus, we particularly requested the participants to follow the nominal trajectory. Nevertheless, our intuitive prediction was that the participants will ly faster and be consistent in multiple trials as they have more experience. In Fig. 13, all groups show decreasing trends in terms of the mean and the standard deviation of the mission completion time. There is no signiicant two-way interaction on the mean mission completion time, F (3.22, 59.65) = 1.32, p = 0.28, η 2 = 0.01. Only Group 2 does not show a signiicant diference between Phase 1 and Phase 3. The results show that Group 3 performed better than other groups in mission completion time reduction. In the bottom of Fig. 13, the standard deviations of the mission completion time decrease over phases. There is no signiicant two-way interaction on the standard deviation of the mission completion time, F (4, 74) = 2.22, p = 0.075, η 2 = 0.04. Nevertheless, Group 3 shows the most signiicant main efects of the phase compared to Group 1 and Group 2 as shown in Fig. 13. Since consistency in time is another feature of expert demonstrations (e.g., the standard deviation of one expert was 2.18 sec), this result indicates that Group 3 successfully emulated expert demonstrations even without a speciic direction in mission completion time.
The SSL-based shared control framework was demonstrated in the UAM vehicle control training task. The human subject experiment results reveal that the proposed framework can expedite human training in a speciic scenario compared to practice-only (no assistance) scheme and the baseline shared control scheme. In particular, similarity to expert performance increased more signiicantly in Group 3 (assisted by the proposed framework) accompanied by an improvement in other metrics like control efort and mission completion time. In subjective cognitive states report, the proposed framework further reduced the training workload compared to the baseline approach. The results reveal that our SSL inference approach and SSL-based shared control scheme can facilitate better training environments by realizing the OCP [23] in complex dynamic control tasks. The human subject experiment results are valuable since the adaptive training environment methods are not necessarily result in improving the human learning rate [3]. Our results present signiicant diferences between the control group, baseline group, and the proposed framework group. This fact shows that the proposed framework can be a foundation for further investigations in SSL-based approaches for human training. However, we note that the proposed framework may not be always efective for general training scenarios. It is fair to say that the proposed framework tested a speciic hypothesis as described in Section 4.1. Indeed, our primary contributions are the design and demonstration of the SSL-based shared control for complex dynamic tasks. Any claim on the human learning rate, on the other hand, may require more experimental data and depend on tasks [49]. We provide several points which can be tested with further investigations in skill training: long-term skill retention, dynamic training environments, and multi-modal feedback. Long-term skill retention can be tested by collecting performance measures after a certain time after the training [35]. It can test whether human subjects maintain and retain their skills. Dynamic training environments can be imposed to test whether human subjects can transfer their capabilities to cope with diferent situations, such as tracking diferent trajectories and avoiding dynamic obstacles [61]. Random errors in the experiment environments (e.g., disturbance) can be imposed to investigate the human learning rate under error-augmented situations [61]. Multi-modal feedback, including haptic feedback [49,61] and auditory feedback [52], has been examined in human training. Especially, various haptic feedback schemes have been implemented to assist human learning [1]. Interesting results might be obtained if an experiment is conducted to examine how the human learning rate changes when all the factors mentioned are involved. Nevertheless, the proposed framework is still valuable since the inferred SSL can be useful for further experiments.
In future work, we may investigate the transparency [13,39] between humans and autonomy for a comprehensive SSL inference. Autonomy may be designed in a better way if it can directly measure, infer, and consider the human cognitive state. Especially, physio-psychological instruments including but not limited to eye-tracker, electroencephalography (EEG), and Galvanic skin response (GSR). Some participants wanted explanations on how autonomy determined the control authority. If autonomy can convey the information concisely, the acceptance of assistance could be further improved. In the human subject experiment, ive participants in Group 2 and Group 3 reported that they want to have more information, for instance, the reasoning for the control authority decision. However, increasing the transparency requires an additional consideration since the higher transparency may induce higher workload to the humans [13]. Finally, we can investigate an online human model update scheme (i.e., update the HSMM online) to cope with potentially fast-changing human characteristics. In our experiments, updating the HSMMs after each trial was enough. However, further fast-changing cases may require online model update [25].
A stochastic-skill-level-based shared control framework is proposed and a human subject experiment was conducted. The purpose of the proposed framework is to help a human novice emulate a human expert in complex dynamic control tasks. The proposed framework consists of three steps to transfer skills from the expert to the novice. In the irst step, the autonomy learns how the expert performs the given task. Then, the autonomy infers the stochastic-skill-level of the novice by comparing the demonstrations of the expert and those of the novice. In the third step, the autonomy allocates the control authority based on the time-varying stochastic-skill-level of the novice. The stochastic-skill-level inference and the control authority allocation are both based on probabilistic approaches which account for the uncertainty of human behavior (i.e., variability). The assistance is given to the novices only when their current stochastic-skill-levels are poor such that the performance measure is not satisfying a stochastic threshold. The human subject experiment results show that the proposed framework can provide a solid foundation for human training in complex dynamic control tasks.
ACM Trans. Hum.-Robot Interact. Stochastic-Skill-Level-Based Shared Control for Human Training in Urban Air Mobility Scenario • 3