skip to main content


Title: Closed-Loop Tracking and Regulation of Emotional Valence State From Facial Electromyogram Measurements
Affective studies provide essential insights to address emotion recognition and tracking. In traditional open-loop structures, a lack of knowledge about the internal emotional state makes the system incapable of adjusting stimuli parameters and automatically responding to changes in the brain. To address this issue, we propose to use facial electromyogram measurements as biomarkers to infer the internal hidden brain state as feedback to close the loop. In this research, we develop a systematic way to track and control emotional valence, which codes emotions as being pleasant or obstructive. Hence, we conduct a simulation study by modeling and tracking the subject's emotional valence dynamics using state-space approaches. We employ Bayesian filtering to estimate the person-specific model parameters along with the hidden valence state, using continuous and binary features extracted from experimental electromyogram measurements. Moreover, we utilize a mixed-filter estimator to infer the secluded brain state in a real-time simulation environment. We close the loop with a fuzzy logic controller in two categories of regulation: inhibition and excitation. By designing a control action, we aim to automatically reflect any required adjustments within the simulation and reach the desired emotional state levels. Final results demonstrate that, by making use of physiological data, the proposed controller could effectively regulate the estimated valence state. Ultimately, we envision future outcomes of this research to support alternative forms of self-therapy by using wearable machine interface architectures capable of mitigating periods of pervasive emotions and maintaining daily well-being and welfare.  more » « less
Award ID(s):
1942585 1755780 2226123
NSF-PAR ID:
10322318
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Frontiers in Computational Neuroscience
Volume:
16
ISSN:
1662-5188
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Hypercortisolism or Cushing's disease, which corresponds to the excessive levels of cortisol hormone, is associated with tiredness and fatigue during the day and disturbed sleep at night. Our goal is to employ a wearable brain machine interface architecture to regulate one's energy levels in hypercortisolism. In the present simulation study, we generate multi-day cortisol profile data for ten subjects both in healthy and disease conditions. To relate an internal hidden cognitive energy state to one's cortisol secretion patterns, we employ a state-space model. Particularly, we consider circadian upper and lower bound envelopes on cortisol levels, and timings of hypothalamic pulsatile activity underlying cortisol secretions as continuous and binary observations, respectively. To estimate the hidden cognitive energy-related state, we use Bayesian filtering. In our proposed architecture, we infer one's cognitive energy-related state using wearable devices rather than monitoring the brain activity directly and close the loop utilizing fuzzy control. To model actuation in the real-time closed-loop architecture, we simulate two types of medications that result in increasing and decreasing the energy levels in the body. Finally, we close the loop using a knowledge-based control approach. The results on ten simulated profiles verify how the proposed architecture is able to track the energy state and regulate it using hypothetical medications. In a simulation study based on experimental data, we illustrate the feasibility of designing a wearable brain machine interface architecture for energy regulation in hypercortisolism. This simulation study is a first step toward the ultimate goal of managing hypercortisolism in real-world situations. 
    more » « less
  2. Resonant tunneling diodes (RTDs) have come full-circle in the past 10 years after their demonstration in the early 1990s as the fastest room-temperature semiconductor oscillator, displaying experimental results up to 712 GHz and fmax values exceeding 1.0 THz [1]. Now the RTD is once again the preeminent electronic oscillator above 1.0 THz and is being implemented as a coherent source [2] and a self-oscillating mixer [3], amongst other applications. This paper concerns RTD electroluminescence – an effect that has been studied very little in the past 30+ years of RTD development, and not at room temperature. We present experiments and modeling of an n-type In0.53Ga0.47As/AlAs double-barrier RTD operating as a cross-gap light emitter at ~300K. The MBE-growth stack is shown in Fig. 1(a). A 15-μm-diam-mesa device was defined by standard planar processing including a top annular ohmic contact with a 5-μm-diam pinhole in the center to couple out enough of the internal emission for accurate free-space power measurements [4]. The emission spectra have the behavior displayed in Fig. 1(b), parameterized by bias voltage (VB). The long wavelength emission edge is at  = 1684 nm - close to the In0.53Ga0.47As bandgap energy of Ug ≈ 0.75 eV at 300 K. The spectral peaks for VB = 2.8 and 3.0 V both occur around  = 1550 nm (h = 0.75 eV), so blue-shifted relative to the peak of the “ideal”, bulk InGaAs emission spectrum shown in Fig. 1(b) [5]. These results are consistent with the model displayed in Fig. 1(c), whereby the broad emission peak is attributed to the radiative recombination between electrons accumulated on the emitter side, and holes generated on the emitter side by interband tunneling with current density Jinter. The blue-shifted main peak is attributed to the quantum-size effect on the emitter side, which creates a radiative recombination rate RN,2 comparable to the band-edge cross-gap rate RN,1. Further support for this model is provided by the shorter wavelength and weaker emission peak shown in Fig. 1(b) around = 1148 nm. Our quantum mechanical calculations attribute this to radiative recombination RR,3 in the RTD quantum well between the electron ground-state level E1,e, and the hole level E1,h. To further test the model and estimate quantum efficiencies, we conducted optical power measurements using a large-area Ge photodiode located ≈3 mm away from the RTD pinhole, and having spectral response between 800 and 1800 nm with a peak responsivity of ≈0.85 A/W at  =1550 nm. Simultaneous I-V and L-V plots were obtained and are plotted in Fig. 2(a) with positive bias on the top contact (emitter on the bottom). The I-V curve displays a pronounced NDR region having a current peak-to-valley current ratio of 10.7 (typical for In0.53Ga0.47As RTDs). The external quantum efficiency (EQE) was calculated from EQE = e∙IP/(∙IE∙h) where IP is the photodiode dc current and IE the RTD current. The plot of EQE is shown in Fig. 2(b) where we see a very rapid rise with VB, but a maximum value (at VB= 3.0 V) of only ≈2×10-5. To extract the internal quantum efficiency (IQE), we use the expression EQE= c ∙i ∙r ≡ c∙IQE where ci, and r are the optical-coupling, electrical-injection, and radiative recombination efficiencies, respectively [6]. Our separate optical calculations yield c≈3.4×10-4 (limited primarily by the small pinhole) from which we obtain the curve of IQE plotted in Fig. 2(b) (right-hand scale). The maximum value of IQE (again at VB = 3.0 V) is 6.0%. From the implicit definition of IQE in terms of i and r given above, and the fact that the recombination efficiency in In0.53Ga0.47As is likely limited by Auger scattering, this result for IQE suggests that i might be significantly high. To estimate i, we have used the experimental total current of Fig. 2(a), the Kane two-band model of interband tunneling [7] computed in conjunction with a solution to Poisson’s equation across the entire structure, and a rate-equation model of Auger recombination on the emitter side [6] assuming a free-electron density of 2×1018 cm3. We focus on the high-bias regime above VB = 2.5 V of Fig. 2(a) where most of the interband tunneling should occur in the depletion region on the collector side [Jinter,2 in Fig. 1(c)]. And because of the high-quality of the InGaAs/AlAs heterostructure (very few traps or deep levels), most of the holes should reach the emitter side by some combination of drift, diffusion, and tunneling through the valence-band double barriers (Type-I offset) between InGaAs and AlAs. The computed interband current density Jinter is shown in Fig. 3(a) along with the total current density Jtot. At the maximum Jinter (at VB=3.0 V) of 7.4×102 A/cm2, we get i = Jinter/Jtot = 0.18, which is surprisingly high considering there is no p-type doping in the device. When combined with the Auger-limited r of 0.41 and c ≈ 3.4×10-4, we find a model value of IQE = 7.4% in good agreement with experiment. This leads to the model values for EQE plotted in Fig. 2(b) - also in good agreement with experiment. Finally, we address the high Jinter and consider a possible universal nature of the light-emission mechanism. Fig. 3(b) shows the tunneling probability T according to the Kane two-band model in the three materials, In0.53Ga0.47As, GaAs, and GaN, following our observation of a similar electroluminescence mechanism in GaN/AlN RTDs (due to strong polarization field of wurtzite structures) [8]. The expression is Tinter = (2/9)∙exp[(-2 ∙Ug 2 ∙me)/(2h∙P∙E)], where Ug is the bandgap energy, P is the valence-to-conduction-band momentum matrix element, and E is the electric field. Values for the highest calculated internal E fields for the InGaAs and GaN are also shown, indicating that Tinter in those structures approaches values of ~10-5. As shown, a GaAs RTD would require an internal field of ~6×105 V/cm, which is rarely realized in standard GaAs RTDs, perhaps explaining why there have been few if any reports of room-temperature electroluminescence in the GaAs devices. [1] E.R. Brown,et al., Appl. Phys. Lett., vol. 58, 2291, 1991. [5] S. Sze, Physics of Semiconductor Devices, 2nd Ed. 12.2.1 (Wiley, 1981). [2] M. Feiginov et al., Appl. Phys. Lett., 99, 233506, 2011. [6] L. Coldren, Diode Lasers and Photonic Integrated Circuits, (Wiley, 1995). [3] Y. Nishida et al., Nature Sci. Reports, 9, 18125, 2019. [7] E.O. Kane, J. of Appl. Phy 32, 83 (1961). [4] P. Fakhimi, et al., 2019 DRC Conference Digest. [8] T. Growden, et al., Nature Light: Science & Applications 7, 17150 (2018). [5] S. Sze, Physics of Semiconductor Devices, 2nd Ed. 12.2.1 (Wiley, 1981). [6] L. Coldren, Diode Lasers and Photonic Integrated Circuits, (Wiley, 1995). [7] E.O. Kane, J. of Appl. Phy 32, 83 (1961). [8] T. Growden, et al., Nature Light: Science & Applications 7, 17150 (2018). 
    more » « less
  3. Humans have an astonishing ability to extract hidden information from the movements of others. For example, even with limited kinematic information, humans can distinguish between biological and nonbiological motion, identify the age and gender of a human demonstrator, and recognize what action a human demonstrator is performing. It is unknown, however, whether they can also estimate hidden mechanical properties of another’s limbs simply by observing their motions. Strictly speaking, identifying an object’s mechanical properties, such as stiffness, requires contact. With only motion information, unambiguous measurements of stiffness are fundamentally impossible, since the same limb motion can be generated with an infinite number of stiffness values. However, we show that humans can readily estimate the stiffness of a simulated limb from its motion. In three experiments, we found that participants linearly increased their rating of arm stiffness as joint stiffness parameters in the arm controller increased. This was remarkable since there was no physical contact with the simulated limb. Moreover, participants had no explicit knowledge of how the simulated arm was controlled. To successfully map nontrivial changes in multijoint motion to changes in arm stiffness, participants likely drew on prior knowledge of human neuromotor control. Having an internal representation consistent with the behavior of the controller used to drive the simulated arm implies that this control policy competently captures key features of veridical biological control. Finding that humans can extract latent features of neuromotor control from kinematics also provides new insight into how humans interpret the motor actions of others. NEW & NOTEWORTHY Humans can visually perceive another’s overt motion, but it is unknown whether they can also perceive the hidden dynamic properties of another’s limbs from their motions. Here, we show that humans can correctly infer changes in limb stiffness from nontrivial changes in multijoint limb motion without force information or explicit knowledge of the underlying limb controller. Our findings suggest that humans presume others control motor behavior in such a way that limb stiffness influences motion. 
    more » « less
  4. Background As a number of vaccines for COVID-19 are given emergency use authorization by local health agencies and are being administered in multiple countries, it is crucial to gain public trust in these vaccines to ensure herd immunity through vaccination. One way to gauge public sentiment regarding vaccines for the goal of increasing vaccination rates is by analyzing social media such as Twitter. Objective The goal of this research was to understand public sentiment toward COVID-19 vaccines by analyzing discussions about the vaccines on social media for a period of 60 days when the vaccines were started in the United States. Using the combination of topic detection and sentiment analysis, we identified different types of concerns regarding vaccines that were expressed by different groups of the public on social media. Methods To better understand public sentiment, we collected tweets for exactly 60 days starting from December 16, 2020 that contained hashtags or keywords related to COVID-19 vaccines. We detected and analyzed different topics of discussion of these tweets as well as their emotional content. Vaccine topics were identified by nonnegative matrix factorization, and emotional content was identified using the Valence Aware Dictionary and sEntiment Reasoner sentiment analysis library as well as by using sentence bidirectional encoder representations from transformer embeddings and comparing the embedding to different emotions using cosine similarity. Results After removing all duplicates and retweets, 7,948,886 tweets were collected during the 60-day time period. Topic modeling resulted in 50 topics; of those, we selected 12 topics with the highest volume of tweets for analysis. Administration and access to vaccines were some of the major concerns of the public. Additionally, we classified the tweets in each topic into 1 of the 5 emotions and found fear to be the leading emotion in the tweets, followed by joy. Conclusions This research focused not only on negative emotions that may have led to vaccine hesitancy but also on positive emotions toward the vaccine. By identifying both positive and negative emotions, we were able to identify the public's response to the vaccines overall and to news events related to the vaccines. These results are useful for developing plans for disseminating authoritative health information and for better communication to build understanding and trust. 
    more » « less
  5. null (Ed.)
    Abstract An autonomous adaptive model predictive control (MPC) architecture is presented for control of heating, ventilation, and air condition (HVAC) systems to maintain indoor temperature while reducing energy use. Although equipment use and occupant changes with time, existing MPC methods are not capable of automatically relearning models and computing control decisions reliably for extended periods without intervention from a human expert. We seek to address this weakness. Two major features are embedded in the proposed architecture to enable autonomy: (i) a system identification algorithm from our prior work that periodically re-learns building dynamics and unmeasured internal heat loads from data without requiring re-tuning by experts. The estimated model is guaranteed to be stable and has desirable physical properties irrespective of the data; (ii) an MPC planner with a convex approximation of the original nonconvex problem. The planner uses a descent and convergent method, with the underlying optimization problem being feasible and convex. A yearlong simulation with a realistic plant shows that both of the features of the proposed architecture—periodic model and disturbance update and convexification of the planning problem—are essential to get performance improvement over a commonly used baseline controller. Without these features, long-term energy savings from MPC can be small while with them, the savings from MPC become substantial. 
    more » « less