skip to main content


Title: Decoding of Facial Strains via Conformable Piezoelectric Interfaces
Devices that facilitate nonverbal communication typically require high computational loads or have rigid and bulky form factors that are unsuitable for use on the face or on other curvilinear body surfaces. Here, we report the design and pilot testing of an integrated system for decoding facial strains and for predicting facial kinematics. The system consists of mass-manufacturable, conformable piezoelectric thin films for strain mapping; multiphysics modelling for analysing the nonlinear mechanical interactions between the conformable device and the epidermis; and three-dimensional digital image correlation for reconstructing soft-tissue surfaces under dynamic deformations as well as for informing device design and placement. In healthy individuals and in patients with amyotrophic lateral sclerosis, we show that the piezoelectric thin films, coupled with algorithms for the real-time detection and classification of distinct skin-deformation signatures, enable the reliable decoding of facial movements. The integrated system could be adapted for use in clinical settings as a nonverbal communication technology or for use in the monitoring of neuromuscular conditions.  more » « less
Award ID(s):
2026344
NSF-PAR ID:
10287639
Author(s) / Creator(s):
; ; ; ; ; ; ; ; ; ;
Date Published:
Journal Name:
Nature biomedical engineering
Volume:
4
ISSN:
2157-846X
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    Conformable electronics are regarded as the next generation of personal healthcare monitoring and remote diagnosis devices. In recent years, piezoelectric‐based conformable ultrasound electronics (cUSE) have been intensively studied due to their unique capabilities, including nonradiative monitoring, soft tissue imaging, deep signal decoding, wireless power transfer, portability, and compatibility. This review provides a comprehensive understanding of cUSE for use in biomedical and healthcare monitoring systems and a summary of their recent advancements. Following an introduction to the fundamentals of piezoelectrics and ultrasound transducers, the critical parameters for transducer design are discussed. Next, five types of cUSE with their advantages and limitations are highlighted, and the fabrication of cUSE using advanced technologies is discussed. In addition, the working function, acoustic performance, and accomplishments in various applications are thoroughly summarized. It is noted that application considerations must be given to the tradeoffs between material selection, manufacturing processes, acoustic performance, mechanical integrity, and the entire integrated system. Finally, current challenges and directions for the development of cUSE are highlighted, and research flow is provided as the roadmap for future research. In conclusion, these advances in the fields of piezoelectric materials, ultrasound transducers, and conformable electronics spark an emerging era of biomedicine and personal healthcare.

     
    more » « less
  2. This paper presents development of a three dimensional finite element model for simulations of a conformable piezoelectric sensor utilizing COMSOL Multiphysics. The sensor has a multi-layer structure composed of four circular piezoelectric elements arranged in an array structure laminated on a soft substrate and is capable of providing a strain mapping of soft tissue surfaces for spatiotemporal biokinematic assessment of the facial skin. Here, we provide the finite element method (FEM) for the sensor to predict its electromechanical behavior. This paper studies the effect of the design parameters such as dimensions of the piezoelectric sensor and the substrate on voltage sensitivity and sensor compliance. The FEM model is established to understand the underlying physics and guide the mechanical characterization of the system. The developed model is experimentally verified through two series of tests. The first set of tests involve comprehensive in vitro mechanical testing to provide accurate measurements of strain during compression, stretching, and bending. The second set of tests present in vivo experiments on healthy and amyotrophic lateral sclerosis subjects. The experimentally verified FEM model provides a detailed insight into analyzing the response of the sensor which establishes new design rules for next generations of conformable piezoelectric sensors. 
    more » « less
  3. Recent advancements in wearable technology have improved lifestyle and medical practices, enabling personalized care ranging from fitness tracking, to real-time health monitoring, to predictive sensing. Wearable devices serve as an interface between humans and technology; however, this integration is far from seamless. These devices face various limitations such as size, biocompatibility, and battery constraints wherein batteries are bulky, are expensive, and require regular replacement. On-body energy harvesting presents a promising alternative to battery power by utilizing the human body’s continuous generation of energy. This review paper begins with an investigation of contemporary energy harvesting methods, with a deep focus on piezoelectricity. We then highlight the materials, configurations, and structures of such methods for self-powered devices. Here, we propose a novel combination of thin-film composites, kirigami patterns, and auxetic structures to lay the groundwork for an integrated piezoelectric system to monitor and sense. This approach has the potential to maximize energy output by amplifying the piezoelectric effect and manipulating the strain distribution. As a departure from bulky, rigid device design, we explore compositions and microfabrication processes for conformable energy harvesters. We conclude by discussing the limitations of these harvesters and future directions that expand upon current applications for wearable technology. Further exploration of materials, configurations, and structures introduce interdisciplinary applications for such integrated systems. Considering these factors can revolutionize the production and consumption of energy as wearable technology becomes increasingly prevalent in everyday life. 
    more » « less
  4. The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness. Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman &Friesen, 1969). Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other based on their own internal state, or situation” (Queiroz et al. 2014, p.2).The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic). The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, the extent to which the presence of micro + macro-expressions affect the perceived expressivity and naturalness of the animated agents, the extent to which exaggerating the micro expressions, e.g. increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional expressivity, and whether there are differences based on the agent’s design characteristics. In the first study, 15 participants watched eight micro-expression animations representing four different emotions (happy, sad, fear, surprised). Four animations featured a stylized agent and four a realistic agent. For each animation, subjects were asked to identify the agent’s emotion conveyed by the micro-expression. In the second study, 234 participants watched three sets of eight animation clips (24 clips in total, 12 clips per agent). Four animations for each agent featured the character performing macro-expressions only, four animations for each agent featured the character performing macro- + micro-expressions without exaggeration, and four animations for each agent featured the agent performing macro + micro-expressions with exaggeration. Participants were asked to recognize the true emotion of the agent and rate the emotional expressivity ad naturalness of the agent in each clip using a 5-point Likert scale. We have collected all the data and completed the statistical analysis. Findings and discussion, implications for research and practice, and suggestions for future work will be reported in the full paper. ReferencesAdamo N., Benes, B., Mayer, R., Lei, X., Meyer, Z., &Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration 2021. IHSI 2021. Advances in Intelligent Systems and Computing, 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-68017-6_33Ekman, P., &Friesen, W. V. (1969, February). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106. https://doi.org/10.1080/00332747.1969.11023575 Queiroz, R. B., Musse, S. R., &Badler, N. I. (2014). Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces. Presence, 23(2), 191-208. http://dx.doi.org/10.1162/

     
    more » « less
  5. Abstract

    Lip‐reading provides an effective speech communication interface for people with voice disorders and for intuitive human–machine interactions. Existing systems are generally challenged by bulkiness, obtrusiveness, and poor robustness against environmental interferences. The lack of a truly natural and unobtrusive system for converting lip movements to speech precludes the continuous use and wide‐scale deployment of such devices. Here, the design of a hardware–software architecture to capture, analyze, and interpret lip movements associated with either normal or silent speech is presented. The system can recognize different and similar visemes. It is robust in a noisy or dark environment. Self‐adhesive, skin‐conformable, and semi‐transparent dry electrodes are developed to track high‐fidelity speech‐relevant electromyogram signals without impeding daily activities. The resulting skin‐like sensors can form seamless contact with the curvilinear and dynamic surfaces of the skin, which is crucial for a high signal‐to‐noise ratio and minimal interference. Machine learning algorithms are employed to decode electromyogram signals and convert them to spoken words. Finally, the applications of the developed lip‐reading system in augmented reality and medical service are demonstrated, which illustrate the great potential in immersive interaction and healthcare applications.

     
    more » « less