More Like this

1. Bio-mimetic high-speed target localization with fused frame and event vision for edge application

   https://doi.org/10.3389/fnins.2022.1010302  

   Lele, Ashwin Sanjay ; Fang, Yan ; Anwar, Aqeel ; Raychowdhury, Arijit ( November 2022 , Frontiers in Neuroscience)

   Evolution has honed predatory skills in the natural world where localizing and intercepting fast-moving prey is required. The current generation of robotic systems mimics these biological systems using deep learning. High-speed processing of the camera frames using convolutional neural networks (CNN) (frame pipeline) on such constrained aerial edge-robots gets resource-limited. Adding more compute resources also eventually limits the throughput at the frame rate of the camera as frame-only traditional systems fail to capture the detailed temporal dynamics of the environment. Bio-inspired event cameras and spiking neural networks (SNN) provide an asynchronous sensor-processor pair (event pipeline) capturing the continuous temporal details of the scene for high-speed but lag in terms of accuracy. In this work, we propose a target localization system combining event-camera and SNN-based high-speed target estimation and frame-based camera and CNN-driven reliable object detection by fusing complementary spatio-temporal prowess of event and frame pipelines. One of our main contributions involves the design of an SNN filter that borrows from the neural mechanism for ego-motion cancelation in houseflies. It fuses the vestibular sensors with the vision to cancel the activity corresponding to the predator's self-motion. We also integrate the neuro-inspired multi-pipeline processing with task-optimized multi-neuronal pathway structure in primates andmore »insects. The system is validated to outperform CNN-only processing using prey-predator drone simulations in realistic 3D virtual environments. The system is then demonstrated in a real-world multi-drone set-up with emulated event data. Subsequently, we use recorded actual sensory data from multi-camera and inertial measurement unit (IMU) assembly to show desired working while tolerating the realistic noise in vision and IMU sensors. We analyze the design space to identify optimal parameters for spiking neurons, CNN models, and for checking their effect on the performance metrics of the fused system. Finally, we map the throughput controlling SNN and fusion network on edge-compatible Zynq-7000 FPGA to show a potential 264 outputs per second even at constrained resource availability. This work may open new research directions by coupling multiple sensing and processing modalities inspired by discoveries in neuroscience to break fundamental trade-offs in frame-based computer vision 1 .« less
2. MOSS—Multi-Modal Best Subset Modeling in Smart Manufacturing

   https://doi.org/10.3390/s21010243  

   Wang, Lening ; Du, Pang ; Jin, Ran ( January 2021 , Sensors)

   Smart manufacturing, which integrates a multi-sensing system with physical manufacturing processes, has been widely adopted in the industry to support online and real-time decision making to improve manufacturing quality. A multi-sensing system for each specific manufacturing process can efficiently collect the in situ process variables from different sensor modalities to reflect the process variations in real-time. However, in practice, we usually do not have enough budget to equip too many sensors in each manufacturing process due to the cost consideration. Moreover, it is also important to better interpret the relationship between the sensing modalities and the quality variables based on the model. Therefore, it is necessary to model the quality-process relationship by selecting the most relevant sensor modalities with the specific quality measurement from the multi-modal sensing system in smart manufacturing. In this research, we adopted the concept of best subset variable selection and proposed a new model called Multi-mOdal beSt Subset modeling (MOSS). The proposed MOSS can effectively select the important sensor modalities and improve the modeling accuracy in quality-process modeling via functional norms that characterize the overall effects of individual modalities. The significance of sensor modalities can be used to determine the sensor placement strategy in smart manufacturing.more »Moreover, the selected modalities can better interpret the quality-process model by identifying the most correlated root cause of quality variations. The merits of the proposed model are illustrated by both simulations and a real case study in an additive manufacturing (i.e., fused deposition modeling) process.« less
3. BioFace-3D: continuous 3d facial reconstruction through lightweight single-ear biosensors

   https://doi.org/10.1145/3447993.3483252  

   Wu, Yi ; Kakaraparthi, Vimal ; Li, Zhuohang ; Pham, Tien ; Liu, Jian ; Nguyen, Phuc ( October 2021 , Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (MobiCom '21))

   Over the last decade, facial landmark tracking and 3D reconstruction have gained considerable attention due to their numerous applications such as human-computer interactions, facial expression analysis, and emotion recognition, etc. Traditional approaches require users to be confined to a particular location and face a camera under constrained recording conditions (e.g., without occlusions and under good lighting conditions). This highly restricted setting prevents them from being deployed in many application scenarios involving human motions. In this paper, we propose the first single-earpiece lightweight biosensing system, BioFace-3D, that can unobtrusively, continuously, and reliably sense the entire facial movements, track 2D facial landmarks, and further render 3D facial animations. Our single-earpiece biosensing system takes advantage of the cross-modal transfer learning model to transfer the knowledge embodied in a high-grade visual facial landmark detection model to the low-grade biosignal domain. After training, our BioFace-3D can directly perform continuous 3D facial reconstruction from the biosignals, without any visual input. Without requiring a camera positioned in front of the user, this paradigm shift from visual sensing to biosensing would introduce new opportunities in many emerging mobile and IoT applications. Extensive experiments involving 16 participants under various settings demonstrate that BioFace-3D can accurately track 53 major facialmore »landmarks with only 1.85 mm average error and 3.38\% normalized mean error, which is comparable with most state-of-the-art camera-based solutions. The rendered 3D facial animations, which are in consistency with the real human facial movements, also validate the system's capability in continuous 3D facial reconstruction.« less
4. Experimental Exploration of Objective Human Pain Assessment Using Multimodal Sensing Signals

   https://doi.org/10.3389/fnins.2022.831627  

   Lin, Yingzi ; Xiao, Yan ; Wang, Li ; Guo, Yikang ; Zhu, Wenchao ; Dalip, Biren ; Kamarthi, Sagar ; Schreiber, Kristin L. ; Edwards, Robert R. ; Urman, Richard D. ( February 2022 , Frontiers in Neuroscience)

   Optimization of pain assessment and treatment is an active area of research in healthcare. The purpose of this research is to create an objective pain intensity estimation system based on multimodal sensing signals through experimental studies. Twenty eight healthy subjects were recruited at Northeastern University. Nine physiological modalities were utilized in this research, namely facial expressions (FE), electroencephalography (EEG), eye movement (EM), skin conductance (SC), and blood volume pulse (BVP), electromyography (EMG), respiration rate (RR), skin temperature (ST), blood pressure (BP). Statistical analysis and machine learning algorithms were deployed to analyze the physiological data. FE, EEG, SC, BVP, and BP proved to be able to detect different pain states from healthy subjects. Multi-modalities proved to be promising in detecting different levels of painful states. A decision-level multi-modal fusion also proved to be efficient and accurate in classifying painful states.
5. Activity recognition in manufacturing: The roles of motion capture and sEMG+inertial wearables in detecting fine vs. gross motion

   https://doi.org/10.1109/ICRA.2019.8793954  

   Kubota, Alyssa ; Iqbal, Tariq ; Shah, Julie A. ; Riek, Laurel D. ( January 2019 , 2019 International Conference on Robotics and Automation (ICRA))

   In safety-critical environments, robots need to reliably recognize human activity to be effective and trust-worthy partners. Since most human activity recognition (HAR) approaches rely on unimodal sensor data (e.g. motion capture or wearable sensors), it is unclear how the relationship between the sensor modality and motion granularity (e.g. gross or fine) of the activities impacts classification accuracy. To our knowledge, we are the first to investigate the efficacy of using motion capture as compared to wearable sensor data for recognizing human motion in manufacturing settings. We introduce the UCSD-MIT Human Motion dataset, composed of two assembly tasks that entail either gross or fine-grained motion. For both tasks, we compared the accuracy of a Vicon motion capture system to a Myo armband using three widely used HAR algorithms. We found that motion capture yielded higher accuracy than the wearable sensor for gross motion recognition (up to 36.95%), while the wearable sensor yielded higher accuracy for fine-grained motion (up to 28.06%). These results suggest that these sensor modalities are complementary, and that robots may benefit from systems that utilize multiple modalities to simultaneously, but independently, detect gross and fine-grained motion. Our findings will help guide researchers in numerous fields of robotics includingmore »learning from demonstration and grasping to effectively choose sensor modalities that are most suitable for their applications.« less