skip to main content


Title: Multi-label activity recognition using activity-specific features and activity correlations
Award ID(s):
1763827
NSF-PAR ID:
10329447
Author(s) / Creator(s):
Date Published:
Journal Name:
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition
ISSN:
1063-6919
Page Range / eLocation ID:
14625 - 14635
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. In clinical settings, most automatic recognition systems use visual or sensory data to recognize activities. These systems cannot recognize activities that rely on verbal assessment, lack visual cues, or do not use medical devices. We examined speech-based activity and activity-stage recognition in a clinical domain, making the following contributions. (1) We collected a high-quality dataset representing common activities and activity stages during actual trauma resuscitation events-the initial evaluation and treatment of critically injured patients. (2) We introduced a novel multimodal network based on audio signal and a set of keywords that does not require a high-performing automatic speech recognition (ASR) engine. (3) We designed novel contextual modules to capture dynamic dependencies in team conversations about activities and stages during a complex workflow. (4) We introduced a data augmentation method, which simulates team communication by combining selected utterances and their audio clips, and showed that this method contributed to performance improvement in our data-limited scenario. In offline experiments, our proposed context-aware multimodal model achieved F1-scores of 73.2±0.8% and 78.1±1.1% for activity and activity-stage recognition, respectively. In online experiments, the performance declined about 10% for both recognition types when using utterance-level segmentation of the ASR output. The performance declined about 15% when we omitted the utterance-level segmentation. Our experiments showed the feasibility of speech-based activity and activity-stage recognition during dynamic clinical events.

     
    more » « less
  2. ABSTRACT At any given time, only a subset of microbial community members are active in their environment. The others are in a state of dormancy, with strongly reduced metabolic rates. It is of interest to distinguish active and inactive microbial cells and taxa to understand their functional contributions to ecosystem processes and to understand shifts in microbial activity in response to change. Of the methods used to assess microbial activity-dormancy dynamics, 16S rRNA/rRNA gene amplicons (16S ratios) and active cell staining with 5-cyano-2,3-ditolyl tetrazolium chloride (CTC) are two of the most common, yet each method has limitations. Given that in situ activity-dormancy dynamics are proxied only by laboratory methods, further study is needed to assess the level of agreement and potential complementarity of these methods. We conducted two experiments investigating microbial activity in plant-associated soils. First, we treated corn field soil with phytohormones to simulate plant soil stress signaling, and second, we used rhizosphere soil from common bean plants exposed to drought or nutrient enrichment. Overall, the 16S ratio and CTC methods exhibited similar patterns of relative activity across treatments when treatment effects were large, and the instances in which they differed could be attributed to changes in community size (e.g., cell death or growth). Therefore, regardless of the method used to assess activity, we recommend quantifying community size to inform ecological interpretation. Our results suggest that the 16S ratio and CTC methods report comparable patterns of activity that can be applied to observe ecological dynamics over time, space, or experimental treatment. IMPORTANCE Although the majority of microorganisms in natural ecosystems are dormant, relatively little is known about the dynamics of the active and dormant microbial pools through both space and time. The limited knowledge of microbial activity-dormancy dynamics is in part due to uncertainty in the methods currently used to quantify active taxa. Here, we directly compared two of the most common methods (16S ratios and active cell staining) for estimating microbial activity in plant-associated soil and found that they were largely in agreement in the overarching patterns. Our results suggest that 16S ratios and active cell staining provide complementary information for measuring and interpreting microbial activity-dormancy dynamics in soils. They also support the idea that 16S rRNA/rRNA gene ratios have comparative value and offer a high-throughput, sequencing-based option for understanding relative changes in microbiome activity, as long as this method is coupled with quantification of community size. 
    more » « less