More Like this

1. Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders

   https://doi.org/10.1371/journal.pcbi.1009439  

   Whiteway, Matthew R.; Biderman, Dan; Friedman, Yoni; Dipoppa, Mario; Buchanan, E. Kelly; Wu, Anqi; Zhou, John; Bonacchi, Niccolò; Miska, Nathaniel J.; Noel, Jean-Paul; et al (September 2021, PLOS Computational Biology)

   Theunissen, Frédéric E. (Ed.)

   Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone. 

   more » « less
2. How do students’ achievement goals relate to learning from well-designed instructional videos and subsequent exam performance?

   https://doi.org/10.1016/j.cedpsych.2023.102162  

   Kuhlmann, Shelbi L.; Bernacki, Matthew L.; Greene, Jeffrey A.; Hogan, Kelly A.; Evans, Mara; Plumley, Robert; Gates, Kathleen; Panter, Abigail (April 2023, Contemporary Educational Psychology)

   Well-designed instructional videos are powerful tools for helping students learn and prompting students to use generative strategies while learning from videos further bolsters their effectiveness. However, little is known about how individual differences in motivational factors, such as achievement goals, relate to how students learn within multimedia environments that include instructional videos and generative strategies. Therefore, in this study, we explored how achievement goals predicted undergraduate students’ behaviors when learning with instructional videos that required students to answer practice questions between videos, as well as how those activities predicted subsequent unit exam performance one week later. Additionally, we tested the best measurement models for modeling achievement goals between traditional confirmatory factor analysis and bifactor confirmatory factor analysis. The bifactor model fit our data best and was used for all subsequent analyses. Results indicated that stronger mastery goal endorsement predicted performance on the practice questions in the multimedia learning environment, which in turn positively predicted unit exam performance. In addition, students’ time spent watching videos positively predicted practice question performance. Taken together, this research emphasizes the availing role of adaptive motivations, like mastery goals, in learning from instructional videos that prompt the use of generative learning strategies. 

   more » « less
3. Semantic Visual Navigation by Watching YouTube Videos

   Chang, Matthew; Gupta, Arjun; Gupta, Saurabh (December 2020, Neural Information Processing Systems (NeurIPS), 2020)

   Semantic cues and statistical regularities in real-world environment layouts can improve efficiency for navigation in novel environments. This paper learns and leverages such semantic cues for navigating to objects of interest in novel environments, by simply watching YouTube videos. This is challenging because YouTube videos do not come with labels for actions or goals, and may not even showcase optimal behavior. Our method tackles these challenges through the use of Q-learning on pseudo-labeled transition quadruples (image, action, next image, reward). We show that such off-policy Q-learning from passive data is able to learn meaningful semantic cues for navigation. These cues, when used in a hierarchical navigation policy, lead to improved efficiency at the ObjectGoal task in visually realistic simulations. We observe a relative improvement of 15-83% over end-to-end RL, behavior cloning, and classical methods, while using minimal direct interaction. 

   more » « less
4. Interpreting Expert Annotation Differences in Animal Behavior

   Tjandrasuwita, Megan; Sun, Jennifer J.; Kennedy, Ann; Yue, Yisong (January 2021, CVPR 2021 Workshop on CV4Animation)

   Hand-annotated data can vary due to factors such as subjective differences, intra-rater variability, and differing annotator expertise. We study annotations from different ex- perts who labelled the same behavior classes on a set of an- imal behavior videos, and observe a variation in annotation styles. We propose a new method using program synthesis to help interpret annotation differences for behavior analysis. Our model selects relevant trajectory features and learns a temporal filter as part of a program, which corresponds to estimated importance an annotator places on that feature at each timestamp. Our experiments on a dataset from behav- ioral neuroscience demonstrate that compared to baseline approaches, our method is more accurate at capturing an- notator labels and learns interpretable temporal filters. We believe that our method can lead to greater reproducibility of behavior annotations used in scientific studies. We plan to release our code. 

   more » « less
5. Long-term Monitoring of Bird Flocks in the Wild

   https://doi.org/10.24963/ijcai.2023/704  

   Kshitiz; Shreshtha, Sonu; Mounir, Ramy; Vatsa, Mayank; Singh, Richa; Anand, Saket; Sarkar, Sudeep; Mali_Parihar, Sevaram (August 2023, International Joint Conferences on Artificial Intelligence Organization)

   Monitoring and analysis of wildlife are key to conservation planning and conflict management. The widespread use of camera traps coupled with AI-based analysis tools serves as an excellent example of successful and non-invasive use of technology for design, planning, and evaluation of conservation policies. As opposed to the typical use of camera traps that capture still images or short videos, in this project, we propose to analyze longer term videos monitoring a large flock of birds. This project, which is part of the NSF-TIH Indo-US joint R&D partnership, focuses on solving challenges associated with the analysis of long-term videos captured at feeding grounds and nesting sites, among other such locations that host large flocks of migratory birds. We foresee that the objectives of this project would lead to datasets and benchmarking tools as well as novel algorithms that would be instrumental in developing automated video analysis tools that could in turn help understand individual and social behavior of birds. The first of the key outcomes of this research will include the curation of challenging, real-world datasets for benchmarking various image and video analytics algorithms for tasks such as counting, detection, segmentation, and tracking. Our recent efforts towards this outcome is a curated dataset of 812 high-resolution, point-annotated, images (4K - 32MP) of a flock of Demoiselle cranes (Anthropoides virgo) taken from their feeding site at Khichan, Rajasthan, India. The average number of birds in each image is about 207, with a maximum count of 1500. The benchmark experiments show that state-of-the-art vision techniques struggle with tasks such as segmentation, detection, localization, and density estimation for the proposed dataset. Over the execution of this open science research, we will be scaling this dataset for segmentation and tracking in videos, as well as developing novel techniques for video analytics for wildlife monitoring. 

   more » « less