An official website of the United States government
Feedback is essential for learning a new skill or improving one’s current skill-level. However, current methods for skill-assessment from video only provide scores or compare demonstrations, leaving the burden of knowing what to do differently on the user. We introduce a novel method to generate actionable feedback (AF) from video of a person doing a physical activity, such as basketball or soccer. Our method takes a video demonstration and its accompanying 3D body pose and generates (1) free-form expert commentary describing what the person is doing well and what they could improve, and (2) a visual expert demonstration that incorporates the required corrections. We show how to leverage Ego-Exo4D’s [29] videos of skilled activity and expert commentary together with a strong language model to create a weakly-supervised training dataset for this task, and we devise a multimodal video-language model to infer coaching feedback. Our method is able to reason across multi-modal input combinations to output fullspectrum, actionable coaching—expert commentary, expert video retrieval, and expert pose generation—outperforming strong vision-language models on both established metrics and human preference studies.
more »
« less
Attention in Skilled Behavior: an Argument for Pluralism
Abstract Peak human performance—whether of Olympic athletes, Nobel prize winners, or you cooking the best dish you’ve ever made—depends on skill. Skill is at the heart of what it means to excel. Yet, the fixity of skilled behavior can sometimes make it seem a lower-level activity, more akin to the movements of an invertebrate or a machine. Peak performance in elite athletes is often described, for example, as “automatic” by those athletes: “The most frequent response from participants (eight athletes and one coach) when describing the execution of a peak performance was the automatic execution of performance” (Anderson et al. 2014). While the automaticity of skilled behavior is widely acknowledged, some worry that too much automaticity in skill would challenge its ability to exhibit human excellence. And so two camps have developed: those who focus on the automaticity of skilled behavior, the “habitualists,” and those who focus on the higher-level cognition behind peak performance, the “intellectualists.” We take a different tack. We argue that skilled behavior weaves together automaticity and higher-level cognition, which we call “pluralism.” That is, we argue that automaticity and higher-level cognition are both normal features of skilled behavior that benefit skilled behavior. This view is hinted at in other quotes about automaticity in skill—while expert gamers describe themselves as “playing with” automaticity (Taylor and Elam 2018), expert musicians are said to balance automaticity with creativity through performance cues: “Performance cues allow the musician to attend to some aspects of the performance while allowing others to be executed automatically” (Chaffin and Logan 2006). We describe in this paper three ways that higher-level cognition and automaticity are woven together. The first two, level pluralism and synchronic pluralism , are described in other papers, albeit under different cover. We take our contribution to be both distinguishing the three forms and contributing the third, diachronic pluralism. In fact, we find that diachronic pluralism presents the strongest case against habitualism and intellectualism, especially when considered through the example of strategic automaticity . In each case of pluralism, we use research on the presence or absence of attention (e.g., in mind wandering) to explore the presence or absence of higher-level cognition in skilled behavior.
more »
« less
- Award ID(s):
- 1633722
- PAR ID:
- 10291355
- Date Published:
- Journal Name:
- Review of Philosophy and Psychology
- Volume:
- 12
- Issue:
- 3
- ISSN:
- 1878-5158
- Page Range / eLocation ID:
- 615 to 638
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract The motor cortex controls skilled arm movement by recruiting a variety of targets in the nervous system, and it is important to understand the emergent activity in these regions as refinement of a motor skill occurs. One fundamental projection of the motor cortex (M1) is to the cerebellum. However, the emergent activity in the motor cortex and the cerebellum that appears as a dexterous motor skill is consolidated is incompletely understood. Here, we report on low-frequency oscillatory (LFO) activity that emerges in cortico-cerebellar networks with learning the reach-to-grasp motor skill. We chronically recorded the motor and the cerebellar cortices in rats, which revealed the emergence of coordinated movement-related activity in the local-field potentials as the reaching skill consolidated. Interestingly, we found this emergent activity only in the rats that gained expertise in the task. We found that the local and cross-area spiking activity was coordinated with LFOs in proficient rats. Finally, we also found that these neural dynamics were more prominently expressed during accurate behavior in the M1. This work furthers our understanding on emergent dynamics in the cortico-cerebellar loop that underlie learning and execution of precise skilled movement.more » « less
-
IntroductionGait automaticity refers to the ability to walk with minimal recruitment of attentional networks typically mediated through the prefrontal cortex (PFC). Reduced gait automaticity (i.e., greater use of attentional resources during walking) is common with aging, contributing to an increased risk of falls and reduced quality of life. A common assessment of gait automaticity involves examining PFC activation using near-infrared spectroscopy (fNIRS) during dual-task (DT) paradigms, such as walking while performing a cognitive task. However, neither PFC activity nor task performance in isolation measures automaticity accurately. For example, greater PFC activation could be interpreted as worse gait automaticity when accompanied by poorer DT performance, but when accompanied by better DT performance, it could be seen as successful compensation. Thus, there is a need to incorporate behavioral performance and PFC measurements for a more comprehensive evaluation of gait automaticity. To address this need, we propose a novel attentional gait index as an analytical approach that combines changes in PFC activity with changes in DT performance to quantify automaticity, where a reduction in automaticity will be reflected as an increased need for attentional gait control (i.e., larger index). MethodsThe index was validated in 173 participants (≥65 y/o) who completed DTs with two levels of difficulty while PFC activation was recorded with fNIRS. The two DTs consisted of reciting every other letter of the alphabet while walking over either an even or uneven surface. ResultsAs DT difficulty increases, more participants showed the anticipated increase in the attentional control of gait (i.e., less automaticity) as measured by the novel index compared to PFC activation. Furthermore, when comparing across individuals, lower cognitive function was related to higher attentional gait index, but not PFC activation or DT performance. ConclusionThe proposed index better quantified the differences in attentional control of gait between tasks and individuals by providing a unified measure that includes both brain activation and performance. This new approach opens exciting possibilities to assess participant-specific deficits and compare rehabilitation outcomes from gait automaticity interventions.more » « less
-
Learners' awareness of their own affective states (emotions) can improve their meta-cognition, which is a critical skill of being aware of and controlling one's cognitive, motivational, and affect, and adjusting their learning strategies and behaviors accordingly. To investigate the effect of peers' affects on learners' meta-cognition, we proposed two types of cues that aggregated peers' affects that were recognized via facial expression recognition:Locative cues (displaying the spikes of peers' emotions along a video timeline) andTemporal cues (showing the positivities of peers' emotions at different segments of a video). We conducted a between-subject experiment with 42 college students through the use of think-aloud protocols, interviews, and surveys. Our results showed that the two types of cues improved participants' meta-cognition differently. For example, interacting with theTemporal cues triggered the participants to compare their own affective responses with their peers and reflect more on why and how they had different emotions with the same video content. While the participants perceived the benefits of using AI-generated peers' cues to improve their awareness of their own learning affects, they also sought more explanations from their peers to understand the AI-generated results. Our findings not only provide novel design implications for promoting learners' meta-cognition with privacy-preserved social cues of peers' learning affects, but also suggest an expanded design framework for Explainable AI (XAI).more » « less
-
null (Ed.)In this paper, we argue in favor of creating robots that both teach and learn. We propose a methodology for building robots that can learn a skill from an expert, perform the skill independently or collaboratively with the expert, and then teach the same skill to a novice. This requires combining insights from learning from demonstration, human-robot collaboration, and intelligent tutoring systems to develop knowledge representations that can be shared across all three components. As a case study for our methodology, we developed a glockenspiel-playing robot. The robot begins as a novice, learns how to play musical harmonies from an expert, collaborates with the expert to complete harmonies, and then teaches the harmonies to novice users. This methodology allows for new evaluation metrics that provide a thorough understanding of how well the robot has learned and enables a robot to act as an efficient facilitator for teaching across temporal and geographic separation.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1007/s13164-021-00529-6
