skip to main content


Title: Towards a Gesture-Based Story Authoring System: Design Implications from Feature Analysis of Iconic Gestures During Storytelling
Current systems that use gestures to enable storytelling tend to mostly rely on a pre-scripted set of gestures or the use of manipulative gestures with respect to tangibles. Our research aims to inform the design of gesture recognition systems for storytelling with implications derived from a feature-based analysis of iconic gestures that occur during naturalistic oral storytelling. We collected story retellings of a collection of cartoon stimuli from 20 study participants, and a gesture analysis was performed on videos of the story retellings focusing on iconic gestures. Iconic gestures are a type of representational gesture that provides information about objects such as their shape, location, or movement. The form features of the iconic gestures were analyzed with respect to the concepts that they portrayed. Patterns between the two were identified and used to create recommendations for patterns in gesture form a system could be primed to recognize.  more » « less
Award ID(s):
1736225 1929599
NSF-PAR ID:
10177573
Author(s) / Creator(s):
; ; ; ; ; ; ;
Date Published:
Journal Name:
Lecture notes in computer science
Volume:
11869
ISSN:
0302-9743
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Effective storytelling relies on engagement and interaction. This work develops an automated software platform for telling stories to children and investigates the impact of two design choices on children’s engagement and willingness to interact with the system: story distribution and the use of complex gesture. A storyteller condition compares stories told in a third person, narrator voice with those distributed between a narrator and first-person story characters. Basic gestures are used in all our storytellings, but, in a second factor, some are augmented with gestures that indicate conversational turn changes, references to other characters and prompt children to ask questions. An analysis of eye gaze indicates that children attend more to the story when a distributed storytelling model is used. Gesture prompts appear to encourage children to ask questions, something that children did, but at a relatively low rate. Interestingly, the children most frequently asked “why” questions. Gaze switching happened more quickly when the story characters began to speak than for narrator turns. These results have implications for future agent-based storytelling system research. 
    more » « less
  2. Effective storytelling relies on engagement and interaction. This work develops an automated software platform for telling stories to children and investigates the impact of two design choices on children’s engagement and willingness to interact with the system: story distribution and the use of complex gesture. A storyteller condition compares stories told in a third person, narrator voice with those distributed between a narrator and first-person story characters. Basic gestures are used in all our storytellings, but, in a second factor, some are augmented with gestures that indicate conversational turn changes, references to other characters and prompt children to ask questions. An analysis of eye gaze indicates that children attend more to the story when a distributed storytelling model is used. Gesture prompts appear to encourage children to ask questions, something that children did, but at a relatively low rate. Interestingly, the children most frequently asked “why” questions. Gaze switching happened more quickly when the story characters began to speak than for narrator turns. These results have implications for future agent-based storytelling system research. 
    more » « less
  3. Sandler, W. ; Aronoff, M. ; Padden, C. (Ed.)
    Variation in the linguistic use of handshapes exists across sign languages, but it is unclear how these iconic handshape preferences arise and become conventionalized. In order to understand the factors that shape such handshape preferences in the earliest stages of language emergence, we examined communication within family homesign systems. Homesigners are deaf individuals who have not acquired a signed or spoken language and who innovate unique gesture systems to communicate with hearing friends and family (“communication partners”). We analyzed how characteristics of participants and stimulus items influence handshape preferences and conventionalization. Participants included 11 deaf homesigners, 24 hearing communication partners (CPs), and 8 hearing non-signing adults from Nicaragua. Participants were asked to label items using gestures or signs. The handshape type (Handling, Object, or combined Handling+Object) was then coded. The participants and groups showed variability in iconic handshape preferences. Adult homesigners’ families demonstrated more conventionalization than did child homesigners’ families. Adult homesigners also used a combined Handling + Object form more than other participants. Younger CPs and those with fewer years of experience using a homesign system showed greater conventionalization. Items that elicited a reliable handshape preference were more likely to elicit Handling rather than Object handshapes. These findings suggest that similarity in terms of handshape type varies even within families, including hearing gesturers in the same culture. Although adult homesigners’ families were more conventionalized than child homesigners’ families, full conventionalization of these handshape preferences do not seem to appear reliably within two to three decades of use in a family when only one deaf homesigner uses it as a primary system. 
    more » « less
  4. Sandler, W. ; Aronoff, M. ; Padden C. (Ed.)
    Variation in the linguistic use of handshapes exists across sign languages, but it is unclear how these iconic handshape preferences arise and become conventionalized. In order to understand the factors that shape such handshape preferences in the earliest stages of language emergence, we examined communication within family homesign systems. Homesigners are deaf individuals who have not acquired a signed or spoken language and who innovate unique gesture systems to communicate with hearing friends and family (“communication partners”). We analyzed how characteristics of participants and stimulus items influence handshape preferences and conventionalization. Participants included 11 deaf homesigners, 24 hearing communication partners (CPs), and 8 hearing non-signing adults from Nicaragua. Participants were asked to label items using gestures or signs. 
    more » « less
  5. We compare the perceived naturalness of character animations generated using three interpolation methods: linear Euler, spherical linear quaternion, and spherical spline quaternion. While previous work focused on the mathematical description of these interpolation types, our work studies the perceptual evaluation of animated upper body character gestures generated using these interpolations. Ninety-seven participants watched 12 animation clips of a character performing four different upper body motions: a beat gesture, a deictic gesture, an iconic gesture, and a metaphoric gesture. Three animation clips were generated for each gesture using the three interpolation methods. The participants rated their naturalness on a 5-point Likert scale. The results showed that animations generated using spherical spline quaternion interpolation were perceived as significantly more natural than those generated using the other two interpolation methods. The findings held true for all subjects regardless of gender and animation experience and across all four gestures. 
    more » « less