More Like this

1. Beyond the Screen: Gestural Perspective-Taking with a Biochemistry Simulation

   https://doi.org/10.1145/3613905.3650816  

   Kim, Taehyun; Nisar, Harris; Lindgren, Robb; Zhang, Jiahao; Tang, Xiaoyu; Lira, Matthew; Talhan, Aishwari (May 2024, ACM)

   NA (Ed.)

   ABSTRACT Significant research has been conducted on how students’ gestures aid in learning scientific concepts, yet there remains a gap in understanding the impact of gesture-based interactions between students and simulations on their interpretation of visualized scientific phenomena. Addressing this, our paper presents a usability test conducted on a dynamic equilibrium visualization simulation developed for introductory college courses. Through a user study involving 40 participants, we conducted a qualitative evaluation to determine how students interpret gesture-controlled simulations. The findings confirm that students generally interpret visualized scientific concepts effectively and that interacting through gestures enhances their interpretation of the simulations. Additionally, this paper discusses the limitations of the current study and suggests directions for future research. 

   more » « less
2. Laminated Layers of Abstraction in Physics Students’ Gesture

   https://doi.org/10.22318/icls2024.200841  

   Zirek, Cagla; Flood, Virginia J; Harrer, Benedikt W (June 2024, Proceedings of the International Conference of the Learning Sciences ICLS 2024)

   Lindgren, R; Asino, T; Kyza, E A; Looi, C-K; Keifert, D T; Suarez, E (Ed.)

   Gestures play a key role for physicists and physics students in representing physics entities, processes, and systems. One affordance of gesture is the ability to laminate or layer together representations of concrete physical features (e.g., objects and their interactions) and symbolic representations (e.g., coordinate systems) to make sense of and model physical scenarios. Using interaction analysis, we illustrate how students can laminate these different layers of abstraction together in gesture to generate complex explanations to solve physics problems. We argue that laminating different layers of abstraction (both the symbolic and concrete) constitute a key form of representational competence in physics. 

   more » « less
3. The Enactment of an Instant: A Study of Gesture in Introductory Physics

   https://doi.org/10.22318/icls2024.490612  

   Booth, Emma Teresa; Flood, Virginia J; Harrer, Benedikt W (June 2024, Proceedings of the International Conference of the Learning Sciences ICLS 2024)

   Lindgren, R; Asino, T; Kyza, E A; Looi, C-K; Keifert, D T; Suarez, E (Ed.)

   Gesture has been shown to play an important role in how learners conceptualize phenomena in physics. However, we know little about how gesture is used to conceptualize instantaneity. Drawing on multimodal microanalysis of interaction, we examine how undergraduate physics students use representational gesture to make sense of instants while modeling energy dynamics. Our analysis describes four different forms of representational gesture used to capture instantaneity: These include (1) Replay loop of scenario, (2) Subinterval on timeline, (3) Freeze frame of scenario, and (4) Indexical location on timeline. 

   more » « less
4. Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

   https://doi.org/10.1145/3427330  

   Williams, Adam S.; Ortega, Francisco R. (November 2020, Proceedings of the ACM on Human-Computer Interaction)

   null (Ed.)

   This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using augmented reality. The work presents a multimodal elicitation study conducted with 24 participants. The canonical referents for translation, rotation, and scale were used along with some abstract referents (create, destroy, and select). In this study time windows for gesture and speech multimodal interactions are developed using the start and stop times of gestures and speech as well as the stoke times for gestures. While gestures commonly precede speech by 81 ms we find that the stroke of the gesture is commonly within 10 ms of the start of speech. Indicating that the information content of a gesture and its co-occurring speech are well aligned to each other. Lastly, the trends across the most common proposals for each modality are examined. Showing that the disagreement between proposals is often caused by a variation of hand posture or syntax. Allowing us to present aliasing recommendations to increase the percentage of users' natural interactions captured by future multimodal interactive systems. 

   more » « less
5. Embodied mixed reality with passive haptics in STEM education: randomized control study with chemistry titration

   https://doi.org/10.3389/frvir.2023.1047833  

   Johnson-Glenberg, Mina C.; Yu, Christine S.; Liu, Frank; Amador, Charles; Bao, Yueming; Yu, Shufan; LiKamWa, Robert (July 2023, Frontiers in Virtual Reality)

   Researchers, educators, and multimedia designers need to better understand how mixing physical tangible objects with virtual experiences affects learning and science identity. In this novel study, a 3D-printed tangible that is an accurate facsimile of the sort of expensive glassware that chemists use in real laboratories is tethered to a laptop with a digitized lesson. Interactive educational content is increasingly being placed online, it is important to understand the educational boundary conditions associated with passive haptics and 3D-printed manipulables. Cost-effective printed objects would be particularly welcome in rural and low Socio-Economic (SES) classrooms. A Mixed Reality (MR) experience was created that used a physical 3D-printed haptic burette to control a computer-based chemistry titration experiment. This randomized control trial study with 136 college students had two conditions: 1) low-embodied control (using keyboard arrows), and 2) high-embodied experimental (physically turning a valve/stopcock on the 3D-printed burette). Although both groups displayed similar significant gains on the declarative knowledge test, deeper analyses revealed nuanced Aptitude by Treatment Interactions (ATIs). These interactionsfavored the high-embodied experimental group that used the MR devicefor both titration-specific posttest knowledge questions and for science efficacy and science identity. Those students with higher prior science knowledge displayed higher titration knowledge scores after using the experimental 3D-printed haptic device. A multi-modal linguistic and gesture analysis revealed that during recall the experimental participants used the stopcock-turning gesture significantly more often, and their recalls created a significantly different Epistemic Network Analysis (ENA). ENA is a type of 2D projection of the recall data, stronger connections were seen in the high embodied group mainly centering on the key hand-turning gesture. Instructors and designers should consider the multi-modal and multi-dimensional nature of the user interface, and how the addition of another sensory-based learning signal (haptics) might differentially affect lower prior knowledge students. One hypothesis is that haptically manipulating novel devices during learning may create more cognitive load. For low prior knowledge students, it may be advantageous for them to begin learning content on a more ubiquitous interface (e.g., keyboard) before moving them to more novel, multi-modal MR devices/interfaces. 

   more » « less