Search for: All records

Physical computing enables learners to create interactive projects using tangible materials and electronic components. These projects commonly utilize microcontroller boards like the micro:bit. In contrast, computer vision (CV) is a powerful technique for detecting input through interaction with everyday materials like paper, and it can be utilized for physical computing projects. However, CV-based toolkits are typically limited to input detection and rely on screen-based or projected outputs. This paper presents a hybrid approach that integrates a CV-based platform called Paper Playground with the micro:bit electronics platform. By combining CV-detected, paper-based inputs with the rich input-output possibilities of microcontroller-based systems, we showcase a multimodal physical computing toolkit. Through three project examples, we explore how this hybrid approach can enhance the creative possibilities in physical computing, and develop a preliminary design space combining CV-based and electronics-based physical computing. 

One of the current limitations in digital educational experiences is the lack of touch. Touch is a critical component in the learning process and in creating inclusive educational experiences for sensorially diverse learners. From haptic devices to tangible user interfaces (TUI), a growing body of research is investigating ways to bring touch back into the digital world, yet many focus on a specific dimension (e.g. haptic feedback or kinesthetic manipulation) of touch. Learning, however, is a multi-dimensional touch experience - it is about moving and being moved. This work presents the Action Quad - a novel haptic-TUI design for teaching geometry (specifically quadrilaterals). The Action Quad is a multi-point-of-contact, reconfigurable tool that synergizes the affordances of both kinesthetic interaction and haptic feedback into a single form factor. We present findings from an initial user study (N=11) investigating how sighted- hearing individuals approach, interact, and experience the Action Quad, and we present a case study with an individual with blindness. We share key takeaways from the design process and participant feedback on interactions with this novel haptic-TUI device, sharing design insights on an emerging area of research that could support a new class of educational learning tools rooted in touch. 

Paper prototyping presents a low-entry barrier method to engaging youth in interaction design. Purely paper-based designs leave a large gap between ideation and implementation. Paper Playground is a prototyping tool that connects physical and virtual papers with JavaScript programs, enabling the creation of multimodal prototypes in both face-to-face and virtual settings. Paper Playground is being designed and developed through iterative co-design activities including youth and adults. Here we present findings from remote co-design sessions with youth, investigating what affordances the participants requested from a multimodal prototyping tool. We reflect on the co-designers desires and remarks on paper use for interactive project design, remote collaborative use, and extensibility for physical computing. 

Sensory extensions enhance our awareness by transforming variations in stimuli normally undetectable by human senses into perceivable outputs. Similarly, interactive simulations for learning promote an understanding of abstract phenomena. Combining sensory extension devices with interactive simulations gives users the novel opportunity to connect their sensory experiences in the physical world to computer-simulated concepts. We explore this opportunity by designing a suite of wearable sensory extension devices that interface with a uniquely inclusive PhET Simulation, Ratio and Proportion. In this simulation, two hands can be moved on-screen to various values, representing different mathematical ratios. Users explore changing hand heights to find and maintain ratios through visual and auditory feedback. Our sensory extension devices translate force, distance, sound frequency, and magnetic field strength to quantitative values in order to control individual hands in the computer simulation. This paper describes the design of the devices and our analysis of feedback from 23 high-school aged youth who used our designs to interact with the Ratio and Proportion simulation.