Physical computing is an engaging domain for learners to explore computer science by creating interactive, tangible projects in an open-ended, hands-on manner. Student and adult makers utilize physical computing to build a diverse range of projects, like robots, home automation, interactive art, etc. To build such projects, makers primarily rely on electronics and microcontroller toolkits, like the Arduinofoot_0 , BBC micro:bit 2 , and Raspberry Pifoot_2 platforms. These toolkits o"er connection to and control of a range of sensors, actuators, LEDs, and other electronic components to program and develop physical computing projects. Computer Vision (CV) o"ers an alternate technique for detecting input and has long been used to build tangible interfaces for various applications [10,18,25]. Instead of using specialized electronics, CV utilizes a single camera feed and everyday materials to sense a range of interactions and environmental variables-making it a versatile tool for physical computing, as showcased recently by Gyory et al [6].
However, CV is inherently a sensing technique, and can only be used for input detection. The tangible input detected using CV is often paired with visual output on a screen or projector, or audio output through a computing device, such as Dynamicland 4 and ReacTIVision [10]. Paper Playground is such a CV prototyping platform that enables building paper-based tangible interfaces and ties them to audiovisual output using a web-based development environment [4] 5 . Projection overlay from overhead (e.g. Paper Playground and Dynamicland) or from under the table (e.g. Re-acTIVision) provides in-place and direct output for the CV-based tangible interfaces, similar to electronic physical computing. However, these are a small portion of the creative space of traditional physical computing, which often integrates richer interactions with the physical world beyond digital displays.
In this work-in-progress, we explore the opportunities a"orded by integrating a CV-based physical computing toolkit, Paper Playground, with electronics-based capabilities of the micro:bit. By connecting these two platforms, we seek to ask, "what creative opportunities might a multimodal physical computing toolkit enable?" This paper showcases three prototypes built by the research team with this toolkit to examine how CV-detected, paper-based input may augment the traditional physical computing paradigm, and how integrating electronic components may provide greater possibilities for CV-based tangible interfaces. Through this exploration, we begin to chart the design space for such multimodal physical computing platforms, setting the stage for future exploration.
In recent years, physical computing, or, computing in the physical world, has been a popular domain for engaging children and adults with computer science and technology. Physical computing ties in closely with constructionist and maker-oriented philosophies of learning [3] because it involves learners creating physical interactive artifacts in an open-ended manner [7,17]. Such an approach to computing education can be more welcoming and positive than screen-based learning because it leverages tangible materials to build meaningful projects in diverse domains such as robotics, wearables, environmental sensing and smart home automation [15,22]. Tangible and physical computing environments also have a positive e"ect on collaboration and active learning because they engage students in working together in a visible way [8,14]. As remarked by Hodges et al [7], physical computing projects are not only limited to computer science education, but connect to other science and engineering domains, and to interactive and performing arts.
Physical computing traditionally goes hand-in-hand with electronics and involves students connecting and programming various electronic components together to build their projects. Over the past few decades, there have been numerous toolkits and devices to engage students in physical computing at di"erent levels of #exibility and complexity. The dominant approach to physical computing is microcontroller boards like the micro:bit, Arduino, and LilyPad [2]. These boards consist of embedded microcontrollers that can be programmed using a block-based or code-based development environment on a computer. They o"er ease of connection with and programming of electronic peripherals for input like buttons, and sensors, and output like LEDs, motors, and speakers. Micro:bit, in particular, is a microcontroller board that integrates components like buttons, 5x5 LED grid, accelerometer, magnetometer, temperature sensor, light-level sensor, microphone, and speakers into one small package, making it easier for learners to get started without needing to handle external circuits and wiring. MakeCodefoot_5 is a web-based development environment for the micro:bit that o"ers block-based graphical programming, as well as text-based programming in JavaScript and Python. In this project, we used the micro:bit and its capabilities for our integrated paper-based physical computing platform.
Aside from electronics, Computer Vision (CV) has often been used for computational learning interfaces, whether it is for describing code-e.g. Quetzal and Tern [9], HyperCubes [5], Kart-ON [19]-or for building other computational artifacts-e.g. Cartoonimator [16] for animation, TADCAD [23] for 3D modeling, Knowledge Net and Creature Features [13] for machine learning. CV-based tangible interfaces o"er the bene!ts of tangible interfaces for learning, such as natural and intuitive interaction, playful and multisensory learning, and collaboration opportunities [1,12]. They are also more practical and accessible in diverse learning environments because they utilize everyday materials like paper and cardboard instead of special electronics [19].
Projects like Beholder [6], Printed Paper Markers [27], DynaTags [21], SLAP Widgets [24] have also explored using CV for physical computing and prototyping input devices for various applications instead of using electronic buttons, sliders, sensors, etc. These projects apply CV techniques like !ducial markers and blob detection on a camera feed to detect changes in physical input devices like a cardboard slider, acrylic button, paper spring, etc., or interactions with the environment, such as light sensing, water drops, etc. Gyory et al. [6] describe how a single type of material-CV !ducial markers-applied di"erently can accomplish the functionalities of several electronic sensors. Further, they discuss how debugging in such CV-driven physical computing can be more intuitive than in electronic physical computing because it relies on visual cues we can see and parse with our eyes, instead of "intangible electrical signals" that we need other tools and libraries to debug.
However, it should be noted that computer vision is by de!nition a sensing mechanism, and can therefore only be a proxy for the inputs in physical computing. CV-based tangible interfaces normally serve as the input for a variety of outputs, such as visual displays, audio, AR/VR, or robots. Projects like ReacTIVision and Dynamicland use a projector to provide visual output in situ with tangible input. Paper Playground (PP) is inspired by Dynamicland and builds upon Paper Programs 7 , an open-source development toolkit for Dynamicland-like experiences. PP developed signi!cant extensions to these existing infrastructures and o"ers a low-code platform to build such CV-based interfaces, as described in the next section. In this paper, we utilized the PP platform to connect the CV-based input to the popular micro:bit physical computing platform to o"er input and output possibilities in physical computing beyond those built into either platform individually.
Paper Playground (PP) is an open-source design tool for makers to build paper-based tangible interfaces based on Computer Vision (CV). The platform involves creating programs associated with physical (or virtual) papers that execute when a camera recognizes !xed colored dot sequences (Fig. 1) on the border of the paper. These paper programs or paper cards can be programmed to perform various actions and produce audio-visual output-on a screen or overlaid on top of the paper using a projector, like in Dynamicland. Multiple paper cards can be placed and interacted with simultaneously. The paper cards can support tracking of tangible interaction through bigger color dots or markers-it can detect 4 di"erent color markers, and their positions on the card, which can be utilized for various forms of input, as described in the next section. The Paper Playground platform was !rst introduced in [4], where the authors discuss !ndings from using PP in remote co-design sessions with youth. This work-in-progress showcases PP as a multimodal physical computing platform with micro:bit and explores the kind of making opportunities it provides.
The Paper Playground platform involves three interfaces: Creator, Camera, and Interactive Display. The Creator provides a low-code interface based on the Model-View-Controller (MVC) architecture [11] for makers to program behaviors for the paper cards. By including components for models (data or variables), controller (paper 7 https://paperprograms.org/ events and logic), and view (display or sound output), PP provides a low-barrier and organized programming interface for makers to quickly develop paper programs with minimal code. The Creator interface also includes Bluetooth components that can connect to a micro:bit v2 for transmitting data or events, such as a button press. The Camera interface helps makers set up and calibrate their webcam using a debugging view that can also be used to simulate virtual paper programs for testing. Lastly, the Interactive Display is the page managing the visual output that can be cast to a projector or a secondary screen. The Paper Playground website 8 documents the requirements, setup, and process of building projects with Paper Playground.
In this section, we demonstrate three physical computing projects we built with Paper Playground and the micro:bit. These showcase the variety of interactive creations possible with this platform, that may otherwise be not possible or di$cult to execute with a traditional electronics-based physical computing kit. These projects are built with common materials like printed paper, cardboard, straws, thumbtacks, and 3D printed objects, along with the micro:bit and electrical components like alligator clips. For each of these projects, we developed a MakeCode project to upload to the micro:bit, which handles Bluetooth communication with PP on the computer. These projects are primarily built in PP using the MVC model, and a small MakeCode project handling the Bluetooth communication on the micro:bit. Please refer to the supplementary video to see these projects in action.
The PP Music Station (Fig. 1A) enables makers to create, play, and mix melodies on the micro:bit speaker/buzzer. The interface includes a note sequence grid (Fig. 2A) upon which players can place markers to represent an 8-note melody, where the rows represent notes (A-G), and the columns represent the notes' position in the melody. The players can place marker pegs on the grid to de!ne a melody. They can further control the pitch (octave) of the notes, and the beats per minute of the melody by manipulating the two cardboard sliders on the melody controller card (Fig. 2B). While the micro:bit is used to play basic notes, a player can use the sound card (Fig. 2C) for additional music or sound clips that are played on the computer. This card features a dial with four markers that can be revealed to trigger the sounds at any point during the melody. PP enables customizing each of these markers to play a di"erent sound that can be uploaded from the computer.
The Music Station provides a spatially collaborative music making interface for the micro:bit. While one can currently program a micro:bit to play a melody using MakeCode, this interface enables players to modify it in real time and mix it with other music or sounds. The paper-based interface enables makers to easily create complex tangible inputs like the note sequence grid with cardboardwhich, using electronics, would have required wiring of 8x7=56 bistable buttons.
With the micro:bit and Paper Playground's projection output, makers can create spatial video games on a tabletop where players directly interact with game elements using tangible interfaces. To explore this possibility, we developed a sur!ng game (Fig. 1B) where one player controls the micro:bit is used to move a character surfing in the sea, while other players can launch waves or enemies from anywhere in the sea using paper cards. The !rst player moves the surfer to catch waves to get points or avoid enemies (Fig. 3A). The wave card is created as a wind sensor with a paper #ap that a player blows on to start the wave from the position of the card (Fig. 3B). The enemy card resembles an underwater cage that a player stretches to release the Kraken enemy (Fig. 3C). Blowing on the paper #ap or stretching the paper card reveals a marker to the camera to trigger the action.
This example project demonstrates the utility of Paper Playground to create custom, context-relevant sensors like the wind and the stretch sensor. The CV-based approach also enables easy tracking of input in the 2D space for spatial interactions, which is signi!cantly more complicated using electronic methods.
The two previous projects demonstrate co-located interactions between the micro:bit and Paper Playground. The Bluetooth connection between the micro:bit and PP also o"ers opportunities to build remote communication projects. This is useful in home automation and Internet-of-Things (IoT) projects which are very common in the physical computing community. Using this platform, we built a plant monitoring system that involves a micro:bit smart planter to monitor the temperature and light using internal and soil moisture using external prongs dug into the plant (Fig. 4C,E). This data is communicated via Bluetooth to PP host running on a computer with a visual dashboard for real-time updates (Fig. 4D). The maker can further use paper-based tangible input to send commands to the micro:bit to electronically control lighting and water to the plant. The light control card gets activated when the user shines a #ashlight on the sun illustration, which makes the marker invisible to the camera due to overexposure (4A). The watering card involves a dial mechanism to "empty" a watering can by rotating it and hiding the blue marker underneath the dial (Fig. 4B).
While such remote home automation projects are common with traditional physical computing, they usually involve a complicated setup of wireless protocols like Bluetooth or WiFi. Paper Playground neatly packages Bluetooth setup with the micro:bit and provides a paper-based interface on the remote side that can be extended easily with fewer resources.
These prototypes were designed and implemented by the !rst author, who has considerable experiences building electronics-based physical computing artifacts. After becoming familiar with the PP system, each of these projects took about 1-2 days for development from conceptualization to iterative testing and the !nished artifacts. Working in this CV-based platform involved thinking about the projects from the perspective of appropriate tangible interactions for the context and applications. While traditionally, one might need knowledge of electronics and di"erent types of sensors and input to incorporate into a project, the key design aspect for these projects was embedding logic into tangible input using CV markers. Consequently, the debugging was primarily visual observation of when and where the markers are detected in the camera view, instead of detecting "invisible" electrical signals from sensors and other electrical components. The PP platform also supported this visual debugging by providing a camera view highlighting detected programs and markers. Being a CV system, however, PP has its set of challenges such as dealing with occlusion by hands, heads, and necessity of calibration to ambient lighting for accurate color detection. PP currently enables detection of only four unique markers, which poses a constraint on the types of interfaces that can be designed. Lastly, these three projects showcase a paradigm of tabletop tangible interfaces with a top-down facing camera. Prior work has shown the potential of using CV with embedded or handheld cameras [20,26] to enable non-stationary and more dynamic interaction. For future, we may look at supporting di"erent modes of tangible interaction using CV and explore the kinds of physical computing projects that that accommodates.
Physical computing with electronics has been widely popular means for students and makers to create computational artifacts in the real world [7]. Electronic components for sensing and output o"er endless possibilities, and microcontroller platforms like the micro:bit and Arduino provide a low barrier to realize those ideas. However, students and novices often struggle with implementing complex inputs with electronics. A CV-based platform like PP can make such input mechanisms much simpler, such as the note sequence grid in Music Station. CV also makes it feasible to detect continuous tracking in the 2D space, which my otherwise require grids of RFID or NFC sensors, or complex LIDAR sensors. Further, as noted in the Multiplayer Spatial Games and Remote Home Automation projects, the paper-based nature of PP o"ers the opportunity to create DIY sensors relevant to the application in mind, whether it is a wind-sensor to generate a wave, or a light-based sensor to turn on lighting. As remarked by Gyory et al. [6], CV also o"ers "visual debugging" for these custom sensors. The simplicity of sensing markers for various input enables greater creativity and #exibility, especially when a maker might not have access to all possible electronic sensors.
On the one hand, CV-based interfaces provide simpler and more creative opportunities for physical computing, and on the other hand, electronic components o"er greater input and output possibilities that are not directly possible with CV. These include complex and/or distributed sensing, and electro-mechanical output. For example, in the remote plant monitoring system, we utilized micro:bit's onboard temperature and light sensors and an external soil moisture sensor to get precise measurements and data about the plant located elsewhere. This project also showcases the use of paper-based interfaces to control electronic components through the micro:bit, such as a watering pump and lighting on the plant. While not demonstrated in these examples, makers can also build robotic projects with outputs like motors or actuators controlled by paper-based interfaces, which goes beyond the traditional audiovisual outputs o"ered by CV-based platforms.
Through this exploration of physical computing projects built by integrating a CV-based platform like Paper Playground with a microcontroller platform like micro:bit, we have begun to chart out a design space of multimodal physical computing. We categorize this design space of physical computing along the axis of input as physical electronic or CV-based, and output as physical electronic or device-based. Traditionally, physical computing platforms o"er electronic components for input like buttons, sliders, sensors for light, distance, motion, etc., and electronic components output, like LEDs, buzzers, motors, etc. Makers also often connect microcontrollers to computing devices like PCs and tablets for digital output on a screen, or sounds and music. Integration of physical computing with a paper-based platform like Paper Playground augments this design space to include a variety of CV-based input using paper to tie to electronic or digital output modalities. Figure 5 illustrates how the projects described in this paper !t in this design space of multimodal physical computing, and highlights the opportunities for further exploration beyond the traditional physical computing paradigm. Augmenting the old paradigm with CV-based paper tangible interfaces, learners and makers may experience greater creative freedom at a lower entry barrier, and through this preliminary design space we seek to engage researchers to examine and discuss these implications.
In this work-in-progress, we examine multimodal physical computing by integrating a CV-based platform, Paper Playground, with an electronics platform, micro:bit. We develop and showcase three prototypes-a music-making interface, a multiplayer spatial game, and a remote home automation system. These example projects help us highlight how utilizing CV-based sensing can simplify DIY input for physical computing, while electronic components can !ll the gaps of CV-based digital input/output. We used these insights to develop a preliminary design space of multimodal physical computing and set the stage for further research in this domain. The authors will bring these project prototypes along with video demonstrations to the conference for the TEI community to interact and engage with this exploration of this new paradigm.
In future work, we plan to further re!ne and explore the PP platform for physical computing with micro:bit and other microcontroller platforms, and provide it as open source to makers to evaluate its possibilities and limitations. We also seek to deepen the preliminary design space from these learnings and expand it to other CV techniques or infrastructures. Physical computing o"ers exciting opportunities for creative computational learning, and we are eager to see how this new paradigm of physical computing impacts learners' experiences.
https://www.arduino.cc/
https://microbit.org/
https://www.raspberrypi.com/
https://dynamicland.org/
https://phetsims.github.io/paper-land/
https://makecode.microbit.org/