An official website of the United States government
We present Beholder, a computer vision (CV) toolkit for building tangible controllers for interactive computer systems. Beholder facilitates designers to build physical inputs that are instrumented with CV markers. By observing the properties of these markers, a CV system can detect physical interactions that occur. Beholder provides a software editor that enables designers to map CV marker behavior to keyboard events; thus connecting the CV-driven tangible controllers to any software that responds to keyboard input. We propose three design scenarios for Beholder—controllers to support everyday work, alternative controllers for games, and transforming physical therapy equipment into controllers to monitor patient progress.
more »
« less
Marking Material Interactions with Computer Vision
The electronics-centered approach to physical computing presents challenges when designers build tangible interactive systems due to its inherent emphasis on circuitry and electronic components. To explore an alternative physical computing approach we have developed a computer vision (CV) based system that uses a webcam, computer, and printed fiducial markers to create functional tangible interfaces. Through a series of design studios, we probed how designers build tangible interfaces with this CV-driven approach. In this paper, we apply the annotated portfolio method to reflect on the fifteen outcomes from these studios. We observed that CV markers offer versatile materiality for tangible interactions, afford the use of democratic materials for interface construction, and engage designers in embodied debugging with their own vision as a proxy for CV. By sharing our insights, we inform other designers and educators who seek alternative ways to facilitate physical computing and tangible interaction design.
more »
« less
- Award ID(s):
- 2040489
- PAR ID:
- 10484711
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9781450394215
- Page Range / eLocation ID:
- 1 to 17
- Format(s):
- Medium: X
- Location:
- Hamburg Germany
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Physical computing enables learners to create interactive projects using tangible materials and electronic components. These projects commonly utilize microcontroller boards like the micro:bit. In contrast, computer vision (CV) is a powerful technique for detecting input through interaction with everyday materials like paper, and it can be utilized for physical computing projects. However, CV-based toolkits are typically limited to input detection and rely on screen-based or projected outputs. This paper presents a hybrid approach that integrates a CV-based platform called Paper Playground with the micro:bit electronics platform. By combining CV-detected, paper-based inputs with the rich input-output possibilities of microcontroller-based systems, we showcase a multimodal physical computing toolkit. Through three project examples, we explore how this hybrid approach can enhance the creative possibilities in physical computing, and develop a preliminary design space combining CV-based and electronics-based physical computing.more » « less
-
Within the CSCL community and in computing and computational making more broadly, issues of equity continue to be under-researched and undertheorized. Here, we examine how FUSE Studios – a set of in-school, choice-based, STEAM learning environments, based around a set of digital and tangible making challenges – supports equitable access to and participation in making and computing. Drawing on web-log data and video-ethnographic data, we argue that four characteristics of FUSE support equity: the design of the challenges; the diverse ways of knowing and doing supported by the activity system; the specific interactions encouraged by the activity system; and the program’s placement inside school. We focus, here, on gender equity, but also discuss implications and planned research on other aspects of equity.more » « less
-
OpenVX is a recently ratified standard that was expressly proposed to facilitate the design of computer-vision (CV) applications used in real-time embedded systems. Despite its real-time focus, OpenVX presents several challenges when validating real-time constraints. Many of these challenges are rooted in the fact that OpenVX only implicitly defines any notion of a schedulable entity. Under OpenVX, CV applications are specified in the form of processing graphs that are inherently considered to execute monolithically end-to-end. This monolithic execution hinders parallelism and can lead to significant processing-capacity loss. Prior work partially addressed this problem by treating graph nodes as schedulable entities, but under OpenVX, these nodes represent rather coarse-grained CV functions, so the available parallelism that can be obtained in this way is quite limited. In this paper, a much more fine-grained approach for scheduling OpenVX graphs is proposed. This approach was designed to enable additional parallelism and to eliminate schedulability-related processing-capacity loss that arises when programs execute on both CPUs and graphics processing units (GPUs). Response-time analysis for this new approach is presented and its efficacy is evaluated via a case study involving an actual CV application.more » « less
-
Computer Vision (CV) is used in a broad range of Cyber-Physical Systems such as surgical and factory floor robots and autonomous vehicles including small Unmanned Aerial Systems (sUAS). It enables machines to perceive the world by detecting and classifying objects of interest, reconstructing 3D scenes, estimating motion, and maneuvering around objects. CV algorithms are developed using diverse machine learning and deep learning frameworks, which are often deployed on limited resource edge devices. As sUAS rely upon an accurate and timely perception of their environment to perform critical tasks, problems related to CV can create hazardous conditions leading to crashes or mission failure. In this paper, we perform a systematic literature review (SLR) of CV-related challenges associated with CV, hardware, and software engineering. We then group the reported challenges into five categories and fourteen sub-challenges and present existing solutions. As current literature focuses primarily on CV and hardware challenges, we close by discussing implications for Software Engineering, drawing examples from a CV-enhanced multi-sUAS system.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3544548.3580643
