skip to main content


Title: Interaction Templates: A Data-Driven Approach for Authoring Robot Programs
Socially interactive robots present numerous unique programming challenges for interaction developers. While modern authoring tools succeed at making the authoring experience approachable and convenient for developers from a wide variety of backgrounds, they are less successful at targeting assistance to developers based on the specific task or interaction being authored. We propose interaction templates, a data-driven solution for (1) matching in-progress robot programs to candidate task or interaction models and then (2) providing assistance to developers by using the matched models to generate modifications to in-progress programs. In this paper, we present the various dimensions that define first how interaction templates might be used, then how interaction templates may be represented, and finally how they might be collected.  more » « less
Award ID(s):
1925043 1924435
NSF-PAR ID:
10340538
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
PLATEAU: 12th Annual Workshop at theIntersection of PL and HCI
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Statistical data manipulation is a crucial component of many data science analytic pipelines, particularly as part of data ingestion. This task is generally accomplished by writing transformation scripts in languages such as SPSS, Stata, SAS, R, Python (Pandas) and etc. The disparate data models, language representations and transformation operations supported by these tools make it hard for end users to understand and document the transformations performed, and for developers to port transformation code across languages. Tackling these challenges, we present a formal paradigm for statistical data transformation. It consists of a data model, called Structured Data Transformation Data Model (SDTDM), inspired by the data models of multiple statistical transformations frameworks; an algebra, Structural Data Transformation Algebra (SDTA), with the ability to transform not only data within SDTDM but also metadata at multiple structural levels; and an equivalent descriptive counterpart, called Structured Data Transformation Language (SDTL), recently adopted by the DDI Alliance that maintains international standards for metadata as part of its suite of products. Experiments with real statistical transformations on socio-economic data show that SDTL can successfully represent 86.1% and 91.6% respectively of 4,185 commands in SAS and 9,087 commands in SPSS obtained from a repository. We illustrate with examples how SDTA/SDTL could assist with the documentation of statistical data transformation, an important aspect often neglected in metadata of datasets.We propose a system called C2Metadata that automatically captures the transformation and provenance information in SDTL as a part of the metadata. Moreover, given the conversion mechanism from a source statistical language to SDTA/SDTL, we show how functional-equivalent transformation programs could be converted to other functionally equivalent programs, in the same or different language, permitting code reuse and result reproducibility, We also illustrate the possibility of using of SDTA to optimize SDTL transformations using rule-based rewrites similar to SQL optimizations. 
    more » « less
  2. Deep Learning (DL) models to analyze source code have shown immense promise during the past few years. More recently, self-supervised pre-training has gained traction for learning generic code representations valuable for many downstream SE tasks, such as clone and bug detection. While previous work successfully learned from different code abstractions (e.g., token, AST, graph), we argue that it is also essential to factor in how developers code day-to-day for general-purpose representation learning. On the one hand, human developers tend to write repetitive programs referencing existing code snippets from the current codebase or online resources (e.g., Stack Overflow website) rather than implementing functions from scratch; such behaviors result in a vast number of code clones. In contrast, a deviant clone by mistake might trigger malicious program behaviors. Thus, as a proxy to incorporate developers' coding behavior into the pre-training scheme, we propose to include code clones and their deviants. In particular, we propose CONCORD, a self-supervised, contrastive learning strategy to place benign clones closer in the representation space while moving deviants further apart. We show that CONCORD's clone-aware contrastive learning drastically reduces the need for expensive pre-training resources while improving the performance of downstream SE tasks. We also empirically demonstrate that CONCORD can improve existing pre-trained models to learn better representations that consequently become more efficient in both identifying semantically equivalent programs and differentiating buggy from non-buggy code. 
    more » « less
  3. Computer-aided design (CAD) programs are essential to engineering as they allow for better designs through low-cost iterations. While CAD programs are typically taught to undergraduate students as a job skill, such software can also help students learn engineering concepts. A current limitation of CAD programs (even those that are specifically designed for educational purposes) is that they are not capable of providing automated real-time help to students. To encourage CAD programs to build in assistance to students, we used data generated from students using a free, open-source CAD software called Aladdin to demonstrate how student data combined with machine learning techniques can predict how well a particular student will perform in a design task. We challenged students to design a house that consumed zero net energy as part of an introductory engineering technology undergraduate course. Using data from 128 students, along with the scikit-learn Python machine learning library, we tested our models using both total counts of design actions and sequences of design actions as inputs. We found that our models using early design sequence actions are particularly valuable for prediction. Our logistic regression model achieved a >60% chance of predicting if a student would succeed in designing a zero net energy house. Our results suggest that it would be feasible for Aladdin to provide useful feedback to students when they are approximately halfway through their design. Further improvements to these models could lead to earlier predictions and thus provide students feedback sooner to enhance their learning. 
    more » « less
  4. In this work-in-progress paper we present emergent recruitment issues encountered during an ongoing design-based project with participants from two-year colleges for an NSF-funded scholarship program. Our hope is to connect with researchers who have previously explored similar issues or may be experiencing them in their current work. Student Pathways in Engineering and Computing for Transfer Students (SPECTRA) is an NSF S-STEM program that provides financial assistance to students transferring from the South Carolina Technical College System into Engineering or Computing majors at Clemson University [1]. SPECTRA also assists students by connecting them with peers at the technical colleges who move together through the transfer process to Clemson and are supported by the SPECTRA program until graduation. In addition to exploring the experiences of current SPECTRA participants, we investigate how the project can be scaled to include more students and sustained after NSF support ends. The 2021-2022 academic year is the third of the five-year program, although, given emergent recruitment issues, we foresee application for a no-cost extension. The primary concern is the low number of students currently supported in comparison to our goals, highlighting recruitment for further examination. We planned to support up to twenty students in year 1, 52 students in year 2, 70 students in year 3, but our actual numbers in the first three years are 7, 12, and 28 students. Given this trend, our concern over how we recruit students into SPECTRA is now at the forefront of our work. The program is not reaching those students who are eligible, and low recruitment has limited the quality of research needed to inform the construction of a sustainable program. To explore recruitment, we have added interviews with potential students at the technical colleges. In addition to this interview process, we have reviewed our internal practices, analysed existing public information and social media from similar programs, and reviewed existing literature from related research and practice. We identified aspects that may have impacted our current situation. The first was explicit, being the impact of COVID-19 on our ability to hold in-person recruitment events. Similar to studies that have identified other COVID-19 impacts to two-year institutions such as “retention rates declined the most in the community college sector (-2.1 pp to 51.6%)” [2], “disparities in upward transfer mobility increased during the pandemic year” [3], and community colleges being hit hardest “with a 9.4 percent decline” in enrollment [4], we intend to further clarify the influence of COVID-19 on our context. COVID-19 also played a role with regard to the need for scholarship funds, as one of the technical colleges in our program used federal relief funds to provide free tuition for all students during the 2020-2021 academic year. Another potential impact is the effectiveness of the SPECTRA webpages and other online materials to meet the needs of potential students considering the program. In this work-in-progress paper, we will share how we are addressing recruitment issues and how new interventions are impacting recruitment. 
    more » « less
  5. We present our work in progress, a real-time mixed reality communication system for remote assistance in medical emergency situations. 3D cameras capture the emergency situation and send volumetric data to a remote expert. The remote expert sees the volumetric scene through mixed reality glasses and guides an operator to the patient. The local operator receives audio and visual guidance augmented onto the mixed reality headset. We compare the mixed reality system against traditional video communication in a user study on a CPR emergency simulation. We evaluate task performance, cognitive load, and user interaction. The results will help to better understand the benefits of using augmented and volumetric information in medical emergency procedures. 
    more » « less