More Like this

1. Self-Annotation Methods for Aligning Implicit and Explicit Human Feedback in Human-Robot Interaction

   https://doi.org/10.1145/3568162.3576986  

   Zhang, Qiping; Narcomey, Austin; Candon, Kate; Vázquez, Marynel (March 2023, HRI '23: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction)

   Recent research in robot learning suggests that implicit human feedback is a low-cost approach to improving robot behavior without the typical teaching burden on users. Because implicit feedback can be difficult to interpret, though, we study different methods to collect fine-grained labels from users about robot performance across multiple dimensions, which can then serve to map implicit human feedback to performance values. In particular, we focused on understanding the effects of annotation order and frequency on human perceptions of the self-annotation process and the usefulness of the labels for creating data-driven models to reason about implicit feedback. Our results demonstrate that different annotation methods can influence perceived memory burden, annotation difficulty, and overall annotation time. Based on our findings, we conclude with recommendations to create future implicit feedback datasets in Human-Robot Interaction. 

   more » « less
2. GenNI: Human-AI Collaboration for Data-Backed Text Generation

   https://doi.org/10.1109/TVCG.2021.3114845  

   Strobelt, Hendrik; Kinley, Jambay; Krueger, Robert; Beyer, Johanna; Pfister, Hanspeter; Rush, Alexander M. (October 2021, IEEE Transactions on Visualization and Computer Graphics)

   null (Ed.)

   Table2Text systems generate textual output based on structured data utilizing machine learning. These systems are essential for fluent natural language interfaces in tools such as virtual assistants; however, left to generate freely these ML systems often produce misleading or unexpected outputs. GenNI (Generation Negotiation Interface) is an interactive visual system for high-level human-AI collaboration in producing descriptive text. The tool utilizes a deep learning model designed with explicit control states. These controls allow users to globally constrain model generations, without sacrificing the representation power of the deep learning models. The visual interface makes it possible for users to interact with AI systems following a Refine-Forecast paradigm to ensure that the generation system acts in a manner human users find suitable. We report multiple use cases on two experiments that improve over uncontrolled generation approaches, while at the same time providing fine-grained control. A demo and source code are available at https://genni.vizhub.ai. 

   more » « less
3. PeTra: A Sparsely Supervised Memory Model for People Tracking

   https://doi.org/10.18653/v1/2020.acl-main.481  

   Toshniwal, Shubham; Ettinger, Allyson; Gimpel, Kevin; Livescu, Karen (July 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics)

   We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots. PeTra is trained using sparse annotation from the GAP pronoun resolution dataset and outperforms a prior memory model on the task while using a simpler architecture. We empirically compare key modeling choices, finding that we can simplify several aspects of the design of the memory module while retaining strong performance. To measure the people tracking capability of memory models, we (a) propose a new diagnostic evaluation based on counting the number of unique entities in text, and (b) conduct a small scale human evaluation to compare evidence of people tracking in the memory logs of PeTra relative to a previous approach. PeTra is highly effective in both evaluations, demonstrating its ability to track people in its memory despite being trained with limited annotation. 

   more » « less
4. Flexible and scalable annotation tool to develop scene understanding datasets

   https://doi.org/10.1145/3546930.3547499  

   Elahi, Md Fazle; Tian, Renran; Luo, Xiao (June 2022, Workshop on Human-In-the-Loop Data Analytics (HILDA ’22))

   Recent progress in data-driven vision and language-based tasks demands developing training datasets enriched with multiple modalities representing human intelligence. The link between text and image data is one of the crucial modalities for developing AI models. The development process of such datasets in the video domain requires much effort from researchers and annotators (experts and non-experts). Researchers re-design annotation tools to extract knowledge from annotators to answer new research questions. The whole process repeats for each new question which is timeconsuming. However, since the last decade, there has been little change in how the researchers and annotators interact with the annotation process. We revisit the annotation workflow and propose a concept of an adaptable and scalable annotation tool. The concept emphasizes its users’ interactivity to make annotation process design seamless and efficient. Researchers can conveniently add newer modalities to or augment the extant datasets using the tool. The annotators can efficiently link free-form text to image objects. For conducting human-subject experiments on any scale, the tool supports the data collection for attaining group ground truth. We have conducted a case study using a prototype tool between two groups with the participation of 74 non-expert people. We find that the interactive linking of free-form text to image objects feels intuitive and evokes a thought process resulting in a high-quality annotation. The new design shows ≈ 35% improvement in the data annotation quality. On UX evaluation, we receive above-average positive feedback from 25 people regarding convenience, UI assistance, usability, and satisfaction. 

   more » « less
5. Remote sensing in support of endangered species management and animal movement research – the Env-DATA tool pack

   Bohrer, G; Kays, R; Davidson, SC; Wienzierl, R; Dodge, S; Mcclain, KM; Wikelski, M. (October 2015, Proceedings of the 66th International Astronautical Congress, International Astronautical Federation)

   The movement of animals is strongly influenced by external factors in their surrounding environment such as weather, habitat types, and human land use. With advances in positioning and sensor technologies, it is now possible to capture animal locations at high spatial and temporal granularities. Likewise, modern space-based remote sensing technology provides us with an increasing access to large volumes of environmental data, some of which changes on an hourly basis. Environmental data are heterogeneous in source and format, and are usually obtained at different scales and granularities than movement data. Indeed, there remain scientific and technical challenges in developing linkages between the growing collections of animal movement data and the large repositories of heterogeneous remote sensing observations, as well as in the developments of new statistical and computational methods for the analysis of movement in its environmental context. These challenges include retrieval, indexing, efficient storage, data integration, and analytic techniques. We have developed a new system - the Environmental-Data Automated Track Annotation (Env-DATA) - that automates annotation of movement trajectories with remote-sensing environmental information, including high resolution topography, weather from global and regional reanalysis datasets, climatology, human geography, ocean currents and productivity, land use, vegetation and land surface variables, precipitation, fire, and other global datasets. The system automates the acquisition of data from open web resources of remote sensing and weather data and provides several interpolation methods from the native grid resolution and structure to a global regular grid linked with the movement tracks in space and time. Env-DATA provides an easy-to-use platform for end users that eliminates technical difficulties of the annotation processes, including data acquisition, data transformation and integration, resampling, interpolation and interpretation. The new Env-DATA system enhances Movebank (www.movebank.org), an open portal of animal tracking data. The aim is to facilitate new understanding and predictive capabilities of spatiotemporal patterns of animal movement in response to dynamic and changing environments from local to global scales. The system is already in use by scientists worldwide, and by several conservation managers, such as the consortium of federal and private institution that manage the endangered Californian Condor populations. 

   more » « less