More Like this

1. Humans in Empirical Software Engineering Studies: An Experience Report

   https://doi.org/10.1109/SANER53432.2022.00154  

   Sharif, Bonita; Mansoor, Niloofar (March 2022, 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER))

   The use of human validation in software engineering methods, tools, and processes is crucial to understanding how these artifacts actually impact the people using them. In this paper, we report our experiences on two methods of data collection we have used in software engineering empirical studies, namely online questionnaire-based data collection and in-person eye tracking data collection using eye tracking equipment. The design and instrumentation challenges we faced are discussed with possible ways to mitigate them. We conclude with some guidelines and our vision for the future in human-centric studies in software engineering. 

   more » « less
2. Capturing Humans’ Mental Models of AI: An Item Response Theory Approach

   https://doi.org/10.1145/3593013.3594111  

   Kelly, Markelle; Kumar, Aakriti; Smyth, Padhraic; Steyvers, Mark (June 2023, FAccT '23: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency)

   Improving our understanding of how humans perceive AI teammates is an important foundation for our general understanding of human-AI teams. Extending relevant work from cognitive science, we propose a framework based on item response theory for modeling these perceptions. We apply this framework to real-world experiments, in which each participant works alongside another person or an AI agent in a question-answering setting, repeatedly assessing their teammate’s performance. Using this experimental data, we demonstrate the use of our framework for testing research questions about people’s perceptions of both AI agents and other people. We contrast mental models of AI teammates with those of human teammates as we characterize the dimensionality of these mental models, their development over time, and the influence of the participants’ own self-perception. Our results indicate that people expect AI agents’ performance to be significantly better on average than the performance of other humans, with less variation across different types of problems. We conclude with a discussion of the implications of these findings for human-AI interaction. 

   more » « less
3. Socially intelligent machines that learn from humans and help humans learn

   https://doi.org/10.1098/rsta.2022.0048  

   Gweon, Hyowon; Fan, Judith; Kim, Been (July 2023, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences)

   A hallmark of human intelligence is the ability to understand and influence other minds. Humans engage in inferential social learning (ISL) by using commonsense psychology to learn from others and help others learn. Recent advances in artificial intelligence (AI) are raising new questions about the feasibility of human–machine interactions that support such powerful modes of social learning. Here, we envision what it means to develop socially intelligent machines that can learn, teach, and communicate in ways that are characteristic of ISL. Rather than machines that simply predict human behaviours or recapitulate superficial aspects of human sociality (e.g. smiling, imitating), we should aim to build machines that can learn from human inputs and generate outputs for humans by proactively considering human values, intentions and beliefs. While such machines can inspire next-generation AI systems that learn more effectively from humans (as learners) and even help humans acquire new knowledge (as teachers), achieving these goals will also require scientific studies of its counterpart: how humans reason about machine minds and behaviours. We close by discussing the need for closer collaborations between the AI/ML and cognitive science communities to advance a science of both natural and artificial intelligence. This article is part of a discussion meeting issue ‘Cognitive artificial intelligence’. 

   more » « less
4. When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans

   https://doi.org/10.1145/3319502.3374832  

   Kwon, Minae; Biyik, Erdem; Talati, Aditi; Bhasin, Karan; Losey, Dylan P.; Sadigh, Dorsa (March 2020, Human-Robot Interaction)

   null (Ed.)
5. Emergent local structures in an ecosystem of social bots and humans on Twitter

   https://doi.org/10.1140/epjds/s13688-023-00406-5  

   Alrhmoun, Abdullah; Kertész, János (December 2023, EPJ Data Science)

   Bots in online social networks can be used for good or bad but their presence is unavoidable and will increase in the future. To investigate how the interaction networks of bots and humans evolve, we created six social bots on Twitter with AI language models and let them carry out standard user operations. Three different strategies were implemented for the bots: a trend-targeting strategy (TTS), a keywords-targeting strategy (KTS) and a user-targeting strategy (UTS). We examined the interaction patterns such as targeting users, spreading messages, propagating relationships, and engagement. We focused on the emergent local structures or motifs and found that the strategies of the social bots had a significant impact on them. Motifs resulting from interactions with bots following TTS or KTS are simple and show significant overlap, while those resulting from interactions with UTS-governed bots lead to more complex motifs. These findings provide insights into human-bot interaction patterns in online social networks, and can be used to develop more effective bots for beneficial tasks and to combat malicious actors. 

   more » « less