An official website of the United States government
[Context] Interviews are the most widely used elicitation technique in requirements engineering. However, conducting effective requirements elicitation interviews is challenging, due to the combination of technical and soft skills that requirements analysts often acquire after a long period of professional practice. Empirical evidence about training the novices on conducting effective requirements elicitation interviews is scarce. [Objectives] We present a list of most common mistakes that novices make in requirements elicitation interviews. The objective is to assist the educators in teaching interviewing skills to student analysts. [Research Method] We conducted an empirical study involving role-playing and authentic assessment with 110 students, teamed up in 28 groups, to conduct interviews with a customer. One researcher made observation notes during the interview while two researchers reviewed the recordings. We qualitatively analyzed the data to identify the themes and classify the mistakes. [Results and conclusion] We identified 34 unique mistakes classified into 7 high level themes. We also give examples of the mistakes made by the novices in each theme, to assist the educationists and trainers. Our research design is a novel combination of well-known pedagogical approaches described in sufficient details to make it repeatable for future requirements engineering education and training research.
more »
« less
Teaching requirements elicitation interviews: an empirical study of learning from mistakes
Interviews are the most widely used elicitation technique in requirements engineering (RE). However, conducting a requirements elicitation interview is challenging. The mistakes made in design or conduct of the interviews can create problems in the later stages of requirements analysis. Empirical evidence about effective pedagogical approaches for training novices on conducting requirements elicitation interviews is scarce. In this paper, we present a novel pedagogical approach for training student analysts in the art of elicitation interviews. Our study is conducted in two parts: first, we perform an observational study of interviews performed by novices, and we present a classification of the most common mistakes made; second, we utilize this list of mistakes and monitor the students’ progress in three set of interviews to discover the individual areas for improvement. We conducted an empirical study involving role-playing and authentic assessment in two semesters on two different cohorts of students. In the first semester, we had 110 students, teamed up in 28 groups, to conduct three interviews with stakeholders. We qualitatively analysed the data to identify and classify the mistakes made from their first interview only. In the second semester, we had 138 students in 34 groups and we monitored and analysed their progress in all three interviews by utilizing the list of mistakes from the first study. First, we identified 34 unique mistakes classified into seven high-level themes, namely question formulation, question omission, interview order, communication skills, analyst behaviour, customer interaction, teamwork and planning. In the second study, we discovered that the students struggled mostly in the areas of question formulation, question omission and interview order and did not manage to improve their skills throughout the three interviews. Our study presents a novel and repeatable pedagogical design, and our findings extend the body of knowledge aimed at RE education and training by providing an empirically grounded categorization of mistakes made by novices. We offer an analysis of the main pain points in which instructors should pay more attention during their design and training.
more »
« less
- Award ID(s):
- 1718377
- PAR ID:
- 10105611
- Date Published:
- Journal Name:
- Requirements Engineering
- ISSN:
- 0947-3602
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
[Context and motivation] Role-playing offers experiential learning through the simulation of real-world scenarios; for this reason, it is widely used in software engineering education. In Requirements Engineering, role-playing is a popular way to provide students hands-on experience with requirements elicitation interviews. [Problem] However, managing a role-playing activity to simulate requirements elicitation interviews in a class is time-consuming, as it often requires pairing students with student assistants or fellow classmates who act as either customers or requirements analysts as well as creating and maintaining the interview schedules between the actors. To make the adoption of role-playing activities in a class feasible, there is a need to develop a solution to reduce instructors’ workload. [Principal ideas] To solve this problem we propose the use of VIrtual CustOmer (VICO), an intent-based, multimodal, conversational agent. VICO offers an interview experience comparable to talking to a human and provides a transcript of the interview annotated with the mistakes students made in it. The adoption of VICO will eliminate the need to schedule interviews as the students can interact with it in their free time. Moreover, the transcript of the interview allows students to evaluate their performance to refine and improve their interviewing skills. [Contribution] In this research preview, we show the architecture of VICO and how it can be developed using existing technologies, we provide an online rule-based initial prototype and show the practicality and applicability of this tool through an exploratory study.more » « less
-
Prior methodological literature on conducting interviews emphasizes the importance of skill development in conducting interviews. However, in contrast to qualitative data analysis, there are few systematic processes in place to guide the interviewer into reflexivity about their role in the interview situation. Here, we present the interview quality reflection tool (IQRT) as a process that we developed from conducting and mentoring semi-structured and unstructured interviews focused on personal lived experiences. The IQRT prompts the interviewer to transcribe each interview question and reflect on how the spoken question served to advance experiential quality in the interview. We illustrate the IQRT itself before demonstrating how we authors used the process to examine experiential quality in three cases conducted in our prior research. Finally, we consider how the IQRT enables researchers to examine the interview situation as a whole, by increasing the self-awareness of the interviewer, and the parts, by commenting on the mechanics of constructing useful questions.more » « less
-
Requirements are elicited from the customer and other stakeholders through an iterative process of interviews, prototyping, and other interactive sessions. Then, requirements can be further extended, based on the analysis of the features of competing products available on the market. Understanding how this process takes place can help to identify the contribution of the different elicitation phases, thereby allowing requirements analysts to better distribute their resources. In this work, we empirically study in which way requirements get transformed from initial ideas into documented needs, and then evolve based on the inspiration coming from similar products. To this end, we select 30 subjects that act as requirements analysts, and we perform interview-based elicitation sessions with a fictional customer. After the sessions, the analysts produce a first set of requirements for the system. Then, they are required to search similar products in the app stores and extend the requirements, inspired by the identified apps. The requirements documented at each step are evaluated, to assess to which extent and in which way the initial idea evolved throughout the process. Our results show that only between 30% and 38% of the requirements produced after the interviews include content that can be fully traced to initial customer’s ideas. The rest of the content is dedicated to new requirements, and up to 21% of it belongs to completely novel topics. Furthermore, up to 42% of the requirements inspired by the app stores cover additional features compared to the ones identified after the interviews. The results empirically show that requirements are not elicited in strict sense, but actually co-created through interviews, with analysts playing a crucial role in the process. In addition, we show evidence that app store-inspired elicitation can be particularly beneficial to complete the requirements.more » « less
-
A key challenge in engineering design problem framing is defining requirements and metrics. This is difficult, in part, because engineers must make decisions about how to treat qualitative and subjective issues, like stakeholder preferences, about how to prioritize different requirements, and about how to maintain tentativeness and ill-structuredness in the solution space. And this is made more challenging in light of the function of requirements in other types of engineering problems, like feasibility analysis, in which the requirements should converge on a decision. Given these challenges, it is unsurprising that there is limited research on how first-year students approach such work, how they make sense of requirements, and how their conceptualizations of requirements change with instruction. Our purpose in this study is to investigate students’ initial understanding and use of engineering requirements in a specific problem solving context. We developed a survey to measure students’ perceptions related to engineering requirements based on constructs derived from the literature on engineering requirements. We implemented the survey in a first-year and in senior courses for the purpose of validating items using factor analysis. Following this, we conducted analysis of survey and interview data restricted to the first-year course, including epistemic beliefs and analysis of students’ agency. Through exploratory factor analysis, we found that factors did not converge around constructs as described in the literature. Rather, factors formed around the forms of information leveraged to develop requirements. Through qualitative analysis of students’ responses on the survey and to interviews, we evaluated the extent to which students expressed agency over their use of requirements to make decisions within a course project. We describe implications of this exploratory study in terms of adapting research instruments to better understand this topic. Further, we consider pedagogical implications for first year programs and beyond in supporting students to develop ownership over decision making related to engineering requirements.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1007/s00766-019-00313-0
