skip to main content


Title: Children and search tools: Evaluation remains unclear
As children search the internet for materials, they o en turn to search engines that, unfortunately, o er children li le support as they formulate queries to initiate the search process or examine resources for relevance. While some solutions have been proposed to address this, inherent to this issue is the need to evaluate the e ectiveness of these solutions. We posit that the evaluation of the diverse aspects involved in the search process – from query suggestion generation to resource retrieval – requires a complex, multi-faceted approach that draws on evaluation methods utilized in human-computer interaction, information retrieval, natural language processing, education, and psychology.  more » « less
Award ID(s):
1763649
PAR ID:
10099389
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
KidRec Workshop co-located with ACM IDC 2019
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Given the more widespread nature of natural language interfaces, it is increasingly important to understand who are accessing those interfaces, and how those interfaces are being used. In this paper, we explore spellchecking in the context of web search with children as the target audience. In particular, via a literature review we show that, while widely used, popular search tools are ill-designed for children. We then use spellcheckers as a case study to highlight the need for an interdisciplinary approach that brings together natural language processing, education, human-computer interaction to address a known information retrieval problem: query misspelling. We conclude that it is imperative that those for whom the interfaces are designed have a voice in the design process. 
    more » « less
  2. People often have difficulty in expressing their information needs. Many times this results from a lack of clarity about the task at hand, or the way an information or search system works. In addition, people may not know what they do not know. The former is addressed by search systems by providing recommendations, whereas there are no good solutions for the latter problem. Even when a search system makes recommendations, they are limited to suggesting objects such as queries and documents only. They do not consider providing suggestions for strategies, people, or processes. This Perspective Paper addresses it by showing how to investigate the nature of the work a person is doing, predicting the potential problems they may encounter, and providing help to overcome those problems. This help could be an object such as a document or a query, a strategy, or a person. This whole process is referred to as Information Fostering. Beyond crafting a general-purpose recommender system, Information Fostering is the idea of providing proactive suggestions and help to information seekers. This could allow them avoid potential problems and capture promising opportunities from a search process before it is too late. The current paper presents this new perspective by outlining desired characteristics of an Information Fostering system, envisioning application scenarios, and proposing a set of potential methods for moving forward. Beyond these details, the primary purpose of this paper is to offer a new viewpoint that looks at the other side of the information seeking coin, by bringing together ideas from human-computer interaction, information retrieval, recommender systems, and education. 
    more » « less
  3. Abstract

    Evaluation metrics such as precision, recall and normalized discounted cumulative gain have been widely applied inad hocretrieval experiments. They have facilitated the assessment of system performance in various topics over the past decade. However, the effectiveness of such metrics in capturing users’ in-situ search experience, especially in complex search tasks that trigger interactive search sessions, is limited. To address this challenge, it is necessary to adaptively adjust the evaluation strategies of search systems to better respond to users’ changing information needs and evaluation criteria. In this work, we adopt a taxonomy of search task states that a user goes through in different scenarios and moments of search sessions, and perform a meta-evaluation of existing metrics to better understand their effectiveness in measuring user satisfaction. We then built models for predicting task states behind queries based on in-session signals. Furthermore, we constructed and meta-evaluated new state-aware evaluation metrics. Our analysis and experimental evaluation are performed on two datasets collected from a field study and a laboratory study, respectively. Results demonstrate that the effectiveness of individual evaluation metrics varies across task states. Meanwhile, task states can be detected from in-session signals. Our new state-aware evaluation metrics could better reflect in-situ user satisfaction than an extensive list of the widely used measures we analyzed in this work in certain states. Findings of our research can inspire the design and meta-evaluation of user-centered adaptive evaluation metrics, and also shed light on the development of state-aware interactive search systems.

     
    more » « less
  4. Children use popular web search tools, which are generally designed for adult users. Because children have different developmental needs than adults, these tools may not always adequately support their search for information. Moreover, even though search tools offer support to help in query formulation, these too are aimed at adults and may hinder children rather than help them. This calls for the examination of existing technologies in this area, to better understand what remains to be done when it comes to facilitating query-formulation tasks for young users. In this paper, we investigate interaction elements of query formulation--including query suggestion algorithms--for children. The primary goals of our research efforts are to: (i) examine existing plug-ins and interfaces that explicitly aid children's query formulation; (ii) investigate children's interactions with suggestions offered by a general-purpose query suggestion strategy vs. a counterpart designed with children in mind; and (iii) identify, via participatory design sessions, their preferences when it comes to tools / strategies that can help children find information and guide them through the query formulation process. Our analysis shows that existing tools do not meet children's needs and expectations; the outcomes of our work can guide researchers and developers as they implement query formulation strategies for children. 
    more » « less
  5. The role that technology plays in supporting children at school and at home is more prominent than ever before due to the global COVID-19 pandemic. This has prompted us to focus the 6th International and Interdisciplinary Perspectives on Children \& Recommender and Information Retrieval Systems (KidRec) workshop on what the lasting changes will be to the design and development of child information retrieval systems. After two years, are information retrieval systems used more in and out of the classroom? Are they more interactive, more or less personalized? What is the impact on the research and business community? Are there long-term and unexpected changes on the design, ethics, and algorithms? The primary goal of our workshop continues to be to build community by bringing together researchers, practitioners, and other stakeholders from various backgrounds and disciplines to understand and advance information retrieval systems for children. 
    more » « less