An official website of the United States government
Search tasks play an important role in the study and development of interactive information retrieval (IIR) systems. Prior work has examined how search tasks vary along dimensions such as the task’s main activity, end goal, structure, and complexity. Recently, researchers have been exploring task complexity from the perspective of cognitive complexity—related to the types (and variety) of mental activities required by the task. Anderson & Krathwohl’s two-dimensional taxonomy of learning has been a commonly used framework for investigating tasks from the perspective of cognitive complexity [1]. A&K’s 2D taxonomy involves a cognitive process dimension and an orthogonal knowledge dimension. Prior IIR research has successfully leveraged the cognitive process dimension of this 2D taxonomy to develop search tasks and investigate their effects on searchers’ needs, perceptions, and behaviors. However, the knowledge dimension of the taxonomy has been largely ignored. In this conceptual paper, we argue that future IIR research should consider both dimensions of A&K’s taxonomy. Specifically, we discuss related work, present details on both dimensions of A&K’s taxonomy, and explain how to use the taxonomy to develop search tasks and learning assessment materials. Additionally, we discuss how considering both dimensions of A&K’s taxonomy has important implications for future IIR research.
more »
« less
Learning assessments in search-as-learning: A survey of prior work and opportunities for future research
People often search for information in order to learn something new. In recent years, the “search-as-learning” movement has argued that search systems should be better designed to support learning. Current search systems (especially Web search engines) are largely designed and optimized to fulfill simple look-up tasks (e.g., navigational or fact-finding search tasks). However, they provide less support for searchers working on complex tasks that involve learning. Search-as-learning studies have investigated a wide range of research questions. For example, studies have aimed to better understand how characteristics of the individual searcher, the type of search task, and interactive features provided by the system can influence learning outcomes. Learning assessment is a key component in search-as-learning studies. Assessment materials are used to both gauge prior knowledge and measure learning during or after one or more search sessions. In this paper, we provide a systematic review of different types of assessments used in search-as-learning studies to date. The paper makes the following three contributions. First, we review different types of assessments used and discuss their potential benefits and drawbacks. Second, we review assessments used outside of search-as-learning, which may provide insights and opportunities for future research. Third, we provide recommendations for future research. Importantly, we argue that future studies should clearly define learning objectives and develop assessment materials that reliably capture the intended type of learning. For example, assessment materials should test a participant’s ability to engage with specific cognitive processes, which may range from simple (e.g., memorization) to more complex (e.g., critical and creative thinking). Additionally, we argue that future studies should consider two dimensions that are understudied in search-as-learning: long-term retention (i.e., being able to use what was learned in the long term) and transfer of learning (i.e., being able to use what was learned in a novel context).
more »
« less
- Award ID(s):
- 1718295
- PAR ID:
- 10326505
- Date Published:
- Journal Name:
- Information processing and management
- Volume:
- 59
- Issue:
- 2
- ISSN:
- 0306-4573
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)While systems engineers rely on systems thinking skills in their work, given the increasing complexity of modern engineering problems, engineers across disciplines need to be able to engage in systems thinking, including what we term comprehensive systems thinking. Due to the inherent complexity of systems thinking, and more specifically comprehensive systems thinking, it is not easy to know how well students (and practitioners) are learning and leveraging systems thinking approaches. Thus, engineering managers and educators can benefit from systems thinking assessments. A variety of systems thinking assessments exist that are relevant to engineers, including some focused on the demonstration of systems thinking knowledge or skills and others measuring attitudes, interests, or values related to systems thinking. Starting with a collection of systems thinking assessments from a systematic literature review conducted by our team, we analyzed in-depth those behavior-based assessments that included the creation of a visual representation and were open-ended, i.e., it did not presuppose or provide answers. The findings from this in-depth analysis of systems thinking behavior-based assessments identified 1) six visualization types that were leveraged, 2) dimensions of systems thinking that were assessed and 3) tensions between the affordances of different assessments. In addition, we consider the ways assessments can be used. For example, using assessments to provide feedback to students or using assessments to determine which students are meeting defined learning goals. We draw on our findings to highlight opportunities for future comprehensive systems thinking behavior-based assessment development.more » « less
-
Recent years have seen great success of large language models (LLMs) in performing many natural language processing tasks with impressive performance, including tasks that directly serve users such as question answering and text summarization. They open up unprecedented opportunities for transforming information retrieval (IR) research and applications. However, concerns such as halluciation undermine their trustworthiness, limiting their actual utility when deployed in real-world applications, especially high-stake applications where trust is vital. How can we both exploit the strengths of LLMs and mitigate any risk caused by their weaknesses when applying LLMs to IR? What are the best opportunities for us to apply LLMs to IR? What are the major challenges that we will need to address in the future to fully exploit such opportunities? Given the anticipated growth of LLMs, what will future information retrieval systems look like? Will LLMs eventually replace an IR system? In this perspective paper, we examine these questions and provide provisional answers to them. We argue that LLMs will not be able to replace search engines, and future LLMs would need to learn how to use a search engine so that they can interact with a search engine on behalf of users. We conclude with a set of promising future research directions in applying LLMs to IR.more » « less
-
Wiebe, E. N.; Harris, C. J.; Grover, S. (Ed.)Efforts to improve instruction frequently focus on fostering meaningful learning—learning based on conceptual understanding—as opposed to knowledge memorized by rote. Consistent with Dewey’s (1963) principle of interaction, fostering meaningful learning entails identifying what children already know and do not know and building on the former to learn (moderately) new knowledge (Claessens & Engel, 2013; Fyfe et al., 2012; Piaget, 1964; Vygotsky, 1978). A learning trajectory (LT) approach to instruction—which includes conceptually and research-based and goals, a research-based learning progression of successive developmental levels, and research-based teaching activities to promote each level—epitomizes such an effort (Clements & Sarama, 2008; Confrey et al., 2012). Formative, classroom-based assessment—ongoing assessment to guide and monitor student learning (Black et al., 2003; Cizek, 2010; Author, 2018a)—is an integral aspect of the LT approach (Daro et al., 2011). In contrast to more commonly used summative assessment strategy (e.g., a unit test given at the end of an instruction unit to assess whether unit content has been mastered and grade progress), formative assessment serves to identify what developmental level a child has already achieved and the next developmentally appropriate level on which instruction should begin (Author, 2018a). Moreover, children are regularly assessed during instruction to gauge whether they–individually or collectively–have mastered a developmental level before instruction proceeds with the next higher level. In sum, “the LT approach involves using formative assessment (National Mathematics Advisory Panel, 2008; Shepard et al., 2018) to provide instructional activities aligned with empirically validated developmental progressions (Fantuzzo, Gadsden, & McDermott, 2011). Although research has shown that LT-based instruction is more efficacious, research is needed to evaluate the add-on value of the formative assessment components of LT-based instruction on student outcomes and the professional development of teachers. This presentation will highlight future lines of research that would provide insight into underlying theory and more productive strategies. Because LTs “need to be supplemented with consideration of obstacles that the student must overcome,” much needs to be learned about the obstacles posed by the content itself, instructional materials, and teachers (Ginsburg, 2009).more » « less
-
Caring assessments is an assessment design framework that considers the learner as a whole and can be used to design assessment opportunities that learners find engaging and appropriate for demonstrating what they know and can do. This framework considers learners’ cognitive, meta-cognitive, intra-and inter-personal skills, aspects of the learning context, and cultural and linguistic backgrounds as ways to adapt assessments. Extending previous work on intelligent tutoring systems that “care” from the field of artificial intelligence in education (AIEd), this framework can inform research and development of personalized and socioculturally responsive assessments that support students’ needs. In this article, we (a) describe the caring assessment framework and its unique contributions to the field, (b) summarize current and emerging research on caring assessments related to students’ emotions, individual differences, and cultural contexts, and (c) discuss challenges and opportunities for future research on caring assessments in the service of developing and implementing personalized and socioculturally responsive interactive digital assessments.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/https://doi.org/10.1016/j.ipm.2021.102821
