An official website of the United States government
Search tasks play an important role in the study and development of interactive information retrieval (IIR) systems. Prior work has examined how search tasks vary along dimensions such as the task’s main activity, end goal, structure, and complexity. Recently, researchers have been exploring task complexity from the perspective of cognitive complexity—related to the types (and variety) of mental activities required by the task. Anderson & Krathwohl’s two-dimensional taxonomy of learning has been a commonly used framework for investigating tasks from the perspective of cognitive complexity [1]. A&K’s 2D taxonomy involves a cognitive process dimension and an orthogonal knowledge dimension. Prior IIR research has successfully leveraged the cognitive process dimension of this 2D taxonomy to develop search tasks and investigate their effects on searchers’ needs, perceptions, and behaviors. However, the knowledge dimension of the taxonomy has been largely ignored. In this conceptual paper, we argue that future IIR research should consider both dimensions of A&K’s taxonomy. Specifically, we discuss related work, present details on both dimensions of A&K’s taxonomy, and explain how to use the taxonomy to develop search tasks and learning assessment materials. Additionally, we discuss how considering both dimensions of A&K’s taxonomy has important implications for future IIR research.
more »
« less
Learning assessments in search-as-learning: A survey of prior work and opportunities for future research
People often search for information in order to learn something new. In recent years, the “search-as-learning” movement has argued that search systems should be better designed to support learning. Current search systems (especially Web search engines) are largely designed and optimized to fulfill simple look-up tasks (e.g., navigational or fact-finding search tasks). However, they provide less support for searchers working on complex tasks that involve learning. Search-as-learning studies have investigated a wide range of research questions. For example, studies have aimed to better understand how characteristics of the individual searcher, the type of search task, and interactive features provided by the system can influence learning outcomes. Learning assessment is a key component in search-as-learning studies. Assessment materials are used to both gauge prior knowledge and measure learning during or after one or more search sessions. In this paper, we provide a systematic review of different types of assessments used in search-as-learning studies to date. The paper makes the following three contributions. First, we review different types of assessments used and discuss their potential benefits and drawbacks. Second, we review assessments used outside of search-as-learning, which may provide insights and opportunities for future research. Third, we provide recommendations for future research. Importantly, we argue that future studies should clearly define learning objectives and develop assessment materials that reliably capture the intended type of learning. For example, assessment materials should test a participant’s ability to engage with specific cognitive processes, which may range from simple (e.g., memorization) to more complex (e.g., critical and creative thinking). Additionally, we argue that future studies should consider two dimensions that are understudied in search-as-learning: long-term retention (i.e., being able to use what was learned in the long term) and transfer of learning (i.e., being able to use what was learned in a novel context).
more »
« less
- Award ID(s):
- 1718295
- PAR ID:
- 10326505
- Date Published:
- Journal Name:
- Information processing and management
- Volume:
- 59
- Issue:
- 2
- ISSN:
- 0306-4573
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)While systems engineers rely on systems thinking skills in their work, given the increasing complexity of modern engineering problems, engineers across disciplines need to be able to engage in systems thinking, including what we term comprehensive systems thinking. Due to the inherent complexity of systems thinking, and more specifically comprehensive systems thinking, it is not easy to know how well students (and practitioners) are learning and leveraging systems thinking approaches. Thus, engineering managers and educators can benefit from systems thinking assessments. A variety of systems thinking assessments exist that are relevant to engineers, including some focused on the demonstration of systems thinking knowledge or skills and others measuring attitudes, interests, or values related to systems thinking. Starting with a collection of systems thinking assessments from a systematic literature review conducted by our team, we analyzed in-depth those behavior-based assessments that included the creation of a visual representation and were open-ended, i.e., it did not presuppose or provide answers. The findings from this in-depth analysis of systems thinking behavior-based assessments identified 1) six visualization types that were leveraged, 2) dimensions of systems thinking that were assessed and 3) tensions between the affordances of different assessments. In addition, we consider the ways assessments can be used. For example, using assessments to provide feedback to students or using assessments to determine which students are meeting defined learning goals. We draw on our findings to highlight opportunities for future comprehensive systems thinking behavior-based assessment development.more » « less
-
Recent years have seen great success of large language models (LLMs) in performing many natural language processing tasks with impressive performance, including tasks that directly serve users such as question answering and text summarization. They open up unprecedented opportunities for transforming information retrieval (IR) research and applications. However, concerns such as halluciation undermine their trustworthiness, limiting their actual utility when deployed in real-world applications, especially high-stake applications where trust is vital. How can we both exploit the strengths of LLMs and mitigate any risk caused by their weaknesses when applying LLMs to IR? What are the best opportunities for us to apply LLMs to IR? What are the major challenges that we will need to address in the future to fully exploit such opportunities? Given the anticipated growth of LLMs, what will future information retrieval systems look like? Will LLMs eventually replace an IR system? In this perspective paper, we examine these questions and provide provisional answers to them. We argue that LLMs will not be able to replace search engines, and future LLMs would need to learn how to use a search engine so that they can interact with a search engine on behalf of users. We conclude with a set of promising future research directions in applying LLMs to IR.more » « less
-
Wiebe, E. N.; Harris, C. J.; Grover, S. (Ed.)Efforts to improve instruction frequently focus on fostering meaningful learning—learning based on conceptual understanding—as opposed to knowledge memorized by rote. Consistent with Dewey’s (1963) principle of interaction, fostering meaningful learning entails identifying what children already know and do not know and building on the former to learn (moderately) new knowledge (Claessens & Engel, 2013; Fyfe et al., 2012; Piaget, 1964; Vygotsky, 1978). A learning trajectory (LT) approach to instruction—which includes conceptually and research-based and goals, a research-based learning progression of successive developmental levels, and research-based teaching activities to promote each level—epitomizes such an effort (Clements & Sarama, 2008; Confrey et al., 2012). Formative, classroom-based assessment—ongoing assessment to guide and monitor student learning (Black et al., 2003; Cizek, 2010; Author, 2018a)—is an integral aspect of the LT approach (Daro et al., 2011). In contrast to more commonly used summative assessment strategy (e.g., a unit test given at the end of an instruction unit to assess whether unit content has been mastered and grade progress), formative assessment serves to identify what developmental level a child has already achieved and the next developmentally appropriate level on which instruction should begin (Author, 2018a). Moreover, children are regularly assessed during instruction to gauge whether they–individually or collectively–have mastered a developmental level before instruction proceeds with the next higher level. In sum, “the LT approach involves using formative assessment (National Mathematics Advisory Panel, 2008; Shepard et al., 2018) to provide instructional activities aligned with empirically validated developmental progressions (Fantuzzo, Gadsden, & McDermott, 2011). Although research has shown that LT-based instruction is more efficacious, research is needed to evaluate the add-on value of the formative assessment components of LT-based instruction on student outcomes and the professional development of teachers. This presentation will highlight future lines of research that would provide insight into underlying theory and more productive strategies. Because LTs “need to be supplemented with consideration of obstacles that the student must overcome,” much needs to be learned about the obstacles posed by the content itself, instructional materials, and teachers (Ginsburg, 2009).more » « less
-
Over the past decades, computer science education (CSEd) research has studied the multitude of factors that may impact student success in introductory programming courses (CS0/CS1). The lack of foundational structure behind how these factors interrelate has made it difficult to gain a thorough understanding of this area of CSEd literature. Gaining a deeper understanding and applying structure to these factors would allow CSEd to adopt better teaching practices, study habits, learning environments, course materials, etc. and to better understand the student experience to better foster success among a broader population of students. Our systematic literature review used search criteria for factors that predicted student success in CS0/CS1, which yielded 311 research articles. We then mapped this body of work under the Biggs' 3P (Presage, Process, Product) educational model, which provides a comprehensive framework for how students engage with learning opportunities. We discovered that although many studies focused on the Presage and Product phases of the model, fewer studies mapped to the Process phase, which describes the students' active learning processes. Our study shows there is a potential gap in the literature and future studies should focus more specifically on how students choose to engage with learning opportunities and what factors may be hindering that engagement throughout a learning period.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/https://doi.org/10.1016/j.ipm.2021.102821
