An official website of the United States government
As demand grows for job-ready data science professionals, there is increasing recognition that traditional training often falls short in cultivating the higher-order reasoning and real-world problem-solving skills essential to the field. A foundational step toward addressing this gap is the identification and organization of knowledge components (KCs) that underlie data science problem solving (DSPS). KCs represent conditional knowledge—knowing about appropriate actions given particular contexts or conditions—and correspond to the critical decisions data scientists must make throughout the problem-solving process. While existing taxonomies in data science education support curriculum development, they often lack the granularity and focus needed to support the assessment and development of DSPS skills. In this paper, we present a novel framework that combines the strengths of large language models (LLMs) and human expertise to identify, define, and organize KCs specific to DSPS. We treat LLMs as ``knowledge engineering assistants" capable of generating candidate KCs by drawing on their extensive training data, which includes a vast amount of domain knowledge and diverse sets of real-world DSPS cases. Our process involves prompting multiple LLMs to generate decision points, synthesizing and refining KC definitions across models, and using sentence-embedding models to infer the underlying structure of the resulting taxonomy. Human experts then review and iteratively refine the taxonomy to ensure validity. This human-AI collaborative workflow offers a scalable and efficient proof-of-concept for LLM-assisted knowledge engineering. The resulting KC taxonomy lays the groundwork for developing fine-grained assessment tools and adaptive learning systems that support deliberate practice in DSPS. Furthermore, the framework illustrates the potential of LLMs not just as content generators but as partners in structuring domain knowledge to inform instructional design. Future work will involve extending the framework by generating a directed graph of KCs based on their input-output dependencies and validating the taxonomy through expert consensus and learner studies. This approach contributes to both the practical advancement of DSPS coaching in data science education and the broader methodological toolkit for AI-supported knowledge engineering.
more »
« less
This content will become publicly available on June 28, 2027
An LLM-Based Agentic AI System for Automated Construction of Knowledge Taxonomies in Data Science Problem Solving
The widespread integration of artificial intelligence into data science practice is fundamentally reconfiguring the roles of data science practitioners. As procedural and execution-oriented tasks such as coding, data preprocessing, and routine analysis become increasingly automated, human contribution is shifting toward complex cognitive task of problem solving including higher-order reasoning and decision making. This shift challenges prevailing models of data science education, which has been focusing on tools and techniques while offering limited support to develop those critical problem solving competencies. A prerequisite for addressing this gap is formalizing the Data Science Problem Solving (DSPS) competency as structured knowledge that can be systematically represented and evaluated, for example, by adopting principled approach of constructing knowledge taxonomies for DSPS. However, constructing a coherent and verifiable taxonomy of such knowledge remains a nontrivial knowledge engineering task. In this paper, we present a multi-agent LLM framework for the automated generation and evaluation of a DSPS knowledge taxonomy, grounded in established taxonomy construction and evaluation methods. The system comprises an Author agent that proposes and revises taxonomies and a Critique agent that evaluates them and provides feedback based on explicit assessment criteria. Additionally, an Orchestrator agent manages iterative refinement by routing feedback and enforcing stopping conditions. Our results demonstrate the potential of organizing LLMs into collaborative-adversarial architectures that support reflective, iterative knowledge construction and refinement rather than single-pass content generation. When further developed, the proposed multi-agent LLM systems can function as a scalable framework for principled knowledge engineering, enabling the systematic design of DSPS curriculum, assessments, instructional scaffolds, and AI-assisted learning environments at scale.
more »
« less
- Award ID(s):
- 2429590
- PAR ID:
- 10702830
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9798400722936
- Page Range / eLocation ID:
- 525 to 529
- Subject(s) / Keyword(s):
- LLM Agentic AI Knowledge Engineering Taxonomy Data Science
- Format(s):
- Medium: X
- Location:
- Seoul Republic of Korea
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
While our world consistently presents complicated, interdisciplinary problems with STEM foundations, most pre-university curricula do not encourage drawing on multidisciplinary knowledge in the sciences and engineering to create solutions. We developed an instructional approach, Iterative Science and Engineering (ISE), that cycles through scientific investigation and engineering design and culminates in constructing a solution to a local environmental challenge. Next, we created, revised, and evaluated a six-week ISE curricular program, Invasive Insects, culminating in 6th–9th-grade students building traps to mitigate local invasive insect populations. Over three Design-Based Research (DBR) cycles, we gathered and analyzed identical pre and post-test data from 554 adolescents to address the research question: what three-dimensional (3D) science and engineering knowledge do adolescents demonstrate over three DBR cycles associated with a curricular program following the Iterative Science and Engineering instructional approach? Results document students’ significant statistical improvements, with differential outcomes in different cycles. For example, most students demonstrated significant learning of 3D science and engineering argument construction in all cycles—still, students only significantly improved engineering design when they performed guided reflection on their designs and physically built a second trap. Our results suggest that the development, refinement, and empirical evaluation of an ISE curricular program led to students’ design, building, evaluation, and sharing of their learning of mitigating local invasive insect populations. To address complex, interdisciplinary challenges, we must provide opportunities for fluid and iterative STEM learning through scientific investigation and engineering design cycles.more » « less
-
Recent advances in LLM agentic systems have improved the automation of offensive security tasks, particularly for Capture the Flag (CTF) challenges. We systematically investigate the key factors that drive agent success and provide a detailed recipe for building effective LLM-based offensive security agents. First, we present CTFJudge, a framework leveraging LLM as a judge to analyze agent trajectories and provide granular evaluation across CTF solving steps. Second, we propose a novel metric, CTF Competency Index (CCI) for partial correctness, revealing how closely agent solutions align with human-crafted gold standards. Third, we examine how LLM hyperparameters, namely temperature, top-p, and maximum token length, influence agent performance and automated cybersecurity task planning. For rapid evaluation, we present CTFTiny, a curated benchmark of 50 representative CTF challenges across binary exploitation, web, reverse engineering, forensics, and cryptography. Our findings identify optimal multi-agent coordination settings and lay the groundwork for future LLM agent research in cybersecurity.more » « less
-
Taxonomies are of great value to many knowledge-rich applications. As the manual taxonomy curation costs enormous human effects, automatic taxonomy construction is in great demand. However, most existing automatic taxonomy construction methods can only build hypernymy taxonomies wherein each edge is limited to expressing the “is-a” relation. Such a restriction limits their applicability to more diverse real-world tasks where the parent-child may carry different relations. In this paper, we aim to construct a task-guided taxonomy from a domain-specific corpus, and allow users to input a “seed” taxonomy, serving as the task guidance. We propose an expansion-based taxonomy construction framework, namely HiExpan, which automatically generates key term list from the corpus and iteratively grows the seed taxonomy. Specifically, HiExpan views all children under each taxonomy node forming a coherent set and builds the taxonomy by recursively expanding all these sets. Furthermore, HiExpan incorporates a weakly-supervised relation extraction module to extract the initial children of a newly expanded node and adjusts the taxonomy tree by optimizing its global structure. Our experiments on three real datasets from different domains demonstrate the effectiveness of HiExpan for building task-guided taxonomies.more » « less
-
LLM-based agents that invoke external tools can cause irreversible harm when their actions diverge from user intent, from deleting critical files to exposing private data or executing untrusted code. Unlike traditional software, where test suites and contracts define expected behavior, LLM agents operate without a specification: there is no artifact against which to check their actions before execution. Existing safeguards rely on prompt engineering or post-hoc evaluation, neither of which closes this gap. We present RefineAct, a runtime verification framework that automatically derives a task-specific specification from the user’s natural-language instruction and enforces it during agent execution. RefineAct operates in three stages: (1) intent formalization translates instructions into first-order Prolog predicates capturing the user’s goal and safety requirements; (2) refinement-based planning decomposes the specification into a concrete action plan with pre-/postcondition chains and risky-action patterns; and (3) runtime verification intercepts each agent-proposed action, queries the Prolog knowledge base to verify precondition satisfaction and constraint compliance, and returns a disposition state that determines whether the action proceeds, requires user confirmation, or is blocked with corrective feedback. We evaluate RefineAct on 144 agent tasks across five domains from the ToolEmu benchmark. RefineAct reduces failure incidence from 77% to 39% while improving task completion quality from 1.0 to 1.9 on a 0–3 scale, demonstrating that deriving and enforcing specifications at runtime can improve both safety and helpfulness of LLM agent behavior.more » « less
