More Like this

1. Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark

   https://doi.org/10.1609/aaai.v40i35.40210  

   Shao, Minghao; Rani, Nanda; Milner, Kimberly; Xi, Haoran; Udeshi, Meet; Aggarwal, Saksham; Putrevu, Venkata_Sai Charan; Shukla, Sandeep K; Krishnamurthy, Prashanth; Khorrami, Farshad; et al (March 2026, Proceedings of the AAAI Conference on Artificial Intelligence)

   Recent advances in LLM agentic systems have improved the automation of offensive security tasks, particularly for Capture the Flag (CTF) challenges. We systematically investigate the key factors that drive agent success and provide a detailed recipe for building effective LLM-based offensive security agents. First, we present CTFJudge, a framework leveraging LLM as a judge to analyze agent trajectories and provide granular evaluation across CTF solving steps. Second, we propose a novel metric, CTF Competency Index (CCI) for partial correctness, revealing how closely agent solutions align with human-crafted gold standards. Third, we examine how LLM hyperparameters, namely temperature, top-p, and maximum token length, influence agent performance and automated cybersecurity task planning. For rapid evaluation, we present CTFTiny, a curated benchmark of 50 representative CTF challenges across binary exploitation, web, reverse engineering, forensics, and cryptography. Our findings identify optimal multi-agent coordination settings and lay the groundwork for future LLM agent research in cybersecurity. 

   more » « less
2. Hybrid agentic AI and multi-agent systems in smart manufacturing

   https://doi.org/10.1016/j.jmsy.2026.04.002  

   Farahani, Mojtaba A; Khan, Md Irfan; Wuest, Thorsten (June 2026, Journal of Manufacturing Systems)

   The convergence of Agentic Artificial Intelligence (AI) and Multi-Agent Systems (MAS) enables a new paradigm for intelligent decision-making in Smart Manufacturing Systems (SMS). Traditional MAS architectures emphasize distributed coordination and specialized autonomy, while recent advances in agentic AI driven by Large Language Models (LLMs) introduce higher-order reasoning, planning, and tool orchestration capabilities. This paper presents a hybrid agentic AI and multi-agent framework for a Prescriptive Maintenance (RxM) use case, where LLM-based agents provide strategic orchestration and adaptive reasoning, complemented by rule-based and Small Language Models (SLMs) agents performing efficient, domain-specific tasks on the edge. The proposed framework adopts a layered architecture that consists of perception, preprocessing, analytics, and optimization layers, coordinated through an LLM Planner Agent that manages workflow decisions and context retention. Specialized agents autonomously handle schema discovery, intelligent feature analysis, model selection, and prescriptive optimization, while a human-in-the-loop interface ensures transparency and auditability of generated maintenance recommendations. This hybrid approach enables dynamic model adaptation, transparent decision-making, and cost-aware maintenance scheduling based on data-driven insights. An initial proof-of-concept implementation is validated on two industrial manufacturing datasets. The developed framework is modular and extensible, allowing new agents or domain-specific modules to be integrated seamlessly as system capabilities evolve. The results demonstrate the system’s capability to automatically detect schema, adapt preprocessing pipelines, optimize model performance through adaptive intelligence, and generate actionable, prioritized maintenance recommendations. The framework shows promise in achieving improved robustness, scalability, and explainability for RxM in smart manufacturing, bridging the gap between high-level agentic reasoning and low-level autonomous execution. 

   more » « less
3. LLM Collaboration with Multi-Agent Reinforcement Learning

   Liu, Shuo; Chen, Tianle; Liang, Zeyu; Lyu, Xueguang; Amato, Christopher (January 2026, The Annual AAAI Conference on Artificial Intelligence (AAAI))

   A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pre-trained independently and not specifically optimized for coordination. For example, existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address this challenge, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, MultiAgent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning multiple LLMs with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using MARL methods for LLM collaboration and highlights the associated challenges. Our code is available at https://github.com/OpenMLRL/CoMLRL. 

   more » « less
4. Generalizability of Large Language Model-Based Agents: A Comprehensive Survey

   https://doi.org/10.1145/3794858  

   Zhang, Minxing; Yang, Yi; Xie, Roy; Dhingra, Bhuwan; Zhou, Shuyan; Pei, Jian (July 2026, ACM Computing Surveys)

   Large Language Model (LLM)-based agents have recently emerged as a new paradigm that extends the capabilities of LLMs beyond text generation to dynamic interaction with external environments. A critical challenge lies in ensuring theirgeneralizability – the ability to maintain consistently high performance across varied instructions, tasks, environments, and domains, especially those different from the agent’s fine-tuning data. Despite growing interest, the concept of generalizability in LLM-based agents remains underdefined, and systematic approaches to measure and improve it are lacking. We provide the first comprehensive review of generalizability in LLM-based agents. We begin by clarifying the definition and boundaries of agent generalizability. We then review existing benchmarks. Next, we categorize strategies for improving generalizability into three groups: methods targeting the backbone LLM, targeting agent components, and targeting their interactions. Furthermore, we introduce the distinction betweengeneralizable frameworks andgeneralizable agents and outline how generalizable frameworks can be translated into agent-level generalizability. Finally, we identify future directions, including the development of standardized evaluation frameworks, variance- and cost-based metrics, and hybrid approaches that integrate methodological innovations with agent architecture-level designs. We aim to establish a foundation for principled research on building LLM-based agents that generalize reliably across diverse real-world applications. 

   more » « less
5. MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming

   https://doi.org/10.1145/3711032  

   Li, Yuan; Sun, Lichao; Zhang, Yixuan (April 2025, Proceedings of the ACM on Human-Computer Interaction)

   Significant advancements have occurred in the application of Large Language Models (LLMs) for social simulations. Despite this, their abilities to perform teaming in task-oriented social events are underexplored. Such capabilities are crucial if LLMs are to effectively mimic human-like social behaviors and form efficient teams to solve tasks. To bridge this gap, we introduce MetaAgents, a social simulation framework populated with LLM-based agents. MetaAgents facilitates agent engagement in conversations and a series of decision making within social contexts, serving as an appropriate platform for investigating interactions and interpersonal decision-making of agents. In particular, we construct a job fair environment as a case study to scrutinize the team assembly and skill-matching behaviors of LLM-based agents. We take advantage of both quantitative metrics evaluation and qualitative text analysis to assess their teaming abilities at the job fair. Our evaluation demonstrates that LLM-based agents perform competently in making rational decisions to develop efficient teams. However, we also identify limitations that hinder their effectiveness in more complex team assembly tasks. Our work provides valuable insights into the role and evolution of LLMs in task-oriented social simulations. 

   more » « less