Search for: All records

Greybox fuzzing relies heavily on high-quality initial seeds, yet preparing such seeds remains a manual bottleneck for programs with custom or poorly documented input formats. Existing LLM-based seed-generation approaches largely rely on one-shot prompting and lack feedback about coverage, format constraints, and generation failures. We present SEEDMIND, an agent-based system that asks LLMs to create seed generators and iteratively improves them using coverage feedback, context pruning, and state-driven realignment. Across OSS-Fuzz targets and the MAGMA benchmark, SEEDMIND generates seeds close in quality to human-curated corpora, significantly outperforms prior LLM-based seed generation, and enables AFL, AFL++, and Honggfuzz to achieve bug-finding performance comparable to human-created seeds. In real-world deployment, SEEDMIND supported our Cyber Reasoning System during the AIxCC competition. After the competition, we further applied SEEDMIND to uncover previously unknown vulnerabilities in widely used open-source projects. 

Large language models (LLMs) are widely used in software development. However, the code generated by LLMs often contains vulnerabilities. Several secure code generation methods have been proposed to address this issue, but their current evaluation schemes leave several concerns unaddressed. Specifically, most existing studies evaluate security and functional correctness separately, using different datasets. That is, they assess vulnerabilities using securityrelated code datasets while validating functionality with general code datasets. In addition, prior research primarily relies on a single static analyzer, CodeQL, to detect vulnerabilities in generated code, which limits the scope of security evaluation. In this work, we conduct a comprehensive study to systematically assess the improvements introduced by four state-of-the-art secure code generation techniques. Specifically, we apply both security inspection and functionality validation to the same generated code and evaluate these two aspects together. We also employ three popular static analyzers and two LLMs to identify potential vulnerabilities in the generated code. Our study reveals that existing techniques often compromise the functionality of generated code to enhance security. Their overall performance remains limited when evaluating security and functionality together. In fact, many techniques even degrade the performance of the base LLM by more than 50%. Our further inspection reveals that these techniques often either remove vulnerable lines of code entirely or generate “garbage code” that is unrelated to the intended task. Moreover, the commonly used static analyzer CodeQL fails to detect several vulnerabilities, further obscuring the actual security improvements achieved by existing techniques. Our study serves as a guideline for a more rigorous and comprehensive evaluation of secure code generation performance in future work.