Software development is a time-consuming and repetitive task. The rapid evolution of large language models (LLMs) has greatly bene!ted software developers. Given task requirements described in natural language, LLMs can generate functional and easy-to-adopt code snippets, signi!cantly accelerating the software development process. Many LLM-based code assistants are already integrated into IDEs, such as Copilot [2] and Cursor [3].
However, just like human developers, LLMs can also make mistakes when producing code. One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers.
To address this, a number of techniques have been proposed to improve the security of LLM-generated code, called secure code generation methods. These techniques typically involve either adjusting LLM weight parameters or manipulating the input to or output from the LLM. For example, SVEN [23] and SafeCoder [24] construct a training dataset containing code snippets with and without vulnerabilities. They use such a dataset to !ne-tune the model such that it can generate vulnerability-free code once trained. CodeGuard+ [21] instead controls the generation process of LLM inference. As LLMs use a decoding algorithm to determine the output, CodeGuard+ modi!es this algorithm to favor outputs that lead to secure code. Another state-of-the-art technique, PromSec [41], iteratively re!nes the task prompt based on the feedback from a vulnerability scanner on the generated code.
The evaluation for these secure code generation techniques typically considers whether the generated code contains vulnerabilities. A common practice is to use a vulnerability scanner, such as Cod-eQL [7], to assess the security. These techniques also measure their impact on normal model utility, that is, whether the enhanced LLM can still generate functional and usable code. A code benchmark like HumanEval [15] is usually adopted for the evaluation.
While the current evaluation schemes make sense, there are a few problems. First, existing works mainly rely on the reported results by a single vulnerability detector, CodeQL, for assessing security, following the security evaluation practice proposed in prior work [45]. This scanner is not the gold standard as it can miss or mis#ag certain vulnerabilities. Our experiments in Section 4.1 show that CodeQL can miss more than 20% vulnerabilities in generated code. Second, the current evaluation scheme examines the security and functional correctness of generated code independently. In other words, it uses a security-related code dataset to measure the security (e.g., [53]) and use another general code dataset to assess functionality (e.g., [44]). Such an evaluation scheme leaves a gap of reported results between the two aspects. Are the generated code from the functionality dataset secure? Are the generated code from the security dataset functionally correct?
A straightforward idea is to use one of those datasets to evaluate both security and functional correctness on the LLM-generated code. However, there are a few issues with the datasets employed in previous works. Most security-related code datasets do not have unit tests [13,53,56], meaning it is not feasible to rigorously evaluate the functionality of the generated code. The functionality benchmarks, such as HumanEval [15] and MBPP [12], on the other hand, contain relatively simple tasks. They do not have the complexity of triggering security issues in the generated code, resulting in nearly 100% security rate [53].
A few works craft new benchmarks for evaluating the security and functionality of LLM-generated code together, such as Code-Guard+ [21] and CWEval [47]. While promising, their sample size is at a small scale, with CodeGuard+ and CWEval containing only 91 and 119 tasks, respectively, which limits a comprehensive and in-depth analysis. In addition, these works focus on either constructing the benchmark for mainstream LLMs or assessing the performance of a speci!c technique (e.g., SVEN [23]). They do not aim to comprehensively evaluate many existing secure code generation techniques.
In this work, we conduct a comprehensive study on the performance of existing secure code generation techniques when considering security and functionality simultaneously. Our goal is to understand not only their overall performance but also what contributes to the improvement or reduction of the two aspects. We focus on function-level code generation, a commonly adopted scenario in LLM-aided development. To achieve this, we leverage two large datasets on code generation tasks: BigCodeBench [70] and SecCodePLT [62], each containing more than 1,000 tasks. This allows for a more comprehensive assessment of existing techniques at a large scale. In addition, to avoid relying on a single vulnerability scanner, we incorporate three static analyzers: CodeQL [7], Bearer [17], and Bandit [48], for evaluating the security of LLMgenerated code. We also consider two LLMs: Llama3.3-70B [22] and Qwen2.5-72B [61], for zero-shot vulnerability detection, as LLMs have shown promise in assessing code security [32,33,55,68].
Our study reveals that while existing secure code generation techniques improve the security of LLM-generated code to some extent, they often come at the cost of sacri!cing functional correctness. A deeper analysis shows that these techniques may simply remove insecure lines of code or produce "garbage code" that is irrelevant to the intended task. However, such problems cannot be discovered with the current evaluation scheme, as the security and functionality scores are reported based on separate evaluations. Furthermore, our study demonstrates that vulnerability scanners have di"erent strengths in detecting certain types of vulnerabilities. No single scanner can cover all potential security issues. Therefore, it is critical to employ more scanners in security evaluations or use other advanced security assessment methods. The contributions of this work are summarized below:
• We introduce a new metric, SAFE, that considers both the security and functionality of LLM-generated code to evaluate state-of-the-art secure code generation techniques and identify gaps in the current evaluation schemes. 1 2 3 4 5 6 7 13 14 15 16 17 18 19 20 21 22 23 24 25 26 50
Overlooked By CodeQL • We construct an enhanced dataset SecCodePLT+ with unit tests, providing a benchmark for secure code evaluation. • We investigate the performance disparity in functionality and security of existing techniques and classify causes of functionality reduction into !ve categories. • We highlight the limited performance improvement of existing techniques under our evaluation framework and call for new approaches to secure code generation.
Code generation aims to obtain a code snippet from an LLM through a prompt describing the desired task. Yet, the generated code can carry vulnerabilities. To enhance security, several techniques have been proposed to guide LLMs in generating code that not only accomplishes its intended tasks but also remains free of vulnerabilities [21,23,24,41,65]. These techniques commonly !ne-tune the model, modify the input prompt, or manipulate the generation process, for which more details are discussed in Section 3.2. Secure code generation needs to be evaluated from two aspects, functionality and security. Functionality measures whether the generated code adheres to the task requirements, while security inspects the existence of vulnerabilities in the generated code. Most existing works evaluate the two aspects independently. That is, they use one dataset to evaluate functionality (e.g., [44]) and another dataset to assess security (e.g., [53]). The results are then combined to represent the technique's performance. The functionality dataset usually comes with unit tests, which can be applied directly for evaluation. Yet, the security dataset o"ers no such measurements. Moreover, the existing works use external vulnerability scanners like CodeQL [7] to detect if the generated code has vulnerabilities.
An intuitive question researchers often have is why the existing works do not directly measure the security of the code generated for the functionality dataset [21,47]. The common reason is that tasks in the functionality dataset are usually simple, which lack the complexity of triggering security issues in the generated code. Thus, they are not suited for systematically assessing security.
import os import random def task_func(directory, n_files): # Create the directory if it doesn't exist if not os.path.exists(directory): os.makedirs(directory) # Generate and write to files for i in range(n_files): file_name = os.path.join(directory, f'file_{i+1}.txt') with open(file_name, 'w') as file: file.write(str(random.randint(0, 9))) # Reset the cursor to the beginning of each file for i in range(n_files): file_name = os.path.join(directory, f'file_{i+1}.txt') with open(file_name, 'r+') as file: file.seek(0) return n_files # Instruction: Create n random txt files in a specific directory, write only a single digit random integer into each file, and then reset the cursor to the beginning of each file. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 import os import random def task_func(directory, n_files): # Create the directory if it doesn't exist -----------------------return n_files 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 DeepSeek-Coder DeepSeek-Coder + CodeGuardPlus
Problem (I). The existing works commonly run CodeQL [7], a static vulnerability scanner, to assess the security of the generated code. CodeQL employs rules referencing the CWE list [1] for detecting security vulnerabilities. However, the rules do not cover the full CWE list. Further, the rules for a single CWE item can be incomplete. As a result, CodeQL can miss vulnerabilities in the generated code, leading to an inaccurate measurement of security.
Figure 1 shows a piece of "secure" code generated by Qwen2.5-Coder-7B-Instruct enhanced by CodeGuard+ [21], following the instruction at the top of the !gure. CodeQL, using both the default rules and GitHub-extended rules [8] for CWE-78 [39], detects no vulnerabilities in the code. However, line 15 in the code directly executes the user-provided shell script without checking for malicious commands, representing a CWE-78 issue. In this case, the security of CodeGuard+ is overestimated.
Problem (II). Independently evaluating functionality and security can obscure the actual performance of secure code generation. For example, during security evaluation, the LLMs can be "encouraged" to generate simple yet task-irrelevant code, passing security tests and leading to one-sided observations.
Figure 2 shows a case where DeepSeek-Coder-V2-Lite [5] is asked for code to create a list of !les with a random number in them and reset the cursor, using the prompt at the top of the !gure. The generated code (on the left) satis!es the task requirements and passes all unit tests. However, without applying techniques to improve the security, the code contains a vulnerability. Line 6 on the left-hand side uses external input to create a directory without proper sanitization. This can be exploited to bypass the limitation to restricted directories (CWE-22 [38]). Similar issues also exist at lines 10 and 16. To secure the code generation, we apply CodeGuard+ [21] on top of DeepSeek-Coder-V2-Lite to rerun the task. The newly generated code, presented on the right of Figure 2, removed all code except for package import and function declaration. Despite its uselessness, this piece of code will pass all security tests and contribute to a high security score.
In fact, independently evaluating functionality and security leads to many other problems like the example above. We categorize them into !ve categories and present their details in Section 4.3.
3 Research Questions and Study Methodology 3.1 Scope and Research Questions Scope of Our Study. In this work, we focus on re-understanding the security and functionality of code generated by LLMs. Instead of considering end-to-end software development by LLMs, we focus on function-level code generation, a commonly adopted scenario in LLM-aided development. Our study does not aim at the performance of the most advanced models but instead targets secure code generation techniques (see Section 3.2) applied to mainstream LLMs, including open-source and proprietary models, such as CodeLlama, DeepSeek-Coder, and GPT-4o.
Research Questions. We focus on the following research questions. We refer to the evaluation of both security and functionality as the combined measure.
• (RQ1) How do vulnerability scanners perform when assessing the security of LLM-generated code? • (RQ2) How do secure code generation techniques perform when security and functionality are evaluated together? • (RQ3) What leads to the disparity (if any) between functionality observed under combined measure and functionality assessed independently? • (RQ4) What contributes to the disparity (if any) between security observed under combined measure and security assessed independently?
Secure Code Generation Methods. We consider four state-of-theart secure code generation techniques published in 2023 and 2024: SVEN [23], SafeCoder [24], CodeGuard+ [21], and PromSec [41]. These methods are among the most recent and widely recognized, including SVEN, the CCS 2023 Distinguished Paper.
Table 1 summarizes these techniques. SVEN and SafeCoder are !ne-tuning-based CodeGuard+ does not require !ne-tuning but instead controls the generation process during inference. Since LLMs produce an output sequence by generating tokens one by one, they rely on a decoding algorithm to determine which token to choose at each step. CodeGuard+ modi!es the decoding algorithm to favor tokens that lead to a secure sequence. As a result, it requires gray-box access to the model, speci!cally to the decoding algorithm during inference. Additionally, CodeGuard+ modi!es the original task prompt by incorporating security-related text, such as "use snprintf" to avoid bu"er over#ow vulnerabilities.
PromSec is a prompt engineering-based technique that iteratively re!nes the task prompt. Speci!cally, it !rst leverages external tools such as Bandit [48] to identify vulnerabilities in the generated code. If any vulnerabilities are detected, PromSec utilizes a generative adversarial network (GAN) model to enhance the security of the code. The updated code is then fed back into the same LLM to generate a new prompt, which is used for the next generation step. This process repeats iteratively until no vulnerabilities are detected in the generated code by the external tool. Code Datasets. There are several datasets available for evaluating the functionality and security of code generation. However, as explained in Section 2, the tasks in the functionality dataset are usually simple, which cannot trigger security issues in the generated code for assessing security. In contrast, security-related datasets often do not include unit tests for assessing functionality. For our study, we use two public datasets: BigCodeBench [70] and SecCode-PLT [62], whose information is shown in Table 2. BigCodeBench includes unit tests, which makes it well-suited for evaluation. However, SecCodePLT does not provide unit tests, limiting its ability to assess functionality.
To ensure a comprehensive evaluation, we construct unit tests for SecCodePLT. Since this dataset provides ground truth code for each generation task, we leverage an LLM, Qwen2.5-Coder-32B [26], to generate unit tests. Speci!cally, we prompt Qwen2.5-Coder-32B to create a set of test inputs based on the ground truth code. We then execute the ground truth code with these inputs to obtain expected outputs. Additionally, we include pre-existing inputs in the prompt to encourage Qwen2.5-Coder-32B to generate diverse test cases. The resulting input-output pairs serve as unit tests for evaluating code generated by various methods. Using this approach, we generated an average of 7.5 unit test cases per task for SecCodePLT, even more than the 5.6 test cases per task provided by BigCodeBench. The enhanced SecCodePLT dataset, which we call SecCodePLT+, can be found here [9].
There are other security-related datasets, such as CyberSecEval [13] and SecurityEval [53]. CyberSecEval provides various task prompts for secure code evaluation across multiple programming languages, along with ground truth code for each task. However, the provided code consists only of function fragments rather than complete programs. As a result, it is challenging to generate unit tests and conduct functional testing using this dataset. SecurityEval contains only 130 samples and lacks ground truth code, making it di$cult to construct unit tests. Additionally, since the data is extracted from example code on the MITRE CWE web page, there is a potential risk of data contamination. LLMs may have already been trained on these examples, which limits the usefulness of this dataset in evaluating their performance.
Measurement. The goal of this study is to evaluate security and functionality together, which has been overlooked by many existing works. One metric introduced by [21,47] considers both aspects. It is called Secure-Pass@𝐿, which calculates the percentage of generated code snippets that pass all unit tests and do not contain any security vulnerabilities. While this is a useful metric, it is too restrictive and overlooks the usefulness of the generated code. For instance, LLMs may not produce a fully functional code snippet that passes all unit tests. However, the generated code snippet may be mostly correct but miss a few lines, such as for handling di"erent data types. Such a snippet can still be useful to developers with minimal e"ort.
Therefore, we propose a new metric, SAFE 1 , which considers both the security and functionality of LLM-generated code while introducing a relaxation on functionality. Speci!cally, it is computed using the following formula:
Here, secure 𝐿 denotes whether the 𝑁-th generated code snippet in the top-𝐿 by an LLM contain vulnerabilities, where 1 indicates no vulnerabilities and 0 otherwise. case-pass 𝐿 represents the average unit test passing rate for the 𝑁-th code in the top-𝐿. That is, we calculate the percentage of passed unit tests for this generated code. Note that we leverage the exponential function to calibrate the unit test passing score, assigning signi!cantly lower scores to samples that pass very few test cases. This metric is more !ne-grained than Secure-Pass@𝐿, as it takes the unit test passing rate into account. While SAFE@𝐿 is designed to be #exible and applicable to various evaluation scenarios, we primarily report results for 𝐿 = 1 in our evaluation. This aligns with common practice in the literature [23,24] and re#ects real-world usage, where users typically see only the top-1 output from LLMs such as ChatGPT or code assistants like Cursor.
LLMs. We employ !ve popular open-source code LLMs for our study: CodeLlama-7B [50], Qwen2.5-Coder-7B [26], DeepSeek-Coder-V2-Lite [5], Mistral-7B [30], and StarCoder-1B [36]. Additionally, we include two commercial APIs: GPT-3.5-Turbo [43] and GPT-4o [27]. We include only smaller-sized LLMs (e.g., Qwen2.5-7B instead of 32B) and older-version models (e.g., StarCoder instead of StarCoder 2) to be consistent with those used by existing works [23,24]. We evaluate only PromSec on the commercial APIs since the other three techniques require white-box or gray-box access to the LLM, which is not available for commercial models. Since PromSec is only applicable to instruction-following models, we exclude Star-Coder from the evaluation, as it is designed for code completion. In addition, when we applied CodeGuard+ to CodeLlama and Mistral, we found that most of the generated code had indentation issues.
To assess the realistic functionality of the code, we correct these issues before conducting unit tests. Speci!cally, we used a Python syntax parser to detect the problem (three space instead of four) and developed a script to automatically correct it.
Vulnerability Scanners. To evaluate the security of generated code, a common practice is to use vulnerability scanners to detect potential security issues. We adopt three widely used static analyzers: CodeQL [7], Bearer [17], and Bandit [48]. Additionally, we consider LLMs for vulnerability detection, as they have shown promising results [32,68]. In our study, we use two LLMs: Qwen2.5-72B [61] and Llama3.3-70B [22], for security evaluation.
4 Evaluation Results
To evaluate the security of LLM-generated code, a common practice is to use a vulnerability scanner to detect potential vulnerabilities. If no vulnerabilities are detected, the code is considered secure.
Prior research [23,24,65] has primarily leveraged the static analyzer CodeQL [7] to assess code security. However, as discussed in Section 2, CodeQL can fail to detect certain vulnerabilities in generated code. Therefore, we also employ two other widely used static analyzers, Bearer [17] and Bandit [48], to measure security. The !rst three groups in each chart in Figure 3 present the security results evaluated by the three vulnerability scanners on the BigCodeBench dataset. CL, BR, and BT correspond to CodeQL, Bearer, and Bandit, respectively. Each chart represents the results for an LLM, and each bar corresponds to a di"erent secure code generation technique. The blue bar represents the base model. We did not report PromSec results for StarCoder, as PromSec requires the base model to generate code solely based on instructions. However, StarCoder only supports code completion tasks.
As shown in Figure 3, CodeQL (!rst group) reports nearly 100% security scores for all techniques, including the base model. (Note that we successfully reproduced CodeQL's results on the dataset used by these techniques, as reported in the original papers.) However, this does not necessarily mean the generated code is truly secure. As we can see the results from Bearer and Bandit (second and third groups), the security scores are approximately 25% lower than those reported by CodeQL for most models. For example, on Qwen2.5-Coder (in the 2nd chart), Bearer reports security scores of around 65%, while Bandit reports around 75% for both the base model and the three secure code generation methods. Although PromSec achieves higher scores, they are still lower than those reported by CodeQL.
Figure 4 reports the results for the SecCodePLT+ dataset. The observations are similar: CodeQL reports nearly 100% security scores for all models, while Bearer and Bandit show lower values. Additionally, we observe that Bearer reports a lower score for PromSec compared to the base model for DeepSeek-Coder and Mistral (in the 3rd and 4th charts). In contrast, Bandit shows the opposite: PromSec improves security over the base model.
The security scores reported by di!erent vulnerability scanners are inconsistent and can even be contradictory. Relying on a single vulnerability scanner is insu"cient for comprehensively assessing the security of generated code.
Figure 5 presents the reported CWE vulnerabilities identi!ed by the three static analyzers for Qwen2.5-Coder enhanced by Safe-Coder on the BigCodeBench dataset. The security scores reported by the scanners are 0.9877, 0.6535, and 0.7482 for CodeQL, Bearer, and Bandit, From the !gure, we observe that Cod-eQL identi!es only a very small number of CWE vulnerabilities (i.e., 4). Bandit detects slightly more vulnerabilities than Bearer but fails to identify certain vulnerabilities, such as CWE-328 and CWE-916, which are detected by either CodeQL or Bearer. The most frequently detected vulnerability is CWE-327, identi!ed by Bearer with 200 occurrences, which relates to the use of a broken or risky cryptographic algorithm.
Finding 2: Di!erent vulnerability scanners have varying strengths in identifying di!erent types of vulnerabilities. No single scanner can cover all potential security issues.
LLMs have shown promising results in detecting vulnerabilities in code. Therefore, we leverage two LLMs, Qwen2.5-72B [61] and Llama3.3-70B [22], to evaluate the security of generated code. Figure 6 presents the results reported by the LLMs and the three static analyzers for the Mistral model on SecCodePLT+. Surprisingly, the security scores reported by the LLMs are signi!cantly lower than those from the static analyzers. Additionally, the relative rankings of di"erent secure code generation techniques vary between the two approaches. To understand this signi!cant disparity, we conduct a manual inspection. In particular, we use SVEN as an example and use a !xed random seed to sample 44 cases from the generated code, ensuring an unbiased selection. We then perform a manual inspection of the sampled code.
Table 3 summarizes our manual analysis. For the three static analyzers, we observe a non-trivial number of false negatives (12-19 out of 44 samples), indicating that they fail to detect certain vulnerabilities. We have discussed such examples in Section 2. Conversely, the LLMs exhibit a high false positive rate (nearly 50%), meaning they tend to over-report vulnerabilities that do not actually constitute security issues. For instance, in one case, Qwen2.5-72B #ags a CWE-532 vulnerability, suggesting a sensitive information leak through logging. However, the generated code does not contain any logging functionality, yet the LLM still reported the issue. These !ndings are consistent with what has been reported in the literature [18,32,37], which indicates that the e"ectiveness of leveraging LLMs in vulnerability detection remains an open research question.
Finding 3: LLMs are helpful in detecting vulnerabilities. However, they also produce a signi#cant number of false positives, which hinders their application in security assessment for generated code.
Existing secure code generation approaches assess the functionality and security of LLM-generated code independently, using di"erent datasets. Here, we evaluate both aspects on the same dataset. Figure 3 presents the results on BigCodeBench. Existing techniques can improve security scores for most models, although di"erent vulnerability scanners may produce inconsistent relative rankings. For Mistral (in the 4th chart), Bearer reports a security reduction with CodeGuard+, while Bandit reports security reductions for three techniques: SVEN, SafeCoder, and CodeGuard+. PromSec consistently enhances security across four models. However, when evaluating the functionality of generated code using existing secure code generation techniques, we observe a notable disparity compared to the results reported in the original papers. The last two groups in each chart in Figure 3 illustrate the functional correctness of the generated code. The "Case" group represents test case-level scores, where we calculate the percentage Figure 7: The overall results of Secure-Pass@1 and SAFE@1 on BigCodeBench. The security evaluation is based on the combined results from the three static analyzers.
Figure 8: The overall results of Secure-Pass@1 and SAFE@1 on SecCodePLT+. The security evaluation is based on the combined results from the three static analyzers.
of passed unit tests for each task and then average the values across all tasks. This provides a !ne-grained view of functional correctness. The "Task" group, on the other hand, represents task-level results, where we count only the samples that pass all unit tests. We observe that most techniques reduce the functionality of generated code at both the test case and task levels. This suggests that these techniques may not actually be !xing vulnerabilities in the generated code but instead sacri!cing functionality to make the code "appear" more secure. We have previously discussed an example case in Section 2 and will further explore this issue in the following RQs. CodeGuard+ improves functionality scores on CodeLlama, Qwen2.5-Coder, and Mistral. However, its security improvements are limited or even decrease on Mistral, indicating that CodeGuard+ struggles to balance security and functionality in generated code.
The observations on the SecCodePLT+ dataset are similar, as shown in Figure 4. All evaluated techniques reduce the functionality of the generated code. SVEN and SafeCoder show a signi!cant drop (over 50%) in functional correctness for Codellama. This is likely due to the challenging nature of SecCodePLT+, where even GPT-4o achieves only 8.74% task-level performance. Fine-tuning-based methods like SVEN and SafeCoder may negatively impact code generation quality, leading to lower functionality scores. Moreover, security improvements are also limited for most models. Notably, PromSec experiences nearly a 9% security degradation on Codellama, as reported by Bearer.
Finding 4: Existing secure code generation techniques can enhance the security of generated code to some extent, but often at the expense of functional correctness.
As observed above, the trade-o" between security and functionality makes direct head-to-head comparisons challenging. Therefore, Table 4: Statistical analysis results for secure code generation methods. T-tests are conducted to assess whether there is a statistically signi!cant di"erence between the base model and each secure code generation method. A p-value (𝑂) < 0.05 indicates statistical signi!cance. Cohen's 𝑃 is reported as the e"ect size (ES), with values around 0.2, 0.5, and 0.8 representing small, medium, and large e"ects, respectively.
Secure-Pass@1 SAFE@1 Secure-Pass@1 SAFE@
1 Model Method 𝑀 ES 𝑀 ES 𝑀 ES 𝑀 ES Codellama SVEN 0.00 0.236 0.00 0.229 0.59 0.021 0.09 0.069 SafeCoder 0.00 0.182 0.00 0.168 0.00 0.119 0.00 0.23 CodeGuard+ 0.00 0.182 0.00 0.202 0.16 0.057 0.00 0.134 PromSec 0.00 0.212 0.00 0.251 0.05 0.079 0.02 0.093 Qwen2.5-Coder SVEN 0.01 0.101 0.05 0.081 0.37 0.036 0.77 0.011 SafeCoder 0.74 0.013 0.92 0.003 0.39 0.034 0.06 0.074 CodeGuard+ 0.12 0.06 0.47 0.029 0.39 0.034 0.05 0.077 PromSec 0.00 0.512 0.00 0.603 0.07 0.073 0.00 0.136 DeepSeek-Coder SVEN 0.00 0.176 0.00 0.17 0.23 0.048 0.03 0.084 SafeCoder 0.00 0.116 0.00 0.157 0.73 0.014 0.84 0.007 CodeGuard+ 0.95 0.002 0.76 0.012 0.56 0.023 0.20 0.051 PromSec 0.00 0.370 0.00 0.411 0.23 0.048 0.26 0.045 Mistral SVEN 0.04 0.083 0.00 0.224 0.00 0.114 0.00 0.207 SafeCoder 0.00 0.206 0.00 0.290 0.40 0.033 0.95 0.002 CodeGuard+ 0.00 0.336 0.00 0.240 0.28 0.043 0.20 0.051 PromSec 0.03 0.087 0.00 0.224 0.09 0.067 0.00 0.140 StarCoder SVEN 0.05 0.079 0.47 0.030 0.29 0.042 0.03 0.086 SafeCoder 0.01 0.102 0.00 0.319 0.54 0.024 0.57 0.022 CodeGuard+ 0.86 0.006 0.60 0.021 0.54 0.024 0.10 0.067 GPT-3.5-Turbo PromSec 0.00 0.171 0.00 0.146 0.90 0.004 0.86 0.007 GPT-4o PromSec 0.29 0.043 0.87 0.006 0.51 0.026 0.13 0.061
we need a metric that evaluates both aspects of generated code in a uni!ed manner. We use the two metrics Secure-Pass@1 and SAFE@1 discussed in Section 3.2 for the evaluation. Figure 7 presents the results using the two metrics on the Big-CodeBench dataset. We also report the t-test and e"ect size (Cohen's Table 5: Results of PromSec using commercial APIs on BigCodeBench and SecCodePLT+.
Secure@1 Pass@1 Secure-Pass@1 SAFE@1 Secure@1 Pass@1 Secure-Pass@ Figure 8 presents the results on SecCodePLT+. Nearly all secure code generation techniques fail to improve Secure-Pass@1 and SAFE@1 scores. This is largely due to the challenging nature of SecCodePLT+. Even GPT-4o, a state-of-the-art commercial LLM, struggles to achieve a high score as we will discuss later. Additionally, on CodeLlama, we observe that CodeGuard+ slightly outperforms PromSec in the Secure-Pass@1 metric with no statistically signi!cant di"erence (𝑂 > 0.05). However, the results are reversed for SAFE@1. This suggests that while PromSec may produce fewer samples that pass all unit tests, the ones it does generate tend to be secure and closely aligned with the intended task. SAFE@1, therefore, provides a more !ne-grained interpretation of the results. While Figure 7 and Figure 8 present results on open-source models, we also evaluate two commercial APIs in our experiments.
As previously discussed, all evaluated techniques except PromSec require white-box or gray-box access to the LLM, which is not available for commercial APIs. Therefore, this experiment primarily focuses on PromSec. Table 5 reports results on the BigCodeBench and SecCodePLT+ datasets. Columns Secure@1 and Pass@1 present individual security scores from di"erent vulnerability scanners along with unit test results, while the following columns show the Secure-Pass@1 and SAFE@1 scores.
PromSec shows a signi!cant improvement in security for GPT-3.5 on BigCodeBench (𝑂 < 0.05, Cohen's 𝑃 = 0.146). However, the Pass@1 scores decrease substantially (𝑂 < 0.05). This is re#ected in the Secure-Pass@1 and SAFE@1 scores, which drop by 23% (from 0.3219 to 0.2447) and 28% (from 0.4445 to 0.381), respectively. GPT-3.5 was the default model used for evaluation in the original PromSec paper [41], but it still cannot improve overall performance when security and functionality are measured together. Security improvements for GPT-4o and on the SecCodePLT+ dataset are limited, with PromSec showing little improvement in Secure-Pass@1 (𝑂 = 0.51) and SAFE@1 scores (𝑂 = 0.13). Another observation is that even GPT-3.5 and GPT-4o exhibit very low Pass@1 scores on the SecCodePLT+ dataset. This highlights the challenging nature of the tasks in this dataset, which require advanced techniques to enhance the functional correctness of generated code.
In RQ2, we observe the degradation in functionality of generated code by existing techniques. We further analyze the results to better understand this phenomenon. Speci!cally, we collect all tasks where the generated code by the base models is functionally correct and inspect the outcomes when applying the secure code generation techniques. Table 6 presents the results on the two datasets.
For each dataset, the !rst two columns show the results for the aforementioned cases.
• The column " to " denotes cases where insecure but functional code by the base model becomes secure but nonfunctional after applying existing techniques.
• The column " to " represents cases where secure and functional code becomes secure but non-functional after applying existing techniques. We observe that around 10% to 30% of tasks fall under the " to " category. Notably, CodeGuard+ with StarCoder on Big-CodeBench leads to 80% of tasks becoming non-functional after Table 7: Case study on the code generated by existing techniques using DeepSeek-Coder on BigCodeBench for the " to " category. The value in parentheses denotes the number of cases for each technique in this category. "NFI" represents cases where the generated code did not follow the instruction. "FN" indicates cases where the code snippet contains a vulnerability that was missed by the static analyzers.
Removed Code Junk Code NFI FN Other SVEN (69) 75.36% 8.7% 0% 2.89% 13.04% SafeCoder (20) 60% 0% 0% 25% 15% CodeGuard+ (31) 67.74% 6.45% 12.9% 3.23% 9.68% PromSec (63) 92.06% 0% 7.94% 0% 0%
improving security. The percentages are generally similar between BigCodeBench (Avg. 9.97%) and SecCodePLT+ (Avg. 11.75%), with no statistically signi!cant di"erence (𝑂 > 0.05). For the " to " category, the percentages are even higher, with most cases ranging from 20% to 60% on BigCodeBench and 20% to 40% on SecCode-PLT+. This indicates that while existing techniques are e"ective in !xing vulnerabilities, they tend to compromise the functionality of generated code for samples that were previously correct. This explains the functionality degradation we observed earlier.
In summary, SVEN, SafeCoder, CodeGuard+, and PromSec can cause originally functional code to become non-functional in up to 76.74%, 84.05%, 80%, and 77.09% of cases, respectively. We perform a two-way ANOVA test to assess whether the secure code generation method has a statistically signi!cant e"ect relative to the base model. The results show that, for all secure code generation methods across both datasets, the p-value is less than 0.001, meaning that existing methods can signi!cantly degrade the performance of the base model. To better understand the issue, we manually inspect the cases in the " to " category. We use DeepSeek-Coder on Big-CodeBench as an example. Table 7 presents the results for the four existing techniques. We classify the cases into !ve categories. These categories are based on our manual inspection of all cases, with discrepancies resolved through majority voting among the authors.
• Removed Code: This category includes cases where techniques simply remove vulnerability-related code to improve security, as shown in the example in Section 2. We observe a large number of cases in this category, with PromSec removing insecure code in 92.06% of the samples. • Junk Code: The generated output is nonsensical, such as the repeated word "task task task." SVEN and CodeGuard+ show around 6%-8% of cases in this category. • Not Following Instruction (NFI): In these cases, the LLM generates code completely irrelevant to the task. CodeGuard+ and PromSec have a non-trivial percentage of cases in this category. • False Negatives (FN): These are cases where static analyzers fail to detect vulnerabilities, which was discussed in RQ1. • Other: In a few cases, the generated code is missing necessary package imports or is otherwise incomplete, leading to functional incorrectness.
Finding 8: The security improvements achieved by existing techniques come at the cost of sacri#cing the functionality of the generated code, primarily by removing vulnerability-related code and generating irrelevant or garbage output.
While existing secure code generation techniques were designed to improve the security of LLM-generated code, they may inadvertently degrade security for certain tasks. We have already observed the performance reduction in Figure 3 and 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 import pandas as pd import os def task_func(original_location, new_location, sheet_name): # Check if the original file exists if not os.path.exists(original_location): raise FileNotFoundError(f"The file does not exist.") # Copy the original file to a new location os.system(f'cp {original_location} {new_location}') # Load the new Excel file try: df = pd.read_excel(new_location, sheet_name) except KeyError: raise ValueError( ) return df 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Qwen-Coder Qwen-Coder + CodeGuardPlus model was originally secure, but vulnerabilities are detected after applying existing techniques. We refer to this as the non-monotonic security improvement of these methods. Table 6 presents the results for the scenario described above. Speci!cally, we consider two cases: • The column " to " denotes cases where secure and functional code by the base model becomes insecure but remains functional after applying existing techniques.
• The column " to " represents cases where secure code becomes insecure after applying existing techniques, regardless of its functionality. Observe that for the !rst case " to ", although it rarely happens, around 1% of samples can still occur, such as CodeGuard+ with Codellama and PromSec with GPT-3.5 on BigCodeBench. The percentage is much higher in SecCodePLT+. In fact, PromSec with DeepSeek-Coder exhibits 24.39% of samples becoming insecure after applying the technique. When functionality is not considered, more instances of secure code becoming insecure are observed after applying existing techniques, as shown in the column " to ". The average percentages are 4.11% for BigCodeBench and 6.87% for SecCodePLT+, respectively.
Finding 9: The security improvement provided by existing techniques is not monotonic; they may introduce vulnerabilities into code that was previously secure. Figure 9 shows an example where the task is to copy data from an Excel spreadsheet into a new Excel !le and then read and return the contents of the new !le. The base model, Qwen2.5-Coder-7B-Instruct, can generate a secure code snippet as shown on the left. However, after applying CodeGuard+ [21], the generated code on the right introduces a vulnerability at line 9. It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection. This vulnerability corresponds to CWE-78 [39].
The internal threat to validity lies in the limitations of static analyzers, which may produce false negatives, as discussed in Section 2.2. To mitigate this, we use three di"erent static analyzers and take the union of all detected CVEs, which helps reduce false negatives.
The external threat to validity primarily lies in the subjects used in our study. The code generation tasks we examine may not be representative. We mitigate this risk by using two recent large datasets covering over 2,000 tasks. Since these datasets focus mainly on Python, our !ndings may not generalize to other programming languages. However, most existing techniques were originally evaluated using Python. Our study re-evaluates these techniques. For the SecCodePLT dataset, we generate unit tests using an LLM. While the test cases may be limited, we address this by manually inspecting the generated inputs in conjunction with the ground truth code and setup !les. We also manually calibrate problematic unit tests. Although we did not manually verify edge case coverage, adding more cases would likely lower the passing rate, which supports our conclusion that existing methods are limited. Additionally, the LLMs and vulnerability scanners used in this study may not be fully representative, especially given the rapid development of LLMs. However, we argue that the general observations regarding existing secure code generation techniques will still hold, as the issues stem from their technical design rather than the speci!c LLMs or tools used.
The construct threat lies in determining a code snippet as vulnerable during manual inspection. Speci!cally, we may misidentify a vulnerability reported by di"erent scanners. To mitigate this threat, we ensure that each vulnerability is examined by at least two authors. A third author will chime in to resolve any disagreements.
Other Limitations. In our study, we do not include the HumanEval dataset as it was primarily designed to measure functionality. It consists of relatively simple tasks that lack the complexity required to trigger security issues in generated code. Additionally, the samples used in our evaluation might have been included in the training data of LLMs, potentially causing data contamination. Our employed two datasets are generally released after the evaluated models. Although this does not guarantee no data contamination, our results show existing works are still limited in generating secure and functional code, even if the test data may have been trained on, which further con!rms the need for better designs.
Due to resource limits, we did not repeat every experiment: a full run costs 430 GPU-hours (↓ 18 days) on a single NVIDIA A100, and larger models (e.g., DeepSeek) require more. As a stability check, we repeated Qwen three times with consistent results, totaling 1,290 GPU-hours (↓ 54 days) on one A100.
Our study highlights key problems in current evaluation schemes:
(1) relying on a single static analyzer for vulnerability detection, and (2) using separate datasets for evaluating security and functionality. Using inappropriate metrics to assess a technique's performance could potentially mislead research progress. To address these limitations, our study introduces a more rigorous framework that uses multiple vulnerability detectors and evaluates both security and functionality on the same set of generated code. Section 4.3 and Section 4.4 provide analysis of failure cases, o"ering insights into which aspects of existing methods require improvement. These contributions are crucial for establishing a proper evaluation standard.
Future research on secure code generation shall follow our evaluation framework by evaluating security and functionality jointly, such that the performance of proposed techniques is rigorously assessed and comparable with one another. Our enhanced SecCode-PLT+ dataset, equipped with unit tests, can serve as a standard benchmark for comprehensively assessing the performance. Additionally, we have categorized various failure cases of existing techniques, such as removing vulnerability-related code to improve security. Future e"orts shall design new approaches that can address these problems and maintain the functionality of generated code while enhancing its security. As we mainly leverage vulnerability detectors for assessing the security of generated code -which themselves may produce false reports -an important future direction is to develop a reliable and theoretically guaranteed vulnerability detection method.
LLM for (Secure) Code Generation. Several Code LLMs have been speci!cally trained for code generation tasks using code datasets, such as CodeGen [42], InCoder [20], SantaCoder [10], CodeLlama [50], Qwen-Coder [26], DeepSeek-Coder [5], Code-Stral [4], and StarCoder [36]. Moreover, various studies have focused on enhancing the performance of Code LLMs by optimizing prompts [31,34,35]. Additionally, some studies use LLMs to synthesize test cases and leverage the augmented datasets to improve the performance of LLM [14,25,64]. Other works aiming to improve the security of LLM-generated code include SVEN [23], SafeCoder [24], CodeGuard+ [21], and PromSec [41]. We elaborate on the details of these methods in Section 3.2. However, all existing code generation studies have either primarily evaluated the functionality of LLMgenerated code or assessed functionality and security separately. Other applications of LLMs include program repair [59,60,67], code analysis [19,40,66], and unit test generation [11,16,63].
LLM for Vulnerability Detection. Another line of research focuses on leveraging LLMs to improve code security. Several studies have utilized LLMs for vulnerability detection [18,32,37,57,68]. Khare et al. [32] examined LLMs' zero-shot capability in detecting vulnerabilities , and their !ndings show that LLMs outperform CodeQL in detecting certain vulnerabilities, such as CWE-22 and CWE-78 [32]. Other works also demonstrated the LLM's promising capability in vulnerability detection [51,54,68]. In addition to leveraging LLMs for vulnerability detection, several studies have also explored their use in vulnerability repair [28,46,69].
Evaluation of LLM-generated Code. Several benchmarks have been proposed for evaluating the functional correctness of LLMgenerated code, including LiveCodeBench [29], BigCodeBench [70], HumanEval [15], MBPP [12], and SWE-Arena [6]. Other benchmarks that consider both functionality and security, such as Sec-CodePLT [62], CodeGuard+ [21], and CWEval [47], are either limited by dataset size [21,47] or lack test cases for nearly half of the samples [62]. Additionally, there are datasets speci!cally designed for code security evaluation, including SecurityEval [53], LLMSe-cEval [56], CyberSecEval [13], and Sallam [52]. However, these datasets do not provide unit tests for functional evaluation. We employ BigCodeBench and SecCodePLT in our study and extend SecCodePLT by augmenting its unit test cases.
Other than using benchmarks for evaluation, several tools and metrics exist for assessing the functionality or security of code. Ren et al. [49] proposed CodeBLEU, which evaluates the accuracy of generated code compared to the ground truth by considering n-gram matches, AST matches, and data-#ow matches. Le et al. [33] prompted LLMs to judge the security and helpfulness of code, while Tong and Zhang [55] examined di"erent prompting techniques for evaluating the semantic correctness of LLM-generated code. Pearce et al. [45], Siddiq and Santos [53] analyzed the safety rate of the generated code using CodeQL. Bhatt et al. [13] proposed an insecure code detector, which leverages a static analyzer and aims to detect insecure coding practices rather than speci!c vulnerabilities. Finally, Wang et al. [58] conducted a study on LLMs' capabilities in secure code generation, vulnerability classi!cation, vulnerability repair, and vulnerability explanation. However, its goal is not to evaluate existing secure code generation techniques.
We conduct a comprehensive study of four secure code generation techniques across two benchmarks. Our study results suggest that future work should employ more than one vulnerability scanner for security evaluation, as di"erent scanners have varying strengths. We also show that existing techniques have limited e"ectiveness in enhancing the security of LLM-generated code when considering functionality simultaneously. These techniques tend to sacri!ce functionality to achieve a higher security score, leading to nonusable generated code. Our study underscores the importance of evaluating both the security and functionality of LLM-generated code simultaneously and provides guidelines for future research.
Security and Functionality Evaluation.