1 Introduction

Large Language Models (LLMs) have been rapidly integrated into products, offering question-answering [30] and code assistance [43] services. While popular in the software domain, increasing efforts [20,27,33] have been made to explore and refine LLM in the hardware domain. Hardware-related benchmarks designed to evaluate LLM have also emerged. However, existing benchmarks [1,16,18,22] have reached a state of saturation in assessing model performance and fail to reflect real-world hardware tasks. These benchmarks do not effectively delineate the performance boundaries of state-of-the-art (SOTA) LLMs.

In the software domain, the SWE-Bench [15] constructed from real-world projects has become the de facto standard for evaluating software engineering performance. Similarly, there is a pressing need to establish a similar benchmark in the hardware domain. Given the rapid advancements in LLMs, existing evaluation benchmarks quickly become obsolete, failing to capture these models' evolving complexity and capabilities. This underscores the urgent need for a forward-looking benchmark to track meaningful progress over the forthcoming years. As a dataset-centric study, HWFixBench sets rigorous and realistic baselines by leveraging real-world hardware repair tasks supported by meticulous data selection.

HWFixBench is derived from high-quality open-source hardware projects. Specifically, we include projects with over 50 citations in academic literature or those that have served as evaluation targets in EDA security competitions, as such projects have rigorous validation. Additionally, we leverage version control data to trace the discovery and resolution of hardware bugs, forming the foundation of HWFixBench. For usability, we modify dependencies in the original designs to ensure they can be simulated as standalone components. The Figure 1 illustrates the three tasks in HWFixBench and their real-world counterparts in version control systems.

(1) Task 1: Bug Detection: The LLM identifies bugs directly from the hardware design, analogous to issues reported by users or test engineers during the hardware design's deployment. (2) Task 2: Bug Fixing: Given a reported issue and the corresponding hardware design, the LLM generates a fix, mirroring the process where a developer submits a pull request to address a specific issue. (3) Task 3: Patch Generation from Design: The LLM generates patches directly from the hardware design without an explicit issue report. While such an approach is uncommon in conventional development workflows, it represents a prevalent application of LLMs in hardware design automation.

To summarize, our key contributions are as follows:

• High-Quality Dataset Construction: We developed a high-quality dataset based on open-source hardware projects with outstanding reputations. Our preprocessing techniques removed design dependencies and enhanced issue descriptions to ensure stability, mitigating the impact of user expertise variability on issue quality. • Comprehensive Model Evaluation: We systematically evaluated various models, including proprietary models, open-source models with general coding capabilities, hardware domain expert fine-tuned models, and prompt-based approaches. Performance was assessed across all three tasks. For fine-tuned models, we measured their performance gains over baseline models. For prompt-based methods, we compared the performance improvement achieved using task-specific prompt formats versus standard evaluation settings within the same model. • Open-Source Benchmark and Testing Framework: We release our benchmark to enable researchers testing their fine-tuned or other accessible models. We also provide a testbench, allowing for more rigorous evaluation endpoints beyond the current version.

Version control documents a project's entire lifecycle, from the initial code creation to deployment, including feature development, bug fixes, and refactoring. These records capture the codebase's evolution and provide contextual information, such as developer discussions, design decisions, and performance optimizations. With the hardware open-source communities' rise, version control has become indispensable in development [10]. Notable open-source hardware groups utilize Git [32] to track modifications throughout the stages of the hardware development, including RTL design, simulation, verification, and synthesis. In this work, version control serves as a raw information source.

• Issue Tracking: Documents design bugs.

• Patch: The patches to correct errors.

Human-processed datasets have become prohibitively expensive with the increasing sizes. Leading AI companies have begun employing PhDs in computer science, mathematics, and other subjects under the title PhD Data Partner to annotate data. However, the limited availability of PhD-level experts constrains the speed, and the high associated costs have sparked interest in fully automated annotation. LLM Judges have been demonstrated as alternatives [5,21,48]. LLMs are considered well-suited for evaluating code semantic correctness. CodeJudge [31], [40], and [31] have explored the feasibility of using LLMs for code evaluation and have found that they can deliver stable performance. This evaluation cannot replace strict functional testing but is the best assessment for current LLM applications in the hardware domain. Currently, LLMs struggle to generate hardware that passes functional verification. However, compared to starting from scratch, having an initial output with flaws is often considered a better start [36].

The assessment of abilities is often conducted in a closed-book manner [21], so HWFixBench exclusively provides the test to LLMs.

The gold standard in the hardware domain remains simulation. To pave the way for fully automated in the future, HWFixBench provides a simulator backend. We expanded the macros, implemented minimal external functions, and developed simulation testbenches.

Our evaluation follows a zero prior knowledge testing approach, meaning that no additional inputs are provided aside from the design (and issue descriptions in Task 2). This test scenario aligns with the current LLM automation trend of turning labor into software.

As discussed in Subsection 2.2, imposing strict functional simulation requirements would result in all LLMs yielding zero performance. This paper does not report detailed metrics for the simulator backend. However, we have open-sourced the benchmark, allowing approaches with non-LLM applications and future powerful LLMs.

We use the pass rate as the performance metric, which measures the success rate of a test given a single attempt. This is a stringent criterion that is not inherently LLM-friendly. LLMs are regarded as experts that produce judgments and conclusions by reading and comparing information in the hardware domain [10].

3.1.1 Preprocessing Pipeline. The Figure 2 black lines represent the data selection process, illustrating the steps to filter data from version control systems for inclusion in HWFixBench. Our benchmark collects all pull requests (PRs) from 12 open-source hardware projects. For each PR, we identify and retrieve the corresponding issue it resolves, selecting only those related to bugs. PRs containing file modifications and aligned patches are filtered, providing the raw dataset. For Bug Detection, the issue reports extracted from version control serve as ground truth. For Bug Fixing, the relationship between issues and PRs provides paired data for testing LLMs in generating fixes given an issue and its associated design. For Patch Generation from Design, PR modifications (patches) serve as reference solutions. Table 1 presents a step-by-step statistical breakdown, detailing the data refinement process-from the total number of collected PRs to the final set of bug-related, issue-resolved file modifications in HWFixBench (excluding projects that were not selected). After filtering, HWFixBench includes nine projects, 500 pull requests, and 1,481 bug-fix-related file modifications. 3.1.2 Post-Processing. We observed that community issues are often vague and do not explicitly describe the bugs, making them unreliable as ground truth. To address this, we manually curated and enhanced the issue descriptions to ensure they clearly articulate the specific bugs being addressed. Additionally, we performed data sanitization and preprocessing on the design files, including de-identifying sensitive information, expanding macros, and minimizing external dependencies referenced within the files. We also give testbenches to facilitate future simulation-based comparisons.

3.1.3 Alpaca Style Dataset Construction. HWFixBench defines a well-defined Alpaca-style instruction-input-output format to ensure interpretability and systematic evaluation of LLM performance.

Task 1: The LLM to identify the corresponding bug report associated with a given hardware design file. The input comprises the full hardware design; the expected output is the relevant issue describing the bug. This task simulates how test engineers or users detect and report hardware issues.

Task 2: The LLM generates a corrective patch based on a reported issue and the corresponding hardware design. The input includes both the bug report and the affected design, while the output is a patch that addresses the issue. This process aligns with real-world debugging workflows, where developers submit fixes via pull requests in version control systems.

Task 3: Unlike the previous task, this task requires the LLM to identify and fix errors autonomously without an explicit bug report. The input consists solely of the hardware design file, and the output is a minimal corrective patch if an issue is detected; otherwise, the model should return "No modification needed." This task evaluates the model's ability to perform self-directed debugging and repair, mimicking an automated hardware verification and correction system.

Table 2 provides two complementary statistical perspectives to comprehensively analyze HWFixBench. Line counts reflect traditional EDA metrics, offering insights into the structural complexity of hardware designs. Token counts, computed using the LLaMA-3 tokenizer, align with the requirements of LLM research by capturing the semantic granularity of the code. Additionally, we present statistical results for the comparative benchmarks discussed in Section 5, enabling a thorough evaluation of HWFixBench in relation to existing benchmarks. The high variability in line/token counts and high skewness underscores the inherent complexity of real-world hardware design changes. HWFixBench contains a large amount, minimal edits, and outlier cases where designs or patches exhibit immense scopes. HWFixBench is a stress hardware domain test for LLM across issue detection, patch generation, and implicit bug resolution.

We designed the prompt based on the assumption in Sub-Section 2.4. The prompt that instructs the evaluator to compare the predicted output with the ground truth. The prompt asks ChatGPT4o whether both results convey the same solution and solve the same issue. If they align, the evaluator returns "MATCHED"; otherwise, "NOT _ MATCHED", accompanied by a short explanation. This structured evaluation ensures consistency in assessing model performance across different tasks.

HWFixBench was evaluated across four categories of LLMs: generalpurpose open-source models, proprietary models, fine-tuned models and prompt methods.

For general-purpose open-source LLMs, we selected a diverse set of models spanning multiple organizations and parameter scales:

• DeepSeek Coder [13] (6.7B, 7B, 33B), R1 Distill Models [7] (Qwen 1.5B, 7B, 14B, 32B, LLaMA 8B, 70B)

• Qwen models [42] (Qwen2.5 14B, 32B, QwQ 32B) For proprietary models, we included:

• OpenAI models (GPT-4o, GPT-4o mini, O1, O1mini, O3mini) For fine-tuned(FT) models, we applied a selection process to ensure compatibility with modern LLM pipelines. Specific models were excluded due to obsolescence (MEIC [41], VeriGen [34], AutoV-Coder [12]) or lack of availability (CraftRTL [19], LLM4SecHW [11], [26]). The final selection included:

• CodeV [46], OriGen [6], RTLCoder [20], and VeriSeek [39] For prompt-based methods, we categorized them into multi-agent systems and normal prompt:

• Multi-Agent Systems: SPEC in [28] were removed. For Ver-ilogCoder [14], we provided additional simulation. • Normal Prompt: We adapted the general prompt format to align with methodologies: PromptV [25], RTLRewriter [44], VerilogReader [23], [4], RTLFixer [38], and [35].

4.2 Overall Performance across models 4.2.1 Task 1. The best-performing model was OpenAI's proprietary GPT-4o, achieving 11.44%. The top-performing open-source model, Microsoft's Phi-4, reached 10.06%, while most other open LLMs achieved between 3% and 8%, a performance level that renders them unusable for real-world applications. Among the finetuned models, OriGen demonstrated a substantial 61.02% relative improvement; however, its absolute performance remained low at 7.71%. This raises questions about whether the observed improvement stems from the weak baseline performance of the underlying DeepSeek Coder series or if fine-tuning genuinely offers meaningful enhancements when applied to powerful models. 4.2.2 Task 2. The strongest performer was o1mini, achieving an impressive accuracy of 83.44%. Among open-source models, DeepSeek R1 Distill Llama 80B obtained the highest score at 43.83%. Notably, models designed for advanced reasoning, often considered superior in general-purpose tasks, did not demonstrate a clear advantage in this project. Model performance positively correlated with scale within the same generation and institution. Only OriGen demonstrated a performance gain among fine-tuned models, while the remaining three models experienced degradation.

This result aligns with the expectation that LLMs can assist in code repair; however, it also highlights a significant gap between proprietary and open-source models. The disparity suggests that OpenAI's closed-source models may have been explicitly trained with hardware-related data, whereas most open-source models lack sufficient exposure to this domain.

Despite being the most challenging task, the models' performances exceeded results from Task 1. QwQ 32B Preview achieved the highest score among all models with 29.95%. Only VeriSeek among the fine-tuned models showed a performance gain. However, it remains unclear whether the relative improvement is genuinely due to the fine-tuning process or reflects the weak baseline performance of the DeepSeek Coder 7B model (12.74% vs. 15.50% from the 6.7B model). The fine-tuned model's score of 13.23% also lags behind the highest-performing fine-tuned model, RTLCoder (14.20%).

A common question in evaluating LLMs is whether their performance reflects genuine understanding or mere memorization of internet-crawled data. Since time progresses linearly and unidirectionally, bug fixes made public after an LLM's knowledge cutoff date could not have been present in its training data. Using each model's reported knowledge cutoff date as a reference, we partition the benchmark into pre-cutoff and post-cutoff subsets. Then, we analyze the performance gap between them to assess whether an LLM truly understands the task as shown in Table 4.

While this approach appears ideal theoretically, hardware data inside LLMs is often actively removed. A notable example is removing hardware code in StarCoder [17] between iterations. This omission creates a fundamental limitation: only a post-cutoff performance drop can reliably indicate that the model lacks proper understanding of hardware tasks. Conversely, an unexpected performance increase cannot be considered evidence of experience, as the model's training data may have contained little hardware knowledge, rendering the experiment's assumptions invalid.

. Only the fine-tuned OriGen model exhibited performance degradation after the knowledge cutoff. Given the overall low baseline for this task, we can infer that LLMs generally perform poorly in directly identifying bugs in hardware designs. Still, their performance does not strongly depend on whether such data was present in the training set. 4.3.2 Task 2. DeepSeek's Coder series maintained stable performance, while the Distill series degraded significantly. Since the latter models did not undergo post-training and only relied on supervised fine-tuning, this suggests a stronger dependence on the training dataset. Similarly, Google's Gemma 2, most Meta AI models, and Microsoft's models exhibited similar post-cutoff performance drops. However, Qwen models improved across iterations-while the Qwen 2.5 series still suffered post-cutoff degradation, the nextgeneration models reversed this trend. For OpenAI models, GPT-4o and GPT-4o mini did not perform poorly, while the "thinking" models exhibited a decline.

Among FT models, those designed for hardware code generation (CodeV and RTLCoder) did not show a drop, whereas debug-specific models suffered a significant decrease. This indicates that the latter depends on explicitly presenting test cases in the training data.

The results were similar to Task 2, except that "thinking" models did not exhibit negative optimization this time. However, fine-tuned models experienced an overall performance decline, suggesting that models trained explicitly for hardware debugging struggle to generalize beyond their pre-existing knowledge.

The ability to extrapolate LLM performance from small-scale to large-scale designs has long been an open question, and this challenge is particularly pronounced for LLMs if they were trained on short designs. Based on two typical hardware LLM datasets [20,45], we identified a typical maximum token length of 2, 567 tokens. To simulate real-world scenarios where additional context is appended, we extended this length by a factor of 1.5×, partitioning the benchmark into a short subset (≤ 3, 852 tokens) and a long subset (> 3, 852 tokens). This evaluation assessed whether LLMs could generalize beyond their maximum training lengths while maintaining robust performance on extended hardware design inputs.

Most models exhibited performance degradation on all tasks when presented with sequences exceeding the estimated training threshold, highlighting a general limitation in handling long hardware tasks. The degradation was particularly pronounced in opensource and fine-tuned models. Only OpenAI's proprietary models demonstrated no degradation on Task 2, suggesting superior long-sequence processing capabilities or a more practical approach in hardware scenarios. presented in Table 6, should be interpreted with caution, as this experiment is not directly comparable to previous evaluations.

A key finding is that the assumption made by prompt method studies, that their effectiveness is independent of the underlying LLM, does not hold. Instead, each prompt method must be strictly tailored to specific LLMs, making them techniques for optimizing particular models rather than universal solutions. While prompting can enhance LLMs that are otherwise poorly suited for hardware tasks by reformatting input information, it cannot substitute for model advancements. For example, Gemma2 27B achieved a twofold performance increase but remained the weakest model. In contrast, QwQ 32B and DeepSeek R1 Distill Qwen 32B maintained relatively strong performance despite not being optimized for these prompt methods, demonstrating excellent model stability. Qwen2.5 32B showed a notable positive gain, making it a promising base model for prompt-based techniques. However, prompt methods alone cannot bridge model size and fundamental performance gaps.

Combining fine-tuned models with prompt methods did not lead to cumulative performance gains. Notably, OriGen, the bestperforming fine-tuned model, suffered a severe performance drop when used in conjunction with prompt methods. VeriSeek exhibited similar degradation. Meanwhile, RTLCoder and CodeV, two hardware design-specific models, demonstrated strong compatibility with prompt-based approaches.

Unlike SWEBench [15], which focuses primarily on Python code, our work extends coverage to SystemVerilog implementations of typical chip functional modules. In practical hardware development, testing and debugging are typically handled by two separate roles: test engineers and develop engineers. Our benchmark reflects this division by structuring tasks into two categories: Task 1 for testing and Task 2 for repair. While mainstream LLM-based approaches often attempt to consolidate these roles or directly generate solutions in a single step, we evaluate this strategy through Task 3, testing the LLM's ability to handle both scenarios simultaneously. We show the SOTA performance of existing benchmarks VerilogEval [18] is a pedagogical tool for digital logic design courses rather than an accurate measure of hardware design capability. Its most complex benchmarks involve simple finite-state machine applications, while its most straightforward cases cover empty designs or constant-zero outputs. Similarly, CreativeEval [8] and CorrectBench [28] are built on the same HDLBits foundation, focusing on basic Verilog-level exercises for educational purposes. This fundamentally differs from HWFixBench, which covers realworld chip functional module designs with practical applications and significantly higher complexity.

While RTLLM [22] shares similarities with our work by including simple module designs, HWFixBench is unique in drawing modules directly from real-world hardware repair workflows, making its tasks more representative of practical engineering challenges. FVEval [16], on the other hand, focuses on specific edge cases in FIFO and Pipeline scenarios, assessing LLM-generated SystemVerilog assertions using JasperGold. While these tasks are not inherently challenging, their low tolerance for error makes them unsuitable as general LLM benchmarks. In contrast, HWFixBench targets more challenging hardware designs but prioritizes providing a better starting point for users rather than enforcing strict simulation pass criteria. With 500 repair instances and more than 1,000 modifications, our benchmark substantially exceeds the combined scale of existing evaluation metrics. Moreover, the designs included in our dataset are of significantly higher practical value, with more intricate testing and repair tasks, making HWFixBench a suitable metric for assessing LLM capabilities in EDA tasks.

HWFixBench is a benchmark designed to evaluate the emerging trend of LLM-driven EDA automation. This work introduces a highvalue benchmark to assess existing and future LLM applications in hardware design and verification. Through comprehensive evaluations of general-purpose LLMs, fine-tuned models, and promptbased methods, we provide a clear performance landscape and identify challenges in adapting LLMs to hardware-related tasks.

Our findings highlight the significant gap between proprietary and open-source models, the task-dependent effectiveness of finetuning, and the model-specific nature of prompt methods. They emphasize that LLM-based automation in EDA requires tailored approaches rather than one-size-fits-all solutions. The benchmark is a foundation for future research, guiding the development of more capable LLMs for hardware applications and providing a standardized evaluation framework for measuring progress in this rapidly evolving domain.