Conference PaperBenchmarking Large Language Models for Automated Verilog RTL Code GenerationThakur, Shailja; Ahmad, Baleegh; Fan, Zhenxing; Pearce, Hammond; Tan, Benjamin; Karri, Ramesh; Dolan-Gavitt, Brendan; Garg, SiddharthAutomating hardware design could obviate a signif-icant amount of human error from the engineering process and lead to fewer errors. Verilog is a popular hardware description language to model and design digital systems, thus generating Verilog code is a critical first step. Emerging large language models (LLMs) are able to write high-quality code in other programming languages. In this paper, we characterize the ability of LLMs to generate useful Verilog. For this, we fine-tune pre-trained LLMs on Verilog datasets collected from GitHub and Verilog textbooks. We construct an evaluation framework comprising test-benches for functional analysis and a flow to test the syntax of Verilog code generated in response to problems of varying difficulty. Our findings show that across our problem scenarios, the fine-tuning results in LLMs more capable of producing syntactically correct code (25.9% overall). Further, when analyzing functional correctness, a fine-tuned open-source CodeGen LLM can outperform the state-of-the-art commercial Codex LLM (6.5% overall). We release our training/evaluation scripts and LLM checkpoints as open source contributions.2023-04-01104197052023 Design, Automation & Test in Europe Conference & Exhibition (DATE)1 to 6https://doi.org/10.23919/DATE56975.2023.101370862039607National Science Foundation