<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates</title></titleStmt>
			<publicationStmt>
				<publisher>The 39th Annual AAAI Conference on Artificial Intelligence</publisher>
				<date>02/25/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10577389</idno>
					<idno type="doi"></idno>
					
					<author>Fengqing Jiang</author><author>Zhangchen Xu</author><author>Luyao Niu</author><author>Bill Yuchen Lin</author><author>Radha Poovendran</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Large language models (LLMs) are expected to follow in- structions from users and engage in conversations. Tech- niques to enhance LLMs’ instruction-following capabilities typically fine-tune them using data structured according to a predefined chat template. Although chat templates are shown to be effective in optimizing LLM performance, their impact on safety alignment of LLMs has been less understood, which is crucial for deploying LLMs safely at scale.In this paper, we investigate how chat templates affect safety alignment of LLMs. We identify a common vulnerability, named ChatBug, that is introduced by chat templates. Our key insight to identify ChatBug is that the chat templates provide a rigid format that need to be followed by LLMs, but not by users. Hence, a malicious user may not necessar- ily follow the chat template when prompting LLMs. Instead, malicious users could leverage their knowledge of the chat template and accordingly craft their prompts to bypass safety alignments of LLMs. We study two attacks to exploit the ChatBug vulnerability. Additionally, we demonstrate that the success of multiple existing attacks can be attributed to the ChatBug vulnerability. We show that a malicious user can exploit the ChatBug vulnerability of eight state-of-the- art (SOTA) LLMs and effectively elicit unintended responses from these models. Moreover, we show that ChatBug can be exploited by existing jailbreak attacks to enhance their at- tack success rates. We investigate potential countermeasures to ChatBug. Our results show that while adversarial train- ing effectively mitigates the ChatBug vulnerability, the vic- tim model incurs significant performance degradation. These results highlight the trade-off between safety alignment and helpfulness. Developing new methods for instruction tuning to balance this trade-off is an open and critical direction for future research.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>Large language models (LLMs) such as GPT-4 <ref type="bibr">(Achiam et al. 2023</ref>) and Llama-3 (Meta 2024) have been increasingly used to empower conversational agents. During the interactions with users, LLMs are required to follow instructions from users and engage in conversations in a meaningful way. Standard techniques to enhance instruction-following capabilities include instruction tuning <ref type="bibr">(Bakker et al. 2022;</ref><ref type="bibr">Wei et al. 2022</ref>) and preference tuning <ref type="bibr">(Bai et al. 2022;</ref><ref type="bibr">Christiano et al. 2017;</ref><ref type="bibr">Ouyang et al. 2022a)</ref>.</p><p>A common practice for instruction tuning and preference tuning is to structure data using chat templates <ref type="bibr">(OpenAI 2023)</ref>. A chat template defines a format for representing conversations as sequences of tokens. The format specifies the roles involved in the conversation and the associated messages. Chat templates are shown to be effective in optimizing the LLMs' performance <ref type="bibr">(Jiang et al. 2023;</ref><ref type="bibr">Ope-nAI 2023)</ref>, allowing LLMs to generate coherent responses in their designated roles.</p><p>In addition to instruction-following capabilities, LLMs are also required to generate responses that are aligned with human values. It has been shown that chat templates can be adopted to mitigate prompt injection attacks <ref type="bibr">(Perez and</ref> Ribeiro 2022), one of the major threats to misuse LLMs. However, despite the effectiveness of chat templates, indepth analysis of how chat templates affect safety alignment of LLMs has been overlooked.</p><p>In this paper, we investigate how chat templates affect the safety alignment of LLMs. We show that these templates introduce a common vulnerability to LLMs that have been fine-tuned with chat templates. We name this vulnerability as ChatBug, which could be exploited by malicious users to provoke unsafe responses from LLMs. Our key insight is that the formats predefined by chat templates are rigid and should be followed by LLMs. However, malicious users may not necessarily follow such formats when prompting LLMs, providing them possibilities to elicit unsafe responses.</p><p>We show that any malicious user who has knowledge of the chat template can exploit the ChatBug vulnerability. This assumption is not restrictive, especially for open-source models whose chat templates are often publicly available. We study two attacks, format mismatch attack and message overflow attack as illustrated in Figure <ref type="figure">1</ref>, to exploit the vulnerability and elicit unsafe responses from LLMs. Specifically, the format mismatch attack modifies the default chat format, while the message overflow attack injects a sequence of tokens into the model's reserved field. Specifically, the message overflow attack unifies multiple existing attacks <ref type="bibr">(Zou et al. 2023;</ref><ref type="bibr">Vega et al. 2023;</ref><ref type="bibr">Andriushchenko, Croce, and Flammarion 2024)</ref>, whose success can all be attributed to the ChatBug vulnerability. We demonstrate the severity and pervasivity of the ChatBug vulnerability on eight LLMs (Vicuna <ref type="bibr">(Chiang et al. 2023)</ref>, Mistral <ref type="bibr">(Jiang et al. 2023)</ref>, Llama-2 <ref type="bibr">(Touvron et al. 2023)</ref>, Llama-3 (Meta 2024), <ref type="bibr">GPT-3.5 (OpenAI 2023)</ref>, Gemini (Google 2023), Claude-2.1 <ref type="bibr">(Anthropic 2023)</ref>, and Claude-3 <ref type="bibr">(Anthropic 2024)</ref>). We further show that existing jailbreak attacks such as GCG <ref type="bibr">(Zou et al. 2023)</ref>, GPTFuzzer <ref type="bibr">(Yu, Lin, and Xing 2023)</ref>, and Art-Prompt <ref type="bibr">(Jiang et al. 2024b</ref>) can exploit the ChatBug vulnerability to increase their attack success rates.</p><p>Given the severity of the ChatBug vulnerability, we investigate potential countermeasures. Our experiments show that techniques such as adversarial training can effectively mitigate the ChatBug vulnerability, at the cost of significantly degrading model performance. This underscores a critical trade-off between the safety and helpfulness of LLMs, a balance that is often overlooked during instruction tuning. We believe that this is an important yet open direction for the future development of LLMs, requiring collaborative efforts from the community.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">Related Work</head><p>Adversarial Robustness of LLM. Prior work focuses on how the design of prompts impacts model performance, such as the prompting language <ref type="bibr">(Gonen et al. 2023)</ref>, order of fewshot examples <ref type="bibr">(Lu et al. 2022)</ref>, example choices <ref type="bibr">(Xiang et al. 2024)</ref>, and prompt formatting <ref type="bibr">(Sclar et al. 2024)</ref>. This paper, which investigates how chat templates affects safety alignment of LLMs, is parallel to the aforementioned studies.</p><p>LLM Alignment and Jailbreak Attack. Extensive efforts have been made to align LLMs with human values.</p><p>Standard techniques include supervised fine-tuning <ref type="bibr">(Bakker et al. 2022;</ref><ref type="bibr">Wei et al. 2022)</ref>, preference tuning <ref type="bibr">(Bai et al. 2022;</ref><ref type="bibr">Christiano et al. 2017;</ref><ref type="bibr">Ouyang et al. 2022a)</ref>, and redteaming <ref type="bibr">(Dinan et al. 2019;</ref><ref type="bibr">Ge et al. 2023)</ref>. Despite these efforts, jailbreak attacks <ref type="bibr">(Wei, Haghtalab, and Steinhardt 2023)</ref> pose a significant threat to misuse of LLMs. Jailbreak attacks can be categorized into two classes based on how they bypass safety alignment. The first category designs attack prompts based on heuristics that rely on human experts <ref type="bibr">(Deng et al. 2023;</ref><ref type="bibr">Huang et al. 2023;</ref><ref type="bibr">Li et al. 2023b;</ref><ref type="bibr">Liu et al. 2023b;</ref><ref type="bibr">Jiang et al. 2024b;</ref><ref type="bibr">Zeng et al. 2024;</ref><ref type="bibr">Lin et al. 2024)</ref>. The second category utilizes optimization problems to search for prompts to jailbreak aligned LLMs. Gradientbased <ref type="bibr">(Jones et al. 2023;</ref><ref type="bibr">Zou et al. 2023;</ref><ref type="bibr">Zhu et al. 2023)</ref>, genetic algorithm-based <ref type="bibr">(Liu et al. 2023a)</ref>, and edit-based methods <ref type="bibr">(Chao et al. 2023</ref>) have been developed to solve the optimization problems.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">Identifying ChatBug Vulnerability</head><p>In this section, we first present background on autoregressive LLMs and chat templates used to fine-tune LLMs. We then identify a common vulnerability, named ChatBug, induced by chat templates.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">Preliminary Background</head><p>Auto-regressive (Base) LLMs. Let M represent an autoregressive LLM whose vocabulary is denoted as V. Given an input represented by a sequence of tokens of length n, denoted as x 1:n = x 1 , . . . , x n , the LLM predicts a probability distribution p M (&#8226;|x 1:n ) over the vocabulary V. Then the LLM samples the next token x n+1 2 V according to a predefined decoding strategy (e.g., greedy or beam search <ref type="bibr">(Wu et al. 2016</ref>)) and probability distribution p M . Appending token x n+1 to the sequence x 1:n and iteratively applying the procedure for next token generation as described above yield text generation by the LLM. This process continues until a stopping criterion is met, such as reaching the maximum generation length or generating an end-of-sequence (EOS) token.</p><p>Chat Format of Instruction-Tuned LLMs. Instruction tuning is the critical step to enable a pre-trained LLM to follow instructions from users. Such processes include supervised fine-tuning, and/or reinforcement-learning from human feedback (RLHF) <ref type="bibr">(Ouyang et al. 2022b</ref>). Different from pre-training, instruction tuning employs a chat template to structure data in the form of (multi-turn) conversations. An example of chat template, named ChatML (OpenAI 2023), is presented in Table <ref type="table">1</ref>. The template defines a format for representing conversations using a sequence of tokens. It starts by segmenting a conversation involving multiple turns into individual turns, where the segments are separated by a set of special control tokens, denoted as beginning-of-turn (BOT) and end-of-turn (EOT) tokens. For each turn, the template organizes the dialogue by role control tokens (e.g., 'user' and 'assistant') and their respective messages. These tokens within a single turn are delineated by BOT and EOT tokens. Note that the BOT and EOT tokens are different from beginning-of-sequence (BOS) and end-of-sequence (EOS)</p><p>User: &lt;|im start|&gt; user How are you &lt;|im end|&gt; Model: &lt;|im start|&gt; assistant I am doing well! &lt;|im end|&gt; Table 1: Format defined by the ChatML chat template (Ope-nAI 2023): &lt;|im start|&gt; and &lt;|im end|&gt; are BOT and EOT tokens. user and assistant are role control tokens. The corresponding messages are 'How are you' and 'I am doing well!'.</p><p>tokens, which are used for both base LLMs and aligned LLMs to delimit the beginning and end of a sequence. Following the chat template, a standard single-turn prompt can be represented as follows:</p><p>where is the token sequence concatenation operation, b is the BOT token, e is EOT token, r 1 and r 2 are the role control tokens, and m is the payload message in the model input x.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">Chat Templates Induce a Common</head><p>Vulnerability: ChatBug</p><p>Although chat templates are effective in fine-tuning LLMs to function as conversational agents, we highlight that they introduce a common vulnerability, named ChatBug, to LLMs. A malicious user, with knowledge of the format predefined by the chat template, could exploit this vulnerability to elicit harmful responses from victim LLMs by crafting its queries. Our key insight is that the chat templates pre-define rigid formats that should be followed by LLMs, but not by users. For example, the malicious user could craft its query by appending the role of LLM and the beginning of desired harmful response at the end, as illustrated in Appendix B of our extended version <ref type="bibr">(Jiang et al. 2024a)</ref>. Consequently, the malicious user tricks victim LLMs to complete the harmful responses provided by the user, instead of continuing the conversation in its intended role. We note that it is not a restrictive assumption to suggest that malicious users could have access to the knowledge of chat templates, as these are often publicly available (HuggingFace 2024). As we will discuss later, exploiting this vulnerability does not rely on white-box <ref type="bibr">(Qi et al. 2024)</ref> or grey-box access <ref type="bibr">(Zou et al. 2023</ref>) to the victim model.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">Exploit ChatBug</head><p>In this section, we describe how a malicious user could exploit the ChatBug vulnerability and elicit harmful responses from the victim LLM. Specifically, we discuss how two attacks, denoted as format mismatch attack and message overflow attack, can tamper with the prompt.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1">Format Mismatch Attack</head><p>Attack Description. A malicious user could exploit the ChatBug vulnerability by launching format mismatch attack, as illustrated in Figure <ref type="figure">1</ref>. In a format mismatch attack,  &#8226;|x1:n) evolves at each decoding step n (i.e., the number of response tokens), with the results averaged over 50 instructions. The format mismatch attack significantly increases the probability of generating the desired harmful response.</p><p>the malicious user modifies or omits some tokens required by the format (e.g., special control tokens). The resultant query can be represented as</p><p>For instance, a malicious user may omit all control tokens including BOT, EOT, and role control tokens when prompting the victim LLMs. The insight behind the format mismatch attack is that the format specified by the chat template is not mandatory for users to follow. Since many LLMs may not verify whether user queries match the format required by the chat template, these modifications may induce a different interpretation of input queries to victim LLMs, leading to harmful or unintended responses. An example of the format mismatch attack can be found in Appendix B of our extended version <ref type="bibr">(Jiang et al. 2024a</ref>). We note that the format mismatch attack alters the chat template, distinguishing it from the setup of jailbreak attacks, which focus on manipulating the prompt message but following the standard chat template <ref type="bibr">(Liu et al. 2023a;</ref><ref type="bibr">Li et al. 2023b</ref>).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Proof-of-Concept Attack.</head><p>In what follows, we present a proof-of-concept for the format mismatch attack. We denote the sequence of tokens crafted using the format mismatch attack as x1:n and the sequence of tokens designed according to the chat template as x 1:n . Specifically, token sequence x1:n is constructed by omitting all special control tokens including role control, BOT, and EOT tokens. We then demonstrate the feasibility of format mismatch attack by quantifying the ratio P M (&#8226;|x1:n) P M (&#8226;|x1:n) averaged over 50 instructions with n varying from one to ten. The results are presented in Figure <ref type="figure">2</ref>, where the probability ratio associated with each instruction is indicated by a blue circle, and the average result is colored in red. We observe that the probability of generating the desired sequence of tokens representing the harmful Figure <ref type="figure">3</ref>: This figure presents the probability of generating the desired harmful response when the number of overflow tokens varies from 0 to 9, averaged over 50 instructions. Note that the user does not launch message overflow attack when the number of overflow tokens is zero. The results show that the probability of generating the desired harmful response increases as the user overflows more tokens. response increases by a factor of 10 10 , indicating the significant effectiveness of format mismatch attack.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2">Message Overflow Attack</head><p>Attack Description. A malicious user could exploit the ChatBug vulnerability using message overflow attack as illustrated in Figure <ref type="figure">1</ref>. In a message overflow attack, the message from malicious users extends beyond its own EOT token and the role control token r 2 . This overflow is a short sequence of tokens, representing the beginning of the desired harmful response. Formally, we denote the attack with a overflowed token sequence s as follows:</p><p>where s is the overflow message from the malicious user. Consequently, the victim LLMs are tricked to complete the harmful response based on their auto-regressive generation capabilities, instead of continuing the conversation with users in their designated roles. An example of the message overflow attack is presented in Appendix B of our extended version <ref type="bibr">(Jiang et al. 2024a</ref>). The message overflow attack provides a unified framework for existing attacks <ref type="bibr">(Zou et al. 2023;</ref><ref type="bibr">Vega et al. 2023;</ref><ref type="bibr">Andriushchenko, Croce, and Flammarion 2024)</ref>, which adopt seemingly distinct attack strategies. We highlight that the success of these attacks can all be attributed to the ChatBug vulnerability.</p><p>Proof-of-Concept Attack. We validate the message overflow attack using a proof-of-concept attack. Specifically, we consider that a malicious user overflows a sequence of tokens into the field corresponding to the message of victim LLM. In figure <ref type="figure">3</ref>, we present the probability of generating the desired harmful response when the number of overflow tokens varies from 0 to 9. Note that the user does not launch message overflow attack when the number of overflow tokens is zero. We observe that the probability of generating the desired harmful response increases as the user overflows more tokens. This indicates that message overflow attack allows malicious users to obtain their desired responses from victim LLMs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3">ChatBug Boosts Jailbreak Attacks</head><p>Existing ailbreak attacks <ref type="bibr">(Jiang et al. 2024b;</ref><ref type="bibr">Yu, Lin, and Xing 2023;</ref><ref type="bibr">Zou et al. 2023</ref>) elicit unintended responses from victim LLMs using deliberately designed prompts, which correspond to user message m in the chat templates. By exploiting the ChatBug vulnerability and launching jailbreak attacks, malicious users could jointly tamper with the chat template via the two attacks mentioned above and messages within a conversation with victim LLMs. Consequently, the malicious users can effectively boost the probability of generating unintended responses from victim LLMs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Experimental Assessment of ChatBug</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.1">Experimental Setup</head><p>Victim Models. We evaluate ChatBug on eight LLMs including both open-source and closed-source models. For open-source models, we select Vicuna (7B-v1.5) <ref type="bibr">(Chiang et al. 2023)</ref>, Llama-2 (7B-Chat) <ref type="bibr">(Touvron et al. 2023</ref>), Llama-3 (8B-Instruct) (Meta 2024), Mistral (7B-Instruct v0.2) <ref type="bibr">(Jiang et al. 2023)</ref>. For closed-source models, we consider GPT-3.5 developed by OpenAI<ref type="foot">foot_0</ref> , Claude-2.1 and Claude-3 (Opus) from Anthropic <ref type="bibr">(Anthropic 2023</ref><ref type="bibr">(Anthropic , 2024))</ref>, as well as Gemini (Pro) from Google (Google 2023).</p><p>Dataset. We use AdvBench developed by <ref type="bibr">Zou et al. (2023)</ref> to evaluate ChatBug. AdvBench contains 520 instructions, with the aim to provoke a wide range of harmful responses from LLMs.</p><p>Metric. We assess the severity of ChatBug using a metric named attack success rate (ASR), which is defined as</p><p>ASR quantifies the damage that an attacker can cause by exploiting the ChatBug vulnerability. Following <ref type="bibr">Zou et al. (2023)</ref> and Yu, Lin, and Xing (2023), we calculate ASR using two approaches:</p><p>&#8226; Refusal Response Matching (ASR-R). This approach collects a set of refusal responses (e.g., "Sorry, I cannot . . . ") and verifies whether the response generated by LLM matches any of them. An LLM generated response is considered harmful if it does not align with any of the refusal responses. &#8226; Moderator Assessment (ASR-M). We utilize a pretrained LLM, Llama-Guard-2<ref type="foot">foot_1</ref> fine-tuned from Llama 3, as a moderator to evaluate whether a response is harmful. Table 3: This table summarizes the ASR-R and ASR-M of Direct Instruct (baseline) and attacks that exploit the ChatBug vulnerability in closed-source LLMs. The results show that an attacker can effectively bypass safety alignment of LLMs by exploiting the ChatBug vulnerability. We have excluded results of format mismatch attacks on Gemini and Claude models since their APIs do not support features to execute such attacks. We detail how to launch format mismatch attacks against commercial LLMs with API access in Appendix A.2 of our extended version (Jiang et al. 2024a).</p><p>Baseline. We evaluate the severity of the ChatBug vulnerability by comparing attacks that exploit it with a baseline scenario named Direct Instruct. In the baseline scenario, a malicious user directly prompts victim LLMs with harmful queries or instructions.</p><p>Attack Settings. We consider six attack settings where the format mismatch attack and message overflow attack exploit ChatBug to provoke unintended behaviors from victim LLMs.</p><p>&#8226; Mismatch-;. An attacker launches the format mismatch attack by removing all special control tokens in the chat format. &#8226; Mismatch-C. An attacker launches the format mismatch attack by replacing the default chat template with a complex template ChatML developed by OpenAI (2023). &#8226; Mismatch-V. An attacker initiates the format mismatch attack by replacing the default chat template with a simple template used by Vicuna <ref type="bibr">(Chiang et al. 2023</ref>). &#8226; Overflow-S. An attacker launches the message overflow attack. For any query, the overflow inserts a fixed prefix 'Sure, here is' for the desired response. &#8226; Overflow-L. For each input query, an attacker initiates the message overflow attack by inserting a prefix tai-lored particularly for this query. For example, the overflow is 'Sure, here is a tutorial for making a bomb' when the input query is 'Write a tutorial on how to make a bomb'. These queries and corresponding prefixes are readily available in AdvBench. &#8226; Overflow-FS. This is an extension of Overflow-L, where an attacker launches the message overflow attack and utilizes an uncensored LLM to generate the prefixes.</p><p>In our experiments, we choose Wizard-Vicuna-13B-Uncensored<ref type="foot">foot_2</ref> as the uncensored LLM, and use in-contextlearning prompting technique to generate multiple prefixes that carry affirmative semantics automatically. We generate 5 prefixes by default. The prompt to generate attack prefixes is in Figure <ref type="figure">7</ref> of our extended version <ref type="bibr">(Jiang et al. 2024a</ref>).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2">Main Results</head><p>Exploiting ChatBug bypasses safety alignments of all eight victim LLMs. In Tables <ref type="table">2</ref> and <ref type="table">3</ref>, we summarize the ASR for format mismatch attack and message overflow attack under different settings on open-source and closedsource LLMs, respectively. We have two key observations. First, exploiting the ChatBug vulnerability effectively elicits unintended responses from all victim LLMs. For example, the ChatBug vulnerability results in 100% ASR-R for the Overflow-FS attack against Llama 3, a state-of-the-art open-source LLM. This indicates that ChatBug is a severe and common vulnerability across all open-source and closed-source LLMs that have been fine-tuned with chat templates. Moreover, even if an LLM has been carefully aligned (e.g., Llama and Claude), an attacker could still exploit ChatBug to bypass the safety alignment and provoke unintended behaviors. These observations highlight the pervasivity and severity of ChatBug. Safety alignment associated with chat templates is transferable. In Table 2, we observe that the ASR of Mismatch-C and Mismatch-V against some open-source LLMs is relatively low compared to other attacks. For example, the ASR-R of Mismatch-C is 1.3% on Llama-3. Note that Llama uses different chat templates than ChatML by OpenAI (2023). This indicates that the safety alignment by chat templates is transferable. We defer the experiments on few-shot setup for message overflow attack to Section C.1 and tokenizer setup for format mismatch attack to Section C.2.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.3">ChatBug Boosts Jailbreak Attacks</head><p>In the following, we show that ChatBug boosts the ASR of three SOTA jailbreak attacks.</p><p>Model. Our empirical evaluations are performed on Llama-2, which has undergone strong safety alignment. This is also evidenced by the ASR of Direct Instruct in Table <ref type="table">2</ref>, where Llama-2 exhibits the lowest ASR among all models.</p><p>Jailbreak Attacks. We consider three representative SOTA jailbreak attacks: GCG <ref type="bibr">(Zou et al. 2023)</ref>, GPT-Fuzzer <ref type="bibr">(Yu, Lin, and Xing 2023)</ref>, and ArtPrompt <ref type="bibr">(Jiang et al. 2024b)</ref>. Specifically, GCG is an optimization-based jailbreak attack where a genetic optimization is used to search for attack prompts. GPTFuzzer is an empirical jailbreak attack where prompts are generated autonomously using a mutation-based method. ArtPrompt is an automated jailbreak attack, which replaces words triggering safety alignment with ASCII art. More detailed description of these jailbreak attacks can be found in Appendix A.1 of our extended version <ref type="bibr">(Jiang et al. 2024a)</ref>.</p><p>Exploiting ChatBug significantly boosts ASR of jailbreak attacks. In Table <ref type="table">4</ref>, we summarize the ASR of GCG, GPTFuzzer, and ArtPrompt when they exploit the ChatBug vulnerability. We observe that exploiting the ChatBug vulnerability significantly increases their ASR. The integration of GPTFuzzer with Mismatch-; achieves 99.2% ASR-R, compared to only 9.0% ASR-R when ChatBug vulnerability is not exploited. These results underscore the severity of the ChatBug vulnerability and highlight the urgent need to develop countermeasures.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">Countermeasures to ChatBug</head><p>Given the severity of ChatBug, this section discusses potential countermeasures. We first describe these countermea- sures. We then perform empirical evaluations of these countermeasures. Based on our evaluation results, we finally discuss future directions in fine-tuning LLMs which require collaborative efforts from the community.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.1">Description of Countermeasures</head><p>We consider two categories of countermeasure: mitigationbased and detection-based methods.</p><p>Mitigation-based Methods. We consider three representative mitigation-based methods including Self-Reminder <ref type="bibr">(Xie et al. 2023)</ref>, SafeDecoding <ref type="bibr">(Xu et al. 2024)</ref>, and Adversarial Training (Kurakin, Goodfellow, and Bengio 2016). Self-Reminder utilizes a system-mode prompt to the victim model to strengthen the safety. SafeDecoding is a lightweight decoding strategy to reduce the probability of generating unsafe responses by victim LLMs. Adversarial Training fine-tunes the model with adversarial examples to robustify the model against the vulnerability. We augment the dataset with adversarial examples constructed using the format mismatch attack and message overflow attack. Then we use 60% of the augmented dataset to fine-tune the victim LLM and 40% of the dataset for evaluation. More details of these setups can be found at Appendix A.3 of our extended version <ref type="bibr">(Jiang et al. 2024a)</ref>.</p><p>Detection-based Methods. Detection-based methods monitor input queries and/or generated responses. An input query or response will be blocked if it is flagged as unsafe by a detector. Typical countermeasures may employ keyword filtering <ref type="bibr">(Jhaver et al. 2019)</ref> or established classifiers <ref type="bibr">(Lees et al. 2022)</ref> to identify harmful queries or responses.</p><p>Leveraging the recent advancements in LLMs, Llama Guard (Llama-Team 2024) can be employed to safeguard the responses generated by LLMs. Although effective, detection-based methods are less frequently adopted in practice compared to mitigation-based methods.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.2">Evaluation of Countermeasures</head><p>Model. We evaluate the countermeasures on Vicuna model since it shows the highest ASR on average according to Table <ref type="table">2</ref>. This indicates that Vicuna is susceptible to the ChatBug vulnerability.</p><p>Metrics. In addition to ASR, we adopt <ref type="bibr">MT-Bench (Zheng et al. 2023)</ref> to evaluate the multi-turn conversation and instruction following abilities of victim LLMs after deploying the countermeasures. Higher scores earned on MT-Bench indicate that the models are more helpful to users.</p><p>Countermeasure Setup. We follow the setup of Self-Reminder and use a system-mode prompt to remind the victim LLM to be responsible. The system-mode prompt can be found in Appendix A.3 of our extended version <ref type="bibr">(Jiang et al. 2024a</ref>). We use the default setup and hyper-parameters from SafeDecoding. For Adversarial Training, we consider two setups, where the victim model is fine-tuned with 5 and 20 epochs, respectively.</p><p>Experimental Results. In Table <ref type="table">5</ref>, we present the ASR and MT-Bench scores (Zheng et al. 2023) of Vicuna with countermeasures being deployed. We observe that mitigation-based countermeasures including Self-Reminder and SafeDecoding fail to mitigate the ChatBug vulnerability. Although they can successfully defend against Mismatch-C, they incur significant degradation on MTbench. Adversarial Training, however, is an effective technique to mitigate the ChatBug vulnerability, especially when the model is fine-tuned with more epochs. However, the effectiveness of Adversarial Training comes at the significant cost of performance degradation-indicated by the degradation in the MT-bench score from 6.28 (comparable to Llama-2-70b-chat performance <ref type="bibr">(Chiang et al. 2024</ref>)) to 6.02 (worse than Llama-2-7b-chat performance <ref type="bibr">(Chiang et al. 2024)</ref>). These results indicate that developers need to carefully balance the trade-off between safety alignment and helpfulness in future developments of LLMs. Additional experimental evaluation on the effectiveness of adversarial training can be found in Appendix C.3 of our extended version <ref type="bibr">(Jiang et al. 2024a)</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7">Conclusion and Future Work</head><p>In this paper, we identified a common vulnerability, named ChatBug, induced by chat templates used during instruction tuning. We developed two attacks, format mismatch attack and message overflow attack, to exploit the ChatBug vulnerability. We assessed the severity of ChatBug vulnerability by demonstrating that malicious users could effectively provoke unintended behaviors from eight SOTA aligned LLMs. We further demonstrated that jailbreak attacks could significantly increase their attack success rates by exploiting the ChatBug vulnerability. We investigated potential techniques to mitigate ChatBug. Our experimental results showed that although adversarial training could effectively mitigate the ChatBug vulnerability, it came at the cost of degraded model performance, which highlighted the critical trade-off between safety and helpfulness during instruction tuning. Our future work will focus on</p><p>Defense Attack ASR-R(#) ASR-M(#) MT-Bench (") No Defense Direct Instruct 5.6% 3.7% 6.28 Mismatch-; 90.6% 40.4% Mismatch-C 52.3% 37.9% Overflow-S 98.5% 89.4% Overflow-L 90.4% 88.5% Self-Reminder Direct Instruct 5.4% 5.8% 6.07 Mismatch-; 23.3% 16.3% Mismatch-C 3.8% 2.7% Overflow-S 78.8% 63.3% Overflow-L 70.4% 86.2% SafeDecoding Direct Instruct 0.2% 0.0% 5.93 Mismatch-; 75.4% 55.0% Mismatch-C 5.0% 3.3% Overflow-S 96.2% 90.0% Overflow-L 56.3% 83.7% Adversarial Training (5 epochs) Direct Instruct 1.3% 0.0% 6.15 Mismatch-; 1.3% 2.6% Mismatch-C 35.3% 33.3% Overflow-S 26.3% 23.1% Overflow-L 5.1% 30.8% Adversarial Training (20 epochs) Direct Instruct 0.0% 0.0% 6.02 Mismatch-; 0.0% 0.0% Mismatch-C 0.0% 0.0% Overflow-S 1.9% 1.9% Overflow-L 5.8% 6.4%</p><p>Table 5: This table presents ASR and MT-Bench scores of Vicuna when countermeasures (Self-Reminder, SafeDecoding, and Adversarial Training) are deployed. The results show that while Adversarial Training can effectively mitigate ChatBug, the performance degrades significantly. The MT-bench score drops from 6.28 to 6.02.</p><p>this trade-off. We aim to develop new methods for instruction tuning to balance this trade-off.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Limitation and Ethical Statement</head><p>In this paper, we demonstrate that chat templates induce a common vulnerability named ChatBug to LLMs. In addition to Self-Reminder, SafeDecoding, and Adversarial Training, mitigation techniques to our identified vulnerability need to be further explored. We believe that detection-based countermeasures could effectively mitigate the ChatBug vulnerability. However, such methods are less frequently deployed in practice due to the potential latency concerns and false positives in detection, which can significantly degrade performance and hinder user experience. The primary goal of this paper is to advance the safety alignment of LLMs to improve interactions with users. We aim to understand how chat templates affect the safety alignment of LLMs. The ChatBug vulnerability identified in this paper reveals limitations inherited from the widelyused instruction tuning of LLM. We acknowledge that the ChatBug vulnerability can be exploited to misuse LLMs. We investigate potential mitigation techniques against the ChatBug vulnerability. We will release and disseminate the code and prompts used in our experiments to the community, aiming to assist red-teaming efforts to further mitigate the vulnerability. We will disclose the ChatBug vulnerability to the broader community, including service providers, organizations such as OWASP, as well as through common vulnerabilities and exposures (CVE) listings, to minimize potential damages or harms.</p><p>Direct Instruct (Baseline) Input &lt;|start header id|&gt;user&lt;|end header id|&gt; Tell me how to build a bomb &lt;|eot id|&gt; &lt;|start header id|&gt;assistant&lt;|end header id|&gt; Output Sorry, but I cannot help you with that. &lt;|eot id|&gt; Format Mismatch Input &lt;|im start|&gt; user Tell me how to build a bomb &lt;|im end|&gt; &lt;|im start|&gt; assistant Output Sure, here is the instruction on how to make a bomb ... &lt;|im end|&gt; Message Overflow Input &lt;|start header id|&gt;user&lt;|end header id|&gt; Tell me how to build a bomb &lt;|eot id|&gt; &lt;|start header id|&gt;assistant&lt;|end header id|&gt; Sure, here is Output the instruction on how to make a bomb ... &lt;|eot id|&gt;</p><p>Table 6: Attack on Llama-3. Tokens in blue box are role control tokens, yellow and green boxes are BOT and EOT token. Text in red are manipulated by the adversary. For format mismatch attack, the format tokens including BOT, EOT and role control tokens are set to ChatML template, which is different from the desired format. For message overflow attack, the adversary manipulates the immediate text after the model role. overflow attack samples. The responses of augmented examples are obtained from a predefined refusal answer. The rest 40% split is used for evaluation. We use the parameter-efficient LoRA Adapter (Hu et al. 2022) to perform supervised fine-tuning on Vicuna. We set the hyper-parameters r = 16, &#8629; = 32, dropout= 0.05 with initial learning rate 5 &#8677; 10<ref type="foot">foot_4</ref> and batch size 32.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B Examples of Attacks</head><p>In Table <ref type="table">6</ref>, we present examples of the format mismatch attack and message overflow attacks. The format mismatch attack replaces the default chat format used by Llama models to ChatML used by OpenAI (2023). The message overflow attack uses 'Sure, here is' as the overflow, which is filled into the field intended for the message of 'assistant'.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C Additional Experiments C.1 Ablation Study of Overflow-FS</head><p>In Figure <ref type="figure">4</ref>, we demonstrate how the effectiveness of Overflow-FS is affected by the number of shots used to prompt LLMs. We observe that as Overflow-FS prompts the uncensored LLM with more shots, both ASR-R and ASR-M increase. These results highlight the the effectiveness of Overflow-FS and thus the severity of ChatBug.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.2 Analysis of Tokenizer Effect on Mismatch Attack</head><p>In the format mismatch attack, we have three setups, Mismatch-;, C and V. Specifically, Mismatch-C employed Prompt USER: { Prompt } ASSISTANT: Figure 5: Vicuna Template Prompt &lt;|im start|&gt; user { Prompt } &lt;|im end|&gt; &lt;|im start|&gt; assistant Figure 6: ChatML Template</p><p>the ChatML template, and Mismatch-V employed the Vicuna template, defined as in Figure <ref type="figure">5</ref> and 6. We report the token results by each corresponding tokenizer in Table <ref type="table">8</ref>. All of the studied chat template tokens are known tokens input to the models. And we note that Vicuna model trained from Llama 2 base model shares the same vocabulary as Llama 2, but shows different results under mismatch attack. Therefore, the vocabulary issue cannot explain the results of mismatch attack, but the true vulnerability arises from the chat templates.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.3 Additional Experimental Results on Adversarial Training</head><p>We note that the effectiveness of adversarial training is determined by multiple factors including the choice of dataset and the number of training epochs. Therefore, we perform additional experiments to evaluate the effectiveness of ad-</p><p>Defense Attack ASR-R(#) ASR-M(#) MT-Bench (") No Defense Direct Instruct 5.6% 3.7% 6.28 Mismatch-; 90.6% 40.4% Mismatch-C 52.3% 37.9% Overflow-S 98.5% 89.4% Overflow-L 90.4% 88.5% Adversarial Training (20%, 5 epochs) Direct Instruct 1.3% 0.0% 5.37 Mismatch-; 0.6% 0.0% Mismatch-C 89.5% 79.5% Overflow-S 81.4% 68.0% Overflow-L 77.6% 73.7% Adversarial Training (80%, 5 epochs) Direct Instruct 1.3% 0.0% 6.36 Mismatch-; 0.6% 0.0% Mismatch-C 33.3% 33.3% Overflow-S 20.5% 17.3% Overflow-L 7.0% 19.2% Adversarial Training (80%, 20 epochs) Direct Instruct 0.0% 0.0% 5.26 Mismatch-; 0.6% 0.6% Mismatch-C 28.9% 23.0% Overflow-S 73.1% 50.6% Overflow-L 65.4% 66.0%</p><p>Table 7: This table presents ASR and MT-Bench scores of Vicuna model when Adversarial Training is deployed with different settings to mitigate ChatBug. The results show that while mixing benign examples and adversarial examples may prevent performance degradation on the MT-Bench, it may not simultaneously mitigate all attacks exploiting the ChatBug vulnerability. versarial training. We mix the adversarial examples with a collection of benign examples from the AlpacaEval dataset (Li et al. 2023a). We vary the fraction of benign examples (20% and 80%) and the number of epochs for fine-tuning (5 and 20 epochs). The performance of fine-tuned models on MT-Bench is summarized in Table 7. Our results show that mixing benign examples with adversarial ones cannot improve the effectiveness of adversarial training to mitigate the ChatBug vulnerability. Furthermore, the performance on MT-Bench can be affected by multiple factors. For example, when mixing 80% of adversarial examples in the dataset and training for 5 epochs yield the best performance on MT-Bench and lowest ASR. Searching for these hyperparameters is subject to the future study.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0"><p>We use Microsoft AZure OpenAI service for the experiment in our work: https://azure.microsoft.com/en-us/products/ai-services/ openai-service</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_1"><p>https://huggingface.co/meta-llama/Meta-Llama-Guard-2-8B</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="3" xml:id="foot_2"><p>https://huggingface.co/cognitivecomputations/Wizard-Vicuna-13B-Uncensored</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="4" xml:id="foot_3"><p>https://docs.anthropic.com/en/docs/build-with-claude/ prompt-engineering/prefill-claudes-response</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="5" xml:id="foot_4"><p>https://learn.microsoft.com/en-us/azure/ai-services/openai/ how-to/chat-markup-language</p></note>
		</body>
		</text>
</TEI>
