More Like this

1. Jailbroken: How Does LLM Safety Training Fail?

   Wei, Alexander Wei; Haghtalab, Nika; Steinhardt, Jacob (December 2023, Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

   Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity -- that safety mechanisms should be as sophisticated as the underlying model -- and argues against the idea that scaling alone can resolve these safety failure modes. 

   more » « less
2. SafeDecoding: Defending against jailbreak attacks via safety-aware decoding

   Xu, Z; Jiang, F; Niu, L; Jia, J; Li, Bo; Poovendran, Radha (May 2024, ICLR Workshop on Secure and Trustworthy Large Language Models (ICLR SeT-LLM))

   As large language models (LLMs) become increasingly integrated into real-world applications such as code generation and chatbot assistance, extensive efforts have been made to align LLM behavior with human values, including safety. Jailbreak attacks, aiming to provoke unintended and unsafe behaviors from LLMs, remain a significant LLM safety threat. In this paper, we aim to defend LLMs against jailbreak attacks by introducing SafeDecoding, a safety-aware decoding strategy for LLMs to generate helpful and harmless responses to user queries. Our insight in developing SafeDecoding is based on the observation that, even though probabilities of tokens representing harmful contents outweigh those representing harmless responses, safety disclaimers still appear among the top tokens after sorting tokens by probability in descending order. This allows us to mitigate jailbreak attacks by identifying safety disclaimers and amplifying their token probabilities, while simultaneously attenuating the probabilities of token sequences that are aligned with the objectives of jailbreak attacks. We perform extensive experiments on five LLMs using six state-of-the-art jailbreak attacks and four benchmark datasets. Our results show that SafeDecoding significantly reduces attack success rate and harmfulness of jailbreak attacks without compromising the helpfulness of responses to benign user queries while outperforming six defense methods. Our code is publicly available at: https://github.com/uw-nsl/SafeDecoding 

   more » « less
3. ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates

   Jiang, Fengqing; Xu, Zhangchen; Niu, Luyao; Lin, Bill Yuchen; Poovendran, Radha (February 2025, The 39th Annual AAAI Conference on Artificial Intelligence)

   Large language models (LLMs) are expected to follow in- structions from users and engage in conversations. Tech- niques to enhance LLMs’ instruction-following capabilities typically fine-tune them using data structured according to a predefined chat template. Although chat templates are shown to be effective in optimizing LLM performance, their impact on safety alignment of LLMs has been less understood, which is crucial for deploying LLMs safely at scale. In this paper, we investigate how chat templates affect safety alignment of LLMs. We identify a common vulnerability, named ChatBug, that is introduced by chat templates. Our key insight to identify ChatBug is that the chat templates provide a rigid format that need to be followed by LLMs, but not by users. Hence, a malicious user may not necessar- ily follow the chat template when prompting LLMs. Instead, malicious users could leverage their knowledge of the chat template and accordingly craft their prompts to bypass safety alignments of LLMs. We study two attacks to exploit the ChatBug vulnerability. Additionally, we demonstrate that the success of multiple existing attacks can be attributed to the ChatBug vulnerability. We show that a malicious user can exploit the ChatBug vulnerability of eight state-of-the- art (SOTA) LLMs and effectively elicit unintended responses from these models. Moreover, we show that ChatBug can be exploited by existing jailbreak attacks to enhance their at- tack success rates. We investigate potential countermeasures to ChatBug. Our results show that while adversarial train- ing effectively mitigates the ChatBug vulnerability, the vic- tim model incurs significant performance degradation. These results highlight the trade-off between safety alignment and helpfulness. Developing new methods for instruction tuning to balance this trade-off is an open and critical direction for future research. 

   more » « less
4. Robust Ensemble Morph Detection with Domain Generalization

   https://doi.org/10.1109/IJCB54206.2022.10007929  

   Kashiani, Hossein; Sami, Shoaib Meraj; Soleymani, Sobhan; Nasrabadi, Nasser M. (October 2022, 2022 IEEE International Joint Conference on Biometrics (IJCB), Abu Dhabi, United Arab Emirates)

   Although a substantial amount of studies is dedicated to morphing detection, most of them fail to generalize for morph faces outside of their training paradigm. Moreover, recent morph detection methods are highly vulnerable to adversarial attacks. In this paper, we intend to learn a morph detection model with high generalization to a wide range of morphing attacks and high robustness against different adversarial attacks. To this aim, we develop an ensemble of convolutional neural networks (CNNs) and Transformer models to benefit from their capabilities simultaneously. To improve the robust accuracy of the ensemble model, we employ multi-perturbation adversarial training and generate adversarial examples with high transferability for several single models. Our exhaustive evaluations demonstrate that the proposed robust ensemble model generalizes to several morphing attacks and face datasets. In addition, we validate that our robust ensemble model gains better robustness against several adversarial attacks while outperforming the state-of-the-art studies. 

   more » « less
5. Protecting Cyber-Physical System Testbeds from Red-Teaming/Blue-Teaming Experiments Gone Awry

   https://doi.org/10.1007/978-3-031-21280-2_8  

   Talukder, M.R.H.; Amin, M.A.; Ray, I (November 2022, Information Security Practice and Experience. ISPEC 2022. Lecture Notes in Computer Science)

   Su, C.; Gritzalis, D.; Piuri, V. (Ed.)

   Many cyber-physical systems (CPS) are critical infrastructure. Security attacks on these critical systems can have catastrophic consequences, putting human lives at risk. Consequently, it is very important to pace CPS systems to red-teaming/blue teaming exercises to understand vulnerabilities and the progression/impact of cyber attacks on them. Since it is not always prudent to conduct such security exercises on live CPS, researchers use CPS testbeds to conduct security-related experiments. Often, such testbeds are very expensive. Since attack scripts used in red-teaming/blue-teaming exercises are, in the strictest sense of the term, malicious in nature, there is a need to protect the testbed itself from these attack experiments that have the potential to go awry. Moreover, when multiple experiments are conducted on the same testbed, there is a need to maintain isolation among these experiments so that no experiment can accidentally or maliciously affect/compromise others. In this work, we describe a novel security architecture and framework to ensure protection of security-related experiments on a CPS testbed and at the same time support secure communication services among simultaneously running experiments based on well-formulated access control policies. 

   more » « less