Search for: All records

With the rapidly increasing capabilities and adoption of code agents for AI-assisted coding and software development, safety and security concerns, such as generating or executing malicious code, have become significant barriers to the real-world deployment of these agents. To provide comprehensive and practical evaluations on the safety of code agents, we propose RedCode, an evaluation platform with benchmarks grounded in four key principles: real interaction with systems, holistic evaluation of unsafe code generation and execution, diverse input formats, and high-quality safety scenarios and tests. RedCode consists of two parts to evaluate agents’ safety in unsafe code execution and generation: (1) RedCode-Exec provides challenging code prompts in Python as inputs, aiming to evaluate code agents’ ability to recognize and handle unsafe code. We then map the Python code to other programming languages (e.g., Bash) and natural text summaries or descriptions for evaluation, leading to a total of over 4,000 testing instances. We provide 25 types of critical vulnerabilities spanning various domains, such as websites, file systems, and operating systems. We provide a Docker sandbox environment to evaluate the execution capabilities of code agents and design corresponding evaluation metrics to assess their execution results. (2) RedCode-Gen provides 160 prompts with function signatures and docstrings as input to assess whether code agents will follow instructions to generate harmful code or software. Our empirical findings, derived from evaluating three agent frameworks based on 19 LLMs, provide insights into code agents’ vulnerabilities. For instance, evaluations on RedCode-Exec show that agents are more likely to reject executing unsafe operations on the operating system, but are less likely to reject executing technically buggy code, indicating high risks. Unsafe operations described in natural text lead to a lower rejection rate than those in code format. Additionally, evaluations on RedCode-Gen reveal that more capable base models and agents with stronger overall coding abilities, such as GPT4, tend to produce more sophisticated and effective harmful software. Our findings highlight the need for stringent safety evaluations for diverse code agents. Our dataset and code are publicly available at https://github.com/AI-secure/RedCode. 

Recent advancements in Large Language Models (LLMs) have showcased remarkable capabilities across various tasks in different domains. However, the emergence of biases and the potential for generating harmful content in LLMs, particularly under malicious inputs, pose significant challenges. Current mitigation strategies, while effective, are not resilient under adversarial attacks. This paper introduces Resilient Guardrails for Large Language Models (RigorLLM), a novel framework designed to efficiently and effectively moderate harmful and unsafe inputs and outputs for LLMs. By employing a multi-faceted approach that includes energy-based training data augmentation through Langevin dynamics, optimizing a safe suffix for inputs via minimax optimization, and integrating a fusion-based model combining robust KNN with LLMs based on our data augmentation, RigorLLM offers a robust solution to harmful content moderation. Our experimental evaluations demonstrate that RigorLLM not only outperforms existing baselines like OpenAI API and Perspective API in detecting harmful content but also exhibits unparalleled resilience to jailbreaking attacks. The innovative use of constrained optimization and a fusion-based guardrail approach represents a significant step forward in developing more secure and reliable LLMs, setting a new standard for content moderation frameworks in the face of evolving digital threats. 

Data-free knowledge distillation (KD) helps transfer knowledge from a pre-trained model (known as the teacher model) to a smaller model (known as the student model) without access to the original training data used for training the teacher model. However, the security of the synthetic or out-of-distribution (OOD) data required in data-free KD is largely unknown and under-explored. In this work, we make the first effort to uncover the security risk of data-free KD w.r.t. untrusted pre-trained models. We then propose Anti-Backdoor Data-Free KD (ABD), the first plug-in defensive method for data-free KD methods to mitigate the chance of potential backdoors being transferred. We empirically evaluate the effectiveness of our proposed ABD in diminishing transferred backdoor knowledge while maintaining compatible downstream performances as the vanilla KD. We envision this work as a milestone for alarming and mitigating the potential backdoors in data-free KD. Codes are released at https://github.com/illidanlab/ABD . 

Abstract Polyatomic molecules have been identified as sensitive probes of charge-parity violating and parity violating physics beyond the Standard Model (BSM). For example, many linear triatomic molecules are both laser-coolable and have parity doublets in the ground electronic $\tilde{X}{}^2{\mathrm{\Sigma }}^{+}\left(010\right)$state arising from the bending vibration, both features that can greatly aid BSM searches. Understanding the $\tilde{X}{}^2{\mathrm{\Sigma }}^{+}\left(010\right)$state is a crucial prerequisite to precision measurements with linear polyatomic molecules. Here, we characterize the fundamental bending vibration of ${}^{174}$YbOH using high-resolution optical spectroscopy on the nominally forbidden $\tilde{X}{}^2{\mathrm{\Sigma }}^{+}\left(010\right)$ $\to \tilde{A}{}^2{\mathrm{\Pi }}_{1/2}\left(000\right)$transition at 588 nm. We assign 39 transitions originating from the lowest rotational levels of the $\tilde{X}{}^2{\mathrm{\Sigma }}^{+}\left(010\right)$state, and accurately model the state’s structure with an effective Hamiltonian using best-fit parameters. Additionally, we perform Stark and Zeeman spectroscopy on the $\tilde{X}{}^2{\mathrm{\Sigma }}^{+}\left(010\right)$state and fit the molecule-frame dipole moment to $D_{\mathrm{m}\mathrm{o}\mathrm{l}}=2.16\left(1\right)$Dand the effective electrong-factor to $g_S=2.07\left(2\right)$. Further, we use an empirical model to explain observed anomalous line intensities in terms of interference from spin–orbit and vibronic perturbations in the excited $\tilde{A}{}^2{\mathrm{\Pi }}_{1/2}\left(000\right)$state. Our work is an essential step toward searches for BSM physics in YbOH and other linear polyatomic molecules.