skip to main content


This content will become publicly available on August 9, 2024

Title: Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models
Recent text-to-image diffusion models such as MidJourney and Stable Diffusion threaten to displace many in the professional artist community. In particular, models can learn to mimic the artistic style of specific artists after “fine-tuning” on samples of their art. In this paper, we describe the design, implementation and evaluation of Glaze, a tool that enables artists to apply “style cloaks” to their art before sharing online. These cloaks apply barely perceptible perturbations to images, and when used as training data, mislead generative models that try to mimic a specific artist. In coordination with the professional artist community, we deploy user studies to more than 1000 artists, assessing their views of AI art, as well as the efficacy of our tool, its usability and tolerability of perturbations, and robustness across different scenarios and against adaptive countermeasures. Both surveyed artists and empirical CLIP-based scores show that even at low perturbation levels (p=0.05), Glaze is highly successful at disrupting mimicry under normal conditions (>92%) and against adaptive countermeasures (>85%).  more » « less
Award ID(s):
2241303 1949650
NSF-PAR ID:
10495502
Author(s) / Creator(s):
; ; ; ; ;
Publisher / Repository:
Proceedings of the 32nd USENIX Security Symposium
Date Published:
Journal Name:
Proceedings of the 32nd USENIX Security Symposium
Format(s):
Medium: X
Location:
Anaheim, CA, USA
Sponsoring Org:
National Science Foundation
More Like this
  1. Recent text-to-image diffusion models such as MidJourney and Stable Diffusion threaten to displace many in the professional artist community. In particular, models can learn to mimic the artistic style of specific artists after "fine-tuning" on samples of their art. In this paper, we describe the design, implementation and evaluation of Glaze, a tool that enables artists to apply "style cloaks" to their art before sharing online. These cloaks apply barely perceptible perturbations to images, and when used as training data, mislead generative models that try to mimic a specific artist. In coordination with the professional artist community, we deploy user studies to more than 1000 artists, assessing their views of AI art, as well as the efficacy of our tool, its usability and tolerability of perturbations, and robustness across different scenarios and against adaptive countermeasures. Both surveyed artists and empirical CLIP-based scores show that even at low perturbation levels (p=0.05), Glaze is highly successful at disrupting mimicry under normal conditions (>92\%) and against adaptive countermeasures (>85\%). 
    more » « less
  2. While cross-disciplinary collaboration continues to be a cornerstone of inventive work in interactive design, the infrastructures of academia, as well as barriers to participation imposed by our professional organizations, make collaboration between particular groups difficult. In this workshop, we will focus specifically on how artist residencies are addressing (or not addressing) the challenges that artists, craftspeople, and/or independent designers face when collaborating with researchers affiliated with DIS. By focusing on the question “what is mutual benefit?”, this workshop seeks to combine the perspectives of artists and academic researchers who collaborate with artists (through residencies or other forms of sustained collaboration) to (1) reflect on benefits or deficiencies in what the residency research model is currently doing and (2) generate resources for our community to effectively structure and evaluate our methods of collaboration with artists. Our hope is to provide recognition of the research contributions of artists and pathways for equitable inclusion of artists as a first step towards broader infrastructural change. 
    more » « less
  3. We propose a methodology, called defender–attacker decision tree analysis, to evaluate defensive actions against terrorist attacks in a dynamic and hostile environment. Like most game‐theoretic formulations of this problem, we assume that the defenders act rationally by maximizing their expected utility or minimizing their expected costs. However, we do not assume that attackers maximize their expected utilities. Instead, we encode the defender's limited knowledge about the attacker's motivations and capabilities as a conditional probability distribution over the attacker's decisions. We apply this methodology to the problem of defending against possible terrorist attacks on commercial airplanes, using one of three weapons: infrared‐guided MANPADS (man‐portable air defense systems), laser‐guided MANPADS, or visually targeted RPGs (rocket propelled grenades). We also evaluate three countermeasures against these weapons: DIRCMs (directional infrared countermeasures), perimeter control around the airport, and hardening airplanes. The model includes deterrence effects, the effectiveness of the countermeasures, and the substitution of weapons and targets once a specific countermeasure is selected. It also includes a second stage of defensive decisions after an attack occurs. Key findings are: (1) due to the high cost of the countermeasures, not implementing countermeasures is the preferred defensive alternative for a large range of parameters; (2) if the probability of an attack and the associated consequences are large, a combination of DIRCMs and ground perimeter control are preferred over any single countermeasure.

     
    more » « less
  4. Abstract

    Part of the reason women are disadvantaged in the labor market is because gender inequalities define social networks of the workplace. In the current project, I consider how gender shapes professional networks by focusing on the R&B/hip hop industry as an empirical case study. By conceptualizing the collaboration patterns between performers of popular R&B/hip hop songs from 2012 to 2020 as a network, I apply exponential random graph models (ERGMs) and find that women tend to occupy marginalized positions when compared to their male peers. Then, I adopt a social exchange framework to argue that critical acclaim is a resource that is associated with higher odds of collaborating for all artists, though gender differences define this process. For instance, the largest gender gaps in collaboration are present among artists who have either won Grammy awards or never received nominations for such honors. These findings suggest that female artists with lower status are often excluded from collaboration opportunities. Once women acquire enough prestige to “make up” for their gender, they may avoid collaborations because gender stereotypes challenge their decision-making power within these interactions.

     
    more » « less
  5. Today, creators of data-hungry deep neural networks (DNNs) scour the Internet for training fodder, leaving users with little control over or knowledge of when their data, and in particular their images, are used to train models. To empower users to counteract unwanted use of their images, we design, implement and evaluate a practical system that enables users to detect if their data was used to train a DNN model for image classification. We show how users can create special images we call isotopes, which introduce ``spurious features'' into DNNs during training. With only query access to a model and no knowledge of the model-training process, nor control of the data labels, a user can apply statistical hypothesis testing to detect if the model learned these spurious features by training on the user's images. Isotopes can be viewed as an application of a particular type of data poisoning. In contrast to backdoors and other poisoning attacks, our purpose is not to cause misclassification but rather to create tell-tale changes in confidence scores output by the model that reveal the presence of isotopes in the training data. Isotopes thus turn DNNs' vulnerability to memorization and spurious correlations into a tool for data provenance. Our results confirm efficacy in multiple image classification settings, detecting and distinguishing between hundreds of isotopes with high accuracy. We further show that our system works on public ML-as-a-service platforms and larger models such as ImageNet, can use physical objects in images instead of digital marks, and remains robust against several adaptive countermeasures.

     
    more » « less