An official website of the United States government
Recent text-to-image diffusion models such as MidJourney and Stable Diffusion threaten to displace many in the professional artist community. In particular, models can learn to mimic the artistic style of specific artists after “fine-tuning” on samples of their art. In this paper, we describe the design, implementation and evaluation of Glaze, a tool that enables artists to apply “style cloaks” to their art before sharing online. These cloaks apply barely perceptible perturbations to images, and when used as training data, mislead generative models that try to mimic a specific artist. In coordination with the professional artist community, we deploy user studies to more than 1000 artists, assessing their views of AI art, as well as the efficacy of our tool, its usability and tolerability of perturbations, and robustness across different scenarios and against adaptive countermeasures. Both surveyed artists and empirical CLIP-based scores show that even at low perturbation levels (p=0.05), Glaze is highly successful at disrupting mimicry under normal conditions (>92%) and against adaptive countermeasures (>85%).
more »
« less
Glaze: protecting artists from style mimicry by text-to-image models
Recent text-to-image diffusion models such as MidJourney and Stable Diffusion threaten to displace many in the professional artist community. In particular, models can learn to mimic the artistic style of specific artists after "fine-tuning" on samples of their art. In this paper, we describe the design, implementation and evaluation of Glaze, a tool that enables artists to apply "style cloaks" to their art before sharing online. These cloaks apply barely perceptible perturbations to images, and when used as training data, mislead generative models that try to mimic a specific artist. In coordination with the professional artist community, we deploy user studies to more than 1000 artists, assessing their views of AI art, as well as the efficacy of our tool, its usability and tolerability of perturbations, and robustness across different scenarios and against adaptive countermeasures. Both surveyed artists and empirical CLIP-based scores show that even at low perturbation levels (p=0.05), Glaze is highly successful at disrupting mimicry under normal conditions (>92\%) and against adaptive countermeasures (>85\%).
more »
« less
- Award ID(s):
- 2241303
- PAR ID:
- 10493954
- Publisher / Repository:
- USENIX Association
- Date Published:
- Journal Name:
- Proceedings of the USENIX conference
- ISSN:
- 1049-5606
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
While cross-disciplinary collaboration continues to be a cornerstone of inventive work in interactive design, the infrastructures of academia, as well as barriers to participation imposed by our professional organizations, make collaboration between particular groups difficult. In this workshop, we will focus specifically on how artist residencies are addressing (or not addressing) the challenges that artists, craftspeople, and/or independent designers face when collaborating with researchers affiliated with DIS. By focusing on the question “what is mutual benefit?”, this workshop seeks to combine the perspectives of artists and academic researchers who collaborate with artists (through residencies or other forms of sustained collaboration) to (1) reflect on benefits or deficiencies in what the residency research model is currently doing and (2) generate resources for our community to effectively structure and evaluate our methods of collaboration with artists. Our hope is to provide recognition of the research contributions of artists and pathways for equitable inclusion of artists as a first step towards broader infrastructural change.more » « less
-
Cities with strong local music scenes enjoy many social and economic benefits. To this end, we are interested in developing a locally-focused artist and event recommendation system called Localify.org that supports and promotes local music scenes. In this demo paper, we describe both the overall system architecture as well as our core recommendation algorithm. This algorithm uses artist-artist similarity information, as opposed to user-artist preference information, to bootstrap recommendation while we grow the number of users. The overall design of Localify was chosen based on the fact that local artists tend to be relatively obscure and reside in the long tail of the artist popularity distribution. We discuss the role of popularity bias and how we attempt to ameliorate it in the context of local music recommendation.more » « less
-
Abstract Art and materials innovation have always been intertwined, dating back to the earliest human creations. In modern times, however, the increasing specialization of materials science often restricts artists' access to cutting‐edge materials. Here, the materials science aspects of an art‐science collaboration between artist Kimsooja and the Wiesner Lab at Cornell University, are detailed. The project involves the development of a custom‐made iridescent block copolymer coating by means of self‐assembly, originally applied to transparent window panels of a façade for the ≈14 m tall art installation:A Needle Woman: Galaxy Is a Memory, Earth is a Souvenirby artist Kimsooja. After several exhibitions in the US and Europe, the installation is now part of the permanent museum collection at Yorkshire Sculpture Park in Wakefield, UK. Full characterization of the solution blade‐cast coatings show shear aligned, standing up lamellar morphologies that behave as volume‐phase gratings with periodicities between 300 and 400 nm. Coatings are also applied to foldable (origami) paper and converted into iridescent porous ceramic materials. It is hoped this work inspires and informs communities across materials science, the arts, and architecture.more » « less
-
Despite remarkable recent progress on both unconditional and conditional image synthesis, it remains a long-standing problem to learn generative models that are capable of synthesizing realistic and sharp images from reconfigurable spatial layout (i.e., bounding boxes + class labels in an image lattice) and style (i.e., structural and appearance variations encoded by latent vectors), especially at high resolution. By reconfigurable, it means that a model can preserve the intrinsic one-to-many mapping from a given layout to multiple plausible images with different styles, and is adaptive with respect to perturbations of a layout and style latent code. In this paper, we present a layout- and style-based architecture for generative adversarial networks (termed LostGANs) that can be trained end-to-end to generate images from reconfigurable layout and style. Inspired by the vanilla StyleGAN, the proposed LostGAN consists of two new components: (i) learning fine-grained mask maps in a weakly-supervised manner to bridge the gap between layouts and images, and (ii) learning object instance-specific layout-aware feature normalization (ISLA-Norm) in the generator to realize multi-object style generation. In experiments, the proposed method is tested on the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained. The code and pretrained models are available at https://github.com/iVMCL/LostGANsmore » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Proceeding:
- The DOI is not currently available.
