An official website of the United States government
Music is one of the most universal forms of communication and entertainment across cultures. This can largely be credited to the sense of synesthesia, or the combining of senses. Based on this concept of synesthesia, we want to explore whether generative AI can create visual representations for music. The aim is to inspire the user’s imagination and enhance the user experience when enjoying music. Our approach has the following steps: (a) Music is analyzed and classified into multiple dimensions (including instruments, emotion, tempo, pitch range, harmony, and dynamics) to produce textual descriptions. (b) The texts form inputs of machine models that can predict the genre of the input audio. (c) The prompts are inputs of generative machine models to create visual representations. The visual representations are continuously updated as the music plays, ensuring that the visual effects aptly mirror the musical changes. A comprehensive user study with 88 users confirms that our approach is able to generate visual art reflecting the music pieces. From a list of images covering both abstract images and realistic images, users considered that our system-generated images can better represent pieces of music than human-chosen images. It suggests that generative arts can become a promising method to enhance users' listening experience while enjoying music. Our method provides a new approach to visualize music and to enjoy music through generative arts.
more »
« less
Exploring Musical Roots: Applying Audio Embeddings to Empower Influence Attribution for a Generative Music Model
Every artist has a creative process that draws inspiration from previous artists and their works. Today, “inspiration” has been automated by generative music models. The black box nature of these models obscures the identity of the works that influence their creative output. As a result, users may inadvertently appropriate, misuse, or copy existing artists’ works. We establish a replicable methodology to systematically identify similar pieces of music audio in a manner that is useful for understanding training data attribution. A key aspect of our approach is to harness an effective music audio similarity measure. We compare the effect of applying CLMR [50] and CLAP [55] embeddings to similarity measurement in a set of 5 million audio clips used to train VampNet [24], a recent open source generative music model. We validate this approach with a human listening study. We also explore the effect that modifications of an audio example (e.g., pitch shifting, time stretching, background noise) have on similarity measurements. This work is foundational to incorporating automated influence attribution into generative modeling, which promises to let model creators and users move from ignorant appropriation to informed creation. Audio samples that accompany this paper are available at https://tinyurl.com/exploring-musical-roots.
more »
« less
- Award ID(s):
- 2222369
- PAR ID:
- 10638223
- Publisher / Repository:
- International Society for Music Information Retrieval Conference
- Date Published:
- Subject(s) / Keyword(s):
- AI Generative Modelling Music
- Format(s):
- Medium: X
- Location:
- San Francisco, CA, USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Generative AI in music (GAIM) technologies are rapidly transforming music production, yet little is known about how working musicians perceive and respond to these changes. This study presents findings from in-depth interviews with 43 musicians, spanning diverse genres, professional roles and experience with music technology. Our analysis, informed by a reflexive thematic analysis approach, suggests complex tensions between perceived benefits and risks of GAIM adoption. Key themes were generated around tensions between (i) fear of reduced job opportunities for professional musicians and appreciation of the potential of AI to make individual musicians more independent and productive; (ii) fear about the exploitation of artists’ work and benefits of open music exchanges; (iii) fear that AI will exacerbate inequities and recognition of AI’s potential to increase access to music production. Our findings highlight the need for careful consideration of justice and fairness in GAIM development and deployment, suggesting that different types of GAIM use (from assistant to replacement) carry distinct ethical implications. This work provides a foundation for understanding how GAIMs can be integrated into music production while respecting artists’ rights and creative agency.more » « less
-
null (Ed.)Most commercial music services rely on collaborative filtering to recommend artists and songs. While this method is effective for popular artists with large fanbases, it can present difficulties for recommending novel, lesser known artists due to a relative lack of user preference data. In this paper, we therefore seek to understand how content-based approaches can be used to more effectively recommend songs from these lesser known artists. Specifically, we conduct a user study to answer three questions. Firstly, do most users agree which songs are most acoustically similar? Secondly, is acoustic similarity a good proxy for how an individual might construct a playlist or recommend music to a friend? Thirdly, if so, can we find acoustic features that are related to human judgments of acoustic similarity? To answer these questions, our study asked 117 test subjects to compare two unknown candidate songs relative to a third known reference song. Our findings show that 1) judgments about acoustic similarity are fairly consistent, 2) acoustic similarity is highly correlated with playlist selection and recommendation, but not necessarily personal preference, and 3) we identify a subset of acoustic features from the Spotify Web API that is particularly predictive of human similarity judgments.more » « less
-
Creating truly original ideas requires extensive knowledge of existing ideas. Navigating prior examples can help people to understand what has already been done and to assess the quality of their own ideas through comparison. The creativity literature has suggested that the conceptual distance between a proposed solution and a potential inspiration can influence one's thinking. However, less is known about how creators might use data about conceptual distance when exploring a large repository of ideas. To investigate this, we created a novel tool for exploring examples called IdeateRelate that visualizes 600+ COVID-related ideas, organized by their similarity to a new idea. In an experiment that compared the IdeateRelate visualization to a simple list of examples, we found that users in the Viz condition leveraged both semantic and categorical similarity, curated a more similar set of examples, and adopted more language from examples into their iterated ideas (without negatively affecting the overall novelty). We discuss implications for creating adaptive interfaces that provide creative inspiration in response to designers' ideas throughout an iterative design process.more » « less
-
Cities with strong local music scenes enjoy many social and economic benefits. To this end, we are interested in developing a locally-focused artist and event recommendation system called Localify.org that supports and promotes local music scenes. In this demo paper, we describe both the overall system architecture as well as our core recommendation algorithm. This algorithm uses artist-artist similarity information, as opposed to user-artist preference information, to bootstrap recommendation while we grow the number of users. The overall design of Localify was chosen based on the fact that local artists tend to be relatively obscure and reside in the long tail of the artist popularity distribution. We discuss the role of popularity bias and how we attempt to ameliorate it in the context of local music recommendation.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
