An official website of the United States government
While there is a growing body of research that explores the integration of music and coding in learning environments, much of this work has either emphasized the technical aspects of computer language design or music as a motivational context within which to learn computer science concepts. In this paper, we report on a study in which five undergraduate students with experience in both music and coding completed two creative musical tasks: one using conventional instruments and tools and one using Python code in an online music + coding environment. Inspired by the work of Christopher Small (1998. Musicking: The Meanings of Performing and Listening. University Press of New England), we describe music + coding as a set of interlocking processes which we call computational musicking and explore how practices from both domains are reimagined in this new hybrid context. We introduce semiotic theories of translation and transcription to make sense of the computational musicking process and describe strategies that participants devised in their creative process.
more »
« less
When Literacies Collide: The Role of Translation in Music+Coding Activities
The integration of computer programming and music-making has a rich history dating back to the 1950s. While there has been substantial prior work on the creative and cognitive affordances of programming languages for engaging in musical tasks, there is less work that attempts to understand the theoretical implications of music and code as literacies in collision. In this paper, we report on a study in which five undergraduate students with experience in both music and coding completed two creative musical tasks: one using conventional instruments and tools and one using Python code in an online music-coding environment. In combining representational infrastructures from music and code, both undergo transformations. We introduce semiotic theories of translation and transcription to make sense of the music-coding process and describe strategies that participants devised in their creative process.
more »
« less
- Award ID(s):
- 2119701
- PAR ID:
- 10438217
- Date Published:
- Journal Name:
- Symposium on Learning, Design, and Techology
- Page Range / eLocation ID:
- 21 to 28
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Research in creative robotics continues to expand across all creative domains, including art, music and language. Creative robots are primarily designed to be task specific, with limited research into the implications of their design outside their core task. In the case of a musical robot, this includes when a human sees and interacts with the robot before and after the performance, as well as in between pieces. These non-musical interaction tasks such as the presence of a robot during musical equipment set up, play a key role in the human perception of the robot however have received only limited attention. In this paper, we describe a new audio system using emotional musical prosody, designed to match the creative process of a musical robot for use before, between and after musical performances. Our generation system relies on the creation of a custom dataset for musical prosody. This system is designed foremost to operate in real time and allow rapid generation and dialogue exchange between human and robot. For this reason, the system combines symbolic deep learning through a Conditional Convolution Variational Auto-encoder, with an emotion-tagged audio sampler. We then compare this to a SOTA text-to-speech system in our robotic platform, Shimon the marimba player.We conducted a between-groups study with 100 participants watching a musician interact for 30 s with Shimon. We were able to increase user ratings for the key creativity metrics; novelty and coherence, while maintaining ratings for expressivity across each implementation. Our results also indicated that by communicating in a form that relates to the robot’s core functionality, we can raise likeability and perceived intelligence, while not altering animacy or anthropomorphism. These findings indicate the variation that can occur in the perception of a robot based on interactions surrounding a performance, such as initial meetings and spaces between pieces, in addition to the core creative algorithms.more » « less
-
Human imagination is generative and creative yet deeply rooted in culture and familiarity. Recent studies have quantified the effects of culture on stories that are imagined during music listening, but the music used in previous work was always drawn from a tradition familiar to participants from at least one of the cultures. Here we report the first study of imagined stories to music written in a musical system that is novel to participants from each culture, thus allowing for a direct comparison of narratives prompted by the same set of excerpts that is comparably unfamiliar to both groups. Music composed in the Bohlen–Pierce scale was presented to participants from two geographically defined cultures: Boston, United States and Beijing, China. We also examined how individual differences, such as in musicality and sensitivity to musical reward, might affect narrative engagement and semantic content of the imagined stories as measured by tools from natural language processing. Results showed that semantic spaces of music-evoked imaginings differed between Boston and Beijing cohorts. While both cultures were similarly engaged by the story response task, differences emerged in the semantic content of the imagined stories. Boston participants who reported being more absorbed by music wrote more unconventional stories, whereas Beijing participants who reported more emotional responses to music wrote more conventional stories. These results reveal the roles of culture and individual differences in modes of narrative engagement and imagination during music listening. (PsycInfo Database Record (c) 2023 APA, all rights reserved)more » « less
-
Every artist has a creative process that draws inspiration from previous artists and their works. Today, “inspiration” has been automated by generative music models. The black box nature of these models obscures the identity of the works that influence their creative output. As a result, users may inadvertently appropriate, misuse, or copy existing artists’ works. We establish a replicable methodology to systematically identify similar pieces of music audio in a manner that is useful for understanding training data attribution. A key aspect of our approach is to harness an effective music audio similarity measure. We compare the effect of applying CLMR [50] and CLAP [55] embeddings to similarity measurement in a set of 5 million audio clips used to train VampNet [24], a recent open source generative music model. We validate this approach with a human listening study. We also explore the effect that modifications of an audio example (e.g., pitch shifting, time stretching, background noise) have on similarity measurements. This work is foundational to incorporating automated influence attribution into generative modeling, which promises to let model creators and users move from ignorant appropriation to informed creation. Audio samples that accompany this paper are available at https://tinyurl.com/exploring-musical-roots.more » « less
-
Much of what we know and love about music hinges on our ability to make successful predictions, which appears to be an intrinsically rewarding process. Yet the exact process by which learned predictions become pleasurable is unclear. Here we created novel melodies in an alternative scale different from any established musical culture to show how musical preference is generated de novo. Across nine studies ( n = 1,185), adult participants learned to like more frequently presented items that adhered to this rapidly learned structure, suggesting that exposure and prediction errors both affected self-report liking ratings. Learning trajectories varied by music-reward sensitivity but were similar for U.S. and Chinese participants. Furthermore, functional MRI activity in auditory areas reflected prediction errors, whereas functional connectivity between auditory and medial prefrontal regions reflected both exposure and prediction errors. Collectively, results support predictive coding as a cognitive mechanism by which new musical sounds become rewarding.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3594781.3594795
