skip to main content


Title: THE SHAPE OF REMIXXXES TO COME: AUDIO TEXTURE SYNTHESIS WITH TIME-FREQUENCY SCATTERING
This article explains how to apply time–frequency scattering, a con- volutional operator extracting modulations in the time–frequency domain at different rates and scales, to the re-synthesis and manip- ulation of audio textures. After implementing phase retrieval in the scattering network by gradient backpropagation, we introduce scale-rate DAFx, a class of audio transformations expressed in the domain of time–frequency scattering coefficients. One example of scale-rate DAFx is chirp rate inversion, which causes each sonic event to be locally reversed in time while leaving the arrow of time globally unchanged. Over the past two years, our work has led to the creation of four electroacoustic pieces: FAVN; Modulator (Scat- tering Transform); Experimental Palimpsest; Inspection (Maida Vale Project) and Inspection II; as well as XAllegroX (Hecker Scat- tering.m Sequence), a remix of Lorenzo Senni’s XAllegroX, released by Warp Records on a vinyl entitled The Shape of RemiXXXes to Come.  more » « less
Award ID(s):
1633206
NSF-PAR ID:
10118932
Author(s) / Creator(s):
;
Date Published:
Journal Name:
Proceedings of the 22nd International Conference on Digital Audio Effects (DAFx-19)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. With the fast development of Fifth-/Sixth-Generation (5G/6G) communications and the Internet of Video Things (IoVT), a broad range of mega-scale data applications emerge (e.g., all-weather all-time video). These network-based applications highly depend on reliable, secure, and real-time audio and/or video streams (AVSs), which consequently become a target for attackers. While modern Artificial Intelligence (AI) technology is integrated with many multimedia applications to help enhance its applications, the development of General Adversarial Networks (GANs) also leads to deepfake attacks that enable manipulation of audio or video streams to mimic any targeted person. Deepfake attacks are highly disturbing and can mislead the public, raising further challenges in policy, technology, social, and legal aspects. Instead of engaging in an endless AI arms race “fighting fire with fire”, where new Deep Learning (DL) algorithms keep making fake AVS more realistic, this paper proposes a novel approach that tackles the challenging problem of detecting deepfaked AVS data leveraging Electrical Network Frequency (ENF) signals embedded in the AVS data as a fingerprint. Under low Signal-to-Noise Ratio (SNR) conditions, Short-Time Fourier Transform (STFT) and Multiple Signal Classification (MUSIC) spectrum estimation techniques are investigated to detect the Instantaneous Frequency (IF) of interest. For reliable authentication, we enhanced the ENF signal embedded through an artificial power source in a noisy environment using the spectral combination technique and a Robust Filtering Algorithm (RFA). The proposed signal estimation workflow was deployed on a continuous audio/video input for resilience against frame manipulation attacks. A Singular Spectrum Analysis (SSA) approach was selected to minimize the false positive rate of signal correlations. Extensive experimental analysis for a reliable ENF edge-based estimation in deepfaked multimedia recordings is provided to facilitate the need for distinguishing artificially altered media content. 
    more » « less
  2. null (Ed.)
    This paper presents Metamorph, a system that generates imperceptible audio that can survive over-the-air trans- mission to attack the neural network of a speech recognition system. The key challenge stems from how to ensure the added perturbation of the original audio in advance at the sender side is immune to unknown signal distortions during the transmission process. Our empirical study reveals that signal distortion is mainly due to device and channel frequency selectivity but with different characteristics. This brings a chance to capture and further pre-code this impact to generate adversarial examples that are robust to the over-the-air transmission. We leverage this opportunity in Metamorph and obtain an initial perturbation that captures the core distortion’s impact from only a small set of prior measurements, and then take advantage of a domain adaptation algorithm to refine the perturbation to further im- prove the attack distance and reliability. Moreover, we consider also reducing human perceptibility of the added perturbation. Evaluation achieves a high attack success rate (90%) over the attack distance of up to 6 m. Within a moderate distance, e.g., 3 m, Metamorph maintains this high success rate, yet can be further adapted to largely improve the audio quality, confirmed by a human perceptibility study. 
    more » « less
  3. Abstract

    There is growing evidence that prey perceive the risk of predation and alter their behavior in response, resulting in changes in spatial distribution and potential fitness consequences. Previous approaches to mapping predation risk across a landscape quantify predator space use to estimate potential predator‐prey encounters, yet this approach does not account for successful predator attack resulting in prey mortality. An exception is a prey kill site that reflects an encounter resulting in mortality, but obtaining information on kill sites is expensive and requires time to accumulate adequate sample sizes.

    We illustrate an alternative approach using predator scat locations and their contents to quantify spatial predation risk for elk(Cervus canadensis) from multiple predators in the Rocky Mountains of Alberta, Canada. We surveyed over 1300 km to detect scats of bears (Ursus arctos/U.americanus), cougars (Puma concolor), coyotes (Canis latrans), and wolves (C.lupus). To derive spatial predation risk, we combined predictions of scat‐based resource selection functions (RSFs) weighted by predator abundance with predictions that a predator‐specific scat in a location contained elk. We evaluated the scat‐based predictions of predation risk by correlating them to predictions based on elk kill sites. We also compared scat‐based predation risk on summer ranges of elk following three migratory tactics for consistency with telemetry‐based metrics of predation risk and cause‐specific mortality of elk.

    We found a strong correlation between the scat‐based approach presented here and predation risk predicted by kill sites and (r = .98,p < .001). Elk migrating east of the Ya Ha Tinda winter range were exposed to the highest predation risk from cougars, resident elk summering on the Ya Ha Tinda winter range were exposed to the highest predation risk from wolves and coyotes, and elk migrating west to summer in Banff National Park were exposed to highest risk of encountering bears, but it was less likely to find elk in bear scats than in other areas. These patterns were consistent with previous estimates of spatial risk based on telemetry of collared predators and recent cause‐specific mortality patterns in elk.

    A scat‐based approach can provide a cost‐efficient alternative to kill sites of quantifying broad‐scale, spatial patterns in risk of predation for prey particularly in multiple predator species systems.

     
    more » « less
  4. Abstract The Van Allen Probes Electric Fields and Waves (EFW) instrument provided measurements of electric fields and spacecraft floating potentials over a wide dynamic range from DC to 6.5 kHz near the equatorial plane of the inner magnetosphere between 600 km altitude and 5.8 Re geocentric distance from October 2012 to November 2019. The two identical instruments provided data to investigate the quasi-static and low frequency fields that drive large-scale convection, waves induced by interplanetary shock impacts that result in rapid relativistic particle energization, ultra-low frequency (ULF) MHD waves which can drive radial diffusion, and higher frequency wave fields and time domain structures that provide particle pitch angle scattering and energization. In addition, measurements of the spacecraft potential provided a density estimate in cold plasmas ( $<20~\text{eV}$ < 20 eV ) from 10 to $3000~\text{cm}^{-3}$ 3000 cm − 3 . The EFW instrument provided analog electric field signals to EMFISIS for wave analysis, and it received 3d analog signals from the EMFISIS search coil sensors for inclusion in high time resolution waveform data. The electric fields and potentials were measured by current-biased spherical sensors deployed at the end of four 50 m booms in the spacecraft spin plane (spin period $\sim11~\text{sec}$ ∼ 11 sec ) and a pair of stacer booms with a total tip-tip separation of 15 m along the spin axis. Survey waveform measurements at 16 and/or 32 S/sec (with a nominal uncertainty of 0.3 mV/m over the prime mission) were available continuously while burst waveform captures at up to 16,384 S/sec provided high frequency waveforms. This post-mission paper provides the reader with information useful for accessing, understanding and using EFW data. Selected science results are discussed and used to highlight instrument capabilities. Science quantities, data quality and error sources, and analysis routines are documented. 
    more » « less
  5. The vibration-assisted atomic force microscope (AFM)-based nanomachining offers a promising opportunity for low-cost nanofabrication with high tunability. However, critical challenges reside in advancing the throughput and the quality assurance of the process due to extensive offline experimental investigations and characterizations, which in turn hinders the wide industry applications of current AFM-based nanomachining process. Hence, it is necessary to create an in-process monitoring for the nanomachining to allow real-time inspection and process characterizations. This paper reports a sensor-based analytic approach to allow real-time estimations of the AFM-based nanomachining process. The temporal-spectral features of collected acoustic emission (AE) sensor signals are applied to predict the depth morphology of nanomachined trenches under different machining conditions. The experimental case study suggests that the most significant frequency domain information from AE sensor can accurately predict (R-squared value around 92%) the nanomachined depth profile. It, therefore, breaks the current limitation of characterization tools at the nanoscale precision level, and opens up an opportunity to allow real-time estimation for quality inspection of vibration-assisted AFM-based nanofabrication process. 
    more » « less