An official website of the United States government
We explore the statistical radio frequency interference (RFI) mitigation technique spectral kurtosis (SK) in the context of simulated realistic RFI signals. SK is a per-channel RFI detection metric that estimates the kurtosis of a collection of M power values in a single channel to discern between human-made RFI and incoherent astronomical signals of interest. We briefly test the ability of SK to flag signals with various representative modulation types, data rates, and duty cycles, as well as accumulation lengths M and multi-scale SK bin shapes. Multi-scale SK uses a rolling window to combine information from adjacent time-frequency pixels to mitigate weaknesses in single-scale SK. High data rate RFI signals with significant sidelobe emission are harder to flag, as well as signals with a 50% effective duty cycle. Multi-scale SK using at least one extra channel can detect both the center channel and side-band interference, flagging most of the signal at the expense of larger false positive rates.
more »
« less
Interference detection in radio astronomy: applying Shapiro–Wilks normality test, spectral entropy, and spectral relative entropy
Abstract Radio-frequency interference (RFI) is becoming an increasingly significant problem for most radio telescopes. Working with Green Bank Telescope data from PSR J1730+0747 in the form of complex-valued channelized voltages and their respective high-resolution power spectral densities, we evaluate a variety of statistical measures to characterize RFI. As a baseline for performance comparison, we use median absolute deviation (MAD) in complex channelized voltage data and spectral kurtosis (SK) in power spectral density data to characterize and filter out RFI. From a new perspective, we implement the Shapiro–Wilks (SW) test for normality and two information theoretical measures, spectral entropy (SE) and spectral relative entropy (SRE), and apply them to mitigate RFI. The baseline RFI mitigation algorithms are compared against our novel RFI detection algorithms to determine how effective and robust the performance is. Except for MAD, we find significant improvements in signal-to-noise ratio through the application of SE, symmetrical SRE, asymmetrical SRE, SK, and SW. These algorithms also do a good job of characterizing broad-band RFI. Time- and frequency-variable RFI signals are best detected by SK and SW tests.
more »
« less
- Award ID(s):
- 2307581
- PAR ID:
- 10540059
- Publisher / Repository:
- Oxford University Press
- Date Published:
- Journal Name:
- RAS Techniques and Instruments
- Volume:
- 3
- Issue:
- 1
- ISSN:
- 2752-8200
- Format(s):
- Medium: X Size: p. 535-547
- Size(s):
- p. 535-547
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract We investigate the effectiveness of the statistical radio frequency interference (RFI) mitigation technique spectral kurtosis ( ) in the face of simulated realistic RFI signals. estimates the kurtosis of a collection ofMpower values in a single channel and provides a detection metric that is able to discern between human-made RFI and incoherent astronomical signals of interest. We test the ability of to flag signals with various representative modulation types, data rates, duty cycles, and carrier frequencies. We flag with various accumulation lengthsMand implement multiscale , which combines information from adjacent time-frequency bins to mitigate weaknesses in single-scale . We find that signals with significant sidelobe emission from high data rates are harder to flag, as well as signals with a 50% effective duty cycle and weak signal-to-noise ratios. Multiscale with at least one extra channel can detect both the center channel and sideband interference, flagging greater than 90% as long as the bin channel width is wider in frequency than the RFI.more » « less
-
ABSTRACT To mitigate the effects of Radio Frequency Interference (RFI) on the data analysis pipelines of 21 cm interferometric instruments, numerous inpaint techniques have been developed. In this paper, we examine the qualitative and quantitative errors introduced into the visibilities and power spectrum due to inpainting. We perform our analysis on simulated data as well as real data from the Hydrogen Epoch of Reionization Array (HERA) Phase 1 upper limits. We also introduce a convolutional neural network that is capable of inpainting RFI corrupted data. We train our network on simulated data and show that our network is capable of inpainting real data without requiring to be retrained. We find that techniques that incorporate high wavenumbers in delay space in their modelling are best suited for inpainting over narrowband RFI. We show that with our fiducial parameters discrete prolate spheroidal sequences (dpss) and clean provide the best performance for intermittent RFI while Gaussian progress regression (gpr) and least squares spectral analysis (lssa) provide the best performance for larger RFI gaps. However, we caution that these qualitative conclusions are sensitive to the chosen hyperparameters of each inpainting technique. We show that all inpainting techniques reliably reproduce foreground dominated modes in the power spectrum. Since the inpainting techniques should not be capable of reproducing noise realizations, we find that the largest errors occur in the noise dominated delay modes. We show that as the noise level of the data comes down, clean and dpss are most capable of reproducing the fine frequency structure in the visibilities.more » « less
-
Abstract Radio Frequency Interference (RFI) is an ever-present limiting factor among radio telescopes even in the most remote observing locations. When looking to retain the maximum amount of sensitivity and reduce contamination for Epoch of Reionization studies, the identification and removal of RFI is especially important. In addition to improved RFI identification, we must also take into account computational efficiency of the RFI-Identification algorithm as radio interferometer arrays such as the Hydrogen Epoch of Reionization Array grow larger in number of receivers. To address this, we present a Deep Fully Convolutional Neural Network (DFCN) that is comprehensive in its use of interferometric data, where both amplitude and phase information are used jointly for identifying RFI. We train the network using simulated HERA visibilities containing mock RFI, yielding a known “ground truth” dataset for evaluating the accuracy of various RFI algorithms. Evaluation of the DFCN model is performed on observations from the 67 dish build-out, HERA-67, and achieves a data throughput of 1.6× 105 HERA time-ordered 1024 channeled visibilities per hour per GPU. We determine that relative to an amplitude only network including visibility phase adds important adjacent time-frequency context which increases discrimination between RFI and Non-RFI. The inclusion of phase when predicting achieves a Recall of 0.81, Precision of 0.58, and F2 score of 0.75 as applied to our HERA-67 observations.more » « less
-
ABSTRACT We explore how chromatic radio frequency interference (RFI) flags affect 21-cm power spectrum measurements. We particularly study flags that are coarser than the analysis resolution. We find that such RFI flags produce excess power in the EoR window in much the same way as residual RFI. We use Fast Holographic Deconvolution (fhd) simulations to explain this as a result of chromatic disruptions in the interferometric sampling function of the array. We also use these simulations in conjunction with Error Propagated Power Spectrum with InterLeaved Observed Noise to show that without modifying current flagging strategies or implementing extremely accurate and complete foreground subtraction, 21-cm EoR experiments will fail to make a significant detection. As a mitigation strategy, we find that circumventing the chromatic structure altogether by flagging the entire analysis band when RFI is detected is simple to implement and highly successful. This demands a detection strategy with a low false-positive rate in order to prevent excessive data loss.more » « less
