skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Automated RFI Flagging for HI Spectra
Manual flagging of RFI is extremely time-consuming and error-prone. We present a machine learning algorithm which automatically identifies radio frequency interference (RFI) in HI spectra. Our algorithm uses the features of polarization asymmetry (defined as |polA - polB|/[polA + polB] ) along with the skew and standard deviation of each channel over time to evaluate the presence of RFI. The algorithm was tested on hundreds of spectra taken by the Undergraduate ALFALFA Team (UAT) as part of the APPSS survey. It outperforms humans not only in speed, but in visually identifying RFI when it is weak or mimics properties of signals. This work has been supported by NSF grants AST-1211005 and AST-1637339.  more » « less
Award ID(s):
1637339
PAR ID:
10097651
Author(s) / Creator(s):
;
Date Published:
Journal Name:
American Astronomical Society, AAS Meeting
Volume:
233
Page Range / eLocation ID:
245.22
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ; ; ; ; et al (, Monthly Notices of the Royal Astronomical Society)
    Abstract Radio Frequency Interference (RFI) is an ever-present limiting factor among radio telescopes even in the most remote observing locations. When looking to retain the maximum amount of sensitivity and reduce contamination for Epoch of Reionization studies, the identification and removal of RFI is especially important. In addition to improved RFI identification, we must also take into account computational efficiency of the RFI-Identification algorithm as radio interferometer arrays such as the Hydrogen Epoch of Reionization Array grow larger in number of receivers. To address this, we present a Deep Fully Convolutional Neural Network (DFCN) that is comprehensive in its use of interferometric data, where both amplitude and phase information are used jointly for identifying RFI. We train the network using simulated HERA visibilities containing mock RFI, yielding a known “ground truth” dataset for evaluating the accuracy of various RFI algorithms. Evaluation of the DFCN model is performed on observations from the 67 dish build-out, HERA-67, and achieves a data throughput of 1.6× 105 HERA time-ordered 1024 channeled visibilities per hour per GPU. We determine that relative to an amplitude only network including visibility phase adds important adjacent time-frequency context which increases discrimination between RFI and Non-RFI. The inclusion of phase when predicting achieves a Recall of 0.81, Precision of 0.58, and F2 score of 0.75 as applied to our HERA-67 observations. 
    more » « less
  2. (, Radio Science)
    Abstract The utilization of the global navigation satellite systems (GNSS) services in both military and civilian applications as well as for scientific investigation has grown exponentially. However, the increasing reliance on GNSS applications has raised concerns about potential risks from intentional radio frequency interference (RFI) transmitters. RFI significantly affects GNSS's environmental monitoring capabilities by inflating the scintillation index and misleading the scientific community with scintillation indices not attributable to ionospheric dynamic events. Consequently, the existing climatological distribution of GNSS scintillations may require careful reevaluation, as it may not adequately filter out RFI induced scintillations. Thus, characterizing the global RFI occurrence regions and developing real‐time detection capabilities to mitigate its effects is critically important. Leveraging GNSS measurements from ground stations and six COSMIC‐2 satellite constellations, we have developed a technique to detect RFI events and identify RFI active regions. Additionally, for the first time, we have implemented techniques that differentiate RFI associated scintillations from scintillations caused by ionospheric turbulence. 
    more » « less
  3. ; ; ; ; ; ; ; ; ; (, The Astrophysical Journal)
    Abstract We present deep upper limits from the 2014 Murchison Widefield Array Phase I observing season, with a particular emphasis on identifying the spectral fingerprints of extremely faint radio frequency interference (RFI) contamination in the 21 cm power spectra (PS). After meticulous RFI excision involving a combination of theSSINSRFI flagger and a series of PS-based jackknife tests, our lowest upper limit on the Epoch of Reionization (EoR) 21 cm PS signal is Δ2≤ 1.61 × 104mK2atk= 0.258h Mpc−1at a redshift of 7.1 using 14.7 hr of data. By leveraging our understanding of how even fainter RFI is likely to contaminate the EoR PS, we are able to identify ultrafaint RFI signals in the cylindrical PS. Surprisingly this signature is most obvious in PS formed with less than 1 hr of data, but is potentially subdominant to other systematics in multiple-hour integrations. Since the total RFI budget in a PS detection is quite strict, this nontrivial integration behavior suggests a need to more realistically model coherently integrated ultrafaint RFI in PS measurements so that its potential contribution to a future detection can be diagnosed. 
    more » « less
  4. ; ; (, American Astronomical Society, AAS Meeting)
    The number of extragalactic sources of HI detected in radio surveys is growing exponentially. It will soon no longer be feasible for human researchers to individually fit spectra. We present algorithms for automatically extracting the typical parameters of interest for the 21 cm HI line—recessional velocity, velocity width, and integrated flux—using neural networks. Features are produced by convolving spectra with templates generated with the Busy Function. We present the results of fitting hundreds of spectra with many different shapes, and at a wide range of signal to noise ratio. Additionally, we compare with prior methods of automated source extraction. This work has been supported by NSF grants AST-1211005 and AST-1637339. 
    more » « less
  5. ; ; ; ; ; ; ; ; (, RAS Techniques and Instruments)
    Abstract Radio-frequency interference (RFI) is becoming an increasingly significant problem for most radio telescopes. Working with Green Bank Telescope data from PSR J1730+0747 in the form of complex-valued channelized voltages and their respective high-resolution power spectral densities, we evaluate a variety of statistical measures to characterize RFI. As a baseline for performance comparison, we use median absolute deviation (MAD) in complex channelized voltage data and spectral kurtosis (SK) in power spectral density data to characterize and filter out RFI. From a new perspective, we implement the Shapiro–Wilks (SW) test for normality and two information theoretical measures, spectral entropy (SE) and spectral relative entropy (SRE), and apply them to mitigate RFI. The baseline RFI mitigation algorithms are compared against our novel RFI detection algorithms to determine how effective and robust the performance is. Except for MAD, we find significant improvements in signal-to-noise ratio through the application of SE, symmetrical SRE, asymmetrical SRE, SK, and SW. These algorithms also do a good job of characterizing broad-band RFI. Time- and frequency-variable RFI signals are best detected by SK and SW tests. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email