skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Automated RFI Flagging for HI Spectra
Manual flagging of RFI is extremely time-consuming and error-prone. We present a machine learning algorithm which automatically identifies radio frequency interference (RFI) in HI spectra. Our algorithm uses the features of polarization asymmetry (defined as |polA - polB|/[polA + polB] ) along with the skew and standard deviation of each channel over time to evaluate the presence of RFI. The algorithm was tested on hundreds of spectra taken by the Undergraduate ALFALFA Team (UAT) as part of the APPSS survey. It outperforms humans not only in speed, but in visually identifying RFI when it is weak or mimics properties of signals. This work has been supported by NSF grants AST-1211005 and AST-1637339.  more » « less
Award ID(s):
1637339
PAR ID:
10097651
Author(s) / Creator(s):
;
Date Published:
Journal Name:
American Astronomical Society, AAS Meeting
Volume:
233
Page Range / eLocation ID:
245.22
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; (, Publications of the Astronomical Society of Australia)
    Radio-frequency interference detection and flagging is one of the most difficult and urgent problems in 21 cm Epoch of Reionisation research. In this work, we present χ2 from redundant calibration as a novel method for RFI detection and flagging, demonstrating it to be complementary to current state-of-the-art flagging algorithms. Beginning with a brief overview of redundant calibration and the meaning of the χ2 metric, we demonstrate a two-step RFI flagging algorithm which uses the values of this metric to detect faint RFI. We find that roughly 27.4% of observations have RFI from digital television channel 7 detected by at least one algorithm of the three tested: 18.0% of observations are flagged by the novel χ2 algorithm, 16.5% are flagged by SSINS, and 6.8% are flagged by AOFlagger (there is significant overlap in these percentages). Of the 27.4% of observations with detected DTV channel 7 RFI, 37.1% (10.2% of the total observations) are detected by χ2 alone, and not by either SSINS or AOFlagger, demonstrating a significant population of as-yet undetected RFI. We find that χ2 is able to detect RFI events which remain undetectable to SSINS and AOFlagger, especially in the domain of long-duration, weak RFI from digital television. We also discuss the shortcomings of this approach and discuss examples of RFI which seems undetectable using χ2 while being successfully flagged by SSINS and/or AOFlagger. 
    more » « less
  2. ; ; ; ; ; ; ; ; ; ; et al (, Monthly Notices of the Royal Astronomical Society)
    Abstract Radio Frequency Interference (RFI) is an ever-present limiting factor among radio telescopes even in the most remote observing locations. When looking to retain the maximum amount of sensitivity and reduce contamination for Epoch of Reionization studies, the identification and removal of RFI is especially important. In addition to improved RFI identification, we must also take into account computational efficiency of the RFI-Identification algorithm as radio interferometer arrays such as the Hydrogen Epoch of Reionization Array grow larger in number of receivers. To address this, we present a Deep Fully Convolutional Neural Network (DFCN) that is comprehensive in its use of interferometric data, where both amplitude and phase information are used jointly for identifying RFI. We train the network using simulated HERA visibilities containing mock RFI, yielding a known “ground truth” dataset for evaluating the accuracy of various RFI algorithms. Evaluation of the DFCN model is performed on observations from the 67 dish build-out, HERA-67, and achieves a data throughput of 1.6× 105 HERA time-ordered 1024 channeled visibilities per hour per GPU. We determine that relative to an amplitude only network including visibility phase adds important adjacent time-frequency context which increases discrimination between RFI and Non-RFI. The inclusion of phase when predicting achieves a Recall of 0.81, Precision of 0.58, and F2 score of 0.75 as applied to our HERA-67 observations. 
    more » « less
  3. (, Radio Science)
    Abstract The utilization of the global navigation satellite systems (GNSS) services in both military and civilian applications as well as for scientific investigation has grown exponentially. However, the increasing reliance on GNSS applications has raised concerns about potential risks from intentional radio frequency interference (RFI) transmitters. RFI significantly affects GNSS's environmental monitoring capabilities by inflating the scintillation index and misleading the scientific community with scintillation indices not attributable to ionospheric dynamic events. Consequently, the existing climatological distribution of GNSS scintillations may require careful reevaluation, as it may not adequately filter out RFI induced scintillations. Thus, characterizing the global RFI occurrence regions and developing real‐time detection capabilities to mitigate its effects is critically important. Leveraging GNSS measurements from ground stations and six COSMIC‐2 satellite constellations, we have developed a technique to detect RFI events and identify RFI active regions. Additionally, for the first time, we have implemented techniques that differentiate RFI associated scintillations from scintillations caused by ionospheric turbulence. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; (, The Astrophysical Journal)
    Abstract We present deep upper limits from the 2014 Murchison Widefield Array Phase I observing season, with a particular emphasis on identifying the spectral fingerprints of extremely faint radio frequency interference (RFI) contamination in the 21 cm power spectra (PS). After meticulous RFI excision involving a combination of theSSINSRFI flagger and a series of PS-based jackknife tests, our lowest upper limit on the Epoch of Reionization (EoR) 21 cm PS signal is Δ2≤ 1.61 × 104mK2atk= 0.258h Mpc−1at a redshift of 7.1 using 14.7 hr of data. By leveraging our understanding of how even fainter RFI is likely to contaminate the EoR PS, we are able to identify ultrafaint RFI signals in the cylindrical PS. Surprisingly this signature is most obvious in PS formed with less than 1 hr of data, but is potentially subdominant to other systematics in multiple-hour integrations. Since the total RFI budget in a PS detection is quite strict, this nontrivial integration behavior suggests a need to more realistically model coherently integrated ultrafaint RFI in PS measurements so that its potential contribution to a future detection can be diagnosed. 
    more » « less
  5. ; ; (, American Astronomical Society, AAS Meeting)
    The number of extragalactic sources of HI detected in radio surveys is growing exponentially. It will soon no longer be feasible for human researchers to individually fit spectra. We present algorithms for automatically extracting the typical parameters of interest for the 21 cm HI line—recessional velocity, velocity width, and integrated flux—using neural networks. Features are produced by convolving spectra with templates generated with the Busy Function. We present the results of fitting hundreds of spectra with many different shapes, and at a wide range of signal to noise ratio. Additionally, we compare with prior methods of automated source extraction. This work has been supported by NSF grants AST-1211005 and AST-1637339. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email