The ionosphere is a region in the upper atmosphere consisting of charged particles, for example, ions and electrons. The charged particles are created when the neutral atmosphere atoms and molecules are ionized by solar radiations as well as through collisions with high energy particles coming from the Sun. The number density of the charged particles exhibits strong diurnal variations as well as significant day-to-day variability (Fang et al., 2018). The ionospheric density variations can significantly impact the radio-wave communications that rely on reflection of radio waves or the higher frequency radio waves that go through the ionosphere. The gradients in the plasma density can diffract radio signals and cause amplitude or phase fluctuations in the Global Navigation Satellite System (GNSS) signals.
In the equatorial ionosphere, plasma instability conditions often occur during the post-sunset period and lead to the formation of plasma bubbles or plasma irregularities. These bubbles and irregularities show strong density fluctuation and their movements are aligned with the background magnetic field structure (Li et al., 2021). When an electromagnetic wave traverses the ionosphere in the presence of such plasma irregularities, the wave experiences interferences different to those when there is no plasma bubble (Sahai et al., 2004). The resulting impact on the signal amplitude and phase can be so significant that the information carried by the wave may be distorted or lost. Studying the ionosphere irregularities helps us to determine the spatial and temporal variations of these changes, estimate the impact of these fluctuations on radio signals, and mitigate their influence on technologies that we rely on.
The ionosonde is one important ground-based instrument that is used to determine ionospheric properties. It sends up a wave of between 1 and 30 MHz that is then reflected back to the ground (Kalita et al., 2019) at different points in the ionosphere. The reflected wave is recorded and processed. One output format is in terms of a visual map called ionogram. On a typical ionogram, the virtual height is shown on the y axis, whereas the frequency of the wave sent by the ionosonde is shown on the x axis. The virtual height approximates the reflection height by multiplying the speed of light by the time delay between the signal transmission and reception and dividing Abstract Spread-F (SF) is a feature that can be visually observed on ionograms when the ionosonde signals are significantly impacted by plasma irregularities in the ionosphere. Depending on the scale of the plasma irregularities, radio waves of different frequencies are impacted differently when the signals pass through the ionosphere. An automated method for detecting SF in ionograms is presented in this study. Through detecting the existence of SF in ionograms, we can help identify instances of plasma irregularities that are potentially affecting the high-frequency radio-wave systems. The ionogram images from Jicamarca observatory in Peru, during the years 2008-2019, are used in this study. Three machine learning approaches have been carried out: supervised learning using Support Vector Machines, and two neural network-based learning methods: autoencoder and transfer learning. Of these three methods, the transfer learning approach, which uses convolutional neural network architectures, demonstrates the best performance. The best existing architecture that is suitable for this problem appears to be the ResNet50. With respect to the training epoch number, the ResNet50 showed the greatest change in the metric values for the key metrics that we were tracking. Furthermore, on a test set of 2050 ionograms, the model based on the ResNet50 architecture provides an accuracy of 89%, recall of 87%, precision of 95%, as well as Area Under the Curve of 96%. The work also provides a labeled data set of around 28,000 ionograms, which is extremely useful for the community for future machine learning studies. by two, thus giving a number in units of distance (kilometers). Two examples of ionograms as taken by Digital Portable Sounder-4D ionosonde are shown in Figure 1.
The colors in the ionogram represent information of the signal, such as the direction where the echo comes from, the Doppler shift, and mode of the echo. For example, in Figure 1, the red points are for a vertical echo of the ordinary mode and the deep blue is for the ordinary echo from the North-North West direction. The size (geometric area) of each rectangular point encodes the magnitude of the amplitude of the echo (Reinisch, 2009). The black curve is for the ion density profile, which is derived by the automatic tracing algorithm called ARTIST version 4. ARTIST (Automatic Real Time Ionogram Scaler with True Height) is a software program for "scaling" ionospheric parameters-where "scaling" entails calculating or estimating the values for the parameters. ARTIST provides the values for peak frequency in the F2 layer, foF2, and the virtual height of the F layer, h'F. These parameters and values are shown in tabular form on the ionograms. The density profile above the ionospheric peak density is provided by the ARTIST tool since the ionosonde can only detect echoes below the density peak. Figure 1 Multiple echoes commonly appear in ionograms. These echoes are caused in part by multiple reflections of a signal bouncing between the ground and ionosphere (Maruyama, 2002). In addition to these multiple echoes, there is another feature that can appear in nighttime ionogram: Spread-F (SF). When a signal from an ionosonde is sent upward, ideally it is expected that only one main echo is received, but may include multiples of the same echo at double the height. The time delay of the echo return provides information about the height of the layer from which it was reflected. There however are cases where different frequencies are reflected at the same height rather than at different heights as is normally expected. Similarly, there are cases where the same frequency is reflected at multiple heights. In the equatorial ionosphere, these behaviors can be associated with plasma irregularities most commonly observed after the post-sunset period. When these cases arise, they show up on an ionogram as a spread in frequency or virtual height (time). The anomalies are thus called SF owing to their spread appearance on ionograms-the echo is literally spread across many more pixels than those in normal cases. This work particularly focuses on the SF in the low magnetic latitude regions. The SF at the low-latitude regions is typically called equatorial SF.
Another phenomenon associated with plasma bubbles and irregularities is ionosphere scintillation, which typically refers to signals at frequencies above the peak plasma frequency of the ionosphere, for example, GNSS or satellite communication signals. Scintillation represents the random fluctuation in the phase and amplitude of received signals (Cander, 2019). The signals propagate through the irregular plasma density structures and the diffraction and/or refraction leads to signal intensity drops and phase shifts. The returned signals then appear to be scintillated. Imagine a planar wave incident on two ionosphere patches of different densities, one patch of normal background density and another of much lower density. They emerge at different times because the speed of a wave in a given medium is a function of the refractive index of that medium; and the refractive index is a function of plasma density (Wiesemann, 2013). Therefore, the wave will move at different speeds in the low-density plasma bubble than in the normal density region. The difference in duration of travel (and thus phase at exit) will result in the vector sum of the two emerging waves being different from the amplitude of the original waves (e.g., Kintner et al., 2009;Maruyama, 2002). The signal received by an instrument would show fluctuations in amplitudes or phases that are not in the original signal. The result is that a receiver may be unable to accurately extract the information that was sent. For example, GNSS satellites transmit L-band signals (∼1.575 GHz) from a height of about 20,200 km to receivers on the ground. The L-band signal has to cross the ionosphere and is therefore affected by the conditions in the ionosphere. The scintillation of a GNSS signal can lead to position errors with the severity of impact of these errors depending on the application (Kintner et al., 2009).
The identification of equatorial SF (ESF) from ionograms is important because ESF is a sign of large-scale ionospheric irregularities (100s of km). The evolution of plasma irregularities largely depends on the background ionosphere and thermosphere conditions and can potentially lead to small-scale (10s km) irregularities that may affect communication and navigation systems. Several studies have shown a strong correlation between scintillation and spread F occurrences. Shi et al. (2011) showed that the occurrence rate of scintillation of the GPS L-band signal was high during times when Strong Spread F occurrence rate was also high at a low-latitude station of Hainan (109.1°E, 19.5°N; dip latitude 9°N). Liang et al. (2015) showed similar conclusions using data from another low-latitude station at Vanimo (2.7°S, 141.3°E; dip latitude 11°S). Therefore, by identifying instances of Spread F, it is possible to estimate the occurrence rate of scintillation events.
Identifying SF has typically been a manual process. It would take a lot of time to label the thousands of ionograms as having SF or not. This work contributes toward automating SF detection. Specifically, the main goal of this work is developing a machine learning (ML) model that improves upon existing models for automatic SF detection. A few generally data-driven and specifically machine learning methodologies have been carried out for SF identification. Scotto et al. (2018) developed a module for automatically detecting SF from the equatorial region. Their software module complements Autoscala, which is software developed for scaling the ionosonde data in order to get parameters of interest, such as critical frequencies, heights, and maximum useable frequencies. However, in the presence of SF, one can no longer trust the data collected. Scotto et al. (2018) therefore built a numerical model that identifies which data Autoscala must exclude. That is, once an SF has been detected, the ionosonde data are no longer used for calculating ionospheric parameters. They used 198 samples from Tucuman station (26.9°S, 294.6°E) to determine a threshold for splitting between SF and no SF. After obtaining the appropriate threshold, they tested using 7,649 manually scaled ionograms for every hour of the year 2016. They obtained a true positive ratio of 81.55%.
Another model has been built with data from a low-latitude station at São José dos Campos (23.2°S, 45.9°W, dip latitude 17.6°S) and an equatorial station at Palmas (10.2°S, 48.2°W, dip latitude 5.5°S). This model was based on fuzzy relation. The experimenters used 4,320 ionograms collected over a period of 30 days between 6 p.m. and 6 a.m. local time. The months studied are January and February 2004, which are summer for Brazil. The module was able to detect the presence of SF in about 92% of the cases that had been labeled as having SF (Pillat et al., 2015). Another study has utilized the ML technique called decision trees to establish a model to identify Spread F (Lan et al., 2018). The inputs for the decision tree model are 47,711 ionograms recorded at Puer station (22.7°N, 101.5°E; Geomagnetic latitude: 12.8°N) in Yunnan province of China. The year 2015 was chosen in this study. Data from the first half of every month-January to December-to build their decision tree and then the data from the latter half of each were used for validation. In order to test their model, 41,471 ionograms from 2015 are used-the test ionograms were different from the training set. Then, they also used 7,367 ionograms from July 2016 because this was a month with significant amount (1,428) of SF. This study obtained 89% accuracy on correctly identifying cases with SF.
These results show some success. However, the models have only been tested for the same location from which the data for training the model were obtained; the models thus may not work for other locations, such as Jicamarca, Peru. While Lan et al. (2018) attempted testing with data from a different year, only 1 month was tested. Machine learning models can be very dependent on the data used for training. This means that one can have an extremely good model based on carefully selected data. It is thus important to include data that are as temporally diverse as possible so that the model does not simply learn the features that apply for one particular year or season. In this work, a large data set with over 28,000 ionograms near equinoxes spanning multiple years (2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015)(2016)(2017)(2018)(2019) is utilized to provide a more comprehensive coverage for SF events.
It is also worth emphasizing that the Jicamarca Radio Observatory (JRO, 12°S, 76.8°W, and dip latitude 1°N) in Peru sits close to the magnetic equator and is the premier scientific facility in the world to study the equatorial ionosphere. JRO hosts multiple radars and ground-based instruments that are used to detect ionosphere properties. It has a collection of scatter radars consisting of three 1.5 MW transmitters and incoherent scatter radar (Milla, 2017). There are also many magnetometers distributed around Peru with which scientists can study changes in the magnetic field associated with ionospheric currents. Some of these measurements are extremely useful and can be combined with ionosonde data to facilitate extensive studies on ionosphere irregularities and the background conditions that lead to the irregularities. As mentioned before, none of the existing machine learning models were built or tested with ionograms from JRO. Therefore, establishing a machine learning model that works with the long-term JRO data and making the labeled data set publicly available are important objectives of this study. Following the descriptions in this paper will also allow other researchers to easily build upon this work.
We describe the data used in this work in Section 2. Section 3 discusses the machine learning methods used in the study, namely the Convolutional Neural Networks (CNNs). The methods include an autoencoder architecture and other architectures based on ResNet50, VGG16, and InceptionV3 architecture from the public domain. The experiments and results are then shown in Section 4, together with a discussion of the results. In Section 5, key findings of the work and potential future work are summarized.
Various long-term observations from multiple instruments from JRO have provided tremendous information and novel insight for understanding the equatorial ionosphere. The ionosonde at JRO (station ID: JI91J) has a probe frequency range between 0.5 and 20 MHz. The calculated virtual height is between 80 and 1,320 km. There are two temporal data collection modes: one mode is to collect every 15 min and the other is every 5 min. Data from both cadences are used in this study. In one of the experiments, the 5-min cadence data were excluded in order to assess the impact that data temporal resolution has on the ML model.
The ionograms are obtained from a NOAA data repository (https://data.ngdc.noaa.gov/instruments/remote-sens- ing/active/profilers-sounders/ionosonde/). Equatorial SF occurs more often during the chosen months of March, April, August, and September (Chapagain et al., 2009). All the ionograms used in this study are from the years 2008-2019 (for 2019 only March). The period between 2008 and 2019 corresponds to the solar cycle 24. One reason for choosing a season where equatorial SF is expected the most is to have sufficient data for both the Normal and SF classes. If the data were not so carefully chosen, there would be a bias since normally there are far more normal ionograms than there are those with SF.
Equatorial SF is a primarily nighttime phenomenon (Chapagain et al., 2009) though it has also been shown to occur, in some rare cases, during the day (Jiang et al., 2016). Therefore, the ionograms from 19:00 local time (LT) of present day to 05:00 LT the following day are chosen in this study. The ionograms were then manually classified and placed into three categories: SF, Normal, and Unsure. SF refers to samples that show SF and Normal refers to those that do not show SF. The SF discussed in this work is the equatorial Spread F. The Unsure samples were those where it was not clear to the human doing the manual classification whether the sample contained SF or not. There are three distinct features of the unsure samples. The largest group is of those where there was too little information for even a human to make sense of it; there were either no echoes at all or just a very small number. The second group of unsure samples had a lot of other noisy echoes that made it hard for the human labeler to confidently determine whether it could be due to SF or not. Finally, a smaller group had echoes all over the place with no clear patterns. Since the unsure samples were hard to place into either Normal or SF class, they were not used for training, validation, or testing. There were about 4,236 unsure samples compared to around 28,000 used in training, validation, and testing. It should be noted that the months chosen are deliberately those with greater likelihood of SF; for example, the months of May and June have been shown to not have a lot of SF so they are omitted from the data set. The data set is also chosen to coincide with the 24th solar cycle, which was from 2008 to 2019, as illustrated by the F10.7 solar flux index. The F10.7 index results from calculating hourly average of the flux of radiation, whose wavelength is 10.7 cm, issuing from the sun (Petrova et al., 2021). F10.7 has a long historical record and has been a good proxy for ultraviolet radiation while also correlating with sunspot number. The F10.7 annual mean is shown in Figure 2 to show the different phases of the solar cycle 24 from which come the ionograms used in this work.
Monthly distribution of samples for the year 2015 is also shown in Figure 3. The month of March for 2015 has a significantly larger number of SF samples (668) than Normal samples (283). In total, there are 1,292 Normal samples and 1901 SF samples. Such skewness in the data distribution, though mild in this case, is important to notice because it restricts what kinds of models one can use. As an example to show how data skewness can affect models, consider the case of a training data set where 900 out 1,000 ionograms are normal and the other 100 are SF. A simple classification ML model that always predicts that a given ionogram is in the normal class will have a 90% accuracy, which is largely due to the unbalanced training data set. Thus, a well-balanced data set is critical to establish a reliable ML model in this study.
The number of SF events as a function of time for day 94 of 2008, 2014, and 2018 is shown in Figure 4. For 2018, there was 1 SF between 8 and 9 p.m. and 2 SFs between 12 a.m. and 1 a.m. Similarly, for 2014, the largest number of SF occurred between 2 a.m. and 3 a.m.-the maximum number of SF events is 4 when the ionosonde is collecting data every 15 min. For 2008, there was at least one SF during each hour between 8 p.m. and 5 a.m. of the selected day.
This work provides and discusses results that use Convolutional Neural Networks (CNNs). However, prior to trying CNNs, we had also tried Support Vector Machines (SVM). After several trials (using five different kernel function options in MATLAB) with SVMs, their accuracy at detecting the presence of ESF in ionograms remained unacceptably low at around 77% on average. This method was thus abandoned. SVMs are known for being mathematically intuitive and are thus a good point of entry for this kind of classification problems. But given their poor performance, CNNs were chosen for further exploration.
Convolutional neural networks (CNN) are a special neural network configuration that have demonstrated good learning ability when presented with input signals where neighboring elements form some semantic unit or feature; this is the case for images. CNNs have thus been extensively used for image classification and detection tasks (Simonyan & Zisserman, 2015). Convolution is a mathematical operation between two matrices that outputs another matrix. To perform the convolution, a kernel (also called mask), which is a matrix of weights, operates on the input signal. The input signal in this case is the ionogram. The ionogram is represented as a three-dimensional tensor of dimensions height (600 pixels), width (700 pixels), and depth (3 color channels: Red, Blue, and Green).
To perform the convolution, the kernel slides across the input image linearly combining the elements of the ionogram that are under the kernel; each element in the ionogram is multiplied by the corresponding element in the kernel and then the products are summed (Choi et al., 2020), giving an output matrix. Properties of the kernel that need to be decided by the network designer include padding, stride, and kernel size. Padding is used to deal with two issues: to handle the pixels on the edge of the input signal and to make up for the fact that the size of the output matrix from the convolution operation is smaller than the original input matrix. A padding of 2 means that two new rows (often with zeros as the values) are added to the top and bottom of the input and two new columns are added to left and right of the input signal (ionogram). Stride refers to the number of rows and columns to skip when moving the kernel across the image. Finally, kernel size (k pixels by r pixels) refers to the size of the kernel matrix. A small kernel looks at more local features, while a larger kernel looks at more global features. The actual values of the kernel matrix (these are called weights) are what the neural network learns during the training phase; this is the unique feature of CNNs in that one does not need to manually define the kernel weights.
A filter can be thought of as a collection of many kernels-a kernel is the 2D matrix that acts on data from one channel (which need not necessarily be color channel), whereas a filter combines these kernels into one 2D matrix via the element-wise addition. One also adds the bias to the output from the convolution filter. Finally, there is an activation function that maps the convolution output to a desired range. A frequently used activation function is the rectified unit function f(x) = max(0, x), where x would be the value at each pixel of the convolution map obtained after the convolution operation (Stuchi et al., 2020). CNNs require far fewer parameters than a regular fully connected neural network. Therefore, training a CNN can be faster than a regular neural network. Even more interestingly is that the kernel weights are learned so that one does not need to manually determine what features to look for in an image-the CNN determines the abstract features that are important (Choi et al., 2020).
There are three key configurations within the CNN approach that had been explored in this work: an autoencoder architecture that enables learning with little supervision, a simple handcrafted CNN model, and the transfer learning approach that uses a more sophisticated network architecture, which had been trained on publicly available data set of images; the data set did not include any ionogram ionograms. The autoencoder approach only achieved a Precision of 63%, Recall of 69%, and Accuracy of 63%. These results were unsatisfactory. The simple handcrafted model was a 7-layer model used for comparison with the more complex network architectures. The focus was then on customizing public neural network architectures that were the basis for the transfer learning approach.
Transfer learning starts with existing architectures that were trained on data that did not include ionograms and then the pretrained model is fine-tuned with the ionogram images. This has been shown to be an effective way to train a model based on a small data set (Weiss et al., 2016). The original model architectures are well-designed and subsequently trained on very large data sets; they have features that can be useful in other classification problems. This is so because the earlier layers of a neural network can act as feature extractors (Stuchi et al., 2020) and can thus recognize patterns that exist across many images, for example, edges and contrasts. Edges exist in images of human faces, cars, and ionograms. Thus, the same model can detect this feature in the many possible scenarios.
The following architectures were tested in this study: VGG16, InceptionV3, and ResNet50. These community models were slightly modified and fine-tuned for the task of classifying SF. The training was carried out using Graphics Processing Units (GPUs) partly through the Google Colaboratory Platform as well as primarily with NVIDIA GeForce GTX 1080i GPUs at the Machine Learning and Data Analytics laboratory under the School of Electrical and Electronic Engineering at Nanyang Technological University Singapore. If training had been done on a specific GPU, evaluation and inference were all done using the on that same GPU.
The VGG16 network architecture was introduced in year 2014. The 16 in VGG stands for 16 weight layers. It was trained on over 1 million images divided across 1,000 classes. In 2014, it achieved 92.7% top-5 test accuracy on the ImageNet data set (Jia et al., 2009). This architecture has convolutional layers that use the rectified linear unit function for introducing nonlinearity. At the top, it has two fully connected layers with each of the two layers having 4,096 neurons. Then, the output is wired to another fully connected with 1,000 neurons (Simonyan & Zisserman, 2015). The 1,000 is because VGG16 was being trained to classify inputs into the 1,000 classes, which are part of the ImageNet Large Scale Visual Recognition Challenge. Finally, this fully connected layer connects to a softmax layer. The nodes at the softmax layer implement the softmax function, which is defined as
where σ is the softmax function, z is the input, the summation (j = 1 to j = K) is over the number of nodes in that layer, and i denotes the ith node. After applying the softmax function, each node will output one real number between 0 and 1. The sum of the outputs from all nodes of this layer has to be equal to 1; so if a certain node were to output 1, then the rest of the nodes will have to output 0. Since each node represents one class, then when node k has the highest value (i.e., σ(z) k is max), the interpretation is that the network has predicted the input as belonging to class k. For a more concrete example, suppose there is fully connected layer with seven neurons, representing seven classes, then when softmax function is applied the result may be:
outFCAfterSoftmax = [0.00, 0.00, 0.63, 0.19, 0.14, 0.025, 0.012]
This identifies the third output as highest. One of the reasons for this softmax layer mapping is to allow interpretation as probability. It can be said that there is a 63% chance that the input belongs to class number 3, and only a 1.2% chance of the input belonging to class 7. Similarly, if there were only two classes (i.e., the softmax layer would have 2 neurons), then the output might be:
This indicates that the sample belongs to class 1 because, in Equation (3) 0.90 is bigger than 0.10. Specifically for the two-class problem, the softmax layer can instead be replaced by a sigmoid layer with just one neuron, but with similar functionality. With only two classes-SF and normal-as opposed to the 1,000 classes for which VGG16 was designed, the top layer (softmax) with the 1,000 neurons can be replaced with a sigmoid-based layer. The sigmoid function, specifically the logistic function, is:
This function is at the very last layer. When the output is greater than 0.5, the input is said to belong to class A, which is SF. When the output of the sigmoid function is less than 0.5, then the sample ionogram is said to belong to the class of Normal ionograms.
First presented in the year 2015, InceptionV3 is designed to have Inception modules. Each module in the series may consist of applying the following operations to the input of the module (Szegedy et al., 2015):
1. 1 × 1 convolution filters 2. 3 × 3 filter for the medium features spread across few pixels 3. 5 × 5 filter for detecting slightly more global features. A large filter size is computationally more expensive. One of the tricks used is to do the same operation-that is, convolution with a 5 by 5 filter-but using a block that is made up of two layers. The two are a 3 by 3 pixel convolution layer, followed by another 3 by 3 layer that takes the output from the previous 3 by 3 (directly without applying an activation function). 4. Maxpooling, an operation that extracts the maximum value from an input region This module can then be used as one structural unit. For ease of computation, the modules use dimension reduction so that 3 × 3 and 5 × 5 filters are computed after applying 1 × 1 filters (Szegedy et al., 2015); this design allows for a deep model but still having computational efficiency. The same minor architectural modifications, as for VGG16, were made to InceptionV3: remove top layer, add new layer with 1,024 units, dropout layer with 0.05 ratio, and a sigmoid layer. The training is also done on GPUs.
It had been recognized that deeper networks are better able to do classification. However, it was also seen that when the network depth crosses some threshold, it is no longer beneficial to make it deeper; in fact, the network's performance degrades. ResNet was created to aim for a deep network architecture that does not sacrifice performance (He et al., 2016). A key feature of the ResNet50 architecture is that of skipping some layers in a proper way.
In the schematic shown in Figure 5, the input (256-d) is going into the module that consists of three layers. The first layer has 64 different kernels of size 1 by 1. This layer operates directly on the input. The output from this layer goes through a nonlinear function rectified linear unit (relu). The output from the relu operation is then the input to the next layer. The next layer consists of kernels of size 3 pixels by 3 pixels; as in the first layer, there are also 64 such kernels. The output is again nonlinearly mapped using relu whose output is the input to the next layer. This next layer has 256 kernels each of size 1 pixel by 1 pixel. Note that all kernel sizes are implicitly of shape n by m by d where d is the number of channels-where the term channel as used here does not refer to the red, blue, and green color channels. So, if the input is an image of 224 by 224 by 3 (i.e., 224 pixels wide by 224 pixels high by 3 channels), then a 3 by 3 pixel kernel that operates on this image implies that its kernel size is actually 3 by 3 by 3.
For the ResNet50 architecture, the input goes through this module made of three layers. But instead of simply passing the output of the three layers to the following set of layers, ResNet adds the original, unmodified input to the output of the three layers (He et al., 2016). This is a distinguishing feature of ResNet and its variants. Once the output from the three layers is summed with the original input, the relu function can be applied. The output from this relu operation is passed to the next set of layers and so on. In this case, the output is passed to another residual block, consisting of three layers with a similar architecture. This led to three residual blocks of the same kind being concatenated. The output from the three residual blocks is then passed to another type of residual block and so on. In the end, the ResNet50 architecture has 50 kernel layers with four types of residual blocks.
In order to train the models, the Keras API was used. The pretrained model had been trained for 1,000 classes. The modification made was to remove the top layer, which is used for 1000-class classification and replace it with a sigmoid layer. However, before the sigmoid layer, two new layers were added: one fully connected layer of 1,024 units (or neurons) and a dropout layer. The dropout layer has a ratio of 0.05; this means that during the training process, this dropout layer deactivates (i.e., sets to 0), at random, 0.05 or 5% of the weights connecting this layer to the previous layer. Table 1 shows the overall changes made for this model. 1 are defined for the training phase, for this (ResNet50) architecture, and similarly for the other two architectures. The value for the Pretrained Model can be "ResNet50," "InceptionV3," or "VGG16." The optimizer is the algorithm that is used for actually finding the weights that minimize the loss. The concept of optimizer algorithm is similar to how the Newton-Raphson algorithm can be used to find zeros/roots of a polynomial, but instead of finding zeros, the optimizer algorithm helps in finding approximate weight values that minimize the loss or cost function. The cost function quantifies the difference between desired output and the output predicted by the model at each training step. The Keras API comes with several choices for the optimizing algorithm, including "Adagrad," "RMSProp," and "SGD." One also can choose to tune the optimizing algorithm itself-for example, in the third row of Table 1, the hyperparameters learning_rate, beta_1, beta_2, epsilon, and amsgrad are specifically for configuring this Adam optimizer. It has been shown that the choice of the optimizing algorithm can affect the final model, but this variable was kept constant in this work.
The data set for training and validation ionograms has 26,330 samples of which 15,770 have SF and 10,560 are Normal. To get the validation data set, the ionogram samples were randomly split: 24% for validation and the rest for training. This resulted in 20,614 samples for training and 5,716 for validation. All three transfer learning experiments (using VGG16, InceptionV3, and ResNet50 base architectures) use the same ionogram data to create three different ML models.
When training and later comparing models, the parameters in Table 1 are kept the same across models. The ionograms are preprocessed in the same way prior to feeding them into the neural network. First, the original image size is rescaled. The original input images are 700 pixels wide and 600 pixels high. However, most CNN algorithms used in Transfer Learning work best with a smaller input shape. The transfer learning base models used here, VGG16, InceptionV3, and ResNet50, respectively, have input tensor shapes of 224 by 224, 299 by 299, and 224 by 224 with a third channel dimension equal to 3 for all models. However, rather than scaling each ionogram data set to its own input shape, all ionograms were scaled to the same shape of 300 pixels by 300 pixels irrespective of the base model. A sample ionogram that has been scaled down and used as an input to the neutral network is shown in Figure 6, on the left. The key information about the ionogram is still visible despite the rescaling. The advantage in scaling down is that the number of parameters whose values need to be learned are significantly reduced, which results in faster training since the number of computations is proportional to the number of weights that need to be learned. Throughout this work, the input samples are images of the Portable Network Graphics (PNG) format. Any preprocessing or modifications performed on the input were thus done on the images. Besides scaling, the other modification to the input that was tested was to crop out the extra information that is on ionograms; this extra information is given by ARTIST. Such information is useful for the human but did not seem necessarily relevant for SF detection. A set consisting of the same number of samples as the original data set but with the ionograms cropped was created; a cropped sample is shown at the right of Figure 6. Then, a model was trained with these cropped ionograms to check whether the model that results is better at detecting SF.
The metrics used in this work are defined as follows, where positive refers to an ionogram with SF:
1. Accuracy of the model with respect to the training and validation set. Accuracy is a measure of how many samples are correctly put into their expected class. 2. False Negatives-This is the number of samples that are incorrectly identified as not having SF when they in fact do have SF.
Parameter Value Pretrained Model ResNet50 Optimizer tf.keras.optimizers.Adam ( learning_rate = 0.001, beta_1 = 0.9, beta_2 = 0.999, epsilon = 1e-07, amsgrad = False) Neurons at first added dense layer 1,024 Retrained layers from original 0 Dropout fraction for dropout layer 0.05 Input shape 300 by 300 Data sets Training: Y2008 ,09,10,11,12,13,14,16,19 (20,614) Validation: 2017, 2015 (5,716 samples)
Testing: 2018 (2,050 samples)
Changes Made to the ResNet50 Architecture to Customize for the Task of Identifying Spread-F 3. False Positives-This is the number of samples that are incorrectly identified as having SF when they do not have SF. 4. True Positives-This is the number of samples that are correctly identified as having SF. 5. True Negatives-This is the number of samples that are correctly identified as not having SF. tive + False Negative). For example, if it is known that there are 10 SF's and the algorithm is able to identify six of them, then the model recall is 6/10 = 0.6. 9. Loss-the loss (also called cost) function for which the training procedure seeks the minimum. The one used here is the binary cross entropy. Ideally, the loss function should be decreasing as the training proceeds. This is because in training a model, the goal is to find the set of parameters/weights that minimizes the value of the cost function.
During the training and validation phases, the above nine metrics were tracked and used as the basis for deciding which is the better model. We consider the increase in the value of a metric as a sign of learning rather than simply the final value. A model is thus considered to be the best based not just on its final metric value but also on how much it has improved with respect to epoch number. performance. The first experiment involved cropping out the autoscaled and other textual data. The ionograms normally contain extra information that are irrelevant to the echoes, such as foF2 and foF1 on the left side of an ionogram; such information is automatically generated by the autoscaling software (e.g., ARTIST). Since this information is not useful for the training, cropped ionograms are tested and one example is demonstrated in Figure 7. In the second experiment, the ionograms collected at a 5-min cadence were removed, leaving only those collected at 15-min intervals. The aim was to evaluate the impact of different time cadences of ionograms on the ML models.
Figure 9 shows that the model based on cropped data starts out with a higher precision (0.75) than the model based on the full size (non-cropped) ionograms, which starts with a precision of 0.60. The model based on the full ionograms shows much more significant learning such that at the 120th epoch, its precision is 0.86, while that from the cropped data are around 0.86. Based on this, using cropped ionograms does not offer much advantage.
Results from the case with images with 15-min cadence show that the precision starts from 0.74 and increases to around 0.88 at the 120th epoch. While this is slightly better than the model based on the full ionograms, it does not seem like a significant enough difference to justify the preprocessing necessary to isolate 5-min cadence from 15-min ones. However, from a physics point of view, it may be worth further exploring the cause for the difference.
Before these experiments, it was expected that a much more complex model, such as ResNet50, would perform better than a simpler model-though this is not always guaranteed. This study also demonstrates that VGG16 and InceptionV3 did not perform well on identifying the SF event as indicated by the smaller change in metric values as a function of training epochs. Since all these models had been established with images that did not contain ionograms, it may be that their great skill at distinguishing features among the images they had seen comes at some cost; they may lack the ability to generalize. There were no ionograms in the ImageNet data set, which had been used to train the ResNet50, VGG16, and InceptionV3 models. Transfer learning works best when there are similarities between the original data set and the new smaller data set-that is, the smaller the difference between samples used in transfer learning and the samples used in the original images, the more effective the transfer learning is (Azizpour et al., 2015). Additionally, network parameters, such as width and depth as well as the choice of the optimization algorithm, seem to affect the efficacy of the transfer learning (Azizpour et al., 2015). These factors may explain why some of the models used here did not perform well on the task of SF prediction.
The data sets used in this work were split into three: the training, the validation, and the test set.
During the training phase, the training data set is used to adjust the weights that make the model. The validation set is useful for checking how well the model performs on an external data set-the validation set is itself not used in adjusting the weights. All the three CNN models therefore saw the training and validation set. But once the best performing model had been determined, only this selected model saw the test set. The test set used for this work consists of 2,050 ionograms from 2018; note on the other hand that training and validation data sets included ionograms from 2008 to 2017 and 2019 and not 2018. The idea behind using a completely different year is to ensure that the test set is very different from the validation set. Of the 2,050 test samples, there are 1,342 SF samples and 708 Normal samples. Using the full ionograms (i.e., not cropped) and the model based on the ResNet50 architecture, the values for the metrics are shown in Table 2. Out of the 1,342 SF samples, the trained model can correctly identify 1,170 samples (true positives). Similarly, out of 708 Normal samples, the algorithm can correctly classify 649 samples (true negatives). There are only 59 samples, which are incorrectly classified as having Spread F when they do not show SF (false positives). There are 172 samples that the model identifies as not having SF when they in fact do (false negatives).
Not all metrics are of equal importance when the system is to be deployed; it depends on the application. If the downstream applications would be severely affected by an error of identifying SF as Normal, then one would give much greater weight to Recall than, for example, Precision. The system would then be built to aggressively identify as many SF as possible, likely resulting in inclusion of some Normal cases as SF. One application that will be built on top of this model is a forecasting model. This forecasting model will use a time series of ionograms in order to construct an ionogram for a future time and classify that constructed ionogram as having SF or not. In this way, one may forecast the SF occurrence. In such an application, it is important to catch as many SF as possible since misclassifying a Normal sample as SF may result in system operators preparing to correct for issues that SF may cause. On the other hand, classifying an SF event as Normal may result in failure to anticipate disruption.
The results in Table 2, while good, may be further improved. We do not know of any other CNN-based model for the SF detection on Jicamarca data with which to compare our metrics, but one may compare with Lan et al. (2020) whose CNN model, based on Wuhan data, had an accuracy of 95%. One cause for the relatively subpar performance of the model presented in this work may be the uncertainty introduced due to the Unsure samples. It was mentioned earlier that aside from training, validation, and test sets, there was also a portion of ionograms that were not classified because the human classifier was too uncertain about the class membership of these ionograms. The goal for setting aside Unsure samples was to ensure only those samples that are clearly SF or Normal are used for building and testing the model. However, since the difference between SF and Normal cases is not as clear as the difference between, for example, a car and a dog, there may still be some Unsure samples that slipped into the training and testing data sets. These may be the cause for the asymptotes noticed during training as well as the performance on the test set. A next step in this work is to build an online web application that retrieves ionograms from Jicamarca and classifies them as SF or Normal. To mitigate some of the issues expected with the likely presence of Unsure samples, a filter will be applied during deployment. This filter will remove as many Unsure samples as possible so that the SF classifier only sees mostly samples that can be classified into SF or Normal classes.
The performance range of the model is relatively good. Assuming that the metrics are normally (Gaussian) distributed, confidence intervals can be obtained (Hazra, 2017)
where n is the number of samples and z is the value from the standard normal distribution corresponding to the desired confidence interval. There are n = 2,050 samples in the test set.
For a 95% significance level, z = 1.96
𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 1.96 * ( 0.89 * (1 -0.89) 2050 ) 1 2 = 0.014
This can be interpreted as saying that at 95% significance level, the "true" accuracy of the model is 0.89 ± 0.014 (uncertainty = 0.014) or between 0.88 and 0.90. This should show that the expected performance of the model is within a reasonable range-mostly above 88%.
To summarize our results, several approaches of ML techniques were carried out in this study. One approach used SVM, while the other is based on CNNs. CNNs with the transfer learning implementation have been described herein. With transfer learning, an existing well-designed architecture was used as the starting point. The model was then modified by removing the top layer then adding a new fully connected layer, a dropout layer, and a sigmoid layer. Performing these same modifications across ResNet50, InceptionV3, and VGG16 showed that ResNet50 is the best based on how much its value for AUC, Precision, and Recall metrics changed with respect to epoch number.
In this study, a total of 28,380 ionograms from Jicamarca and primarily during equinoxes are labeled. There were 17,112 samples with SF events and 11,268 normal samples from 2008 to 2019. The filenames and the corresponding label for each ionogram are available to the community and can be accessed at this link: https:// doi.org/10.21979/N9/BHGVGH.
Two types of ML approaches are adopted to create an automatic algorithm for identifying the SF event from these ionograms: SVM and CNN. This study has primarily discussed models based on the CNN architectures that were fine-tuned in a process of transfer learning. The model based on the ResNet50 architecture appears to be the most effective model based on the evolution of its AUC, which improved from 0.64 to 0.86 in 120 epochs.
The study further investigates the impact that other information, besides the echo dots and traces, has on the final ML model. Results show that the model trained with the cropped images is not significantly better based on these evaluation metrics. The model with only 15-min cadence images also does not show better overall performance vis-à-vis the model trained with both the 15-min and 5 min data. Therefore, one can simply use the full ionograms with the ResNet50 base architecture and with the architectural modifications made in this study (i.e., remove top layer and replace with a dense layer of 1,024 neurons, add a dropout layer with rate of 0.05, and a sigmoid classification layer) to achieve the same results.
The next steps of this study will include creating a web app that streams real-time ionograms from JRO in order to automatically label the real-time images and to establish a catalog of SF events for future use. The ML model developed here significantly reduces the need for the manual process in identifying the SF events and is an extremely important step toward exploring SF forecasting based on time-series ionograms.