Cardiovascular diseases are the leading cause of death in the USA [7]. An electrocardiogram (ECG) records the electrical activity of the heart, thereby providing the summative evaluation of the cardiac electrical activity. It has been estimated that up to 300 million ECGs are recorded annually in Europe alone [23], these enormous amounts of ECG data highlights the importance of computer-aid interpretation. A high-accuracy computer-aid interpretation can save expert clinicians considerable time and efforts, as well as reducing the number of misdiagnoses.
Deep neural networks (DNN) [37], inspired by information processing and distributed communication nodes in biological systems, has been receiving massive interest in both academia and industry for a decade. They are computational models comprising of multiples layers, in which output of a layer is the input of the successive layer. The hierarchy of layers enables the network to learn the increasingly abstract, higher-level representations of the input data. DNNs have been showing their dominating performances in various intelligent tasks including biomedical [8] and health informatics [3] [17]. In the last decade, various DNNs-based methods have been employed in ECG-based automatic arrhythmia classification. Convolutional Neural Networks (CNNs) is the most favorable method [4] and could be categorized into 2 main groups: 1D CNNs in time series and 2D CNNs on time-frequency spectrograms. The former uses raw ECGs as the input [2] [25] [15] [16], split each ECG signal into multiple smaller segments which are then classified into labels in prediction step. The second approach focuses on frequency characteristic of the ECG signal, using its time-frequency spectrogram as the input of a 2D CNN for classification [12] [14] [36] [35]. Although the CNNsbased approaches have proven to be effective for arrhythmia classification, they suffer following limitations Lack of temporal relationship: Either 1D CNN on time series or 2D CNNs on spectrogram first partitions an ECG signal into a set of 1D segments or 2D spectrograms at different time. Then, a CNN-based network is applied into each 1D segments or 2D spectrograms. There is no mechanism to model the temporal relations between these segments or spectrograms within the same sample coming from one patient. Meta-data is not taken into consideration: ECG signal is presented in a high dimensional space while meta-data is given in a binary number (i.e. gender) or scale (age). Combining a high dimensional space of ECG signal (either times series or spectrogram) and very low dimensional space of meta-data is challenging. Most existing works do not take meta-data into account.
Single module: Most of the existing works is single module, i.e. they target at either time series with 1D CNNs frameworks or spectrogram with 2D CNNs frameworks. None of the previous works explores how to fuse multiple modules to inherit the merits from both time series and spectrogram.
To address the aforementioned limitations, we proposed a multi-module Recurrent Convolution Neural Networks (RC-NNs) with transformer encoder. Our network makes use of LSTM [11] as a RNNs and contains four modules as follows. (i) time series module by a 1D RCNNs: In this module, 1D CNNs is first utilized to extract spatio-information from time series segment and LSTM then is used to model the temporal relations between 1D segments. (ii) spectrogram module by a 2D RCNNs: Given an ECG signal, spectrograms at different times are extracted by Short Time Fourier Transform (STFT). A 2D CNNs is used to learn visual representation in spatial domain and a LSTM network is applied to model the tempoinformation between spectrograms within an ECG signal; (iii) meta-data module: An autoencoder to featurize/vectorize the metadata to learn semantic information from both sex and gender; (iv) fusion module: the information from three modules is then fused under a transformer encoder. The entire network is illustration in Fig. 1, each module is presented in one colored block.
Recently, researchers have made major efforts in using DLbased techniques to outperform specialist cardiologist in ECG interpretation. Various ideas have been proposed and Convolutional Neural Network (CNN) has been widely implemented in automatic arrhythmia diagnosis. Yildirim in [39] proposed a novel approach to classify 10-second ECG signal fragments involving 17 classes. Hannun [2] also proposed an end-to-end DL approach to classify 12 rhythm classes using single-lead ECG recordings. Although the work achieved good results, it raised a question if DNN would be useful in a realistic clinical setting, where 12-lead ECGs are the clinical standard. Ribeiro [25] partially addressed the question by presenting a DNN model using 12-lead ECG recordings to classify 6 types of abnormalities. Recurrent neural network (RNN) is also widely applied for arrhythmia diagnosis due to their highly dynamic behavior. Wang [33] proposed a global and updatable classification scheme named Global Recurrent Neural Network (GRNN). Zhang [40] introduced a patient-specific ECG classification using RNN to learn time correlation among ECG signal points. Long short-term memory (LSTM) and its improved version, gated recurrent unit (GRU) are among best DNN candidates in ECG classification [6], [26] and [31].
The aforementioned studies show that an end-to-end DNN can successfully learn complex representative features of ECG signals with less or without excessive dependencies on manual feature extraction. Although the end-to-end approach extracts the "deep features" automatically along the network layers, it neglects one important feature of ECG, the frequency response. The importance of ECG frequency content was recognized from the beginning of the 20 th century [5] [34], and has been studied in various medical research nowadays, such as [28] and [29]. There are various time-frequency transformation methods used for ECG feature extraction, Shorttime Fourier Transform (STFT) is extensively used to achieve ECG's spectral content. To exploit frequency characteristic of ECGs, several efforts have been made. Huang [12] used STFT-based spectrogram and 2D CNN for ECG arrhythmia classification. Each ECG signal is transformed into 2D-image of spectrogram to be subsequently fed into 2D CNN for image classification. Xia [36] [35] proposed using STFT and stationary wavelet transform (SWT) transformations to obtain two-dimensional (2-D) matrix input suitable for deep CNNs. Yildirim [38] proposed a novel wavelet sequence based on deep bidirectional LSTM network model.
The work mentioned above merely focused on ECG signal characteristics. Other important characteristics such as patients' physical state (e.g. age, gender) are not considered [4]. Macfarlane [20] showed that ECG interval measurements, including QRS duration, heart rate, QT dispersion, and selected Q-wave durations are highly influenced by patients' gender, age and race. Therefore, age and gender differences in the ECG should be incorporated into a variety of criteria for ECG interpretation [21]. In this paper, we propose an ECG arrhythmia classification method using multimodality -ECG signal, its frequency response and demographic factors (age and gender).
Our proposed network consisting of four modules, i.e. Time Series Module, Spectrogram Module, Metadata Module and Fusion Module is detailed as follows:
This module aims to extract spatio-temporal information given an ECG time series signal. Let X be a recording ECG signal, X is partitioned into n segments i.e. X = {x i } i=n i=1 . Each segment length is set as l. There are two steps in this module. At the first step, the spatio-feature of each segment is extracted by 1D CNNs. We use function F to represent 1D CNNs, which transforms input segment {x i } i=n i=1 into spatial representation vector:
In the second step, a bidirectional LSTM (BLSTM) [9] is applied to model the temporal relations between 1D segments.
w h e r e f i ∈ R L .I no u re x p e r im e n t s , w es e t l=3 6 0, n=
b e t t e rs em a n t i c a l l yf u s e t h e s ef e a t u r e s .T h efi n a lf e a t u r efi s g e n e r a l l yc om p u t e da s f o l l o w s
w h e r ew 0 , { w i }a n d{ w i }a r e l e a r n tb yT r a n s f o rm e rE n c o d e r [ 3 2 ] . F i n a l l y ,w eem p l o ya f u l l yc o n n e c t e d l a y e rw i t hs o f tm a x t o c o n v e r t t h eo u t p u t sf∈R L i n t o i n t oc a t e g o r i c a lp r o b a b i l i t i e s , where f = θ(f ) and f ∈ R K (6)
IV. EXPERIMENTAL RESULTS Datasets MIT-BIH Arrhythmia dataset consists of 48 thirty minutes long two-lead ECG recordings of 47 subjects. The recordings are digitized using a sampling frequency of 360Hz. The database consists of a total 20 labels. In our experiments, we follow similar experiment setup in [12], i.e. we choose five most common labels i.e. Normal beat (N), Left bundle branch block beat (L), Right bundle branch block beat (R), Premature ventricular contraction (V), Atrial premature beat (A) and Others ( ) as all the other beats. This dataset has 2 leads and lead V5 is used. The split of training:validation is 90:10 and label that occurs most was used as the sample class. Metrics F 1 -score is computed as the harmonic mean of the precision and recall:
Accuracy is the measure of how well the model could perform classification. It is the fraction of correct predictions among the total number of predictions.
In this section, we first examine the effectiveness of RCNNs compared to CNNs as show in Table I CNNs, 2D CNNs, RNNs. In this experiment, different methods are conducted on different number of classes while our approach is conducted on the most common classes i.e. five most common classes and 1 class other for all the other 15 labels.
In this paper, we proposed an ECG arrhythmia classification method based multi-module Recurrent Convolutional Neural Networks (RCNNs). The experiment has been conducted on six classes (five most common classes and the other classes) from MIT-BIH arrhythmia database. Our network takes all time-series, spectrogram and metadata into consideration. The proposed multi-module RCNNs is able to model both spatial information through CNNs and temporal information through LSTM. Our experiments have shown that metadata plays an important role to improve the classification performance. Our multi-module network outperforms most SOTA approach on the same dataset, with F1-score = 99.14%, and accuracy = 98.29%.