Ads1298 Multi Lead Ecg

Deep Neural Network

Antˆonio H. Ribeiroa, b, 1, Manoel Horta Ribeiroa, Gabriela M.M. Paix˜aoa, c, Derick M. Oliveiraa, Paulo R. Gomesa, c, J´essica A. Canazarta, c, Milton P. S. Ferreiraa, c, Carl R. Anderssonb, Peter W. Macfarlaned, Wagner Meira Jr.a, Thomas B. Sch¨onb, 2, and Antonio Luiz P. Ribeiroa, c, 3

Abstract

The role of automatic electrocardiogram (ECG) analysis in clinical practice is limited by the accuracy of existing models. Deep Neural Networks (DNNs) are models composed of stacked transformations that learn tasks by examples. This technology has recently achieved striking success in a variety of task and there are great expectations on how it might improve clinical practice. Here we present a DNN model trained in a dataset with more than 2 million labeled exams analyzed by the Telehealth Network of Minas Gerais and collected under the scope of the CODE (Clinical Outcomes in Digital Electrocardiology) study. The DNN outperform cardiology resident medical doctors in recognizing 6 types of abnormalities in 12-lead ECG recordings, with F1 scores above 80% and specificity over 99%. These results indicate ECG analysis based on DNNs, previously studied in a single-lead setup, generalizes well to 12-lead exams, taking the technology closer to the standard clinical practice.

Preprint. The final version of this paper was published in Nature Communications – volume: 11, article number: 1760 (2020). https://doi.org/10.1038/s41467-020-15432-4. title = {Automatic diagnosis of the 12-lead {{ECG}} using a deep neural network}, author = {Ribeiro, Ant{\^o}nio H. and Ribeiro, Manoel Horta and Paix{\~a}o, Gabriela M. M. and Oliveira, Derick M. and Gomes, Paulo R. and Canazart, J{\’e}ssica A. and Ferreira, Milton P. S. and Andersson, Carl R. and Macfarlane, Peter W. and Meira Jr., Wagner and Sch{\"o}n, Thomas B. and Ribeiro, Antonio Luiz P.},

Doi = {10.1038/S41467-020-15432-4},

url = {https://doi.org/10.1038/s41467-020-15432-4},

C

ardiovascular diseases are the leading cause of death worldwide and the electrocardiogram (ECG) is a major tool in their diagnoses. As ECGs transitioned from analog to digital, automated computer analysis of standard 12-lead electrocardiograms gained importance in the process of medical diagnosis [2, 3]. However, limited performance of classical algorithms [4, 5] precludes its usage as a standalone diagnostic tool and relegates them to an ancillary role [6, 3].

Deep neural networks (DNNs) have recently achieved striking success in tasks such as image classification and speech recognition , and there are great expectations when it comes to how this technology may improve health care and clinical practice [9, 10, 11]. So far, the most successful applications used a supervised learning setup to automate diagnosis from exams.

Supervised learning models, which learn to map an input to an output based on example input-output pairs, have achieved better performance than a human specialist on their routine work-flow in diagnosing breast cancer and detecting retinal diseases from three-dimensional optical coherence tomography scans . While efficient, training DNNs in this setup introduces the need for large quantities of labeled data which, for medical applications, introduce several challenges, including those related to confidentiality and security of personal health information .

A convincing preliminary study of the use of DNNs in ECG analysis was recently presented in . For single-lead ECGs, DNNs could match state-of-the-art algorithms when trained in openly available datasets (e.g. 2017 PhysioNet Challenge data ) and, for a large enough training dataset, present superior performance when compared to practicing cardiologists. However, as pointed out by the authors, it is still an open question if the application of this technology would be useful in a realistic clinical setting, where 12-lead ECGs are the standard technique .

The short-duration, standard, 12-lead ECG (S12L-ECG) is the most commonly used complementary exam for the evaluation of the heart, being employed across all clinical settings, from the primary care centers to the intensive care units. While long-term cardiac monitoring, such as in the Holter exam, provides information mostly about cardiac rhythm and repolarization, the S12L-ECG can provide a full evaluation of the cardiac electrical activity.

This includes arrhythmias, conduction disturbances, acute coronary syndromes, cardiac chamber hypertrophy and enlargement and even the effects of drugs and electrolyte disturbances. Thus, a deep learning approach that allows for accurate interpretation of S12L-ECGs would have the greatest impact.

S12L-ECGs are often performed in settings, such as in primary care centers and emergency units, where health professionals have limited diagnostic abilities in interpreting S12-ECGs [17, 18]. The need for an accurate automatic interpretation is most acute in low and middle-income countries, which are responsible for more than 75% of deaths related to cardiovascular disease , and where the population, often, do not have access to cardiologists with full expertise in ECG diagnosis.

The use of DNNs for S12L-ECG is still largely unexplored. A contributing factor for this is the shortage of full digital S12L-ECG databases, since most recordings are still registered only on paper, archived as images, or stored in PDF format . Most available databases comprise a few hundreds of tracings and no systematic annotation of the full list of ECG diagnoses , limiting their usefulness as training datasets in a supervised learning setting. This lack of systematically annotated data is unfortunate, as training an accurate automatic method of diagnosis from S12L-ECG would be greatly beneficial.

In this paper, we demonstrate the effectiveness of DNNs for automatic S12L-ECG classification. We build a large-scale dataset of labelled S12L-ECG exams for clinical and prognostic studies (the CODE - Clinical Out- comes in Digital Electrocardiology study) and use it to develop a DNN to classify 6 types of ECG abnormalities considered representative of both rhythmic and morphologic ECG abnormalities.

Model Specification And Training

We collected a dataset consisting of 2,322,513 ECG records from 1,676,384 different patients of 811 counties in the state of Minas Gerais/Brazil from the Telehealth Network of Minas Gerais (TNMG) . The dataset characteristics are summarized in Table 1.

The acquisition and annotation procedures of this dataset are described in Methods. We split this dataset into a training set and a validation set. The training set contains 98% of the data. The validation set consists of the remaining 2% (˜50,000 exams) of the dataset and it was used for hyperparameter tuning.

We train a DNN to detect: 1st degree AV block (1dAVb), right bundle branch block (RBBB), left bundle branch block (LBBB), sinus bradycardia (SB), atrial fibrillation (AF) and sinus tachycardia (ST). These 6 abnormalities are displayed in Figure 1.

We used a DNN architecture known as the residual network , commonly used for images, which we here have adapted to unidimensional signals. A similar architecture has been successfully employed for detecting abnormalities in single-lead ECG signals . Furthermore, in the 2017 Physionet challenge , algorithms for

2

Figure 1: (Abnormalities examples) A list of all the abnormalities the model classifies. We show only 3 representative leads (DII, V1 and V6).

506 (61.2 % )

Table 1: (Dataset summary) Patient characteristcs and abnormalities prevalence, n (%). detecting AF have been compared in an open dataset of single lead ECGs and, both the architecture described in and other convolutional architectures [24, 25] have achieved top scores.

The DNN parameters were learned using the training dataset and our design choices were made in order to maximize the performance on the validation dataset. We should highlight that, despite using a significantly larger training dataset, we got the best validation results with an architecture with, roughly, one quarter the number of layers and parameters of the network employed in .

Testing And Perfomance Evaluation

For testing the model we employed a dataset consisting of 827 tracings from distinct patients annotated by 3 different cardiologists with experience in electrocardiography (see Methods). The test dataset characteristics are summarized in Table 1. Table 2 shows the performance of the DNN on the test set. High-performance measures were obtained for all ECG abnormalities, with F1 scores above 80% and specificity indexes over 99%.

We consider our model to have predicted the abnormality when its output — a number between 0 and 1 — is above a threshold. Figure 2 shows the precision-recall curve for our model, for different values of this threshold. Neural networks are initialized randomly, and different initialization usually yield different results. In order to show the stability of the method, we have trained 10 neural networks with the same set of hyperparameters and different initializations. The range between the maximum and minimum precision among these realizations, for different values of thereshold, are the shaded regions displayed in Figure 2. These realizations have micro average precision (mAP) between 0.946 and 0.961, we choose the one with mAP imediatly above the median value of all executions (the one with mAP = 0.951)1. All the analysis from now on will be for this realization of the neural network, which correspond both to the strong line in Figure 2 and to the scores presented in Table 2.

For this model, Figure 2 shows the point corresponding to the maximum F1 score for each abnormality. The threshold corresponding to this point is used for producing the DNN scores displayed in Table 2. The same dataset was evaluated by: i) two 4th year cardiology residents; ii) two 3rd year emergency residents; and, iii) two 5th year medical students. Each one annotated half of the exams in the test set. Their average performances are given, together with the DNN results, in the Table 2 and their precision-recall scores are plotted on Figure 2. Considering the F1 score, the DNN matches or outperforms the medical residents and students for all abnormalities. The confusion matrices and the inter-rater agreement (kappa coefficients) for the DNN, the resident medical doctors and students are provided, respectively, in Supplementary Tables 1 and 2(a).

Additionally, in Supplementary Table 2(b) we compare the inter-rater agreement between the neural network and the certified cardiologists that annotated the test set. A trained cardiologist reviewed all the mistakes made by the DNN, the medical residents and the students, trying to explain the source of the error. The cardiologist had meetings with the residents and students where they together agreed on which was the source of the error. The results of this analysis are given in Table 3.

In order to compare of the performance difference between the DNN and resident medical doctors and students, we compute empirical distributions for the precision (PPV), recall (sensitivity), specificity and F1 score using bootstraping . The boxplots corresponding to these bootstrapped distributions are presented in Supplementary Figure 1.

We have also applied the McNemar test to compare the misclassification 1We couldn’t choose the model with mAP equal to the median value because 10 is even number, hence there is no single middle value.

Dnn

cardio. emerg. stud.

Dnn

cardio. emerg. stud.

Dnn

cardio. emerg. stud.

Dnn

cardio. emerg. stud.

0.873

Table 2: (Performance indexes) Scores of our DNN are compared on the test set with the average performance of: i) 4th year cardiology resident (cardio.); ii) 3rd year emergency resident (emerg.); and, iii) 5th year medical students (stud.). (PPV = positive predictive value)

Dnn

cardio. emerg. stud.

meas.

Noise

unexplain. meas.

Noise

concep. atte. meas.

Noise

concep. atte. meas.

Noise

concep. atte.

5

Table 3: (Error analysis) Present the analysis of misclassified exams. The errors were classified into the following categories: i) measurements errors (meas.)

Were Ecg Interval Measurements Preclude The Given

diagnosis by its textbook definition ; ii) errors due to noise, were we believe that the analyst or the DNN failed due to a lower than usual signal quality; and, iii) other type of errors (unexplain.). Those were further divided, for the medical residents and students, into two categories: conceptual errors (concep.), where our reviewer suggested that the doctor failed to understand the definitions of each abnormality, and attention errors (atte.), where we believe the error could be avoided if the reviewer had been more careful.

distribution of the DNN, the medical residents and the students. Supplementary Table 3 show the p-values of the statistical test. Both analyses do not indicate a statistically significant difference in performance among the DNN and the medical residents and students for most of the classes.

Finally, to asses the effect of how we structure our problem, we have considered alternative scenarios where we use the 2,322,513 ECG records in 90%-5%-5% splits, stratified randomly, by patient or in chronological order. Being the splits used, respectively, for training, validation and as a second larger test set. The results indicate no statistically significant difference between the original DNN used in our analysis and the alternative models developed in the 90%-5%-5% splits. The exception is the model developed using the chronologically order split, for which the changes along time in the telehealth center operation have affected the splits (cf. Supplementary Figure 2).

Discussion

This paper demonstrates the effectiveness of ”end-to-end” automatic S12L-ECG classification. This presents a paradigm shift from the classical ECG automatic analysis methods .

These Classical Methods, Such

using traditional signal processing techniques and then use these features as inputs to a classifier. End-to-end learning presents an alternative to these two-step approaches, where the raw signal itself is used as an input to the classifier which learns, by itself, to extract the features. This approach have presented, in a emergency room setting, performance superior to commercial ECG software based on traditional signal processing techniques Neural networks have previously been used for classification of ECGs both in a classical — feature-based — setup [31, 32] and in an end-to-end learn setup [33, 34, 15]. Hybrid methods combining the two paradigms are also available: the classification may be done using a combination of handcrafted and learned features or by using a two-stage training, obtaining one neural network to learn the features and another to classify the exam according to these learned features .

The paradigm shift towards end-to-end learning had a significant impact on the size of the datasets used for training the models. Many results using classical methods [28, 34, 36] train their models on datasets with few examples, such as the MIT-BIH arrhythmia database , with only 47 unique patients. The most convincing papers using end-to-end deep learning or mixed approaches, on the other hand, have constructed large datasets, ranging from 3,000 to 100,000 unique patients, for training their models [35, 16, 15, 30].

Large datasets from previous work [35, 16, 15], however either were obtained from cardiac monitors and Holter exams, where patients are usually monitored for several hours and the recordings are restricted to one or two leads. Or, consist of 12-lead ECGs obtained in a emergency room setting [38, 30]. Our dataset with well over

Dnn

cardio. emerg. stud.

(F) St

Figure 2: (Precision-recall curve) Show precision-recall curve for our nominal prediction model on the test set (strong line) with regard to each ECG abnormalities. The shaded region show the range between maximum and minimum precision for neural networks trained with the same configuration and different initialization.

Points corresponding the performance of resident medical doctors and students are also displayed, together with the point corresponding to the DNN performance for the same threshold used for generating Table 2. Gray dashed curves in the background correspond to iso-F1 curves (i.e. curves in the precision-recall plane with constant F1 score).

2 million entries, on the other hand, consists of short duration (7 to 10 seconds) S12L-ECG tracings obtained from in-clinic exams and is orders of magnitude larger than those used in previous studies. It encompasses not only rhythm disorders, like AF, SB and ST, as in previous studies , but also conduction disturbances, such as 1dAVb, RBBB and LBBB. Instead of beat to beat classification, as in the MIT-BIH arrhythmia database, our dataset provides annotation for S12L-ECG exams, which are the most common in clinical practice.

The availability of such a large database of S12L-ECG tracings, with annotation for the whole spectrum of ECG abnormalities, opens up the possibility of extending initial results of end-to-end DNN in ECG automatic analysis to a system with applicability in a wide range of clinical settings. The development of such tech- nologies may yield high-accuracy automatic ECG classification systems that could save clinicians considerable time and prevent wrong diagnoses. Millions of S12L-ECGs are performed every year, many times in places where there is a shortage of qualified medical doctors to interpret them. An accurate classification system could help to detect wrong diagnoses and improve the access of patients from deprived and remote locations to this essential diagnostic tool of cardiovascular diseases.

The error analysis shows that most of the DNN mistakes were related to measurements of ECG intervals. Most of those were borderline cases, where the diagnosis relies on a consensus definitions that can only be ascertained when a measurement is above a sharp cutoffpoint. The mistakes can be explained by the DNN failing to encode these very sharp thresholds. For example, the DNN wrongly detecting a SB with a heart rate slightly above 50 bpm or a ST with a heart rate slightly below 100 bpm. Supplementary Figure 3 illustrate this effect. Noise and interference in the baseline are established causes of error and affected both automatic and manual diagnosis of ECG abnormalities. Nevertheless, the DNN seems to be more robust to noise and it made fewer mistakes of this type compared to the medical residents and students. Conceptual errors (where our reviewer suggested that the doctor failed to understand the definitions of each abnormality) were more frequent for emergency residents and medical students than for cardiology residents. Attention errors (where we believe that the error could have been avoided if the manual reviewer were more careful) were present at a similar ratio for cardiology residents, emergency residents and medical students.

Interestingly, the performance of the emergency residents is worse than medical students for many abnormal- ities. This might seem counter-intuitive because they do have less years of medical training. It might, however,

6

be justified by the fact that emergency residents, unlike cardiology residents, do not have to interpret these exams on a daily basis, while medical students still have these concepts fresh from their studies. Our work is perhaps best understood in the context of its limitations. While we obtained the highest F1 scores for the DNN, the McNemar statistical test and bootstrapping suggest that we do not have confidence enough to assert that the DNN is actually better than the medical residents and students with statistical significance.

We attribute this lack of confidence in the comparison to the presence of relatively infrequent classes, where a few erroneous classifications may significantly affect the scores. Furthermore, we did not test the accuracy of the DNN in the diagnosis of other classes of abnormalities, like those related to acute coronary syndromes or cardiac chamber enlargements and we cannot extend our results to these untested clinical situations. Indeed, the real clinical setting is more complex than the experimental situation tested in this study and, in complex and borderline situations, ECG interpretation can be extremely difficult and may demand the input of highly specialized personnel. Thus, even if a DNN is able to recognize typical ECG abnormalities, further analysis by an experienced specialist will continue to be necessary to these complex exams.

This proof-of-concept study, showing that a DNN can accurately recognize ECG rhythm and morpholog- ical abnormalities in clinical S12L-ECG exams, opens a series of perspectives for future research and clinical applications. A next step would be to prove that a DNN can effectively diagnose multiple and complex ECG abnormalities, including myocardial infarction, cardiac chamber enlargement and hypertrophy and less com- mon forms of arrhythmia, and to recognize a normal ECG. Subsequently, the algorithm should be tested in a controlled real-life situation, showing that accurate diagnosis could be achieved in real-time, to be reviewed by clinical specialists with solid experience in ECG diagnosis. This real-time, continuous evaluation of the algo- rithm, would provide rapid feedback that could be incorporated as further improvements of the DNN, making it even more reliable.

The TNMG, the large telehealth service from which the dataset used was obtained , is a natural laboratory for these next steps, since it performs more than 2,000 ECGs a day and it is currently expanding its geographical coverage over a large part of a continental country (Brazil). An optimized system for ECG interpretation, where most of the classification decisions are made automatically would imply that the cardiologists would only be needed for the more complex cases. If such a system is made widely available, it could be of striking utility to improve access to health care in low and middle-income countries, where cardiovascular diseases are the leading cause of death and systems of care for cardiac diseases are lacking or not working well .

In conclusion, we developed an end-to-end DNN capable of accurately recognizing six ECG abnormalities in S12L-ECG exams, with a diagnostic performance at least as good as medical residents and students. This study shows the potential of this technology, which, when fully developed, might lead to more reliable automatic diagnosis and improved clinical practice. Although expert review of complex and borderline cases seems to be necessary even in this future scenario, the development of such automatic interpretation by a DNN algorithm may expand the access of the population to this basic and useful diagnostic exam.

Dataset Acquisition

All S12L-ECGs analyzed in this study were obtained by the Telehealth Network of Minas Gerais (TNMG), a public telehealth system assisting 811 out of the 853 municipalities in the state of Minas Gerais, Brazil . Since September 2017, the TNMG has also provided telediagnostic services to other Brazilian states in the Amazonian and Northeast regions.

The S12L-ECG exam was performed mostly in primary care facilities using a tele-electrocardiograph manufactured by Tecnologia Eletrnica Brasileira (So Paulo, Brazil) model TEB ECGPC - or Micromed Biotecnologia (Brasilia, Brazil) - model ErgoPC 13. The duration of the ECG recordings is between 7 and 10 seconds sampled at frequencies ranging from 300 to 600 Hz. A specific software developed in-house was used to capture ECG tracings, to upload the exam together with the patients clinical history and to send it electronically to the TNMG analysis center. Once there, one cardiologist from the TNMG experienced team analyzes the exam and a report is made available to the health service that requested the exam through an online platform.

January 2014) in the in-house software since December 2017. The analysis program was used to automatically identify waves and to calculate axes, durations, amplitudes and intervals, to perform rhythm analysis and to give diagnostic interpretation [42, 29]. The Uni-G analysis program also provides Minnesota codes , a standard ECG classification used in epidemiological studies . Since April 2018 the automatic measurements are being shown to the cardiologists that give the medical report. All clinical information, digital ECGs tracings and the cardiologist report were stored in a database. All previously stored data was also analyzed by Uni-G software in order to have measurements and automatic diagnosis for all exams available in the database, since the first recordings. The CODE study was established to standardize and consolidate this database for clinical and

7

epidemiological studies. In the present study, the data (for patients above 16 years old) obtained between 2010 and 2016 was used in the training and validation set and, from April to September 2018, in the test set.

Labelling Training Data From Text Report

For the training and validation sets, the cardiologist report is available only as a textual description of the abnormalities in the exam. We extract the label from this textual report using a three-step procedure. First, the text is pre-processed by removing stop-words and generating n-grams from the medical report. Then, the Lazy Associative Classifier (LAC) , trained on a 2800-sample dictionary created from real diagnoses text reports, is applied to the n-grams. Finally, the text label is obtained using the LAC result in a rule-based classifier for class disambiguation. The classification model reported above was tested on 4557 medical reports manually labeled by a certified cardiologist who was presented with the free-text and was required to choose among the pre-specified classes. The classification step recovered the true medical label with good results, the macro F1 score achieved were: 0.729 for 1dAVb; 0.849 for RBBB; 0.838 for LBBB; 0.991 for SB; 0.993 for AF; 0.974 for ST.

Training And Validation Set Annotation

To annotate the training and validation datasets, we used: i) the Uni-G statements and Minnesota codes obtained by the Uni-G automatic analysis (automatic diagnosis); ii) automatic measurements provided by the Uni-G software; and, iii) the text labels extracted from the expert text reports using the semi-supervised methodology (medical diagnosis). Both the automatic and medical diagnosis are subject to errors: automatic classification has limited accuracy [4, 5, 3, 6] and text labels are subject both to errors of the practicing expert cardiologists and the labeling methodology.

Hence, we combine the expert annotation with the automatic analysis to improve the quality of the dataset. The following procedure is used for obtaining the ground truth

1. We:

(a) Accept a diagnosis (consider an abnormality to be present) if both the expert and either the Uni-G statement or the Minnesota code provided by the automatic analysis indicated the same abnormality. (b) Reject a diagnosis (consider an abnormality to be absent) if only one automatic classifier indicates the abnormality in disagreement with both the doctor and the other automatic classifier.

After this initial step, there are two scenarios where we still need to accept or reject diagnoses. They are: i) both classifiers indicate the abnormality, but the expert does not; or ii) only the expert indicates the abnormality, whereas none of the classifiers indicates anything.

2. We used the following rules to reject some of the remaining diagnoses: (a) Diagnoses of ST where the heart rate was below 100 (8376 medical diagnoses and 2 automatic diagnoses) were rejected.

(b) Diagnoses of SB where the heart rate was above 50 (7361 medical diagnoses and 16427 automatic diagnosis) were rejected. (c) Diagnoses of LBBB or RBBB where the duration of the QRS interval was below 115 ms (9313 medical diagnoses for RBBB and 8260 for LBBB) were rejected.

(d) Diagnoses of 1dAVb where the duration of the PR interval was below 190 ms (3987 automatic diagnoses) were rejected. 3. Then, using the sensitivity analysis of 100 manually reviewed exams per abnormality, we came up with the following rules to accept some of the remaining diagnoses: (a) For RBBB, d1AVb, SB and ST, we accepted all medical diagnoses. 26033, 13645, 12200 and 14604 diagnoses were accepted in this fashion, respectively.

(b) For AF, we required not only that the exam was classified by the doctors as true, but also that the standard deviation of the NN intervals was higher than 646. 14604 diagnoses were accepted using this rule.

According to the sensitivity analysis, the number of false positives that would be introduced by this procedure was smaller than 3% of the total number of exams.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.