Gyroscope Balance Assessment

1

CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant

Interpretation Methods

The Critical Assessment of Genome Interpretation Consortium

Abstract

The Critical Assessment of Genome Interpretation (CAGI) aims to advance the state of the art for computational prediction of genetic variant impact, particularly those relevant to disease. The five complete editions of the CAGI community experiment comprised 50 challenges, in which participants made blind predictions of phenotypes from genetic data, and these were evaluated by independent assessors. Overall, results show that while current methods are imperfect, they have major utility for research and clinical applications. Missense variant interpretation methods are able to estimate biochemical effects with increasing accuracy. Performance is particularly strong for clinical pathogenic variants, including some difficult-to-diagnose cases, and extends to interpretation of cancer-related variants. Assessment of methods for regulatory variants and complex trait disease risk is less definitive, and indicates performance potentially suitable for auxiliary use in the clinic. Emerging methods and increasingly large, robust datasets for training and assessment promise further progress ahead.

Introduction

Rapidly accumulating data on individual human genomes hold the promise of revolutionizing our understanding and treatment of human disease.1, 2 Effectively leveraging these data requires reliable methods for interpreting the impact of genetic variation. The DNA of unrelated individuals differs at millions of positions,3 most of which make negligible contribution to disease risk and phenotypes. Therefore, interpretation approaches must be able to identify the small number of variants with phenotypic significance, including those causing rare disease such as cystic fibrosis,4 those contributing to increased risk of cancer5 or acting as cancer drivers,6 those contributing to complex traits such as type II diabetes,7 and those affecting the response of individuals to drugs such as warfarin.8 Identifying the relationship between variants and phenotype can also lead to new biological insights and new therapeutic strategies. Until recently, interpretation of the role of specific variants has either been acquired by empirical observations in the clinic, thus slowly accumulating robust knowledge,9 or by meticulous and often indirect in vitro experiments whose interpretation may be challenging. Computational methods offer a third and potentially powerful approach, and over one hundred have been developed,10 but their power, reliability, and clinical utility have not been established.

The Critical Assessment of Genome Interpretation (CAGI) is an organization that conducts community experiments to assess the state of the art in computational interpretation of genetic variants. CAGI experiments are modeled on the protocols developed in the Critical Assessment of Structure Prediction (CASP) program,11 adapted to the genomics domain. The process is

2

designed to assess the accuracy of computational methods, highlight methodological innovation and reveal bottlenecks, guide future research, contribute to the development of new guidelines for clinical practice, and provide a forum for the dissemination of research results. Participants are periodically provided with sets of genetic data and asked to relate these to unpublished phenotypes. Independent assessors evaluate the anonymized predictions, promoting a high level of rigor and objectivity. Assessment outcomes together with invited papers from participants have been published in special issues of the journal Human Mutation.12, 13 Since CAGI has stewardship of genetic data from human research participants, an essential part of the organizational structure is its Ethics Forum composed of ethicists and researchers, together with patient advocates. Further details are available at https://genomeinterpretation.org/.

Over a period of a decade, CAGI has conducted five rounds of challenges, 50 in all, attracting 738 submissions worldwide (Figure 1, Supplementary Table 1, and Supplementary Text). Challenge datasets have come from studies of variant impact on protein stability14, 15 and functional phenotypes such as enzyme activity,16, 17 cell growth,18 and whole-organism fitness,19 with examples relevant to rare monogenic disease,20 cancer,21 and complex traits.22, 23 Variants in these datasets have included those affecting protein function, gene expression, and splicing, and have comprised single base changes, short insertions or deletions (indels), as well as structural variation. Genomic scale has ranged from single nucleotides to complete genomes, with inclusion of some complementary multiomic and clinical information (Figure 1).

Results are analyzed from three perspectives to provide (i) the clinical community with an assessment of the usefulness and limitations of computational methods, (ii) the biomedical research community with information on the current state of the art for predicting variant impact on a range of biochemical and cellular phenotypes, and (iii) the developers of computational methods with data on method performance with the aim of spurring further innovation. This latter perspective is particularly important because of the recent successes of artificial

Intelligence Approaches In Related Fields.24, 25

The results are presented under a set of themes that have emerged over the decade of CAGI challenges. For each theme, specific examples of performance are provided, based on particular ranking criteria. As always in CAGI, these should not be interpreted as identifying winners and losers––other criteria might result in different selections. Further, these selections were made by authors of this paper, some of whom have been CAGI participants, rather than by independent assessors. However, the examples shown are consistent with the assessors’ earlier rankings.

Results

Biochemical effect predictions for missense variants are strongly correlated with the experimental data, but individual predicted effect size accuracy is limited. The pathogenicity of missense variants implicated in monogenic disease and cancer is often supported by in vitro experiments that measure effects on protein activity, cell growth, or various biochemical properties.26 Thirteen CAGI challenges have assessed the ability of computational methods to estimate these functional effects using datasets from both high- and low-throughput experimental assays, and ten of these have been reanalyzed here. Figure 2 shows selected results for two

3

challenges, each with a different type of biochemical effect. In the NAGLU challenge,16 participants were asked to estimate the relative enzyme activity of 163 rare missense variants in N-acetyl-glucosaminidase found in the ExAC database.27 In the PTEN challenge,14 participants were asked to estimate the effects of a set of 3,716 variants in the phosphatase and tensin homolog on the protein’s stability as measured by relative intracellular protein abundance in a high-throughput assay.28 Detailed descriptions of these and other challenges are provided in Supplementary Materials.

Figure 1: CAGI timeline, participation, and range of challenges. (A) Stages in a round of CAGI, typically extending over two years. Each round includes a set of challenges with similar timelines. (B) Number of participating unique groups (in blue) and submissions (in orange) across CAGI rounds. (C) Genetic scale (top) and phenotypic characterization (bottom) of CAGI challenges. Some challenges belong to more than one category and are included more than once. (D) CAGI challenges, listed by round. C: cancer, CT: complex trait, M: Mendelian, O: other (including multiomics), RD: rare disease. See Supplementary Table 1 for more details.

Challenge development Prediction season Assessment season Dissemination

4

Figure 2: Predicting the effect of missense variants on protein properties: Results for two representative CAGI challenges. Each required estimation of continuous phenotype values, enzyme activity in a cellular extract for NAGLU and intracellular protein abundance for PTEN, for a set of missense variants. Selection of methods is based on the average ranking over four metrics for each participating method: Pearson’s correlation, Kendall’s tau, ROC AUC and truncated ROC AUC; see Methods for definitions. (A) Relationship between observed and predicted values for the selected method in each challenge. “Benign” variants are yellow and “pathogenic” are purple (see text). The diagonal line represents exact agreement between predicted and observed values. Dashed lines show the thresholds for pathogenicity for observed (horizontal) and predicted biochemical values (vertical). For NAGLU, below the pathogenicity threshold, there are 12 true positives (lower left quadrant) and three false positives (upper left quadrant), suggesting a clinically useful performance. Bars below each plot show the boundaries for accuracy meeting the threshold for Supporting (green), Moderate (blue) and Strong (red) clinical evidence, with 95%

5

For both challenges, the relationship between estimated and observed phenotype values shows high scatter (Figure 2A). There is modest improvement with respect to a well-established older method, PolyPhen-2,29 which we consider a baseline. This is a trend consistently seen in other missense challenges (Supplementary Table 2). How much of this improvement is due to the availability of larger and more reliable training sets rather than methodological improvements is unknown. Consistent with the scatter plots, there is moderate agreement between predicted and experimental values as measured by Pearson’s correlation and Kendall’s tau (Figure 2B).

Over all ten analyzed missense functional challenges (Supplementary Table 3, Supplementary Figures 1-6), Pearson’s correlation for the selected methods ranges between 0.24 and 0.84 (average correlation 𝑟̅ = 0.55) and Kendall’s tau ranges between 0.17 and 0.63 (𝜏̅ = 0.40), both showing strong statistical significance over the random model (𝑟̅ = 0, 𝜏̅ = 0). The PolyPhen-2 baseline achieves 𝑟̅ = 0.36 and 𝜏̅ = 0.23. Direct agreement between observed and predicted values is measured by 𝑅! (Supplementary Table 3); see Methods. 𝑅! is 1 for a perfect method and zero for a control method that assigns the mean of the experimental data for every variant.

For NAGLU, the highest 𝑅! achieved is 0.16, but for PTEN it is only -0.09. Over the ten biochemical challenges, the highest 𝑅! value ranges between -0.94 and 0.40, with an average of -0.19. The relatively poorer showing by this criterion compared with Pearson’s and Kendall’s correlation metrics suggests that the methods are often not well calibrated to the experimental value, reflecting the fact that they are rarely designed for predictions of continuous values and scales of this kind. Overall, performance is far above random but modest in terms of absolute accuracy.

Diversity of methods. A diverse set of methods was used to address the biochemical effect challenges, varying in the type of training data, input features, and statistical framework. Most were trained on pathogenic versus presumed benign variants.10, 30 At first glance, a binary classification approach appears ill-suited to challenges which require prediction across a full range of phenotype values. In practice, function and pathogenicity are related,31 and so these methods performed as well as the few trained specifically to identify alteration of function.32 confidence intervals. (B) Two measures of overall agreement between computational and experimental results, for the two selected performing methods and positive and negative controls, with 95% confidence intervals. An older method, PolyPhen-2, provides a negative control against which to measure progress over the course of the CAGI experiments. Estimated best possible performance is based on experimental uncertainty and provides an empirical upper limit positive control. The color code for the selected methods is shown in panel C. (C) ROC curves for the selected methods with positive and negative controls, using estimated pathogenicity thresholds. (D) Truncated ROC curves showing performance in the high true positive region, most relevant for identifying clinically diagnostic variants. The true positive rate and false positive rate thresholds for the Supporting, Moderate, and Strong evidential support is shown for one selected method. (E) Estimated probability of pathogenicity (left y-axis) and positive local likelihood ratio (right y-axis) as a function of one selected method’s score. Predictions with probabilities over the red, blue, and green thresholds provide Strong, Moderate, and Supporting clinical evidence, respectively. Solid lines show smoothed trends. Prior probabilities of pathogenicity are the estimated probability that any missense variant in these genes will be pathogenic. For NAGLU, the probabilities of pathogenicity reach that needed for a clinical diagnosis of “likely pathogenic”. For predicted enzyme activity less than 0.11, the probability provides Strong evidence, below 0.17 Moderate evidence, and below 0.42, Supporting evidence. The percent of variants encountered in the clinic expected to meet each threshold are also shown. Performance for PTEN shows that the results are consistent with providing Moderate and Supporting evidence levels for some variants.

6

Many methods are based on measures that reflect the evolutionary fitness of substitutions and population dynamics, rather than pathogenicity or functional properties. The relationship between fitness, pathogenicity and function is complex, perhaps limiting performance. To partly address this, some methods also exploit functional roles of specific sequence positions, particularly by utilizing UniProtKB annotations and predicted structural and functional

Properties.32-35

Despite the broad range of algorithms, training data, features and the learning setting there is a strong correlation between results of the leading methods (Pearson’s correlation ranges from 0.6 to 0.9), almost always stronger than the correlation between specific methods and experiment (Supplementary Figure 8). The level of inter-method correlation is largely unrelated to the level of correlation with experiment, which varies widely from about 0.24 (CALM1) to 0.6 (NAGLU).

Why correlation between methods is stronger than with experiment is unclear, though it may be affected by the relatedness of functional disruption, evolutionary conservation, and pathogenicity as well as common training data and experimental bias. The assessor for the NAGLU challenge identified 10 variants where experiment disagrees strongly with predicted values for all methods.16 When these are removed, the correlation between the leading methods’ results and experiment increases from 0.6 to 0.73 (Supplementary Figure 8), although it is still lower than the correlation between the two leading methods (0.82), of which, surprisingly, one is supervised34 and the other is not.36 It could be that these 10 variants are cases where the computational methods systematically fail, or it could be that most are some form of experimental artifact. In situations like this, follow-up experiments are needed.

Structure-informed approaches. Some methods use only biophysical input features, and in some cases are trained on the effect of amino acid substitutions on protein stability, rather than pathogenicity or functional impact. Benchmarking suggests that a large fraction of rare disease- causing and cancer driver missense mutations act through destabilization,37, 38 so there is apparently considerable potential for these approaches. These methods have been effective on challenges directly related to stability, being selected as first and second for the PTEN and TMPT protein abundance challenges and first for the Frataxin change of free energy of folding challenge. They have been amongst top performers in a few other challenges, sometimes in combination with sequence feature methods, for example, cancer drivers in CDKN2A and rescue mutations in TP53.15 Generally, however, these methods, along with the structure-based machine learning methods, have not been as successful as expected compared to the methods that are primarily sequence-based. Three factors may improve their performance in future. First, better combination with the sequence methods will likely mitigate the problem of false negatives; that is, pathogenic variants that are not stability related. Second, until recently, use of structure has been restricted by low experimental structural coverage of the human proteome (only about 20% of residues). Because of recent dramatic improvements in the accuracy of computed protein structures,39 variants in essentially all residues in ordered structural regions are now amenable to this approach. Third, better estimation of the effect of missense mutations on stability40 should improve accuracy. An advantage of biophysical and related methods is that they can sometime provide greater insight into underlying molecular mechanisms (Supplementary Figure 13).

Domain-level information had the potential to deliver improved performance in other instances, such as CBS, where the heme-binding domain present in humans was absent from the yeast

7

ortholog, and RAD50, for which assessment showed that restricting predictions of deleteriousness to the specific domain involved in DNA repair would have substantially improved the accuracy of several methods.

Computational methods can substantially enhance clinical interpretation of missense variants. The most direct test of the clinical usefulness of computational methods is to assess their ability to correctly assign pathogenic or benign status for clinically relevant variants. CAGI challenges have addressed this for rare disease variant annotations and for germline variants related to cancer risk.

Results for predicting the biochemical effects of missense mutations inform clinical applications. For some biochemical challenges, it is possible to relate the results to clinical utility of the methods. For NAGLU, some rare variants in the gene cause recessive Sanfilippo B disease. Disease strongly correlates with variants conferring less than 15% enzyme activity,16 allowing variants in the study to be classified as pathogenic or benign on that basis (purple and yellow circles in Figure 2A). Figure 2A shows that 12 out of the 15 variants with less than 15% predicted activity using the selected method also have less than 15% experimental activity, suggesting high positive predictive value and clinical usefulness for assigning pathogenicity. On the other hand, 28 of the 40 variants with measured activity below 15% are predicted to have higher activity so there are also false negatives. For PTEN, information on the relationship to disease is less well established, but data fall into low and high abundance distributions,14 and the assessor suggested a pathogenicity threshold at the distribution intersection.

Performance in correctly classifying variants as pathogenic is often represented by ROC curves (Figure 2C), showing the tradeoff between true positive (y-axis) and false positive (x-axis) rates as a function of the threshold used to discretize the phenotype value returned by a prediction method, and summarized by the area under that curve (AUC). The selected methods return AUCs greater than 0.8 for both challenges. Over all reanalyzed biochemical effect challenges, the top AUC ranges from 0.68 to 1.0, with an average of AUC

(((((( = 0.83, And With High Statistical

significance over a random model (AUC = 0.5). The PolyPhen-2 baseline has AUC

(((((( = 0.74; See

Supplementary Table 3. However, all models fell well short of the empirical limit (AUC

(((((( = 0.98)

estimated from variability in experimental outputs. Because the experimental uncertainties are based on technical replicates, the experimental AUCs are likely overestimated, so it is difficult to judge how much further improvement might be possible. The full ROC curve areas provide a useful metric to measure the ability of the methods to separate pathogenic from other variants. In a clinical setting though, the left portion of the curve is often the most relevant; that is, the fraction of pathogenic variants identified without incurring too high a level of false positives, where the level of tolerated false positives is application dependent. Figure 2D uses truncated ROC curves to show the performance in this region, with the selected methods’ AUCs reaching 0.55 for NAGLU and 0.47 for PTEN. The smaller value for the PTEN truncated ROC curve AUC reflects the higher fraction of false positives at the left of the PTEN scatter plot, particularly those variants predicted to have near-zero protein abundance but with high observed values.

For use in the clinic, the quantity of most interest is the probability that a variant is diagnostic of disease (i.e., can be considered pathogenic), given the available evidence. In addition to the

8

information provided by a computational method, initial evidence is also provided by knowledge of how likely any variant in a particular gene is to be diagnostic of the disease of interest.41 For example, for NAGLU, about 25% of the rare missense variants in ExAC were found to have less than 15% enzyme activity,42 suggesting that there is an approximately 25% prior probability that any rare missense variant found in the gene will be pathogenic (a prior odds of pathogenicity of 1:3). To obtain the desired posterior probability of pathogenicity, which is also the local positive predictive value at the score 𝑠 returned by a method, one can use a standard Bayesian odds

Formulation43

posterior odds of pathogenicity = lr" × prior odds of pathogenicity, where the local positive likelihood ratio, lr", is the slope of the ROC curve at the score value 𝑠; see Methods for a formal discussion.

Figure 2E shows lr" and the posterior probability of pathogenicity for NAGLU and PTEN. For NAGLU, at low predicted enzyme activities the positive likelihood ratio rises sharply to about 15. The corresponding posterior probability is 0.8. For PTEN, lr" reaches a value of about 6.

Using a pathogenicity prior of 0.21 wildtype protein abundance (Supplementary Materials) the corresponding posterior probability of pathogenicity is 0.6. ACMG/AMP sequence variant interpretation guidelines recommend a probability of pathogenicity of 0.99 to label a variant “pathogenic” and 0.90 to label one “likely pathogenic”, the thresholds for clinical action.26, 44, 45 So for these and other biochemical challenges, the computational evidence alone is not sufficiently strong to classify the variants other than as Variant of Uncertain Significance.

However, the clinical guidelines integrate multiple lines of evidence to contribute to meeting an overall probability of pathogenicity threshold, so that it is not necessary (or indeed possible) for computational methods alone to provide a pathogenicity assignment. The guidelines provide rules that classify each type of evidence as Very Strong, Strong, Moderate, and Supporting.26 For example, a null mutation in a gene where other such loss-of-function mutations are known to cause disease is considered Very Strong evidence, while at the other extreme, a computational assignment of pathogenicity for a missense mutation is currently considered only Supporting.

Although these guidelines were originally defined in terms of evidence types, Tavtigian et al.45 have shown that the rules can be approximated using a Bayesian framework, with each threshold corresponding to reaching a specific positive likelihood ratio; e.g., lr" = 2.08 for Supporting evidence when the prior probability is 0.1 (Methods). The resulting thresholds for each level of evidence are shown below the scatter plots in Figure 2A and in the posterior probability of pathogenicity plots in Figure 2E. For NAGLU, for the selected method, predicted enzyme activities lower than 0.11 correspond to Strong evidence, below 0.17 to Moderate, and below 0.42 to Supporting. These thresholds correspond to approximately 31% of rare variants in this gene providing Supporting evidence, 12% Moderate, and 6% Strong. The top performing methods for the ten biochemical missense challenges all reach Supporting, and sometimes Moderate and Strong evidential support (Supplementary Figures 1-6, Supplementary Tables 2 and 3). These results are encouraging in that they suggest this framework can supply a means of quantitatively evaluating the clinical relevance of computational predictions and that under appropriate circumstances, computational evidence can be given more weight than at present.

The next section explores these properties further.

9

Identifying rare disease variants. The ClinVar9 and HGMD46 databases provide an extensive source of rare disease-associated variants against which to test computational methods. A limitation is that most methods have used some or all of these data in training, making it difficult to perform unbiased assessments. The prospective “Annotate All Missense” challenge assessed the accuracy of those predictions on all missense variants that were annotated as pathogenic or benign in ClinVar and HGMD after May 2018 when predictions were recorded, through December 2020, so avoiding training contamination. All predictions directly submitted for the challenge as well as all precomputed predictions deposited in the dbNSFP database47 before May 2018 (dbNSFP v3.5) were evaluated, predictions from a total of 26 groups.

All selected methods, including PolyPhen-2, achieved high AUCs, ranging from 0.85 to 0.92 for separating “pathogenic” from “benign” variants, and only slightly lower values (maximum AUC 0.88) when “likely pathogenic” and “likely benign” are included (Figure 3A). The two metapredictors, REVEL48 and Meta-LR,49 tools that incorporate predictions from multiple other methods, perform slightly better than primary methods, although VEST3 and VEST433 outperformed Meta-LR. There is a substantial improvement over the performance of PolyPhen-2, especially in the left part of the ROC curve (Figure 3B), though as with the biochemical effect challenges, some of that may be due to the availability of larger and more reliable training sets.

Supplementary Table 4 shows slightly higher performance on ClinVar pathogenic variants than HGMD; however, these resources use different criteria for assigning pathogenicity. The lower panels in Figure 3 show positive likelihood ratios and posterior probability of pathogenicity for a selected metapredictor, REVEL.48 With a prior probability of pathogenicity of 0.1 (approximating the prior when examining possible pathogenic variants in one or a few genes in a diagnostic setting; see Methods) 14, 10, and 4% of variants reach the Supporting, Moderate, and Strong evidence thresholds, respectively. With the much smaller prior of 0.01, representative of screening for possible secondary variants, about 6% of variants will provide Supporting evidence and 2% reach Moderate. These estimates are not exact since there may be significant differences between the distribution and properties of variants in these databases and those encountered in the clinic. For example, some genes have only benign variant assignments in the databases, and these might be excluded from consideration in the clinic. Performing the analysis on only genes with both pathogenic and benign assignments slightly reduced performance—highest AUC on the confident set of variants is 0.89 instead of 0.92. The selected methods also change slightly; see Supplementary Figure 9. In spite of this and other possible caveats, the overall performance of the computational methods is encouraging and, as with biochemical effect challenges, suggests that the computational methods can provide greater benefit in the clinic than recognized by the current standards.

Identifying germline cancer risk variants. About a quarter of CAGI experiments have involved genes implicated in cancer (Figure 1), and have included variants in BRCA1, BRAC2, PTEN, TPMT, NSMCE2 (coding for SUMO-ligase), CHEK2, the MRN complex (RAD50, MRE11, and NBS1), FXN, NPM-ALK, CDKN2A and TP53. An additional challenge addressed breast cancer pharmacogenomics. From a cancer perspective, the most informative of these is a challenge provided and assessed by members of the ENIGMA consortium,21 using a total of 321 germline BRCA1/BRCA2 missense and in-frame indel variants. Performance on this challenge was impressively high, with four groups providing submissions that gave AUCs greater than 0.9 and

10

Figure 3: Performance of computational methods in correctly identifying pathogenic variants in the two principal rare disease variant databases, HGMD and ClinVar. The left panels show data for variants labeled as “pathogenetic” in ClinVar and “DM” in HGMD together with “benign” in ClinVar. The right panels add variants labelled as “likely pathogenic” and “likely benign” in ClinVar as well as “DM?” in HGMD. Meta and single method examples were selected on the basis of the average ranking of each method for the ROC and truncated ROC AUCs.

See Supplementary Materials for more details and selection criteria. (A) ROC curves for the selected metapredictors and single methods, together with a baseline provided by PolyPhen-2. Particularly for pathogenic variants alone, impressively high ROC areas are obtained, above 0.9, and there is a substantial improvement over the older method’s performance. (B) Blowup of the left-hand portion of the ROC curves, most relevant to high confident identification of pathogenic variants. Clinical thresholds for Supporting, Moderate and Strong clinical evidence are shown. (C) Local positive likelihood ratio as a function of the confidence score returned by REVEL. Very high values (>100) are obtained for the most confident pathogenic assignments. (D) Local posterior probability of pathogenicity; that is, probability that a variant is pathogenic as a function of the REVEL score for the two prior probability scenarios. For a prior probability of 0.1, typical of a single candidate gene situation (solid line) and database pathogenic and benign variants (left panel) the highest scoring variants reach posterior probability above 0.9, strong enough evidence for a clinical assignment of “likely pathogenic”. In both panels, variants with a score greater than 0.45 provide Supporting clinical evidence (green threshold), and scores greater than 0.8 provide Strong

11

two with AUCs exceeding 0.97. In the other BRCA1/BRCA2 variant challenge, the highest AUC is 0.88 on a total of 10 missense variants. The strong results may reflect the fact these are highly studied genes. More and larger scale challenges with a variety of genes are required in order to draw firm conclusions. Further details of cancer challenges are provided in Supplementary Figure 10 and Supplementary Table 5.

Assessing methods that estimate the effect of variants on expression and splicing is difficult, but results show these can contribute to variant interpretation. Variants that regulate the abundance and isoforms of mRNA either through altered splicing or through altered rates of transcription play a significant role in disease, particularly complex traits. CAGI has included four challenges using data from high-throughput assays of artificial gene constructs, two for splicing and two for expression. For all four, evaluation of the results is limited by a combination of small effect sizes (changes larger than two-fold in splicing or expression are rare in these challenges) and experimental uncertainty, but some interesting properties can be identified.

Splicing. The CAGI splicing challenges used data from high-throughput minigene reporter

Assays.50

The MaPSy challenge asked participants to identify which of a set of 797 exonic single- nucleotide HGMD disease variants affect splicing and by how much. Two experimental assays were available, one in vitro on a cell extract and the other by transfection into a cell line. Only variants that produced a statistically robust change of at least 1.5-fold were considered splicing changes. Figure 4 summarizes the results. The top-performing groups achieved moderately high AUCs of 0.84 and 0.79 and the highest lr" is about 6. Notably, very few variants qualify as significant splicing changes, and there are inconsistences between the two assays, with a number of variants appearing to have a fold change substantially greater than 1.5 in one assay but not the other. Additionally, the experimental noise significantly overlaps with many splicing differences.

For these reasons, it is unclear what maximum AUC could be achieved by a perfect method. The Vex-Seq challenge required participants to predict the extent of splicing change introduced by 1,098 variants in the vicinity of known alternatively spliced exons.50 Supplementary Figure 11 shows that performance was rather weak for identification of variants that increase splicing (top AUC = 0.71), but that may be because many points classified as positive are experimental noise. There are more variants that show a statistically robust decrease in splicing, and prediction performance is correspondingly stronger (top AUC = 0.78). The assessor noted additional

Nuances In Performance.50

The selected high-performing method for both challenges (MMSplice51) decomposes the sequence surrounding alternatively spliced exons into distinct regions and evaluates each region using separate neural networks.51 More detailed splicing results are provided in Supplementary Table 6.

evidence (red threshold). The estimated % of variants encountered in a clinical setting expected to meet each threshold are also shown. For example, about 14% of variants provide Supporting evidence. Dotted lines show results obtained with a prior probability of 0.01.

12

Transcription. Several CAGI challenges have assessed the ability of computational methods to identify single base changes that affect the expression level of specific genes. The CAGI4 eQTL challenge assessed whether methods could find causative variants in a set of eQTL loci, using a massively parallel reporter assay.52 Because of linkage disequilibrium and sparse sampling, variants associated with an expression difference in an eQTL screen are usually not those directly causing the observed expression change. Rather, a nearby variant will be. The challenge had two parts. Participants were asked to predict whether insertion of the section of genomic DNA around each variant position into the experimental construct produced any expression.

Supplementary Figure 12A shows that the top performing methods were effective at this¾the largest AUC is 0.81. The second part of the challenge required participants to predict which variants affect expression levels. Here the results are much less impressive (Supplementary Figure 12B). The scatter plot shows a weak relationship between observed and predicted expression change and Pearson’s or Kendall’s correlation are also small. The best AUC is only 0.66 and the maximum lr" is about 5. Most of the experimental expression changes are small (less than 2-fold), and may be largely experimental noise, partly accounting for the apparent poor performance. But as the scatter plot shows, a subset of the variants with largest effects could not be identified by the top performing method. A combination of experimental and computational factors contributed to poor performance, and more challenges of this sort are needed.

The CAGI5 regulation saturation mutagenesis challenge examined the impact on expression of every possible base pair variant within five disease-associated human enhancers and nine disease-associated promoters.53 As shown in Figure 5, performance is stronger in promoters than Figure 4: Performance of computational methods in identifying variants that affect splicing in the MaPSy challenge. Methods were selected based on the average ranking over three metrics: Pearson’s correlation, Kendall’s tau, and ROC AUC. Scatter plots, Kendell’s tau, and Pearson’s correlation results are shown for in vivo (A, D) and in vitro assays (B, E) separately. The small number of purple points in the scatter plots represent splicing fold changes greater than 1.5-fold. The ROC curve (C) shows performance in variant classification for the two selected methods. The maximum local positive likelihood ratio (lr!, F) may be large enough for use as auxiliary information, see Discussion (solid line is smoothed fit to the data).

13

enhancers and stronger for decreases in expression compared with increases. Fewer variants show experimental increases and these tend to be less well distinguished from noise. Performance for small expression changes is hard to evaluate because of overlap with experimental noise. Nevertheless, the highest AUC for promoter impact prediction is 0.81 while the highest AUC for enhancer impact prediction is 0.79, relatively respectable values. In addition, the scatter plots show that large decreases in expression are well predicted, suggesting the methods are quite informative for the most significant effects.

The CAGI splicing and expression challenges are not as directly mappable to disease and clinical relevance as in some other challenge areas. Variants have been a mixture of common and rare and the use of artificial constructs in high-throughput experiments limits relevance of challenge performance in the whole-genome context. Nevertheless, the results do have potential applications. In complex trait disease genome-wide association studies (GWAS), the variants found to be associated with a phenotype are usually not those causing the effect. Identifying the functional variants is not straightforward, and current regulatory prediction methods can provide hypotheses as to possible effects on expression or splicing.

CAGI participants identified diagnostic variants that were not found by clinical laboratories. A major goal of CAGI is to test the performance of computational methods under as close to clinical conditions as possible. In the area of rare disease diagnosis, four challenges have addressed this by requiring participants to identify diagnostic variants in sets of clinical data. The Johns Hopkins and intellectual disability (ID) challenges employed diagnostic panels, covering a limited set of candidate genes in particular disease areas. As compared with genome-wide data, diagnostic panels inherently restrict the search to only variants belonging to a known set of relevant genes.

For a number of genetically undiagnosed cases in the Johns Hopkins panel, CAGI participants found high-confidence deleterious variants in genes associated with a different disease from that reported, suggesting physicians may have misdiagnosed the symptoms.54 However, because of the clinical operating procedures of the diagnostic laboratory, it has not been possible to further investigate these cases. In the ID panel, some plausible calls were made on novel variants that had not been reported to the patient partly because the majority of standard computational

Methods Returned Assignments Of “Benign”.55

The two SickKids challenges (SickKids4 and SickKids5) were based on whole-genome sequence data for children with rare diseases from the Hospital for Sick Children in Toronto. These are all cases that were undiagnosed by the state-of-the-art SickKids pipeline,20 and so were particularly challenging compared with those normally encountered in the clinic. In the SickKids4 challenge, variants proposed by challenge participants were deemed diagnostic by the referring physicians for two of the cases in part due to matching detailed phenotypes. This was the first instance of the CAGI community directly contributing in the clinic. In SickKids5, two of the highest confidence nominated diagnostic variants provided correct genome-patient matches. While not meeting ACMG/AMP criteria for pathogenicity26 these were considered interesting candidates for further investigation, again potentially resolving previously intractable cases.

14

These clinical challenges required participants to develop full analysis pipelines, including quality assessment for variant calls, proper inclusion of known pathogenetic variants from databases such as HGMD46 and ClinVar,9 and an evaluation scheme for weighing the evidence.

The SickKids challenges also required compilation of a set of candidate genes or some equivalent approach. Varying success in addressing these factors will have influenced the results, so it is not possible to effectively compare the core computational methods. Overall, current approaches have limitations in this setting––they tend to ignore or fail to reliably evaluate synonymous and noncoding variants; if the relevant gene is not known its variants will usually not be examined; and data for epigenetic causes are not available. Nevertheless, the CAGI results Figure 5: Performance on the Regulation Saturation expression challenge. The two left columns show performance in predicting increased (left) and decreased (right) expression in a set of enhancers (purple points represent variants that significantly change expression). The right pair of columns show equivalent results for promoters. The scatter plots (A) show strong performance in identifying decreases in expression (purple points), but weaker results for expression increases. Performance on promoters is stronger than on enhancers. Overlap of changed and non-changed experimental expression points suggests that experimental uncertainty reduces the apparent performance of the computational methods. Panel (B) shows correlation coefficients for selected methods. Panel (C) shows ROC curves for predicting under and overexpression. Panel (D) shows local lr!, where the solid lines are smoothed fits to the data.

15

for these challenges again make it clear that current state-of-the-art computational approaches can make valuable contributions in real clinical settings. Complex trait interpretation is often complicated by confounders in the data. Many common human diseases, such as Alzheimer’s disease, asthma, and type II diabetes, are complex traits and as with monogenic disorders, genetic information should in principle be useful for both diagnosis and prognosis. Individual response to drugs (pharmacogenomics) also often has a complex trait component. As opposed to monogenic disease, where one or two genetic variants are judged responsible for a phenotype, and cancer, where a small number (4-5 on average) of driver variants are key in each individual,6 complex traits have relatively small contributions from many variants. Environmental factors usually also play a substantial role so that methods using genome information alone have limited maximum accuracy. Further complicating interpretation, variants have relatively small effects on gene expression, splicing, and molecular function. Also, most CAGI complex trait challenges have been based on exome data, whereas at many GWAS risk loci lie outside coding regions.56 To some extent, the status of relevant common variants not present in the exome data can be imputed on the basis of linkage disequilibrium, but this places an unclear limit on achievable accuracy. Limited or no availability of training data also restricted method performance. Phenotypes tend to be less precise than for other types of disease; for example, phenotypes underlying a Crohn’s disease diagnosis are quite varied. Altogether, these factors make this a difficult CAGI area. Nevertheless, these challenges have been informative and have drawn new investigators into the rapidly developing area of Polygenic Risk Score (PRS) estimation.57 One challenge, CAGI4 Crohn’s, has yielded apparently robust conclusions on the performance of methods in this area.

Crohn’s disease (CAGI4). Participants were provided with exome data for 111 individuals and asked to identify the 64 who had been diagnosed with Crohn’s disease. A variety of computational approaches were used, including clustering by genotypes, analysis of variants in pathways related to the disease, and evaluation of SNPs in known disease-associated loci. The highest-scoring method (AUC 0.72; Figure 6A) used the latter approach together with conventional machine learning, and trained on data from an earlier GWAS.58 Figure 6B shows case and control score distributions for that method. A perfect method would have no distribution overlap. These results are far from that, but there is clear signal at the extremes, and as Figure 6C shows, that translates into a positive likelihood ratio with an approximately twenty- fold range (0.3 to 6), only a little lower than that obtained for the biochemical effect and clinical missense challenges. With a prior probability of disease of 1.3%,59 relative risk (see Methods) also has a range of about twenty-fold (Figure 6D), with the highest-risk individuals having six- fold higher risk compared to that estimated for the population average. For some complex trait diseases, for example coronary heart disease,60 this is discriminatory enough to support clinical action, and for many diseases would provide a valuable additional factor to more standard risk measures such as age and sex. Newer PRS methods, which aim to incorporate many weak contributions from SNPs, were not evaluated in this CAGI challenge.

Other complex trait CAGI challenges (Supplementary Materials) revealed batch effects (CAGI2 Crohn’s and Bipolar Disorder) and population structure effects (CAGI3 Crohn’s) in the data, leaked clinical data (Warfarin and VET challenges), or discrepancies between self-reported traits and those predicted from genetic data (PGP challenges), thereby complicating assessment.

16

Performance in matching genomes and disease phenotypes, an additional component in some challenges, was poor. Additional complex trait challenge results are provided in Supplementary Table 7.

The CAGI Ethics Forum has guided responsible data governance. Data used in CAGI challenges are diverse in terms of sensitivity (e.g., with respect to participant reidentification risk, potential for stigmatization, potential impact of pre-publication data disclosure), collected under a broad variety of participant consent understandings and protection frameworks, and analyzed by predictors with varying degrees of familiarity with local and international biomedical regulations. This heterogeneity calls for a nuanced approach to data access and the tailored vetting of CAGI experiments. The CAGI Ethics Forum was launched in 2015 to proactively address these concerns. Incorporating input from bioethicists, researchers, clinicians, and patient advocates, it has developed policies for responsible data governance (e.g., assisting in revision of the general CAGI data use agreement, to safeguard human data and also protect all CAGI participants, including data providers for unpublished data), cautioned against overinterpretation of findings (e.g., highlighting the contribution of social and environmental risk factors to disease, and the potential negative consequences, such as stigma, of associating particular disease variants with a specific population), and provided input on a variety of guidelines and procedures, including CAGI’s participant vetting process (e.g., how to identify a bona fide researcher) and a system of tiered access conditions for datasets, depending on their sensitivity. Future directions include investigating the scalability of current user validation and Figure 6: Identifying which of a set of individuals are most at risk for Crohn’s disease, given exome data. Examples were selected on the basis of ranking by ROC AUC. (A) ROC curves for two selected methods. Statistically significant but relatively low ROC areas are obtained. (B) Distributions of disease prediction scores for individuals with the disease (red) and without (green) for the method with the highest AUC (kernel density representation of the data). (C) Local positive likelihood ratio (lr!) as a function of prediction score for the method with the highest AUC. (D) Relative risk of disease (log2 scale), compared to that in the general population) as a function of prediction score. Individuals with the lowest risk scores have approximately 1/3 the average population risk, while those with the highest scores have risk exceeding six-fold the average, a twenty-fold total range. Depending on the disease, identifying individuals with higher than three-fold the average risk may be sufficient for clinical action.

17

data access models, exploring implications for family members of unexpected challenge findings, discussing policies to ensure proper credit attribution for constituent primary methods used by metapredictors, and identifying additional means of ensuring accountability options with respect to responsible data sharing.

Discussion

Principal findings. Over CAGI’s first decade, five rounds of CAGI challenges have provided a picture of the current state of the art in interpreting the impact of genetic variants on a range of phenotypes and provided a basis for the development of improved methods as well as for more calibrated use in clinical settings.

A key finding is that for most missense challenges it is possible to relate phenotype values to a pathogenicity threshold, and so deduce potential performance in a clinical setting, particularly for rare Mendelian diseases. The results suggest that computational methods are generally more reliable than recognized in the current clinical use guidelines.26 The Annotate All Missense challenge most directly probed clinical utility, assessing pre-deposited computational annotations of pathogenicity against subsequent clinical annotation. Encouragingly, the results show that of recently classified variants, up to 15% can have computational annotations that are sufficiently reliable to be considered Supporting evidence under the present guidelines,26 with some reaching Moderate and Strong thresholds of reliability (Figure 3). Together with the recent results from the ClinGen’s Sequence Variant Interpretation group,43 these results suggest that computational methods can play a beneficial larger role in clinical variant interpretation.

The plurality of challenges has required prediction of the impact of missense variants on biochemical properties as varied as protein stability, enzyme activity or cell growth rate. Although most methods were not trained for this type of use, there are some informative overall results albeit with limited accuracy for individual variants (Figure 2, Supplementary Tables 2 and 3). Results are also modestly but consistently better than for an older method trained on smaller datasets. Some challenges are still beyond the current state of the art. Examples are the liver pyruvate kinase challenge, which required prediction of allosteric regulation effects,61 NPM- ALK for prediction of protein binding changes, and the asthma challenge where comparison of RNASeq data between twins could not be used to assign disease status.

Several challenges have directly assessed the usefulness of missense impact and other methods under clinical conditions. Three have been particularly informative. The two SickKids challenges required participants to identify diagnostic variants using whole-genome sequence data for a set of pediatric rare disease cases where a state-of-the-art pipeline was not successful. For two of these difficult cases, CAGI participants were able to provide interpretations accepted by physicians, and some others may also be correct. The Johns Hopkins pipeline challenge required participants to match gene panel data to a set of heart and lung disease classes that individuals had been diagnosed with. Despite the handicap of not knowing the phenotype/genomic relationships, participants were apparently able to find diagnostic variants for a number of cases where the conventional pipeline had not succeeded, and to suggest that in some cases the initial diagnosis may be incorrect. Not all diagnostic variants in these three challenges are missense and there was limited success with other types of variation. The challenges also highlighted the fact

18

that successful application in the clinic requires integration of the computational methods into a more comprehensive pipeline, creating a barrier to participation for some research groups. Computational methods for identifying variants that introduce small splicing or gene expression effects have so far been less well represented in CAGI, and issues with data availability and appropriateness have limited insights. However, the results do show that these methods have potential for providing auxiliary evidence for assigning pathogenicity and may also help interpret GWAS results.

Missense and splicing variants also play major roles in cancer, and the challenges have provided insight into potential clinical use there. In addition, CAGI has included a set of challenges probing the ability of computational methods to identify germline variants that predispose to cancer. Results for the two BRCA gene challenges suggest that computational methods are sufficiently accurate to contribute to clinical assessment of variants, but more data are needed.

Up to now, complex traits have proven a difficult area in which to obtain large and robust datasets to test the computational methods in CAGI, partly because of concerns about data privacy (most datasets are subject to strict permission rules) and partly because of limited reliability for early datasets. Nevertheless, the challenges have provoked experimentation with a range of interesting computational approaches.23 For the one Crohn’s disease challenge with clear outcomes, the best methods were able to identify some high-risk individuals (Figure 6).

Future challenges may also include other types of omics data. Most current computational methods for scoring single variants are general; i.e., they are typically applied to any variant of the same type in any gene without adjustment, and are primarily based on sequence conservation, with limited inclusion of functional and structural properties. CAGI results show there is potential for more bespoke approaches together with contributions from functional and structural analysis. Protein structure properties, particularly effects on stability, have helped in several challenges (CDKN2A and TP53), and it appears that knowledge of domain functional properties would have helped in two more (CBS and RAD50).

Similarly, although of limited accuracy, methods that identify variants with small splicing and expression effects have potential to strengthen pathogenicity predictions. These and other structural and functional properties provide information that is largely independent of sequence conservation, so there is scope for combination with other properties into more comprehensive models. It is likely that future methods will be more gene-specific and will combine multiple information sources. In general, methods integrating predictions from multiple separate methods in a calibrated manner tend to perform better than single methods. Strong inter-method calibration, usually achieved by machine learning, is key here and simple consensus approaches should be avoided, unless specifically evaluated first.43 Participants have sometimes used unanticipated information to improve performance on challenges. For example, in a PGP challenge requiring matching of full genome sequences to extensive phenotype profiles, a participant made use of information in the PGP project blog.62 Though these sometimes subvert the intended challenge, in some ways this reflects what happens in a clinical setting¾all relevant information is used, however obscure.

19

Critical assessment for genome interpretation methods. CAGI relies on the same factors as other critical assessment community experiments: a willingness of the relevant research groups to participate, clearly defined metrics for success, the availability of large enough and accurate enough sets of experimental data to provide a gold standard, and independent objective assessment of the results. Participation in CAGI has been strong in most areas, particularly for missense variants, and a vibrant and interactive community has developed. New researchers have been attracted to the field and new collaborations have resulted in the development of creative

Algorithms With Broad Applicability.63-65

The regulatory variants research community is smaller but has participated, with encouraging results.53 Complex trait interpretation methods, particularly PRS,57 are rapidly growing, so we expect more participation in future, with challenges perhaps including other types of omics data.

For clinical challenges such as SickKids and the Johns Hopkins panel, CAGI has not yet succeeded in attracting much participation from professionals in that area. CAGI has been fortunate in its assessors, with many excellent analyses.12, 13 CAGI has not committed to any evaluation metric as a primary performance indicator. Instead, methods have been characterized using descriptive and useful metrics for both real-valued and discretized predictions, spanning the interests of the CAGI community and its intended audience.

Open questions remain regarding a set of metrics that can give more complete insight into the prediction performance, and new metrics, such as balanced precision,66 log ROC curves67 and the LC index68 are under consideration (Supplementary Figures 1, 2, 3, 4, and 7 provide log ROCs).

The biggest obstacle to clear assessment has been and continues to be data diversity and quality, a key difference between CAGI and related community endeavors. Other initiatives, such as CASP,69 deal primarily with one type of data (protein structure) and the data are usually of very high quality and directly relevant to the goals of the computational methods. By contrast, CAGI deals with many different settings, including studies of biochemical effects with a broad range of phenotypes, the pathogenicity of variants both germline and somatic, clinical phenotypes, and statistical relationships. Also, while genome variant calling is reliable, it does have limits.70 For example, in the SickKids challenges, some variants suggested as diagnostic by CAGI participants had been found to be incorrect calls and so eliminated in the clinical pipeline, using sequence validation data CAGI participants did not have access to. Adapting available data to form suitable challenges is difficult, and compromises are sometimes needed to devise a challenge where the results can be objectively assessed. For example, in one of the SickKids and in the Johns Hopkins clinical challenges, assessment hinged on requiring participants to match genomes to phenotypes. But that makes it much harder to identify diagnostic variants than in the real-life situation. Conversely, challenge providers have sometimes benefited from the detailed scrutiny of their data by CAGI participants prior to its publication. In some cases, interpretation of clinical challenge results is also complicated by there being no conclusive diagnoses.

For biochemical effect experiments there has been a trend towards high-throughput methods producing data for 1000s of variants.14 These surveys of the variant/phenotype landscape are providing major insights.71 But there are usually biases and compromises in the experimental design, as well as limited evaluation of uncertainty and error that make them less suitable for CAGI. For example, in the SUMO-ligase challenge, there is evidence that different interactions

20

between the human protein in the yeast milieu compared to that of human contributed to odd results for some variants.17 For the Annotate All Missense challenge, very high ROC areas are obtained by some methods. Since ClinVar and HGMD both contain an unknown level of misannotation,34 it is not clear how much of the remaining disagreement is due to such factors versus imperfect methods. Similar caveats apply to the high-throughput methods used for regulatory variants which usually use artificial gene constructs that may not fully reflect the genome context. For most challenges, experimental error estimates are almost always based on technical replicates, so likely underestimating true uncertainty. Some disagreement between predicted and experimental values may also be due to the complicated nature of the biochemical effects. For example, a variant may alter a post-translational modification that affects protein abundance.14 In clinical data provided to CAGI, all phenotypes ascribed to a patient appear to have equal weight, while the physician managing a case will have knowledge of which are major versus minor features.

Potential and prospects. Results from the first ten years of CAGI have not only highlighted current abilities and limitations but have pointed to the way forward. As already noted, one of the most important implications is the need for greater utilization of computational methods in the clinic.

Experimental biotechnology platforms have become widely available71 and the genomic data collection in the clinic has greatly increased. While the next generation of computational tools should benefit from these developments, they also pose new challenges. Deeper characterization of experimental approaches is needed to address data uncertainty and biases. Potential circularity between computation-assisted variant annotation and method assessment also needs to be considered. Future rounds of CAGI will address these issues by using assessment methods that mitigate or eliminate problems with data, by the development and promotion of practices and standards for application of methods, by working with experimental groups to provide sufficiently large and high-quality datasets, and by effectively following up on disagreements at the intersection of high-throughput functional experiments, genetic association studies, and outputs of computational prediction methods. CAGI also plans to increase evaluation of the combinatorial effect of variants, either in single-molecule biochemical assays or in clinical applications with whole-genome studies for both rare and complex phenotypes.

The current performance levels for exonic variation in Mendelian disorders, combined with rapidly accumulating data and a recent breakthrough in protein structure prediction,69 suggest that upcoming methods should more consistently achieve Strong and Very Strong clinical evidence levels. On the other hand, accurately modeling the biology of variant impact on protein function, such as catalysis, binding and allostery, will probably require the development of a different class of methods capable of generating meaningful scores on a continuous scale.

Although this will take time, it may be reasonable to expect improved calibration in specific cases in the short term, which could translate to an increased use of variant interpretation methods in biomedical research.

So far CAGI has had few large-scale challenges for regulatory variants, but it is clear that because of many small effect sizes, future challenges will need very high-quality experimental data to enable a clear assessment. Complex trait challenge data quality has also been a limiting

21

factor but given that data quality and availability are improving, future challenges are expected to resolve this issue. A question still to be addressed with both regulatory and complex trait challenges is that the extent to which the computational methods can provide mechanistic insight rather than just statistical association, and future challenges should provide answers.

Genomic data relevant to disease are increasingly complemented by other data types, particularly expression data, proteomics, and metabolomics. CAGI has already included such multiomics challenges (Figure 1) and will do so more in future.

Improvements in overall variant impact prediction and disease gene prioritization will also lead to the development of new tools with increased automation in clinical diagnostics. This will require careful integration of the component tools and pipelining, including improved standardization and interoperability practices. As of now, CAGI has not evaluated the seamlessness of use of these methods and may consider it as these platforms mature.

Finally, owing to complex institutional and national regulatory policies and practices, obstacles to data sharing remain significant. CAGI has been addressing such issues by developing robust data use agreements and will seek to increase closed-environment assessments.72 Through the activities of its Ethics Forum, it will also seek to create data use agreement standards and practices to that will simplify responsible data sharing while promoting responsible and respectful research uses.

Methods

We describe different evaluation scenarios considered in the Critical Assessment of Genome Interpretation (CAGI), motivate the selection of performance measures, and discuss ways to interpret the results.

Terminology and notation. Let (𝑥, 𝑦) ∈ 𝒳× 𝒴 be a realization of an input-output random vector (𝑋, 𝑌). The input space 𝒳 may describe variants, gene panels, exomes, or genomes in different CAGI scenarios. Similarly, the output space 𝒴 can describe discrete or continuous targets; e.g., it can be a binary set {pathogenic, benign} when the task is to predict variant pathogenicity, or a continuous set [0, ∞) representing a percent of enzymatic activity of the wildtype protein (a mutated molecule can have increased activity compared to the wildtype) or a cell growth rate relative to that with the wildtype gene.

Let 𝑠: 𝒳→ℝ be a predictor that assigns a real-valued score to each input and 𝑓: 𝒳→𝒴 be a predictor that maps inputs into elements of the desired output space; i.e., 𝑓 is real-valued when predicting a continuous output and discrete when predicting a discrete output. When predicting binary outcomes (e.g., 𝒴= {0, 1}), 𝑠(𝑥) is often a soft prediction score between 0 and 1, whereas 𝑓(𝑥) can be obtained by discretizing 𝑠(𝑥) based on some decision threshold 𝜏. Scores 𝑠(𝑥) can also be discretized based on a set of thresholds {𝜏#}#$%

& , As Discussed Later. In A Binary

classification scenario, 𝑓(𝑥) = 1 (pathogenic prediction) when 𝑠(𝑥) ≥𝜏 and 𝑓(𝑥) = 0 (benign prediction) otherwise. We shall sometimes denote a discretized binary model as 𝑓'(𝑥) to emphasize that the predictor was obtained by thresholding a soft scoring function 𝑠(𝑥) at 𝜏. The target variable 𝑌 can similarly be obtained by discretizing the continuous space 𝒴 using a set of

Thresholds {𝜏(

) }, that are different from {𝜏#} used for the scoring function. In one such case, discretizing the continuous space 𝒴 of functional impact of an amino acid variant into {damaging, nondamaging} transforms a regression problem into classification, which may provide additional insights during assessment. With a minor abuse of notation, we will refer to both continuous and the derived discrete space as 𝒴. The exact nature of the target variable 𝑌 and the output space will be clear from the context.

Finally, Let 𝒟= {(𝑥*, 𝑦*)}*$%

+ be a test set containing 𝑛 input-output pairs on which the predictors are evaluated. Ideally, this data set is representative of the underlying data distribution and non- overlapping with the training data for each evaluated predictor. Similarly, we assume the quality of the measurement of the ground truth values {𝑦*}*$%

+ Is High Enough To Ensure Reliable

evaluation. While we took multiple steps to ensure reliable experiments and blind assessments, it is difficult to guarantee complete enforcement of either of these criteria. For example, an in vitro assay may be an imperfect model of an in vivo impact or there might be uncertainty in collecting experimental read-outs. Additionally, the notion of a representative test set may be ambiguous and cause difficulties when evaluating a model that was developed with application objectives different from those used to assess its performance in CAGI.

Evaluation for continuous targets. Evaluating the prediction of continuous outputs is performed using three primary measures (𝑅!, Pearson’s correlation coefficient, and Kendall’s tau) and two secondary measures (root mean square error and Spearman’s correlation coefficient). 𝑅! is defined as the difference between the variance of the experimental values and the mean-squared error of the predictor, normalized by the variance of the experimental values. It is also referred to as the fraction of variance of the target that is explained by the model. 𝑅! ∈

"%&

, (1)

*$%

is the mean of the target values in 𝒟 (observe that each value 𝑦* may itself be an average over technical or biological replicates, if available in the experimental data). The 𝑅! values above 0 indicate that the predictor is better than the trivial predictor—one that always outputs the mean of the target variable—and values close to 1 are desired. The values below 0 correspond to models with inferior performance to the trivial predictor. Maximizing the 𝑅! metric requires calibration of output scores; that is, a high correlation between predictions and target values as well as the proper scaling of the prediction outputs. For example, a predictor outputting a linear transformation of the target such as 𝑓(𝑋) = 10 ⋅𝑌 or a monotonic nonlinear transformation of the target such as 𝑓(𝑋) = log 𝑌 may have a high correlation, but a low 𝑅!. 𝑅!, therefore, can be seen as the strictest metric used in CAGI. However, this metric can adversely impact methods outputting discretized prediction values. Such methods are preferred by some tool developers as they simplify interpretation by clinicians, experimental scientists, or other users.

In some cases, it may be useful to also report the root mean squared error (RMSE), estimated

*$%

. (2)

23

RMSE can offer a useful interpretation of the performance and is provided as a secondary measure in CAGI evaluations. The correlation coefficient between the prediction 𝑓(𝑋) and target 𝑌 is defined as a normalized covariance between the prediction output and the target variable. Pearson’s correlation

"%&

, (3)

*$%

is the mean of the predictions. Pearson’s correlation coefficient does not depend on the scale of the prediction, but it is affected by the extent of a linear relationship between predictions and the target. That is, a predictor outputting a linear transformation of the target such as 𝑓(𝑋) = 10 ⋅𝑌 will have a perfect correlation. However, a monotonic nonlinear transformation of the target such as 𝑓(𝑋) = log 𝑌 may have a relatively low 𝑟. Although not our main metric, we also explored Spearman’s rank correlation as a secondary metric. Spearman’s correlation is defined as Pearson’s correlation on the rankings.

We also computed Kendall’s tau, which is the probability of a concordant pair of prediction- target points linearly scaled to the [−1, 1] interval instead of [0, 1]. Assuming that all prediction and target values are distinct, a pair of points (𝑓(𝑥*), 𝑦*) and (𝑓(𝑥#), 𝑦#) is concordant if either (𝑓(𝑥*) > 𝑓(𝑥#) and 𝑦* > 𝑦#) or (𝑓(𝑥*) < 𝑓(𝑥#) and 𝑦* < 𝑦#). Otherwise, a pair of points is

. (4)

It ranges between −1 and 1, with 1 indicating that all pairs are concordant, 0 indicating half of the concordant pairs (e.g., a random ordering) and −1 indicating that all pairs are discordant. A predictor outputting a linear transformation of the target 𝑓(𝑋) = 10 ⋅𝑌 and a monotonic nonlinear transformation of the target 𝑓(𝑋) = log 𝑌 will both have a perfect tau of 1. Compared to Pearson’s correlation, Kendall’s tau can be seen as less sensitive to the scale but more sensitive to the ordering of predictions. Eq. 4 is defined under the assumption that both the predictions and the outputs are unique. However, this assumption is not satisfied by all biological datasets and predictors. To address this issue, we use Kendall’s tau-b, a widely accepted

?

, (5) where 𝛽(𝑛) = 𝑛(𝑛−1)/2, 𝑢* (𝑣*) is the size of the 𝑖th group of ties in the predictions (outputs) and 𝑇 (𝑆) is the number of such groups in the predictions (outputs).73 Evaluation for binary targets. Evaluating binary outputs is performed using standard protocols in binary classification.74 We compute the Receiver Operating Characteristic (ROC) curve, which is a 2D plot of the true positive rate 𝛾= 𝑃(𝑓'(𝑋) = 1|𝑌= 1) as a function of the false positive rate 𝜂= 𝑃(𝑓'(𝑋) = 1|𝑌= 0), where 𝜏 is varied over the entire range of prediction scores. The area under the ROC curve can be mathematically expressed as AUC = ∫𝛾 𝑑𝜂

And Is

the probability that a randomly selected positive example 𝑥" will be assigned a higher score than

24

a randomly selected negative example 𝑥1 by the model.75 That is, assuming no ties in prediction scores AUC = 𝑃(𝑓(𝑋") > 𝑓(𝑋1)). In the presence of ties, AUC is given by 𝑃(𝑓(𝑋") >

%

! 𝑃(𝑓(𝑋") = 𝑓(𝑋1)).76 The AUC is estimated on the test set 𝒟 using the standard numerical computation that allows for ties.77 Although AUC does not serve as a metric that directly translates into clinical decisions, it is useful in that it shows the degree of separation of the examples from the two groups of data points (positive vs. negative). Another useful property of the AUC is its insensitivity to class imbalance.

Though AUC is a useful measure for capturing the overall performance of a classifier’s score function, it has limitations when applied to a decision-making setting such as the one encountered in the clinic. Typically, clinically relevant score thresholds that determine the variants satisfying Supporting, Moderate or Strong evidence26 lie in a region of low false positive rate (FPR). A measure well-suited to capture clinical significance of a predictor ought to be sensitive to the variations in the classifier’s performance in the low FPR region (when predicting pathogenicity). However, the contribution of the low FPR region to AUC is relatively small. This is because it not only represents a small fraction of the entire curve, but also because the TPR values in that region are relatively small. Thus, AUC is not sensitive enough to the variation in a predictor’s performance in the low FPR region. To mitigate this problem, we also provide area under the ROC curve truncated to the [0, 0.2] FPR interval. What constitutes low FPR is not well defined; however, it appears that the [0, 0.2] FPR interval combined with the [0, 1] TPR interval is a reasonable choice in CAGI applications; see Figures 2-3. We normalize the truncated AUC to span the entire [0, 1] range by dividing the observed value by 0.2, the maximum possible area below the ROC truncated at FPR = 0.2.

CAGI evaluation of binary classifiers also involves calculation of the Matthews correlation coefficient.78 The Matthews correlation coefficient (MCC) was computed as a Pearson’s correlation coefficient between binary predictions and target values on the test set 𝒟. Efficient MCC estimation was carried out from the confusion matrix.78 Evaluation for clinical significance. Current guidelines from the American College for Medical Genetics and Genomics (ACMG) and Association for Molecular Pathology (AMP) established a qualitative framework for combining evidence in support of or against variant pathogenicity for clinical use.26 These guidelines point to five different levels of pathogenicity and (effectively) nine distinct types of evidence in support of or against variant pathogenicity. The five pathogenicity levels involve classifications into pathogenic, likely pathogenic, variant of uncertain significance (VUS), likely benign, and benign variants, whereas the nine levels of evidential support are grouped into Very Strong, Strong, Moderate, and Supporting for either pathogenicity or benignity, as well as indeterminate evidence that supports neither pathogenicity nor benignity.

Richards et al.26 have manually categorized different types of evidence and also listed twenty rules for combining evidence for a variant to be classified into one of the five pathogenicity- benignity groups. For example, variants that accumulate one Very Strong and one Strong line of evidence of pathogenicity lead to the classification of the variant as pathogenic; variants that accumulate one Strong and two Supporting lines of evidence lead to the classification of the variant as likely pathogenic, etc.26 The guidelines allow for the use of computational evidence

25

such that a computational prediction of pathogenicity can be considered as the weakest (Supporting) line of evidence. Thus, combined with other evidence, these methods can presently contribute to a pathogenicity assertion for a variant, but in a restricted and arbitrary way.43 Since Supporting evidence is qualitatively the smallest unit of contributory evidence in the ACMG/AMP guidelines, we refer to any computational model that reaches the prediction quality equivalent of Supporting evidence and higher as a model that provides contributory evidence in the clinic.

Numerically, a variant that is classified as pathogenic should have at least a 99% probability of being pathogenic given all available evidence, whereas a variant that is likely pathogenic should have at least a 90% probability of being pathogenic given the evidence.26, 45 Variants that cross the 90% probability threshold for pathogenicity are considered clinically actionable.26 Analogously, variants with sufficient support for benignity will typically be ruled out from being diagnostic in a clinical laboratory. Note that, though the guidelines provide a probabilistic interpretation of the pathogenicity assertions, they do not provide any general quantitative interpretation of the evidence. Consequently, any framework designed to express the evidence levels quantitatively, must tie such quantitative evidential support to the pathogenicity probabilities, mediated by the ACMG/AMP rules for combining evidence.

The possibility of incorporating computational methods into clinical decision making in a properly calibrated manner presents interesting opportunities and unique challenges. In particular, since the evidence levels are only described qualitatively, it is not obvious how to determine what values of a predictor’s output score qualify as providing a given level of evidence. Thus, to apply a computational line of evidence in the clinic in a principled manner, and consistent with the guidelines, there is a need for a framework that assigns a quantitative interpretation to each evidence level.

Tavtigian et al.45 proposed such a framework to provide numerical support for each type of evidential strength for its use in ACMG/AMP guidelines for or against variant pathogenicity. This approach is based on the relationship between prior and posterior odds of pathogenicity as well as on independence of all lines of evidential support for a given variant. We briefly review this approach.

Let 𝐸 be a random variable indicating evidence that can be used in support of or against variant pathogenicity. The positive likelihood ratio (LR") given concrete evidence 𝑒 is defined as

Bc7Def8Cf Chh7 Ci Bjdkc9E:8L8Dm

BF8CF CHH7 CI BJDKC9E:8L8DM (6)

%1N(O$%)

N(O$%) , (7) where the first term on the right corresponds to the posterior odds of pathogenicity given the evidence and the second term on the right corresponds to the reciprocal of the prior odds of pathogenicity. The prior odds of pathogenicity depend solely on the class prior 𝑃(𝑌= 1); that is, the fraction of pathogenic variants in the selected reference set. The expression for LR" also allows for an easy interpretation as the increase in odds of pathogenicity given evidence 𝑒

26

compared to the situation when no evidence whatsoever is available. The likelihood ratio of 2, for example, states that a variant with evidence 𝑒 is expected to have twice as large odds of being pathogenic than a variant picked uniformly at random from a reference set. As CAGI only considers computational evidence, we will later replace the posterior probability 𝑃(𝑌= 1|𝐸= 𝑒) by 𝑃(𝑌= 1|𝑓(𝑋) = 1) for discretized predictors or by 𝑃(𝑌= 1|𝑠(𝑋) = 𝑠) for the predictors that output a soft numerical score 𝑠. The probability 𝑃(𝑌= 1|𝑓(𝑋) = 1) is the positive predictive value (or precision) of a binary classifier, whereas the probability 𝑃(𝑌= 1|𝑠(𝑋) = 𝑠) can be seen as the local positive predictive value, defined here in a manner analogous to the local

False Discovery Rate.79

It can be shown80 that the positive likelihood ratio can also be stated as

N(Q$R|O$%)

N(Q$R|O$A) (8) thus clarifying that LR" can be seen as the ratio of the true positive rate and false positive rate when 𝑃(𝐸= 𝑒|𝑌= 1) is replaced by 𝑃(𝑓(𝑋) = 1|𝑌= 1) and 𝑃(𝐸= 𝑒|𝑌= 0) by 𝑃(𝑓(𝑋) = 1|𝑌= 0).

Tavtigian et al.45 give an expression relating the posterior 𝑃(𝑌= 1|𝐸= 𝑒) to LR" and the prior

St((R)N(O$%)

(ST((R)1%)N(O$%)"% (9) which itself is obtained from Eq. 7. They also present a framework that allows for assigning probabilistic interpretations to different types of evidential strength (Supporting, Moderate, Strong, and Very Strong) and combining them in a manner consistent with the rules listed in Richards et al.26 and the probabilistic interpretation of likely pathogenic and pathogenic classes.

Their formulation is given in terms of the positive likelihood ratio LR" in an exponential form. We restate this model using a notion of the total (or combined) positive likelihood ratio LRU

",

based on all available evidence, 𝐸U, of a variant that is expressed as a product of LR" factors

# "$3,

& , (10) where 𝑛7V, 𝑛WC, 𝑛7D, and 𝑛X7 are the counts of Supporting (su), Moderate (mo), Strong (st), and Very Strong (vs) lines of evidence present in 𝐸U, and 𝑐 is the LR" value assigned to a single line of Very Strong evidence. It is easy to show that √𝑐 .

Correspond To The Lr" For A

single line of Supporting, Moderate, and Strong line of evidence, respectively. In other words, the model from Eq. 10 enforces that if a Very Strong piece of evidence increases LRU

" By A

multiplicative factor of 𝑐, then a Supporting, Moderate, or a Strong piece of evidence increases

" By A Factor Of √𝑐

.

#

, respectively. For a reasonable consistency with Richards et al.26 this model also explicitly encodes that one line of Very Strong evidence is equal to the two lines of Strong evidence, four lines of Moderate evidence, and eight lines of Supporting evidence.

The appropriate value of 𝑐, however, depends on the class prior. It is the smallest number for

Which The Lru

" values computed for the qualitative criteria from the likely pathogenic class in Richards et al.26 reach 𝑃(𝑌= 1|𝐸U = 𝑒) values of at least 0.9 and, similarly, for those in the

27

pathogenic class, reach a 𝑃(𝑌= 1|𝐸U = 𝑒) value of at least 0.99. The dependence on the class

Class Prior Is Small, A Larger Value Of Lru

" will be required to achieve the same posterior level, thereby requiring a larger value of 𝑐 (Supplementary Figure 14). Tavtigian et al.45 also proposed that two rules from Richards et al.26 be revised; that is, one of the rules was proposed to be “demoted” from pathogenic to likely pathogenic, whereas another rule was proposed to be “promoted” from the likely pathogenic to pathogenic. For a class prior of 0.1 that was selected based on the experience from the clinic, the value 𝑐= 350 was found to be suitable. This, in turn, suggests that the Supporting, Moderate, and Strong lines of evidence

Should Require The Likelihood Ratio Values Of √𝑐

.

= 18.7,

respectively. However, note again that for different priors, these values will be different; see next Section and Supplementary Figure 14. Moreover, while the level of posterior for the combined evidence (Eq. 13) is required to be at least 0.9 to satisfy the likely pathogenic rule and 0.99 for pathogenic, this does not mean that the posterior level for a single line of evidence is the same for all values of 𝑐. This is a consequence of the fact that the framework provides intuitive interpretation only at the level of the combined posterior.

When drawing evidence from a pathogenicity predictor, it is necessary to further clarify what evidence is in the first place. At least two options are available: (i) the evidence is the score 𝑠(𝑥); that is, a raw prediction of pathogenicity, or (ii) the evidence is a discretized prediction 𝑓'(𝑥), obtained by thresholding 𝑠(𝑥). These approaches, referred to here as local and global, respectively, lead to different interpretations because all evaluation metrics hold only on average, either over all variants with a score 𝑠(𝑥) or all variants satisfying 𝑓'(𝑥) = 1; i.e., having a score above 𝜏. When both 𝑠(𝑥) and 𝑓'(𝑥) are available, this leads to difficulties in interpreting the results of the global approach because all scores 𝑠(𝑥) that map into 𝑓'(𝑥) = 1 will be treated identically. Unfortunately, this implies that scores 𝑠 greater than but still close to 𝜏 most likely do not meet the levels of evidential strength for the interval. At the same time, scores close to the high end of the range almost certainly make the levels of evidential strength above the designated level. This means that a clinician seeing a variant with score slightly above 𝜏 would have to interpret this prediction as contributory to pathogenicity, yet this interpretation would almost certainly be incorrect. Based on the recommendations from the ClinGen’s Sequence Variant Interpretation group43 we focus on the local view as well as local performance criteria to define levels of evidential strength and assess whether methods achieve these levels. In the end, however, we also provide global estimates to understand the performance of each tool more comprehensively.

Y(Z|O$%)

Y(Z|O$A) , (11) where 𝑝(𝑠|𝑌= 𝑦), for 𝑦∈{0, 1} are class-conditional densities. We obtain an estimate of the local positive likelihood ratio lr{" from the test data as described in the Section titled “Computing the clinically relevant measures”. Now, the threshold to determine the variants with Supporting level of evidence is given as the minimum score above which all variants achieve local positive likelihood value greater than or equal to √𝑐 .

𝜏7V = Min}𝜏: ∀𝑠≥𝜏, Lr{ "(𝑠) ≥ √𝑐

. •, (12) though we note that Pejaver et al.43 incorporated an additional factor based on the confidence interval for lr{ "(𝑠) to result in more stringent recommendations for score thresholding. Similarly, the thresholds for variants with Moderate, Strong and Very Strong evidence are given by 𝜏WC =

• And 𝜏x7 = Min}𝜏: ∀S ≥𝜏,

lr{ "(𝑠) ≥ c•. Once the threshold set {𝜏7V, 𝜏WC, 𝜏7D, 𝜏X7} is determined, we can compute either the global LR" (e.g., 𝑠≥𝜏7V) or the LR" corresponding to an interval of scores (e.g., 𝜏7V ≤𝑠< 𝜏WC) by computing the true positive rate and false positive rate for a given set of scores. A global positive predictive value can be similarly estimated once the class prior is known.

In all CAGI evaluations, a predictor is considered to provide contributory evidence in a clinical setting if it reaches any one of the evidence levels according to the ACMG/AMP guidelines, and according to the model by Tavtigian et al.45 and recommendations by Pejaver et al.43 Among predictors that reach the desired levels of evidential support, the ones that reach higher levels are generally considered favorably. However, we have not considered any criterion to rank the predictors that reach the same levels of evidential support.

Selection of class priors for variant pathogenicity. Different clinical scenarios require the use of different class priors of variant pathogenicity. We generally distinguish between two clinical situations.

In the first setting, a clinician is presented with a proband with specific phenotypic expression and the objective is to find variants responsible for the clinical phenotype. In certain monogenic disorders with Mendelian inheritance patterns, the fraction of rare variants found to be pathogenic can be as high as 25%, as in the case of the NAGLU challenge. Similarly, Tavtigian et al.45 report an experience-based prior of 10% based on their work with BRCA genes, which we adopted in this work.

The second setting reflects situations such as screening for potential secondary variants. Here we have used an estimate by Pejaver et al.34 that up to 1.5% of missense variants in an apparently healthy individual could be disease-causing.

Overall, prior probability of pathogenicity was set to 1% and 10% to demonstrate the distinction in the level of evidential support necessary. These resulted in 𝑐= 8511 and 𝑐= 351, respectively (note that 𝑐= 351 was selected instead of 𝑐= 350 to avoid rounding errors in finding a 𝑐 that best models ACMG/AMP rules). In each functional missense challenge, the level of prior probability observed for each gene based on experimental data was further considered.

For large class priors such as 50% or above, the Tavtigian et al.45 framework holds only when an additional rule from Richards et al.26 is removed; that is, we ignored that two Supporting lines of evidence for benignity assert a likely benign variant.

Performance measures for clinical application. Diagnostic odds ratio. The diagnostic odds ratio (DOR) is commonly used in biomedical sciences to measure the increase in odds of

29

pathogenicity in the presence of evidence 𝑒 compared to the odds of pathogenicity in the absence

%1N(O$%|Q[R)

N(O$%|Q[R) . (13) The difference between Eq. 7 and Eq. 13 is that the prior odds, those governed by the prior 𝑃(𝑌= 1) and used in Eq. 7, are replaced by the odds governed by the probability 𝑃(𝑌= 1|𝐸≠𝑒); that is, odds of pathogenicity when the evidence 𝑒 was not the one that was observed. The quantity 𝑃(𝑌= 0|𝐸≠𝑒) = 1 −𝑃(𝑌= 1|𝐸≠𝑒) is referred to as the negative predictive value when the observed evidence is 𝑓(𝑋) = 0. DOR ∈[0, ∞) can also be expressed

St((R)

ST)(R) , (14)

N(Q[R|O$%)

N(Q[R|O$A) . (15) In contrast to typical studies of variant risk assessment81 and polygenic risk scores,57 DOR was calculated without adjustments for usual confounders such as race, ethnicity, etc. that are generally not available in CAGI challenges and, technically, produce conditional odds ratios.80 However, the DOR values estimated in our experiments have an identical interpretation as the results of logistic regression run with a single independent variable (co-variate) at a time. Glas et al.80 give a broader coverage of diagnostic odds ratios that further connect some of the quantities discussed here (e.g., AUC vs. DOR).

We only consider DOR with the computational evidence of the “global” type; that is, when 𝑠(𝑥) ≥𝜏. Consequently, DOR at 𝜏 can be expressed as

N(Z(\)^'|O$A)

N(Z(\)^'|O$%). (16) Unlike positive likelihood ratio, DOR does not have a “local” version. This is because one cannot define a local negative likelihood ratio.

Percent of variants predicted as pathogenic. In addition to finding whether a method reaches Supporting, Moderate or Strong levels of evidence, it is important to also quantify the proportion of variants in the reference set for which a given evidence level is reached. To this end, for a given score threshold 𝜏, we define the percent of variants in the reference set that the method assigns a score as high as or higher than 𝜏, and refer to it as “probability of pathogenic (positive) predictions”, or PPP. Mathematically, it can be expressed as the following probability PPP(𝜏) = 𝑃(𝑠(𝑋) ≥𝜏). (17) The probability (equivalently, percent) of variants reaching a given level of evidence can now be quantified as PPP(𝜏), where 𝜏 is the score threshold at which a variant is declared to meet the desired evidential support.

30

Posterior probability of pathogenicity. Given a method, the posterior probability of pathogenicity or the absolute risk for a variant is defined as the probability that the variant is pathogenic based on the score it is assigned by the method. It is expressed as 𝜌(𝑠) = 𝑃(𝑌= 1|𝑠(𝑋) = 𝑠). (18) We also refer to this quantity as a local positive predictive value or local precision.

Relative risk. Given a method, the relative risk (RR) of pathogenicity of a variant is defined as the posterior probability of pathogenicity (based on the score assigned by the method) relative to the prior probability of pathogenicity. It is expressed as the following ratio

N(O$%)

. (19) The prior probability of pathogenicity can also be interpreted as the average of the posterior probability over all variants in the reference set; that is

(20)

where the last step follows since 𝑝(𝑠|𝑌= 1) is a density function and its integral over ℝ is 1. Observe that our definition of relative risk is an extension of the “global” version used in clinical applications where the denominator would be 𝑃(𝑌= 1|𝑠(𝑋) ≠𝑠), which effectively equals 𝑃(𝑌= 1) for all predictors outputting continuous scores.

Computing clinically relevant measures. We show here how the measures for evaluation of binary targets and clinically relevant measures are computed from the test data 𝒟. It is necessary to be cautious when making decisions on a reference (target) population based on the measures computed on the test set 𝒟. Some of the measures computed on 𝒟 accurately represent the corresponding values on the target population. However, other measures are biased because the test data set for many challenges is not representative of the target population. In particular, the proportions of positives (e.g., pathogenic variants) in the test set 𝛼𝒟= 𝑃𝒟(𝑌= 1) may be vastly different from that in the target population 𝛼= 𝑃(𝑌= 1). Consequently, the class-prior dependent measures, when estimated directly from the test set, are incorrectly calibrated to the test set class priors.

Fortunately, the class-prior dependent measures can be corrected using an estimate of the target population’s class priors if known or if estimated using a principled approach.82, 83 The correction is derived under the assumption that the reference population and the test set are distributionally identical, except for the differences in class priors. To elaborate, the target distribution of inputs 𝑝(𝑥) can be expressed in terms of the class-conditional distributions, 𝑝(𝑥|𝑌= 𝑦) for 𝑦∈{0, 1},

And The Class Priors As Follows

𝑝(𝑥) = 𝛼⋅𝑝(𝑥|𝑌= 1) + (1 −𝛼) ⋅𝑝(𝑥|𝑌= 0). (21) We assume that the test set distribution of inputs might have different class priors, but the same class-conditional distributions as the target population. Precisely,

=

𝛼𝒟𝑝(𝑥|𝑌= 1) + (1 −𝛼𝒟)𝑝(𝑥|𝑌= 0).

(22)

It is easy to see that any of the clinical and non-clinical measures that only depend on the class- conditional distributions, but not class priors, when computed on the test set is an unbiased estimate of the measure on the target population. However, if a measure also depends on the class priors, it needs to be corrected to reflect the reference population’s class prior. All the class- prior independent measures used in this paper can be expressed in terms of class-conditional derived quantities such as the true positive rate (TPR), the false positive rate (FPR) and the local positive likelihood ratio lr"(𝑠). The class-prior dependent measures additionally have the class- prior in their expressions.

TPR, FPR and 𝒍𝒓"(𝒔). Formally, TPR is defined as the proportion of positive inputs that are correctly predicted to be positive. Mathematically, TPR(𝜏) = 𝑃(𝑠(𝑥) ≥𝜏|𝑌= 1), (23) where 𝑠(𝑥) is a continuous score function of a classifier and 𝜏 is a threshold such that an input scoring above 𝜏 is predicted to be positive. Similarly, FPR is defined as the proportion of negative inputs that are incorrectly predicted to be positive. Mathematically, FPR(𝜏) = 𝑃(𝑠(𝑥) ≥𝜏|𝑌= 0), (24) TPR and FPR can be computed from the test data as the proportion of positive and negative test

(25)

where 𝒟" and 𝒟1 are the subsets of points in the test set 𝒟 labeled as positive and negative, respectively. Some of the clinically relevant measures used in our study are “local” in nature in the sense that they are derived from a local neighborhood around a score value instead of the entire range of scores above (or below) the threshold. Such measures can be expressed in terms of the local positive likelihood ratio lr"(𝑠). To compute lr"(𝑠), we exploit its relationship to the posterior

Ef((Z)N(O$%)

(eF((Z)1%)N(O$%)"% .

(26)

Similarly, the test data posterior probability can be expressed as

Ef((Z)N𝒟(O$%)

(eF((Z)1%)N𝒟(O$%)"%. (27) Note that since lr"(𝑠) only depends on the class-conditional distribution, it does not change when defined on the target population. Unlike the target population’s posterior, the test data posterior can be estimated from the test data as described below. Once the test posterior is estimated, the equation above can be inverted to estimate lr" as

%1N𝒟(O$%)

N𝒟(O$%) , (28) where the 𝑃•𝒟(𝑌= 1|𝑠(𝑋) = 𝑠) is an estimate of the test data posterior and 𝑃𝒟(𝑌= 1) is the proportion of positives in the test data, which may differ from the true prior for a randomly picked variant in the gene of interest or another reference sample. Note that though the formula above expresses lr{ "(𝑠) in terms of the prior odds, suggesting a dependence on the class prior, lr{ "(𝑠) is class-prior independent, as discussed earlier. In theory, 𝑃𝒟(𝑌= 1|𝑠(𝑋) = 𝑠), is the proportion of pathogenic variants among all variants in 𝒟 having a score 𝑠. Therefore, estimating the local posterior efficiently would require observing the same score many times in the set of variants with known labels. This is unlikely since we only have scores for a finite set of variants and thus the posterior cannot be estimated without making further assumptions. However, assuming that the posterior is a smooth function of the score—similar scores correspond to similar local posterior values—we estimate the posterior as the proportion of pathogenic variants in a small window around the score; that is, [𝑠−𝜖, 𝑠+ 𝜖], where 𝜖 was selected to be 5% of the range of the predictor’s outputs, with the range considered to be an interval between the 5th and 95th percentile of predicted values on the dataset, selected as such to minimize the influence of outliers. In addition, for stable estimates, we required that at least 10% of the variants, up to a maximum of 50 variants, from the data set are within a window; therefore, the final window size was dependent on score 𝑠 and data set 𝒟.

Measures that do not require correction. Among the measures considered in this paper, TPR, FPR, ROC curve, AUC, LR", LR1, DOR and lr" do not require correction. Class-prior independence of TPR, FPR and lr+ is obvious from their definitions as discussed earlier. ROC curve is obtained by plotting TPR against FPR and consequently, it is also class-prior independent. By extension AUC, being the area under the ROC curve, is also class-prior independent. The global positive likelihood ratio LR", formulated with the evidence of the type 𝑠(𝑥) ≥𝜏, is given by TPR(τ)/FPR(τ). Similarly, the global LR1 is given by (1 − TPR(τ))/(1 −FPR(τ)). Since DOR is the ratio of LR" and LR1, it is by extension class-prior independent.

Measures that require correction. Among the measures considered in this paper, probability of pathogenic predictions (PPP), positive predictive value (PPV), posterior probability (𝜌) and relative risk (RR), being class-prior dependent, require corrections to be properly applied to the target population. To show that the measures are indeed class-prior dependent, we re-formulate them by separating the class-prior from the class-conditional dependent terms.

(31)

where the derivation is the same as that for Eq. 26.

Ef((Z)

g(eF((Z)1%)"% .

(32)

We use the expressions above to correctly calculate class-prior dependent metrics on the target domain by first computing the class-conditional dependent terms (TPR, FPR or lr") using the test data 𝒟 and then using an estimate of the class prior of the target distribution in the corresponding expression.

Statistical significance and confidence interval estimation. All p-values and confidence intervals in CAGI evaluations were estimated using bootstrapping with 1,000 iterations.84 The Critical Assessment of Genome Interpretation Consortium

Analysis And Writing

Shantanu Jain1,+, Constantina Bakolitsa2,+, Steven E. Brenner2,$,*, Predrag Radivojac1,3,*, John

+Contributed Equally

$This author was unable to fully contribute to the paper writing due to an injury and its sequelae

Cagi Organizers

John Moult4, Susanna Repo2, Roger A. Hoskins2, Gaia Andreoletti2, Daniel Barsky2, Steven E.

Informatics Infrastructure And Support

Ajithavalli Chellapan5, Hoyin Chu1,6-7, Navya Dabbiru5, Naveen K. Kollipara5, Melissa Ly2, Andrew J. Neumann2, Lipika R. Pal4, Gaurav Pandey2, Robin C. Peters-Petrulewicz2, Rajgopal Srinivasan5, Stephen F. Yee2, Sri Jyothsna Yeleswarapu5, Maya Zuhl4,8

Predictors

Ogun Adebali9-10, Ayoti Patra11-12, Michael A. Beer11, Raghavendra Hosur13-14, Jian Peng13, Brady M. Bernard15-16, Michael Berry9, Shengcheng Dong17, Alan P. Boyle17, Aashish Adhikari2,18,

34

Jingqi Chen2,19, Zhiqiang Hu2, Robert Wang2,20, Yaqiong Wang2,19, Maximilian Miller21, Yanran Wang21-22, Yana Bromberg21, Paola Turina23, Emidio Capriotti23, James J. Han24, Kivilcim Ozturk24, Hannah Carter24, Giulia Babbi23, Samuele Bovo23, Pietro Di Lena23, Pier Luigi Martelli23, Castrense Savojardo23, Rita Casadio23, Melissa S. Cline25, Greet De Baets26, Sandra Bonache27-28, Orland Díez27,29, Sara Gutiérrez-Enríquez27, Alejandro Moles-Fernández27,29, Gemma Montalban27,30, Lars Ootes31, Selen Özkan31, Natàlia Padilla31, Casandra Riera31, Xavier De la Cruz31, Mark Diekhans25, Peter Huwe32-33, Qiong Wei32,34, Qifang Xu32, Roland L.

Dunbrack32, Valer Gotea35, Laura Elnitski35, Gennady Margolin35, Piero Fariselli36-37, Ivan V. Kulakovskiy38-39, Vsevolod J. Makeev38, Dmitry D. Penzar38,40, Ilya E. Vorontsov38-39, Alexander V. Favorov11,38, Julia R. Forman41-42, Marcia Hasenahuer43-44, Maria S. Fornasari43, Gustavo Parisi43, Ziga Avsec45, Muhammed H. Çelik45-46, Thi Yen Duong Nguyen45, Julien Gagneur45, Fang-Yuan Shi47, Matthew D. Edwards13,48, Yuchun Guo13,49, Kevin Tian13,50, Haoyang Zeng13,51, David K. Gifford13, Jonathan Göke52, Jan Zaucha53-54, Julian Gough55, Graham R.S. Ritchie44,56, Adam Frankish44,57, Jonathan M. Mudge44,57, Jennifer Harrow57-58, Erin L. Young59, Yao Yu60, Chad D. Huff60, Katsuhiko Murakami61-62, Yoko Nagai61,63, Tadashi Imanishi61,64, Christopher J.

Mungall65, Julius O.B. Jacobsen57,66, Dongsup Kim67, Chan-Seok Jeong67-68, David T. Jones69, Mulin Jun Li70-71, Violeta Beleva Guthrie11,54, Rohit Bhattacharya11,72, Yun-Ching Chen11,73, Christopher Douville11, Jean Fan11, Dewey Kim7,11, David Masica11, Noushin Niknafs11, Sohini Sengupta11,74, Collin Tokheim6,11,75, Tychele N. Turner11,76, Hui Ting Grace Yeo11,52, Rachel Karchin11, Sunyoung Shin77-78, Rene Welch77, Sunduz Keles77, Yue Li13,79, Manolis Kellis7,13, Carles Corbi-Verge80-81, Alexey V. Strokach80, Philip M. Kim80, Teri E. Klein50, Rahul Mohan82- 83, Nicholas A. Sinnott-Armstrong50, Michael Wainberg82,84, Anshul Kundaje50, Nina Gonzaludo85- 86, Angel C.Y. Mak85,87, Aparna Chhibber88-89, Hugo Y.K. Lam88,90, Dvir Dahary91, Simon Fishilevich92, Doron Lancet92, Insuk Lee93, Benjamin Bachman94, Panagiotis Katsonis94, Rhonald C. Lua94, Stephen J. Wilson94-95, Olivier Lichtarge94, Rajendra R. Bhat96, Laksshman Sundaram50, Vivek Viswanath96, Riccardo Bellazzi97, Giovanna Nicora97-98, Ettore Rizzo98, Ivan Limongelli98, Aziz M. Mezlini80, Ray Chang99, Serra Kim99, Carmen Lai99, Robert O'Connor99-100, Scott Topper99, Jeroen van den Akker99, Alicia Y. Zhou99, Anjali D. Zimmer99, Gilad Mishne99, Timothy R. Bergquist101-102, Marcus R. Breese85,103, Rafael F. Guerrero3,104, Yuxiang Jiang3, Nikki Kiga101, Biao Li103,105, Matthew Mort103,106, Kymberleigh A. Pagel3, Vikas Pejaver101,107, Moses H.

Stamboulian3, Janita Thusberg103, Sean D. Mooney101, Predrag Radivojac1,3, Nuttinee Teerakulkittipong4,108, Chen Cao4,109, Kunal Kundu4,110, Yizhou Yin4, Chen-Hsin Yu4, Maya Zuhl4,8, Lipika R. Pal4, John Moult4, Michael Kleyman4,111, Chiao-Feng Lin20,112, Mary Stackpole4,113, Stephen M. Mount4, Gökcen Eraslan7,114, Nikola S. Mueller114, Tatsuhiko Naito115, Aliz R. Rao30, Johnathan R. Azaria116-117, Aharon Brodie116, Yanay Ofran116, Aditi Garg118, Debnath Pal118, Alex Hawkins-Hooker41,69, Henry Kenlay41,119, John Reid41,120, Eliseos J.

Mucaki121, Peter K. Rogan121, Jana M. Schwarz122, David B. Searls123, Gyu Rie Lee101,124, Chaok Seok124, Andreas Krämer125, Sohela Shah125-126, ChengLai V. Huang2,127, Jack F. Kirsch2, Maxim Shatsky65, Yue Cao128, Haoran Chen128-129, Mostafa Karimi127-128, Oluwaseyi Moronfoye128, Yuanfei Sun128, Yang Shen128, Ron Shigeta130-131, Colby T. Ford132, Conor Nodzak132, Aneeta Uppal132-133, Xinghua M. Shi132,134, Thomas Joseph5, Sujatha Kotte5, Sadhna Rana5, Aditya Rao5, V.G. Saipradeep5, Naveen Sivadasan5, Uma Sunderam5, Rajgopal Srinivasan5, Mario Stanke135, Andrew Su136, Ivan Adzhubey137-138, Daniel M. Jordan107,139, Shamil Sunyaev139, Frederic Rousseau26, Joost Schymkowitz26, Joost Van Durme26, Sean V. Tavtigian59, Marco Carraro36, Silvio E. Tosatto36, Orit Adato116, Liran Carmel140, Noa E. Cohen140, Tzila Fenesh116, Tamar Holtzer116, Tamar Juven-Gershon116, Ron S. Unger116, Abhishek Niroula141, Ayodeji

35

Olatubosun142, Jouni Väliaho142, Yang Yang143, Mauno Vihinen141-142, Mary E. Wahl34,139, Billy Chang144, Ka Chun Chong144, Inchi Hu145-146, Rui Sun144,147, William Ka Kei Wu144, Xiaoxuan Xia144, Benny C. Zee144, Maggie H. Wang144, Meng Wang47, Chunlei Wu136, Yutong Lu147, Ken Chen147, Yuedong Yang148-150, Christopher M. Yates151-152, Anat Kreimer2,21, Zhongxia Yan2,13, Nir Yosef2, Huying Zhao150, Zhipeng Wei153, Zhaomin Yao153, Fengfeng Zhou153, Lukas

Challenge Data Providers

Roxana Daneshjou50, Russ B. Altman50, Fumitaka Inoue85,156, Nadav Ahituv85, Adam P. Arkin2, Federica Lovisa36,157, Paolo Bonvini36,121, Sarah Bowdin158, Stefano Gianni159, Elide Mantuano160, Velia Minicozzi161, Leonore Novak159, Alessandra Pasquo162, Annalisa Pastore163, Maria Petrosino164-165, Rita Puglisi42, Angelo Toto159, Liana Veneziano160, Roberta Chiaraluce164, Mad P.

Ball139,166, Jason R. Bobe139,167, George M. Church139, Valerio Consalvi159, Matthew Mort103,106, David N. Cooper106, Bethany A. Buckley126, Molly B. Sheridan11, Garry R. Cutting168, Maria C. Scaini169, Kamil J. Cygan110,170, Alger M. Fredericks170, David T. Glidden170, Christopher Neil170- 171, Christy L. Rhine170-171, William G. Fairbrother170, Aileen Y. Alontaga172, Aron W. Fenton172, Kenneth A. Matreyek101,173, Lea M. Starita101, Douglas M. Fowler101, Britt-Sabina Löscher174, Andre Franke175, Scott I. Adamson176, Brenton R. Graveley176, Joe W. Gray177, John P. Kane85, Maria Kousi178, Nicholas Katsanis179-180, Max Schubach122, Martin Kircher122, Nina Gonzaludo85- 86, Angel C.Y. Mak85,87, Paul L.F. Tang85,181, Pui-Yan Kwok85, Wyatt T. Clark182, Guoying K.

Yu182-183, Jonathan H. LeBowitz182, Francesco Benedicenti184, Elisa Bettella36, Stefania Bigoni185, Federica Cesca36, Isabella Mammi186, Cristina Marino-Buslje187, Donatella Milani188, Angela Peron189-191, Roberta Polli36, Stefano Sartori36,157, Franco Stanzial184, Irene Toldo36, Licia Turolla192, Maria C. Aspromonte36, Mariagrazia Bellini36, Emanuela Leonardi36, Xiaoming Liu193- 194, Christian Marshall80, Richard McCombie195, Lisa Elefanti169, Chiara Menin169, M. Stephen Meyn77,196, Alessandra Murgia36, Kari C.Y. Nadeau50, Lipika R. Pal4, John Moult4, Susan L.

Neuhausen197, Robert L. Nussbaum126,198, Mehdi Pirooznia73,199, James B. Potash168, Dago F. Dimster-Denk2, Jasper D. Rine2, Jeremy R. Sanford25, Michael Snyder50, Sean V. Tavtigian59, Atina G. Cote80,200, Song Sun80,200, Marta W. Verby80,200, Jochen Weile80,200, Frederick P. Roth80,200, Ryan Tewhey201, Pardis C. Sabeti7, Joan Campagna85, Marwan M. Refaat85,202, Julianne Wojciak85, Soren Grubb203, Jay Shendure101, Amanda B. Spurdle204, Dimitri J. Stavropoulos80, Nephi A. Walton205-206, Peter P. Zandi168, Elad Ziv85

Ethics Forum

Wylie Burke101, Flavia Chen85,139, Lawrence R. Carr123, Selena Martinez123, Jodi Paik123, Julie Harris-Wai85, Mark Yarborough207, Stephanie M. Fullerton208, Barbara A. Koenig85

Assessors

Roxana Daneshjou50, Gregory McInnes50,209, Russ B. Altman50, Dustin Shigaki11, Michael A. Beer11, Aashish Adhikari2,18, John-Marc Chandonia2,65, Mabel Furutsuki2, Zhiqiang Hu2, Laura Kasak2,210, Changhua Yu2,211, Vikas Pejaver101,107, Yana Bromberg21, Castrense Savojardo23, Rui Chen50,88, Melissa S. Cline25, Qifang Xu32, Roland L. Dunbrack32, Iddo Friedberg212, Gad A.

Getz7,138,213, Qian Cong214, Lisa N. Kinch214, Jing Zhang214, Nick V. Grishin214, Alin Voskanian215, Maricel G. Kann215, Wyatt T. Clark182, Elizabeth Tran82, Nilah M. Ioannidis2, Maria C. Aspromonte36, Mariagrazia Bellini36, Emanuela Leonardi36, Jesse M. Hunter77,216, Rupa Udani77,217, M. Stephen Meyn77,196, Binghuang Cai101, Sean D. Mooney101, Alexander A.

36

Morgan50,218, Lipika R. Pal4, John Moult4, Stephen M. Mount4, Alessandra Murgia36, Robert L. Nussbaum126,198, Jeremy R. Sanford25, Artem Sokolov25,138, Joshua M. Stuart25, Shamil Sunyaev139, Sean V. Tavtigian59, Marco Carraro36, Manuel Giollo36,127, Giovanni Minervini36, Alexander M.

Monzon36, Silvio E. Tosatto36, Anat Kreimer2,21, Nir Yosef2

Advisory Board And Scientific Council

Russ B. Altman50, Serafim Batzoglou18,219, Yana Bromberg21, Atul J. Butte50,85, George M. Church139, Garry R. Cutting168, Laura Elnitski35, Marc S. Greenblatt220, Reece K. Hart221, Ryan Hernandez79,85, Tim J.P. Hubbard42, Scott Kahn18,222, Rachel Karchin11, Anne O’Donnell-Luria7, M. Stephen Meyn77,196, Sean D. Mooney101, Alexander A. Morgan50,218, Pauline C. Ng52, Robert L.

Nussbaum126,198, John Shon18,223, Michael Snyder50, Shamil Sunyaev139, Sean V. Tavtigian59, Scott

John Moult4, Steven E. Brenner2

College Park, Maryland, USA; 5. Tata Consultancy Services, Hyderabad, India; 6. Dana-Farber Cancer Institute, Boston, Massachusetts, USA; 7. Broad Institute of MIT and Harvard, Cambridge, Massachusetts, USA; 8. ICF International, Cambridge, Massachusetts, USA; 9.

California, USA; 13. Massachusetts Institute of Technology, Cambridge, Massachusetts, USA; 14. Encoded Genomics, South San Francisco, California, USA; 15. Institute for Systems Biology, Seattle, Washington, USA; 16. Earle A. Chiles Research Institute, Providence Health & Leuven, Leuven, Belgium; 27. Vall d’Hebron Institute of Oncology, Barcelona, Spain; 28.

Barcelona, Spain; 30. Université Laval, Québec, Québec, Canada; 31. Vall d’Hebron Institute of Research, Barcelona, Spain; 32. Fox Chase Cancer Center, Philadelphia, Pennsylvania, USA; 33. Washington, USA; 35. National Human Genome Research Institute, Bethesda, Maryland, USA; Institute of General Genetics, Moscow, Russia; 39. Institute of Protein Research, Pushchino, Cambridge, Cambridge, UK; 42. King’s College London, London, UK; 43. Universidad Nacional de Quilmes, Bernal, Argentina; 44. European Molecular Biology Laboratory - Beijing, China; 48. Verily Life Sciences, South San Francisco, California, USA; 49. CAMP4

37

USA; 51. Insitro, South San Francisco, California, USA; 52. Genome Institute of Singapore, Laboratory of Molecular Biology, Cambridge, UK; 56. Amazon Web Services, Seattle, Washington, USA; 57. Wellcome Sanger Institute, Hinxton, UK; 58. ELIXIR, Hinxton, UK; 59.

Center, Houston, Texas, USA; 61. National Institute of Advanced Industrial Science and Technology, Tokyo, Japan; 62. Fujitsu Ltd., Tokyo, Japan; 63. Varinos Inc., Tokyo, Japan; 64. Advanced Institute of Science and Technology, Daejeon, South Korea; 68. Korea Institute of Tianjin, China; 72. Williams College, Williamstown, Massachusetts, USA; 73. Johnson & Johnson, New Brunswick, New Jersey, USA; 74. PierianDx, Creve Coeur, Missouri, USA; 75.

Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA; 76. Washington California, USA; 83. Dashworks, Boston, Massachusetts, USA; 84. Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health, Toronto, Ontario, Canada; 85.

Biosciences, Menlo Park, California, USA; 87. CytomX Therapeutics, South San Francisco, California, USA; 88. Personalis, Menlo Park, California, USA; 89. Bristol Myers Squibb, Redwood City, California, USA; 90. HypaHub, Sunnyvale, California, USA; 91. LifeMap Sciences Inc., Alameda, California, USA; 92. Weizmann Institute, Rehovot, Israel; 93. Yonsei Seattle, Washington, USA; 102. Sage Bionetworks, Seattle, Washington, USA; 103. Buck Raleigh, North Carolina, USA; 105. Gilead, Foster City, California, USA; 106. Cardiff California, USA; 110. Regeneron, Tarrytown, New York, USA; 111. Moderna, Cambridge, Massachusetts, USA; 112. DNANexus, Mountain View, California, USA; 113.

EarlyDiagnostics, Los Angeles, California, USA; 114. Helmholtz Zentrum Muenchen, Aviv, Israel; 117. Imperva, San Mateo, California, USA; 118. Indian Institute of Science, Universitätsmedizin Berlin, Berlin, Germany; 123. No affiliation; 124. Seoul National Francisco, California, USA; 127. Amazon, Seattle, Washington, USA; 128. Texas A&M

38

Pennsylvania, USA; 130. IndieBio, San Francisco, USA; 131. iAccelerate, North Wollongong, Diego, California, USA; 137. Brigham and Women’s Hospital, Boston, Massachusetts, USA; Australia; 150. Sun Yat-sen Memorial Hospital, Guangzhou, China; 151. Imperial College, Changchun, China; 154. CeMM Research Center for Molecular Medicine of the Austrian Academy of Sciences, Vienna, Austria; 155. Shenzhen Bay Laboratory, Shenzhen, China; 156.

Carolina, USA; 167. PersonalGenomes.org, Sanford, North Carolina, USA; 168. Johns Hopkins Therapeutics Inc., Durham, North Carolina, USA; 181. AccuraGen Inc., Menlo Park, California, USA; 182. BioMarin, Novato, California, USA; 183. Global Blood Therapeutics, South San Francisco, California, USA; 184. Regional Hospital of Bolzano, Bolzano, Italy; 185. Ferrara Utah School of Medicine, Salt Lake City, Utah, USA; 192. AULSS 2 Marca Trevigiana, Treviso, South Florida, Tampa, Florida, USA; 195. Cold Spring Harbor Laboratory, Cold Spring Harbor, New York, USA; 196. Hospital for Sick Children, Toronto, Ontario, Canada; 197. Beckman Research Institute of City of Hope, Duarte, California, USA; 198. Myriad, Salt Lake City, Utah, USA; 199. National Institutes of Health, Bethesda, Maryland, USA; 200. Mount Sinai Health System, New York City, New York, USA; 201. Jackson Laboratory, Bar Harbor, Maine, USA;

39

Copenhagen, Copenhagen, Denmark; 204. QIMR Berghofer Medical Research Institute, Brisbane, Australia; 205. Geisinger Genomic Medicine Institute, Danville, Pennsylvania, USA; Tartu, Tartu, Estonia; 211. California Institute of Technology, Pasadena, California, USA; 212.

Nationwide Children’s Hospital, Columbus, Ohio, USA; 217. Sema4, Stamford, Connecticut, USA; 218. Khosla Ventures, Menlo Park, California, USA; 219. Seer.bio, Redwood City, Palo Alto, California, USA; 222. LunaPBC, San Diego, California, USA; 223. SerImmune, Institute of Standards and Technology, Portland, Oregon, USA

Conflict Of Interest

Principal authors of this paper participated as predictors in many of the CAGI challenges reported. The unified numerical framework employed for reanalysis of the challenges yields results that are consistent with those obtained by the independent assessors of each challenge and in particular selected methods are the highest ranked in the independent assessments.

Nevertheless, while every care was taken to mitigate any potential biases in this work, the authors’ participation in CAGI may have affected the presentation of findings, including the selection of challenges, metrics, assessment criteria and emphasis given on particular results.

VBG is a current employee and shareholder of AstraZeneca; RB is a shareholder of enGenome; AJB is a co-founder and consultant to Personalis and NuMedii as well as a consultant to Samsung, Mango Tree Corporation and in the recent past, 10x Genomics, Helix and Pathway; Carles Corbi-Verge is a computational scientist at the drug discovery company; Cyclica INC and is compensated with income and equity; KC is one of the Regeneron authors and owns options and/or stock of the company; DD is Chief Scientist at Geneyx Genomex Ltd; CD is a consultant to Exact Sciences and is compensated with income and equity; GAG receives research funds from IBM and Pharmacyclics, and is an inventor on patent applications related to MSMuTect, MSMutSig, MSIDetect, POLYSOLVER and SignatureAnalyzer-GPU, and is a founder, consultant and holds privately held equity in Scorpion Therapeutics; NG is an employee and stockholder at Pacific Biosciences; RH is a paid consultant for Invitae and Scientific Advisory Board member for Variant Bio; AK is a consultant at Illumina Inc., Scientific Advisory Board member of OpenTargets; KK is one of the Regeneron authors and owns options and/or stock of the company; IL is an employer and stockholder of enGenome; MSM owns stock in PhenoTips; GN is an employee of enGenome; AOD-L is a member of the Scientific Advisory Board of Congenica; ER is a shareholder of enGenome; PKR is the founder of CytoGnomix; FPR is a shareholder in Ranomics and SeqWell, an advisor for SeqWell, BioSymetrics, and Constantiam BioSciences, and has received research sponsorships from Biogen, Alnylam, Deep Genomics and Beam Therapeutics; PCS is the co-founder and shareholder of Sherlock

40

Biosciences, a board member and shareholder of Danaher Corporation, and has filed patents related to this work; PLFT is an employer and stockholder in AccuraGen; RT has filed patents related to this work; MHW is a shareholder of Beth Bioinformatics Co., Ltd.; JZ is an employee agreement from TCS.

Acknowledgments

The authors are grateful for the contributions of Talal Amin, Patricia Babbitt, Eran Bachar, Stefania Boni, Kirstine Calloe, Ombretta Carlet, Shann-Ching Chen, Chien-Yuan Chen, Jun Cheng, Luigi Chiricosta, Alex Colavin, Qian Cong, Emma D’Andrea, Carla Davis, Xin Feng, Carlo Ferrari, Yao Fu, Alessandra Gasparini, David Goldgar, Solomon Grant, Steve Grossman, Todd Holyoak, Rick Lathrop, Xiaolin Li, Quewang Liu, Mary Malloy, Beth Martin, Zev Medoff, Nasim Monfared, Susanna Negrin, Eric Odell, Michael Parsons, Nathan Pearson, Alexandra Piryatinska, Catherine Plotts, Jennifer Poitras, Clive Pulinger, Francesco Reggiani, Melvin M.

Scheinman, Nicole Schmitt, George Shackelford, Vasily Sitnik, Fiorenza Soli, Qingling Tang, Nancy Mutsaers Thomsen, Jing Wang, Chenling Xiong, Lijing Xu, Shuhan Yang, Lijun Zhan, and Huiying Zhao.

Funding

The content of this work is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. The CAGI experiments and conferences were supported by National Institutes of Health (NIH) awards U41HG007346, U24HG007346, and R13HG006650 to SEB, as well as a Research Agreement with Tata Consultancy Services (TCS) to SEB, and a supplement to NIH U19HD077627 to RLN. NIH U41HG007346 supported ANA, GA, CB, SEB, J-MC, RAH, ZH, LK, BAK, MKL, ZM, JM, AJN, RCP, YW, SFY; NIH U24HG003467 supported CB, SEB, J-MC, ZH, SMF, RCP, PR, SJ; the TCS research agreement supported CAGI activities of ANA, CB, DB, J-MC, ZH, LK, and SFY; NIH U19HD077627 supported RAH and SEB; NIH R13HG006650 provided CAGI travel support to SA, ANA, GA, JRA, CB, DB, MAB, BB, SEB, BAB, YC, EC, MC, HC, J-MC, JC, MSC, RD, DD, RLD, MDE, BF, AWF, IF, SMF, MF, NG, MSG, NVG, JJH, RH, ZH, CLVH, TJPH, SK, RK, MK, PK, DK, BAK, AK, RHL, MKL, DM, GM, MSM, SDM, AAM, JM, SMM, AJN, AO, KAP, LRP, VRP, PR, SR, GS, AS, JMS, SS, RST, CT, AV, MEW, RW, SJW, ZY, SFY, JZ, MZ; NIH U41HG007346 provided travel support to RK, ZM, SDM; UC Berkeley funds additionally provided CAGI travel support for YB, LS. CAGI projects and participants were further supported as follows: OA by an EMBO Installation Grant (No: 4163), TÜBİTAK (Grant IDs: 118C320, 121E365), TÜSEB (Grant ID: 4587); IA by NIH R01GM078598, R35GM127131, R01HG010372; RBA by NIH GM102365; APA by the Office of Science, Office of Biological 05CH11231; ZA, MHÇ, JC, JG and TYDN by a Competence Network for Technical, Scientific High Performance Computing in Bavaria KONWIHR, a Deutsche Forschungsgemeinschaft fellowship through the Graduate School of Quantitative Biosciences Munich and an NVIDIA hardware grant; MAB, AP, DS by NIH U01HG009380; SB by the Asociación Española contra el

41

Cáncer; PB, FL by the Fondazione CARIPARO, Padova, Italy; APB, SD by NIH U24HG009293; WB by the NIH and the Greenwall Foundation; AJB by the Barbara and Gerson Bakar Foundation, and Priscilla Chan and Mark Zuckerberg; KC, SG, NS, NMT by the Danish National Research Foundation Centre for Cardiac Arrhythmia; EC, PT by MIUR 201744NR8S; RC, VC by the 201744NR8S/Ministero Istruzione, Università e Ricerca, PRIN project; MSC by NIH U01CA242954; MM, DNC by Qiagen through a License Agreement with Cardiff Innovación (PID2019-111217RB-I00) and Ministerio de Economía y Competitividad (SAF2016- 80255-R and BIO2012-40133) and a European Regional Development Fund (Pirepred- EFA086/15); MD by NIH U41HG007234; OD by FIS PI12/02585 and PI15/00355 from the Spanish Instituto de Salud Carlos III (ISCIII) funding, an initiative of the Spanish Ministry of Economy and Innovation partially supported by European Regional Development FEDER Funds; MDE, DKG, YG, KT, HZ by NIH R01HG008363, U01HG007037, U41HG007346, R13HG006650; LE, VG, MSG by a grant from the Intramural Research Program of the NHGRI to LE (1ZIAHG200323-14); WGF by NIH R01GM127472; PF by MIUR 201744NR8S and 20182022D15D18000410001; MSF, MH, GP by the Comision Nacional de Investigaciones Cientificas y Tecnicas (CONICET) [Grant ID: PIP 112201101–01002] ; Universidad Nacional de Quilmes [Grant ID: 1402/15]; DMF by NIH R01GM109110, R01HG010461; AF, JH, JMM by Wellcome Trust grant ; and NIH U54HG004555; AG, DP by intramural funding; NVG by NIH GM127390 and the Welch Foundation I-1505; SG-E by PI13/01711 and PI16/01218 to SGE from the Spanish Instituto de Salud Carlos III (ISCIII) funding, an initiative of the Spanish Ministry of Economy and Innovation partially supported by European Regional Development FEDER Funds. SGE was also supported by the Miguel Servet Progam [CP16/00034]; TI by JSPS KAKENHI Grant Number 16HP8044; JOBJ, CJM by NIH R24OD011883; SK by NIH U01HG007019, RO1HG003747; MK by NIH AG054012, AG058002, AG062377, NS110453, NS115064, AG062335, AG074003, NS127187, AG067151, MH109978, MH119509, HG008155, DA053631; IVK by RSF 20-74-10075; CL, JVdA by Color Genomics; KAM by NIH R35GM142886; GM by NIH T32LM012409; JM by NIH R01GM120364, R01GM104436; LO, SÖ, NP, CR by the Spanish Ministerio de Ciencia e Innovación (PID2019-111217RB-I00) and Ministerio de Economía y Competitividad (SAF2016- 80255-R and BIO2012-40133); European Regional Development Fund (Pirepred-EFA086/15); VP by NIH K99LM012992; SDM and PR by R01LM009722 and R01MH105524; PR by Curie International Outgoing Fellowship PIOF-GA-2009-237751; PKR by Natural Sciences and Engineering Research Council of Canada 371758-2009, Canadian Breast Cancer Foundation, Canada Foundation For Innovation, Canada Research Chairs, Compute Canada and Western HG004233, HG010461, the Canada Excellence Research Chairs, and a Canadian Institutes of Health Research Foundation Grant; PCS by NIH UM1HG009435; JRS by NIH R35GM130361; CS by MUR PRIN2017 2017483NH8_002; CS by NRF-2020M3A9G7103933; YS by NIH R35GM124952; LMS by NIH RM1HG010461; RT by NIH 1UM1HG009435; JMS by NCI R01CA180778; SVT by NCI R01 CA121245; MV by Finnish Academy, Swedish Research Council, Swedish Cancer Society; MEW by NSF GRF, NIH GM068763; MHW by the National Natural Science Foundation of China (NSFC) [31871340, 71974165]; FZ by The Senior and Junior Technological Innovation Team (20210509055RQ), the Jilin Provincial Key Laboratory of Big Data Intelligent Computing (20180622002JC), and the Fundamental Research Funds for

42

the Central Universities, JLU; YZ by the Australia Research Council [DP210101875]; EZ by NCI K24CA169004, California Initiative to Advance Precision Medicine.

Supplementary Materials

Supplementary materials are available at https://github.com/genomeinterpretation/CAGI50 and the CAGI web site https://genomeinterpretation.org/.

Data And Code Avilability

CAGI challenge datasets, predictions, and assessment standards are available on the CAGI website for public access or under controlled access for registered CAGI participants, unless ethical concerns about sensitive human data preclude such sharing. Where possible, such sensitive human datasets are available via repositories such as dbGaP and European Genome- Phenome Archive. The code used for the analysis of this work is available at the GitHub repository https://github.com/genomeinterpretation/CAGI50 and the CAGI web site https://genomeinterpretation.org/.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.