Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> Accurate diagnosis of disease is a critical part of health care. New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased from nondiseased states. The partial area under the receiver operating characteristic (ROC) curve is a measure of diagnostic test accuracy. We present an interpretation of the partial area under the curve (AUC), which gives rise to a nonparametric estimator. This estimator is more robust than existing estimators, which make parametric assumptions. We show that the robustness is gained with only a moderate loss in efficiency. We describe a regression modeling framework for making inference about covariate effects on the partial AUC. Such models can refine knowledge about test accuracy. Model parameters can be estimated using binary regression methods. We use the regression framework to compare two prostate-specific antigen biomarkers and to evaluate the dependence of biomarker accuracy on the time prior to clinical diagnosis of prostate cancer. <s> BIB001 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> ∗Harvard University, tcai@hsph.harvard.edu †University of Washington, mspepe@u.washington.edu ‡University of Washington, tlumley@u.washington.edu ∗∗Fred Hutchinson Cancer Research Center, yzheng@fhcrc.org ††University of Vermont, Nancy.Jenny@uvm.edu This working paper is hosted by The Berkeley Electronic Press (bepress) and may not be commercially reproduced without the permission of the copyright holder. <s> BIB002 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver-operating-characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers. Copyright © 2007 John Wiley & Sons, Ltd. <s> BIB003 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> The performance of clinical tests for disease screening is often evaluated using the area under the receiver-operating characteristic (ROC) curve (AUC). Recent developments have extended the traditional setting to the AUC with binary time-varying failure status. Without considering covariates, our first theme is to propose a simple and easily computed nonparametric estimator for the time-dependent AUC. Moreover, we use generalized linear models with time-varying coefficients to characterize the time-dependent AUC as a function of covariate values. The corresponding estimation procedures are proposed to estimate the parameter functions of interest. The derived limiting Gaussian processes and the estimated asymptotic variances enable us to construct the approximated confidence regions for the AUCs. The finite sample properties of our proposed estimators and inference procedures are examined through extensive simulations. An analysis of the AIDS Clinical Trials Group (ACTG) 175 data is further presented to show the applicability of the proposed methods. The Canadian Journal of Statistics 38:8–26; 2010 © 2009 Statistical Society of Canada <s> BIB004 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> To assess the classification accuracy of a continuous diagnostic result, the receiver operating characteristic (ROC) curve is commonly used in applications. The partial area under the ROC curve (pAUC) is one of widely accepted summary measures due to its generality and ease of probability interpretation. In the field of life science, a direct extension of the pAUC into the time-to-event setting can be used to measure the usefulness of a biomarker for disease detection over time. Without using a trapezoidal rule, we propose nonparametric estimators, which are easily computed and have closed-form expressions, for the time-dependent pAUC. The asymptotic Gaussian processes of the estimators are established and the estimated variance-covariance functions are provided, which are essential in the construction of confidence intervals. The finite sample performance of the proposed inference procedures are investigated through a series of simulations. Our method is further applied to evaluate the classification ability of CD4 cell counts on patient's survival time in the AIDS Clinical Trials Group (ACTG) 175 study. In addition, the inferences can be generalized to compare the time-dependent pAUCs between patients received the prior antiretroviral therapy and those without it. <s> BIB005 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> In biomedical studies, statistical approaches based on the Receiver Operating Characteristic (ROC) analysis have been extensively used in the evaluation of classification performance of markers and construction of classifiers. In this article, we investigate time-dependent ROC approaches for censored survival data. While most existing studies have been focused on uncensored and right-censored data, insufficient attention has been paid to other censoring schemes. This study advances from existing studies by investigating more diverse censoring schemes and developing ROC measurements under such censoring. Both estimation and inference are investigated. We conduct simulation and find satisfactory performance of the proposed approaches. We apply the proposed approaches to two real data sets, compare the prognostic power of markers, and investigate whether their linear combinations have better prognostic performance. We also explore graphical tools that can assist diagnostics and efficiently monitor the classification performance. Copyright © 2011 John Wiley & Sons, Ltd. <s> BIB006 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> Summary Recently meta-analysis has been widely utilized to combine information across multiple studies to evaluate a common effect. Integrating data from similar studies is particularly useful in genomic studies where the individual study sample sizes are not large relative to the number of parameters of interest. In this article, we are interested in developing robust prognostic rules for the prediction of t-year survival based on multiple studies. We propose to construct a composite score for prediction by fitting a stratified semiparametric transformation model that allows the studies to have related but not identical outcomes. To evaluate the accuracy of the resulting score, we provide point and interval estimators for the commonly used accuracy measures including the time-specific receiver operating characteristic curves, and positive and negative predictive values. We apply the proposed procedures to develop prognostic rules for the 5-year survival of breast cancer patients based on five breast cancer genomic studies. <s> BIB007 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> Risk prediction models have been widely applied for the prediction of long-term incidence of disease. Several parameters have been identified and estimators developed to quantify the predictive ability of models and to compare new models with traditional models. These estimators have not generally accounted for censoring in the survival data normally available for fitting the models. This paper remedies that problem. The primary parameters considered are net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We have previously similarly considered a primary measure of concordance, area under the ROC curve (AUC), also called the c-statistic. We also include here consideration of population attributable risk (PAR) and ratio of predicted risk in the top quintile of risk to that in the bottom quintile. We evaluated estimators of these various parameters both with simulation studies and also as applied to a prospective study of coronary heart disease (CHD). Our simulation studies showed that in general our estimators had little bias, and less bias and smaller variances than the traditional estimators. We have applied our methods to assessing improvement in risk prediction for each traditional CHD risk factor compared to a model without that factor. These traditional risk factors are considered valuable, yet when adding any of them to a risk prediction model that has omitted the one factor, the improvement is generally small for any of the parameters. This experience should prepare us to not expect large values of the risk prediction improvement evaluation parameters for any new risk factor to be discovered. Copyright © 2010 John Wiley & Sons, Ltd. <s> BIB008 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> In this paper we investigate the addition of new variables to an existing risk prediction model and the subsequent impact on discrimination quantified by the area under the receiver operating characteristics curve (AUC of ROC). Based on practical experience, concerns have emerged that the significance of association of the variable under study with the outcome in the risk model does not correspond to the significance of the change in AUC: that is, often the variable is significant, but the change in AUC is not. This paper demonstrates that under the assumption of multivariate normality and employing linear discriminant analysis (LDA) to construct the risk prediction tool, statistical significance of the new predictor(s) is equivalent to the statistical significance of the increase in AUC. Under these assumptions the result extends asymptotically to logistic regression. We further show that equality of variance–covariance matrices of predictors within cases and non-cases is not necessary when LDA is used. However, our practical example from the Framingham Heart Study data suggests that the finding might be sensitive to the assumption of normality. Copyright © 2011 John Wiley & Sons, Ltd. <s> BIB009 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> To develop more targeted intervention strategies, an important research goal is to identify markers predictive of clinical events. A crucial step toward this goal is to characterize the clinical performance of a marker for predicting different types of events. In this article, we present statistical methods for evaluating the performance of a prognostic marker in predicting multiple competing events. To capture the potential time-varying predictive performance of the marker and incorporate competing risks, we define time- and cause-specific accuracy summaries by stratifying cases based on causes of failure. Such definition would allow one to evaluate the predictive accuracy of a marker for each type of event and compare its predictiveness across event types. Extending the nonparametric crude cause-specific receiver operating characteristics curve estimators by Saha and Heagerty (2010), we develop inference procedures for a range of cause-specific accuracy summaries. To estimate the accuracy measures and assess how covariates may affect the accuracy of a marker under the competing risk setting, we consider two forms of semiparametric models through the cause-specific hazard framework. These approaches enable a flexible modeling of the relationships between the marker and failure times for each cause, while efficiently accommodating additional covariates. We investigate the asymptotic property of the proposed accuracy estimators and demonstrate the finite sample performance of these estimators through simulation studies. The proposed procedures are illustrated with data from a prostate cancer prognostic study. <s> BIB010 </s> Time-dependent AUC with right-censored data: a survey study <s> Discussion <s> Authors have proposed new methodology in recent years for evaluating the improvement in prediction performance gained by adding a new predictor, Y, to a risk model containing a set of baseline predictors, X, for a binary outcome D. We prove theoretically that null hypotheses concerning no improvement in performance are equivalent to the simple null hypothesis that Y is not a risk factor when controlling for X, H0 : P(D = 1 | X,Y ) = P(D = 1 | X). Therefore, testing for improvement in prediction performance is redundant if Y has already been shown to be a risk factor. We also investigate properties of tests through simulation studies, focusing on the change in the area under the ROC curve (AUC). An unexpected finding is that standard testing procedures that do not adjust for variability in estimated regression coefficients are extremely conservative. This may explain why the AUC is widely considered insensitive to improvements in prediction performance and suggests that the problem of insensitivity has to do with use of invalid procedures for inference rather than with the measure itself. To avoid redundant testing and use of potentially problematic methods for inference, we recommend that hypothesis testing for no improvement be limited to evaluation of Y as a risk factor, for which methods are well developed and widely available. Analyses of measures of prediction performance should focus on estimation rather than on testing for no improvement in performance. Copyright © 2013 John Wiley & Sons, Ltd. <s> BIB011
While the AUC is uniquely defined in the context of the evaluation of diagnostic tests, its extension to prognostic markers has led to the development of a variety of definitions: these definitions vary according to the underlying definitions of cases (incident or cumulative) and controls (static or dynamic), and also depend on the study characteristics (the marker can be measured only once or repeatedly and competing events may be considered, or not). Regarding the choice of the retained definition for cases and controls, no clear guidance has really emerged in the literature. It seems however that the cumulative/dynamic definition may be more appropriate for clinical decisions making (enrollment in clinical trials for instance) while the incident/static definition may be more appropriate for "pure" evaluation of the marker (if interpretation of trends of AUC values over time is of particular interest for instance). Once this definition has been chosen, appropriate estimators are available, depending on various assumptions (independence of the marker and censoring, proportional hazards, ...), and we presented most of them in this review article. For the sake of brevity, we were not able to cover some interesting extensions of time-dependent AUCs. In particular, covariate specific time-dependent ROC curves and AUCs have been studied in order to adjust the discrimination of a marker for external covariates (age, gender, ...). We refer the reader to BIB004  for the standard setting, BIB002 for the longitudinal setting and BIB010 for the competing events setting. In addition, some authors advocate that not the entire ROC curve is of interest and the area under only a portion of it should be computed, leading to the so-called partial AUC BIB001 . In the context of prognostic markers, Hung and Chiang BIB005 proposed a nonparametric estimator of the cumulative/dynamic time-dependent version of the partial AUC. Other interesting extensions include diverse censoring patterns BIB006 (only right-censoring was considered in this review) and the combination of results from multiple studies BIB007 which is particularly useful in genomic studies. Another closely related topic is the evaluation of the added predictive ability of a new marker: for instance, we may wonder how better a risk score would be if we added some biological markers (SNPs, genes, ...). We refer the reader to the works in BIB008 BIB009 BIB003 BIB011 for some insights, noticing though that most of these works do not cover the right-censored setting considered in our review.