Once a segmentation method is deployed in routine little is known about its real performance on new data. Due the absence of GT, it is not possible to assess performance using traditional evaluation measures. However, it is critical to be able to assess the level of accuracy on clinical data [1]}, and in particular, it is important to detect when an automatic segmentation method fails. Especially when the segmentation is an intermediate step within a larger automated processing pipeline where no visual quality control of the segmentation results is feasible. This is of high importance in large-scale studies such as the UK Biobank Imaging Study [2]} where automated methods are applied to large cohorts of several thousand images, and the segmentation is to be used for further statistical population analysis. In this work, we are asking the question whether it is possible to assess segmentation performance and detect failure cases when there is no GT available to compare with. One possible approach to monitor the segmentation performance is to occasionally select a random dataset, obtain a manual expert segmentation and compare it to the automatic one. While this can merely provide a rough estimate about the average performance of the employed segmentation method, in clinical routine we are interested in the per case performance and want to detect when the automated method fails. The problem is that the performance of a method might be substantially different on clinical data, and is usually lower than what is found through cross-validation on annotated data carried out beforehand due to several reasons. First, the annotated database is normally used during incremental method development for training, model selection and fine tuning of hyper-parameters. This can lead to overfitting [3]} which is a potential cause for lower performance on new data. Second, the clinical data might be different due to varying imaging protocols or artefacts caused by pathology. To this end, we propose a general framework for predicting the real performance of deployed segmentation methods on a per case basis in the absence of GT.
