Evaluating the Clinical Realism of Synthetic Chest X-Rays Generated Using Progressively Growing GANs

Abstract: Chest x-rays are a vital tool in the workup of many patients. Similar to most medical imaging modalities, they are profoundly multi-modal and are capable of visualising a variety of combinations of conditions. There is an ever pressing need for greater quantities of labelled data to develop new diagnostic tools, however this is in direct opposition to concerns regarding patient confidentiality which constrains access through permission requests and ethics approvals. Previous work has sought to address these concerns by creating class-specific GANs that synthesise images to augment training data. These approaches cannot be scaled as they introduce computational trade offs between model size and class number which places fixed limits on the quality that such generates can achieve. We address this concern by introducing latent class optimisation which enables efficient, multi-modal sampling from a GAN and with which we synthesise a large archive of labelled generates. We apply a PGGAN to the task of unsupervised x-ray synthesis and have radiologists evaluate the clinical realism of the resultant samples. We provide an in depth review of the properties of varying pathologies seen on generates as well as an overview of the extent of disease diversity captured by the model. We validate the application of the Fréchet Inception Distance (FID) to measure the quality of x-ray generates and find that they are similar to other high resolution tasks. We quantify x-ray clinical realism by asking radiologists to distinguish between real and fake scans and find that generates are more likely to be classed as real than by chance, but there is still progress required to achieve true realism. We confirm these findings by evaluating synthetic classification model performance on real scans. We conclude by discussing the limitations of PGGAN generates and how to achieve controllable, realistic generates.

\section{Introduction}
\label{sec:introduction}
Radiological investigations form an integral component of the diagnostic workup of many patients. Plain x-rays in particular, are often the most common initial modality used to screen, diagnose or monitor clinical conditions. These investigations typically require expert radiologist review for both quality control and diagnosis. Recent advances in machine learning, however, have allowed for techniques to automate or augment these processes~\cite{kermanyIdentifyingMedicalDiagnoses2018, majkowskaChestRadiographInterpretation2020,  rajpurkarCheXNetRadiologistLevelPneumonia2017}. This has achieved a level of performance surpassing individual expert radiologists in certain domains with a faster overall turnaround due to a reduction in the analysis time per image~\cite{majkowskaChestRadiographInterpretation2020}. In contrast to the opacity associated with the predictions made by earlier deep learning networks, newer techniques have made it far simpler to explain decisions and localise regions of importance for predictions~\cite{selvarajuGradCAMVisualExplanations2020, smilkovSmoothGradRemovingNoise2017} thus allowing for model-expert hybridisation. These models rely on large archives of labelled images that are costly to produce and are constrained by significant ethical restrictions relating to patient confidentiality.

This study serves to evaluate if medically plausible x-ray images can be created using Generative adversarial networks (GANs) for the production of teaching material and data augmentation for machine learning studies that does not compromise confidentiality. This is achieved by training these algorithms with open source, publicly accessible x-ray data.~\footnote{An ethics waiver was issued for this work by the Human Research Ethics Committee (Medical) of the University of the Witwatersrand, Johannesburg on 11/08/2020.} In doing so successfully, this would address challenges in machine learning and in education and training, by:

\begin{itemize}
    \item Providing data sets that do not compromise patient confidentiality, but provide the same outcomes;
    \item Providing augmented/boosted datasets (especially to address the issue of rare classes); and
    \item Producing x-rays with specified target pathologies.
\end{itemize}

This paper evaluates the background of the state of the art, the dataset used, and the development of a system capable of producing x-rays which appear to be medically viable. The results are presented and discussed.

\section{Background}

GANs form a class of generative methods rooted in game theory with the objective of training a generator to produce samples indistinguishable from a source distribution~\cite{goodfellowGenerativeAdversarialNetworks2014}. GANs are formulated by two networks in opposition to one another: a generator ($G$) whose goal is to generate plausible samples, and a discriminator ($D$) whose goal is to detect such samples. The generator operates by transforming a sample from a noise vector ($z$) into a sample which is similar to a chosen reference distribution, with the training signal provided by how effectively such a generate can fool the discriminator into classifying it as real. This configuration optimally reaches a Nash equilibrium when the generator produces samples indistinguishable from the reference set and the discriminator can no longer learn to detect generates~\cite{meschederWhichTrainingMethods2018, salimansImprovedTechniquesTraining2016}.
\newline \newline
The training of GANs to reach the Nash equilibrium remains an evolving problem as the optimisation of such networks requires the minimisation of a cost function in the hopes that a solution will be similar to the equilibrium. Numerous alterations to the training setup and loss functions have been utilised to improve sample quality and variability as well as the reliability of GAN training convergence. These improvements have enabled networks to reliably learn more complex distributions, however, convergence to the Nash equilibrium is still not guaranteed~\cite{meschederWhichTrainingMethods2018, salimansImprovedTechniquesTraining2016}.
The GAN formulation can be applied to a variety of distributions, one common use is that of image synthesis. The generation of images has progressed from the original, fully connected GAN used to produce low resolution images to the application of convolutional networks with the DCGAN architecture ~\cite{radfordUnsupervisedRepresentationLearning2016}. Recent work has expanded on this and has resulted in techniques capable of producing images at megapixel resolutions while retaining much of the quality and diversity  of the reference images~\cite{karrasStyleBasedGeneratorArchitecture2019, karrasAnalyzingImprovingImage2020, karrasProgressiveGrowingGANs2018}.
\newline\newline
Yi \emph{et al.}~\cite{yiGenerativeAdversarialNetwork2019} in their review of GANs for medical imaging, describe the ability to synthesise images as one of the most important uses of GANs due to the ability to generate images free of concerns regarding patient confidentiality as well as the benefits inherent in generating rare image categories to improve classification performance. Moradi \emph{et al.} \cite{moradiChestXrayGeneration2018} demonstrate a performance improvement by generating a specific class of x-rays using a DCGAN architecture \cite{radfordUnsupervisedRepresentationLearning2016} as part of their classification workflow. These images are of limited resolution (128x128) and focus entirely on cardiac abnormalities.
\newline \newline
Beers \emph{et al.}~\cite{beersHighresolutionMedicalImage2018} applied the progressive growing technique, wherein a GAN learns to generate images of increasing resolution~\cite{karrasProgressiveGrowingGANs2018} to images of the retinal fundus and MRI images of gliomas with a focus on retaining segmentation fields of generated images. They found that images could be produced that closely matched reference images and that segmentation maps from the generative network were of a similar quality to those of standard algorithms. Their analysis focuses on images from quite a constrained domain with limited variation given that the images are sourced from small tissue regions from a small number of classes.
\newline \newline
Chest X-Rays (CXRs) by comparison, cover a far broader region of the body and as such have significantly broader variation at both global and local levels. This density of information is in keeping with the number of abnormalities that can be detected, however, it may impair the performance of GANs as the capacity of the network may be exceeded by such variance.
\newline \newline
To evaluate the applicability to CXRs, we implement the Progressively Grown GAN (PGGAN) architecture~\cite{karrasProgressiveGrowingGANs2018} on the ChestX-ray14 dataset~\cite{wangChestXray8HospitalScaleChest} in a fully unsupervised manner and have a practicing radiologist evaluate the clinical plausibility of the various images. We note that the model captures global structure, such as the overall thoracic structure, but at times fails to reproduce finer details such as cardiac pacemakers or electrocardiograph leads. We examine the distribution of disease labels generated by the model in comparison to the source distribution through the use of a common classifier trained on the source images. We find that the model captures all the relevant disease classes in the correct proportion yet differs significantly from the prevalence in the source set. We further evaluate individual class performance by sampling from the PGGAN latent space and optimising to produce an image that maximises the classifier's score of the class of interest.

\section{Dataset}
\label{sec:dataset}
The ChestX-ray14 dataset is an update to the \emph{ChestX-ray8} dataset~\cite{wangChestXray8HospitalScaleChest}, a large, open medical X-Ray dataset published without restriction by the National Institutes of Health (NIH) Clinical Center. The dataset comprises \num{112120} x-rays from \num{30805} unique patients. The images comprise \num{15} classes, viz.:
\begin{AutoMultiColItemize}
    \item No Finding
    \item Atelectasis
    \item Cardiomegaly
    \item Consolidation
    \item Edema
    \item Effusion
    \item Emphysema
    \item Fibrosis
    \item Hernia
    \item Infiltration
    \item Mass
    \item Nodule
    \item Pleural Thickening
    \item Pneumonia
    \item Pneumothorax
\end{AutoMultiColItemize}
These images were collected from a clinical archive and should broadly reflect the typical clinical prevalence of these conditions within the community served by the NIH. 75\% of images are normal investigations. The remainder are made up of the various labels ranging from the most prevalent, infiltration (10\%), to the least, hernia (0.5\%). The diagnostic labels are accompanied by bounding boxes for feature localisation. These labels were created through natural language processing (NLP) extraction and were estimated to be over 90\% accurate. This accuracy has been disputed, with the visual content reviewed not adequately matching the proposed labels for a number of investigations~\cite{oakden-raynerExploringLargeScale2019}. A modified set of labels was made available by Rajpurkar \emph{et al.}~\cite{rajpurkarDeepLearningChest2018} as part of their work on pathology detection, wherein a network was trained to classify images based on the original labels and then subsequently used to relabel the original dataset. These labels are available for the majority of images with a residual manually labelled test set which has not been released publicly.
\newline \newline
The NIH dataset, license and publication can be found here: \url{https://nihcc.app.box.com/v/ChestXray-NIHCC}

\section{Development}
\label{sec:development}
\subsection{GAN Training}
\label{sec:gan-train}
We implement a modified PGGAN model~\cite{karrasProgressiveGrowingGANs2018} in Pytorch Lightning~\cite{falconPyTorchLightning2019} based on the open-source implementation produced by Facebook Research and available through the Pytorch Model Zoo (\url{https://github.com/facebookresearch/pytorch_GAN_zoo}). We start with a randomly initialised model in the configuration suggested by Karras \emph{et al.}~\cite{karrasProgressiveGrowingGANs2018}, namely generating 4x4 images and progressively doubling the resolution after a number of images by adding the appropriate layers and mixing the upsampled prior resolutions with the subsequent outputs in a linearly decreasing fashion. We utilising WGAN-GP loss~\cite{gulrajaniImprovedTrainingWasserstein2017}, minibatch discrimination, pixel normalisation and an exponential moving average of generator weights for evaluation. We train up to a resolution of 1024x1024 with \num{800000} images shown during each period of layer mixing and a further \num{800000} for training of the added layers. All training was performed on an Amazon Web Services (AWS) p3.8xlarge instance. Examples of uncurated, random samples from the trained generated can be seen in figure \ref{fig:collage}.

\begin{figure}[!h]
    \centering
    \includegraphics[height=18cm]{Images/Generate_Grid.png}
    \caption{A random selection of generated images from the final PGGAN model.}
    \label{fig:collage}
\end{figure}

\subsection{Classifier Training}
\label{sec:classifier-train}
For labelling the various CXRs, we implement a Densenet-121~\cite{huangDenselyConnectedConvolutional2018} pre-trained on ImageNet~\cite{russakovskyImageNetLargeScale2015} and replace the final fully connected layer with the number of classes in the ChestX-ray14 dataset. The model is trained end-to-end in a multi-label configuration to predict all classes simultaneously while making use of the weighted cross entropy function (\ref{eq:BCE-weight}) implemented by Guendel \emph{et al.}~\cite{guendelLearningRecognizeAbnormalities2018}. The modified loss balances the frequency of positive and negative classes per label based on the frequency within the dataset. This adjusts the loss to require the model to discriminate between all classes equally which improves performance for rarer classes.

\begin{equation} \label{eq:BCE-weight}
\begin{split}
D_{CS}(y_n,\ \hat{y}_n) = w_P \cdot y_n \log(\hat{y}_n) + w_N \cdot (1 - y_n) \log(1 - \hat{y}_n),  \\
w_P = \frac{N_p + N_n}{N_p}, \quad
w_N = \frac{N_p + N_n}{N_n}
\end{split}
\end{equation}

For training, we extract subsets of the dataset to include only images with modified labels from Rajpurkar \emph{et al.}~\cite{rajpurkarDeepLearningChest2018}, we group images at the patient level and average the labels across all images for an individual to produce a summary of the average set of conditions per patient. The dataset is then split into training, validation and test sets through iterative stratification of the average patient labels to maintain label proportions across sets while ensuring no patient overlap occurs. We augment images with a 10\textdegree \ rotation, random horizontal flip probability of 50\%, and colour jitter for brightness, contrast, saturation, and hue of \num{0.1}. The model is trained using an ADAM optimiser~\cite{kingmaAdamMethodStochastic2017} using the default settings with an initial learning rate of $10^{-3}$, which is reduced by a factor of ten if the validation micro-averaged Reciever Operating Characteristic (ROC) Area Under Curve (AUC) fails to improve for several epochs. Once model performance plateaus, it is evaluated on the test set. We repeat the same process for the final trained discriminator, replacing the final binary linear layer with the number of classes and train the subsequent classification model end-to-end.

\section{Analysis and Results}
\label{sec:analysis}
\subsection{Image Quality}
\label{sec:img_quality}
The generated images, as seen in figure~\ref{fig:collage}, appear to broadly reflect the NIH source images with global features varying similarly across both sets. Sex, posture, exposure, and positioning (AP vs PA) are all represented within the generates. The images accurately reflect the standard anatomical features found in CXRs with realistic alterations in perspective seen with changes in patient posture or positioning of the x-ray detector. Soft tissues such as the heart, liver, and stomach are faithfully reproduced with normal variability in their relative positioning. Bony structures are correctly placed but suffer from inconsistent profiles, with the ribs in particular tending to reveal a degree of undulation. Closer inspection often reveals slight curves or alterations in calibre that are seldom explained by the perspective of the image. Beyond the x-ray itself, the model has learnt to include various markup elements included in the reference images. Most images have a symbol or tag demonstrating the left side of the image, in addition, text such as 'PORTABLE' and 'AP' are included on numerous images. Some images are correctly labelled, however numerous images do not match the text included, with some having multiple copies of each label. A white arrow typically used to demonstrate locations of interest on a study has also been reproduced and is often in similar positions to the reference images. Some reproductions include elements such as white borders, poor exposure or image cropping, all of which are features present in the source dataset and reproduced by the PGGAN.
\newline \newline
The main distinction between distributions is that the generated images broadly lack certain smaller scale features of the source set. Jewelry, ECG leads, pacemakers, and various bone screws and plates are largely absent with occasional partially formed objects in locations where these should appear. The elements of the missing structures form only a fraction of each overall image and are often fairly detailed objects themselves. A potential explanation for this phenomenon is the progressive growing technique itself, as these objects would only be generated near the end of the training process as the resolution of the images allowed for the details to be appreciated. Examining the generator in the final model without the moving weight average, supports this as a larger proportion of the images possess partially formed objects, implying the network was slowly incorporating these features. The appearance of these elements failed to improve despite training beyond the recommended total number of images. This is probably due to the reduced batch size at higher resolutions which would result in exceedingly slow convergence for rarer, more subtle features.
\newline \newline
The Fréchet Inception Distance (FID) is a metric intended for evaluating the quality of generated images~\cite{heuselGANsTrainedTwo2018}. It works by embedding a set of real and synthetic images in the final average pooling layer of an Inception Net~\cite{szegedyRethinkingInceptionArchitecture2015} pre-trained on ImageNet~\cite{russakovskyImageNetLargeScale2015}. The sets are assumed to be multivariate Gaussian distributions with the average and covariance of each utilised to calculate the Fréchet distance. This distance reflects the difference in features extracted from each set based on the kernels from the Inception Net. The distance has been demonstrated to be consistent with human judgement of visual quality and more resistant to noise than prior approaches~\cite{heuselGANsTrainedTwo2018, szegedyRethinkingInceptionArchitecture2015}. The FID is sensitive to class mode dropping, with distances increasing with greater class discrepancies between the two sets. Despite this sensitivity, the FID reports ideal results if a model reproduces the training samples perfectly~\cite{szegedyRethinkingInceptionArchitecture2015}. The use of the FID for x-rays requires the underlying Inception model be capable of extracting features to adequately represent and compare sets against each other. This may be problematic given that x-rays are quite distinct from the classes found in ImageNet. To evaluate the applicability of the FID, we split the NIH dataset at the patient level and use the average labels from each patient to iteratively stratify the patients into two groups. The groups are of a similar disease distribution and are without patient overlap. We calculate the FID and get a low value of \num{0.53}, indicating that the process extracts similar features from both sets. These features may be sub-optimal given the underlying network has not been trained on biomedical data, yet the small distance value demonstrates that the images are embedded in a similar manner. To evaluate the extent to which the FID can discriminate between individual pathologies we utilise the \emph{No Finding} label as a baseline and evaluate the FID between it and each label, the results of which can be seen in table~\ref{table:FIDs}. The results demonstrate that the FID is able to distinguish between all the labelled pathologies with the distance seemingly recapitulating the relative scale of the pathological change on the image. Very large distances are associated with the \emph{Edema} and \emph{Consolidation} labels, which are findings that may distort major segments of the lung fields, while \emph{Mass} and \emph{Nodule} labels produce significantly reduced distances as they generally occupy only small segments of the study. We use the FID to evaluate the quality of synthetic x-rays compared to the NIH images, with the distance between the full source dataset and \num{100000} random generates being \num{8.02}. This value is comparable with FIDs reported for PGGAN on other high resolution image generation tasks \cite{karrasProgressiveGrowingGANs2018}.

\begin{table}[ht]
\caption{Fréchet Inception Distance (FID) per Dataset Split}
\centering
\begin{tabular}{l c l c}
\hline\hline
Split & FID & Split & FID\\ [0.5ex]
\hline
Stratified & 0.53 & Hernia & 15.96 \\
Sex & 7.87 & Infiltration & 20.05 \\
No Finding & 0.00 & Mass & 10.06 \\
Atelectasis & 19.90 & Nodule & 6.22 \\
Cardiomegaly & 14.23 & Pleural Effusion & 23.90 \\
Consolidation & 42.45 & Pleural Thickening & 13.05 \\
Edema & 59.40 & Pneumonia & 32.05 \\
Emphysema & 19.56 & Pneumothorax & 18.00 \\
Fibrosis & 9.72 & Synthetic & 8.02\\ [1ex]
\hline
\end{tabular}
\label{table:FIDs}
\end{table}

\begin{figure}[!htb]
    \centering
    \includegraphics[width=10cm]{Images/Prevalence.png}
    \caption{Label prevalence of both the original NIH dataset and a random selection of \num{130000} generates. Point estimates are the average number of each label across all images. Confidence intervals are bootstrapped by resampling the full set of labels with replacement \num{10000} times.}
    \label{fig:prevalence}
\end{figure}

\subsection{Variability in Pathology}
\label{sec:variability}
The proportion of the labels from the NIH dataset and a random selection of \num{130000} generated images can be seen in Figure \ref{fig:prevalence}. The labels were determined by utilising a common classifier to label both the NIH and generated sets with confidence intervals derived by bootstrapping the predictions \num{10000} times. Overall it can be seen that the generates cover the range of labels with each represented in broadly similar proportions to the NIH set. Despite the similarity, the majority of labels are significantly less prevalent in the generates compared to reference with the exception of \emph{Atelectasis}, \emph{Cardiomegaly}, \emph{Infiltration} and \emph{No Finding}. We hypothesize that this is likely to be due to a continuation of the phenomenon noted with the absence of certain smaller objects, the objects in this case being features of disease. This is supported by the label distribution, as more common conditions with larger disease features tend to be over represented, while diseases with progressively finer features tend to be increasingly sparse in the generated samples. \emph{No Finding} for example, is largely defined by the absence of smaller features and can be seen to be significantly in excess beyond the NIH set. \emph{Atelectasis} and \emph{Cardiomegaly} in comparison, define several characteristics that are present at a moderate resolution and see a smaller increase in prevalence. This is in stark contrast to \emph{Emphysema} and \emph{Pleural Thickening}, which are quite fine on x-ray and similarly show the greatest reduction in prevalence.

\subsection{Class Example Generation}
\label{sec:discriminator}

\begin{figure}[!htb]
    \centering
    \includegraphics[width=8cm]{Images/Class_Optimisation.png}
    \caption{The class optimisation methods employed to generate images representative of a particular disease label.}
    \label{fig:class-optim}
\end{figure}

\begin{figure}[!htb]
    \centering
    \includegraphics[height=16cm]{Images/Class_Generates.png}
    \caption{Example classes generated by maximising the classifier class logit. Images often have multiple findings, the finding optimised is the label given.}
    \label{fig:class-pos}
\end{figure}

To examine individual disease classes, we optimise for samples in the latent space ($z$) which produce images that maximise the classification score for the class of interest. We attempt two methods for this that can be seen in Figure \ref{fig:class-optim}. Examples of generates can be seen in Figure~\ref{fig:class-pos}.
\newline\newline
Optimising the Densenet-121 model proved to be far slower, more difficult and less reliable overall. The process frequently plateaus and subsequently fails to produce a sample for a particular class. Successes often rely on favourable sampling from the latent space, with such samples typically converging rapidly. The same process applied to the re-purposed discriminator proved far more successful as samples almost always converge to maximise the class of interest, with far higher potential class logits than those seen with the initial method.
\newline\newline
The ease with which the latent space may be optimised with regard to disease classes implies a degree of orientation with respect to the labels which may hold semantic value. We leave a full exploration of the properties of the latent space to future work.
Attempts to produce examples of isolated labels often fail to converge. Optimisation by minimising other class scores alongside the maximisation of the class of interest typically results in a profound limitation to the score that can be achieved. This phenomenon probably results from the multi-label nature of the disease classes and the medical relationship between them.
\newline\newline
Attempting to produce extreme examples of \emph{Cardiomegaly} begins to include \emph{Pleural Effusion} as the logit increases. Similar concordance can be seen with other classes. As an example: The \emph{Emphysema} class similarly produces \emph{Pneumothorax} labels, while \emph{Nodule} findings produce associated \emph{Mass} labels with sufficiently large logit values. These findings reflect a correlation with the underlying pathophysiology of the conditions themselves, as extreme examples of classes often include labels related to complications of the conditions.

\section{Discussion}

There is substantial variety in available choices for GAN architectures and training configurations that may be utilised for this work. The selection of the PGGAN architecture allows for megapixel resolution image generation with a more typical generator architecture despite known issues with micro-scale details~\cite{karrasProgressiveGrowingGANs2018}. Subsequent work on PGGAN has seen significant strides in the form of StyleGAN~\cite{karrasStyleBasedGeneratorArchitecture2019} and StyleGAN2~\cite{karrasAnalyzingImprovingImage2020}. These improvements introduce modifications inspired from work on style transfer techniques which provides the model with a latent sample and noise at each image scale. This significantly improves small scale detail by preventing the model from consuming its generative capacity to preserve stochastic variation at the expense of a more convoluted and atypical GAN setup~\cite{karrasStyleBasedGeneratorArchitecture2019}. The selection of PGGAN over a StyleGAN variant is a necessary initial foray into the generation of high resolution synthetic x-rays as it provides a meaningful baseline of the traditional GAN formulation, against which future works may be compared.
\newline \newline
Generated x-rays were produced as portable network graphic (PNG) images in the same format as the source images provided by the NIH. This is not ideal as radiological investigations are typically stored as digital imaging and communications in medicine (DICOM) files. DICOM studies are stored with a 16-bit colour depth and are typically windowed to view different features of an image that are visible in varying colour ranges. This is largely lost with conversion to 8-bit PNGs or other file formats. The loss of colour depth and windowing results in difficulty visualising features for certain pathologies even if the classifier can detect the features necessary to classify the image. Future work should strive to produce images compatible with the various features of DICOM studies which will improve the ability to demonstrate pathological features at different views as well as potential improvements to classifier performance.

\section{Conclusion}
\label{sec:conclusion}
We apply recent advances in generative modelling to the task of synthesising high-resolution medical chest x-ray images for the purpose of training and teaching. We demonstrate that it is possible to produce highly realistic, clinically plausible images that capture much of the variation in standard x-rays, however, there remains a need for improvement in the reproduction of small-scale details. The model is capable of reproducing all of the abnormalities and in similar proportions to the source image distribution. Future work should examine architectural and training improvements that will better capture micro variations in images to more faithfully reproduce both pathological features as well as the features of support devices such as pacemakers. We anticipate that such improvements will enable the production of high quality teaching and training images without the concern of breaching patient confidentiality.

\printbibliography

\end{document}

Image and Video Processing (eess.IV); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)