INTRODUCTION

The use of deep generative models (DGMs), such as generative adversarial networks (GANs), in medical imaging pipelines has been an active area of research recently. Realizations from a DGM represent variates drawn from an unknown high-dimensional distribution that describes the ensemble of training images. Evaluation of these realizations often involves comparison of statistics derived from either the grayscale intensity distribution or pairs of intensities at a fixed distance. In the context of medical images, expert knowledge is required to perceive contextual differences, such as whether an organ is correct in shape, size and general appearance relative to other organs and given a known pathology. Thus, "high-order" spatial statistics are defined here as those conveying the contextual information not readily or adequately expressible via pairwise pixel correlations alone. 1 In other words, here, "high-order" is not to be confused with the high-degree moments of a first-order image statistic such as the skewness or kurtosis. To our knowledge, no objective method exists that provides a direct assessment of the reproducibility of statistics representative of the high-order spatial information in diagnostic images.

The aim of this work is to provide a method for evaluating the ability of a DGM to reproduce specified spatial arrangements within an image. In particular, how global measures of training relate to individual image errors is explored using algorithmically specified rules in the designed SOMs. This is demonstrated on two popular architectures but is not restricted to them. It is further noted that no claim is made about either architecture being superior in any regard because the goal of this work is not to do a comprehensive assessment of all instances of a particular architecture but only to demonstrate the methodology for use of the proposed SOMs.

Send correspondence to Frank J. Brooks. E-mail:fjb@illinois.edu, Telephone:

Reproducibility of radiomic features from a clinical dataset

Two GAN architectures (described in Sec.2.4) were trained on the fastMRI brains dataset. 2 A total of 17357 slices were extracted from volumes with T2 contrast at 3T magnet strength. Slices were selected such that the area occupied by the foreground was at least half of the maximum foreground area over all slices in the ensemble. These were resized to 256x256 and converted to 8-bit after data cleaning. For radiomic feature analysis, PyRadiomics 2.2.0 3 was employed with the following settings: histogram binned to 32 gray levels, distances 1-3 and sigma 1-2, for all 2D features and image classes available in the library, giving a total of 1023 features.

Designed SOMs

The first SOM, henceforth referred to as "Voronoi", is a set of eight classes of varying spatial order. Each realization within a class is a Voronoi diagram with a predetermined number of regions ranging from 12 to 96, in multiples of 12. Most importantly, the grayscale intensity of each tile within a realization is rank-correlated with its area such that larger areas have higher grayscale intensity. This represents a high-order arrangement rule that tests learning of information beyond typical second-order correlations. In the second SOM, henceforth referred to as "alphabet", each realization is a panel of 64 equally sized letters placed equidistantly on a zero intensity background. Within each realization, the following 8 letters: Z, H, Y, W, V, K, L, X have fixed prevalence.

Here, an additional high-order arrangement rule is that each training realization has exactly 4 H-V and 8 W-Y pairs such that the second letter in the pair always follows the first. This enables the measurement of per-image frequencies as well as ensemble frequencies of the letters.

The Voronoi ensemble consists of 65536 realizations per each of the 8 total classes while the alphabet ensemble consists of 131072 realizations. Each realization is an 8-bit image of size 256x256. Sample realizations for each ensemble are shown in

Post-processing of generated images

For the analysis of generated images from Voronoi, a statistical classifier is designed to extract the number of regions and corresponding shades. In the case of alphabets, template matching is employed to recognize letters and provide the corresponding uncertainty in recognition of each letter in the generated images. Image class prevalence and per-image feature prevalence are then computed based on these results, while also accounting for the uncertainty in the post-processing methods.

Network trainings

Two popular network architectures: ProGAN 4 and StyleGAN2(config-e) 5 were chosen to demonstrate the use of the proposed tests. For ProGAN, the network was trained for 7M images (with transitions set to 400k images) for Voronoi and 12M for alphabet and the clinical dataset, using the default training scheme. For StyleGAN2 (config-e), all trainings were performed for 4M images. The regularization parameter R 1 was set to the default value of 100 for the chosen training configuration and truncation was set to ψ=0.5. For both architectures, the last model was chosen post-training and 10240 realizations were generated for analysis. The trainings were performed on Tesla V100, GeForce GTX 1080 and 1080Ti GPUs and took between 4 and 14 days per GPU. It is noted that for the purpose of this work, the goal was not to achieve the best performance for a network in terms of a chosen metric but to demonstrate the use of the proposed SOMs as a tool for testing a chosen instance of any architecture.

RESULTS

Reproducibility of radiomic features of a clinical dataset

Results from the Voronoi SOM

As shown in Fig. 3, the equal prevalence of all eight classes in the training data is not respected by either network, even after accounting for the errors in the post-hoc classifier. Empirical testing shows that the specified rank correlation between area and shade in the true data (ρ=0.9) is not maintained in the network-generated ensembles (ProGAN: ρ=0.8 and StyleGAN2: ρ=0.7). In particular, a non-negligible proportion of realizations have poor rank correlation. Lastly, unrealistic realizations with ambiguous class membership are also present in the generated ensemble. Other visually observed errors include presence of high-frequency, low-magnitude artifacts in regions expected to have a constant value (refer Fig. 3), unrealistic curvature of boundaries, and smudged regions. The FID-10k scores for ProGAN and StyleGAN2 are 16.5 and 26.7 respectively.

Results from the alphabet SOM

Statistics upto and including second-order are preserved in the generated images to the extent that individual letters generally appear well formed and are clearly distinguishable, which is also represented in the low FID-10k scores for both architectures (Progan: 5.46; Stylegan2: 8.94). The extent of malformation or uncertainty of a letter can be quantified and letters that are malformed beyond visual recognition are excluded from further analyses (about 1 in 6400 letters for ProGAN and 1 in 128 letters for StyleGAN2). As the expected frequency of each letter in a realization is known, the χ 2 goodness-of-fit statistic is plotted as shown in Fig. 4. The number of realizations beyond the 95% critical value threshold is 203 and 118 for ProGAN and StyleGAN2 respectively within an ensemble of 6000 images, indicating that most images in an ensemble lie within the expected ensemble variation. However, the number of "perfect" realizations (χ 2 =0) is only 1 for ProGAN and 3 for Stylegan2 within the entire ensemble, suggesting that if compliance with the rule is critical for use of the image, then essentially none of the generated images are acceptable, although the ensemble prevalence of letters is largely respected by both networks. Lastly, the high-order arrangement rule for the occurrence of letter-pairs is tested. The expected frequency for the pairs H-V and W-Y is exactly 4 and 8. However, as seen in Fig. 4, a wide range of frequencies is observed for realizations from both networks. Furthermore, it was observed that the letters V and Y usually occurred without their preceding partner, even this never occurs in the training data. Together, these observations indicate that high-order information rules are not learnt in training.

It is reiterated here that these tests serve as tools for evaluating the capacity of any generative model and no

CONCLUSIONS

The altered radiomic feature distributions highlight the need for careful evaluation of the statistical properties of an ensemble generated from a DGM, before it is employed for any medical imaging application. In this direction, the Voronoi SOM serves as an important tool to quantify the capacity of a given DGM to reproduce statistics representative of properties such as the prevalence of distinct pathologies in the training ensemble and their characteristic features. The alphabet SOM allows for assessing individual realizations for properties such as relative shapes and locations of objects and structures (e.g., organs or blood vessels) through per-image high-order statistics, especially when ensemble statistics are respected.

The proposed SOMs provide a general method for the evaluation of reproducibility of high-order information through specific pixel arrangement rules relevant to ensembles of medical images. Such an architectureindependent evaluation of the capacity of a generative model provides more information than conventional tests or FID scores. In the context of the properties tested by the proposed SOMs, an informed choice of a network can be made based on the relative importance of such properties for a given task through comprehensive testing of multiple architectures and their variations. Thus, designed SOMs can be employed for quantifying the utility of a DGM architecture when high-order information is of relevance.