Are synthetic data rewards distributed to parties and their downstream ML task performances commensurate to their contributions?
We firstly assess whether our CGM framework can distribute synthetic data points \(G_i\)  to each party \(i\)  as reward such that the closeness of the empirical distributions associated with \(D_i \cup G_i\)  vs. reference dataset \(T\)  correlates well with its expected marginal contribution via the normalized Shapley value \(\alpha _i\) . We quantify such a closeness using
4 metrics (which we take the negative of so that higher is better): (a) unbiased MMD estimate (REF ), (b) an estimate of reverse Kullback-Leibler divergence based on \(k\) -nearest neighbors [1]} averaged over \(k=2,\ldots ,6\) , (c) Wasserstein-2 distance between multivariate Gaussians fit to \(D_i \cup G_i\)  vs. \(T\)  (i.e., how Fréchet Inception distance for evaluating GANs is computed [2]}), and (d) class imbalance \(\rho \)  calculated with \(\rho _i := (1/m) \sum _{y=1}^{m}p^2_y\)  where \(m\)  is the no. of classes and \(p_y\)  is the proportion of data points in party \(i\) 's combined dataset \(D_i \cup G_i\)  belonging to class \(y\) . In all datasets, \(T\)  is equally distributed among the classes and hence achieves a minimum for \(\rho \) . We also measure the correlation of the no. \(|G_i|\)  of synthetic data points as reward to party \(i\)  with \(\alpha _i\) . Fig. REF  shows results of the mean and standard error of the correlations over varying \(\beta = 1, 2, 4, 8\)  in the weighted sampling. It can be observed that across all splits, datasets, and metrics, the negative of the metrics and \(|G_i|\)  mostly display highly positive correlations with \(\alpha _i\) , as desired. We defer the discussion of the few negative correlations to Appendix REF .
