For the remainder of this section, to simplify notation and maintain generality, we consider generic loss functions \(\ell \colon \mathcal {Z}\rightarrow \mathbb {R}_+\)  which
contain the model \(h\)  and take inputs from a generic input space \(\mathcal {Z}\) . For example in the context of supervised learning, \(\mathcal {Z}= \mathcal {X}\times \mathcal {Y}\)  can be the product space of features and labels and the loss \(\ell (z) = \tilde{\ell }(h_\theta (x),\,y)\)  can be seen as a concatenation of loss function and model \(h\) .
We denote the set of probability measures on the space \(\mathcal {Z}\)  by \(\mathcal {P}(\mathcal {Z})\) .
For two measures \(\mu ,\,\nu \)  on \(\mathcal {Z}\) , we say that \(\nu \)  is absolutely continuous with respect to \(\mu \) , denoted \(\nu \ll \mu \)  if \(\mu (A) = 0\)  implies that \(\nu (A) = 0\)  for any measurable set \(A\subseteq \mathcal {Z}\) .
Among the plethora of distances between probability measures, such as total variation and Wasserstein distance, a particularly popular choice is the family of \(f\) -divergences which has been extensively studied in the context of distributionally robust optimization [1]}, [2]}, [3]}, [4]}. In this paper, we focus on the Hellinger distance, which is a specific \(f\) -divergence.
