This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC- 2092 CASA – 390781972. We also thank the anonymous reviewers for their careful reading and their useful
suggestions, which will help us extending this work to a full paper.

Examples of Copula Distributions
There is a wealth of different copula distributions, ranging from parametric distributions to semi-parametric and completely non-parametric copula models. In this Section, we give some insights into the construction of one class of copula distributions and illustrate some basic properties.
Example A.1 (Construction using Sklar's Theorem) 
There is a generic way to construct a copula according to Definition REF . Consider any multivariate continuous and invertbile CDF \(\Phi \)  with marginal CDFs \(\Phi _1,\dots , \Phi _D\) . Then, following (REF ) in Sklar's Theorem REF , we know that
\(C({u}):= \Phi \bigl (\Phi _1^{-1}({u}_1), \dots ,\Phi _D^{-1}({u}_D)\bigr )\) 
defines a valid copula.
Setting \(\Phi \)  to be the multivariate Gaussian distribution with correlation matrix \(R\) , we obtain the Gaussian Copula. If \(R\)  is the identity matrix we obtain the Independence Copula, which is simply the product of independent uniform distributions. Both copulae are visualized in Figure REF . Following the construction in (REF ) and replacing the CDF, we can obtain other copulae,such as the t-Copula. Figure REF  visualizes one example for a distribution based on the Independence Copula and based on the Gaussian Copula.

Example A.2 (Copulae that induce Tail-Dependency) 
A crucial property of models in financial risk analysis is tail dependency. Roughly said, a tail dependency casts marginals to be dependent in a tail event. To give an example for a tail dependency, consider two essentially independent marginal distributions, such as \({\mathbf {x}}_1=\lbrace \text{debt status of your bank}\rbrace \)  and \({\mathbf {x}}_2=\lbrace \text{trust for your bank}\rbrace \) . Usually, our trust for the bank is not essentially determined by the amount of debts it has. However, in a marginal tail event \(\lbrace {\mathbf {x}}=\text{bankrupt}\rbrace \) , of course, our trust drops rapidly.
Mathematically, the upper tail dependency, i.e. the dependency in upper-tail events, for a random variable \(({\mathbf {x}}_1, {\mathbf {x}}_2)\)  is given by
\(\lambda _U = \lim _{u\rightarrow 1^-}\mathbb {P}\bigl ({\mathbf {x}}_2> F_{{\mathbf {x}}_2}^{-1}(u)\, \vert \, {\mathbf {x}}_1 > F_{{\mathbf {x}}_1}^{-1}(u)\bigr )\hspace{5.0pt}.\) 
Similarly, we can define the lower tail dependency \(\lambda _L\) .
One such copula that accounts for upper tail dependency is the Gumbel Copula, which is given by
\(C({u}):= \exp \bigl (-((-\log ({u}_1))^\rho + (-log({u}_2))^\rho )^{1/\rho }\bigr ) \hspace{5.0pt}\) 
for \(\rho \ge 1\) . Figure REF  shows a visualization of the Gumbel copula.
One can show that \(\lambda _U=2-2^{1/\theta }\)  [1]}. Figure REF  visualizes the Gumbel copula distribution with Gaussian and Gamma marginals. We can observe a decent dependency—indicated by the peak pointing to the upper right—in the upper-tail events, which is mathematically described by \(\lambda _U\) .

<FIGURE><FIGURE>One could think of many more different properties that might be included using our copula approach, such as multi-modality and symmetry.
However, it is still to be researched whether properties, such as the tail-dependency, are being preserved by the NF. Nonetheless, we think that fixing specific known properties in the base distribution facilitates training in acting as a type of regularization towards these given properties.

Supplementary Experiments
In the following, we give some further empirical results that underpin our findings from the main paper.
Figure REF  shows the investigated base distributions and Figure REF  visualizes the target distribution.
<FIGURE><FIGURE>Figure REF  and Figure REF  supplement the findings about the learned quantiles: Even the other heavy-tailed distributions—the cases heavierTails and correctFamily—are able to successfully approximate the quantiles. This observation suggests that it is sufficient to use a broad surrogate of the base distribution that captures the true tail behavior.
<FIGURE><FIGURE>To address the invertibility and numerical stability of the learned transformation, we investigate the local Lipschitz constant as derived in Section REF  in the main text. Figure REF  shows the supplementary Lipschitz surfaces of the cases heavierTails and correctFamily, which indicate a more regular transformation than the normal case. Still, in the case correctFamily we find large local Lipschitz constants in \(T_\theta ^{-1}\) , which, however, are structured in accordance to the base distribution. In the vanilla NF this irregularity is less structured and, since these irregularities occur in high-probability areas of the base distribution, more relevant.
<FIGURE>These results support our claim that choosing appropriately tailed base distribution can help in learning a numerically robust transformation.

Computational Details
In all of our experiments, we employed Masked Autoregressive Flows [2]} with 3 layers. Each layer contains a reverse permutation, followed by an autoregressive transformation with 4 hidden features. The code is based on the nflows package [3]}.
We trained the NFs on a sample of size \(10\,000\) . Optimization was carried out using the Adam optimizer with the PyTorch default settings and a batch size of 128. Test losses are evaluated based on \(10\,000\)  test samples.
The reported training and test losses (Figure REF ) have been averaged over 100 runs and the depicted confidence intervals correspond to a confidence of \(95\%\) .
To estimate the Lipschitz surface we rely on an estimation of
\(\sup _{\Vert {v}\Vert _2=1} \frac{1}{\varepsilon } \Vert T({x}) - T({x}+ \varepsilon {v}) \Vert _2\hspace{5.0pt}, \) 
for some small constant \(\varepsilon \) . We chose \(\varepsilon :=10^{-3}\) . Further, we approximate (REF ) by
\(\max _{j\in \lbrace 1,\dots ,100\rbrace } \frac{1}{\varepsilon } \Bigl \Vert T({x}) - T({x}- \varepsilon \frac{{v}^{(j)}}{\Vert {v}^{(j)}\Vert _2} \Bigr \Vert _2,\) 
where \({v}^{(1)}, \dots {v}^{(100)}\)  are i.i.d. samples from \(\mathcal {N}(0, I)\) , see [4]} for further details.
All code is provided with the submission and can further be accesses through https://github.com/MikeLasz/Copula-Based-Normalizing-Flows.
