How much different between $D_{KL}[P(X)||Q(X)]$ and $D_{KL}[Q(X)||P(X)]$? While Kullback Leibler divergence is not symmetric: $D_{KL}[P(X)||Q(X)] \neq D_{KL}[Q(X)||P(X)]$, how much different $D_{KL}[P(X)||Q(X)]$ from $D_{KL}[Q(X)||P(X)]$?
Does the quantity $D_{KL}[P(X)||Q(X)] - D_{KL}[Q(X)||P(X)]$ have some interesting interpretation?

Let $\epsilon \in (0,1)$ and suppose that $P(X)=(1-\epsilon)\delta_{0}+\epsilon\delta_{1}$ and $Q(X)=\frac 12\delta_{0}+\frac 12\delta_{1}$.
Then $D_{KL}[P(X)||Q(X)]=(1-\epsilon)\ln(2(1-\epsilon))+\epsilon \ln(2\epsilon)$ and $$D_{KL}[Q(X)||P(X)] = \frac 12 \ln(\frac 1{2(1-\epsilon)})+\frac 12\ln(\frac{1}{2\epsilon})$$
As $\epsilon \to 0$, $D_{KL}[P(X)||Q(X)]\to \ln 2$ and $D_{KL}[Q(X)||P(X)] \to \infty$, thus $$D_{KL}[P(X)||Q(X)] - D_{KL}[Q(X)||P(X)] \xrightarrow[\epsilon \to 0]{}-\infty$$
The rationale is that the KL divergence $D_{KL}[\mu||\nu]$ blows up when $\mu$ is not absolutely continuous w.r.t $\nu$ (e.g. when the support of $\mu$ is not a subset of the support of $\nu$). This is a common pitfall for $f$-divergences.
