Special probability distribution If $p(x)$ is a probability distribution with non-zero values on $[0,+\infty)$, what conditions on $p(x)$ yield $\int_0^{\infty}p(x)\log{\frac{ p(x)}{(1+\epsilon)p({x}(1+\epsilon))}}dx \leq c \epsilon^2$, where $c$ is a constant and $0&lt;\epsilon&lt;1$.


The inequality above is actually a Kullback-Leibler Divergence between distribution $p(x)$ and stretched version of it ${(1+\epsilon)}p({x}{(1+\epsilon)})$. I have found out that this inequality holds for Exponential, Gamma, and Weibull distributions and I am interested to know if that works for a larger class of probability distributions.

Any idea what does that inequality mean?



Before, I thought this inequality holds for all distributions and I was looking for proofs, but I realized this is not true in general by counter examples such as the one introduced as an answer bellow in which lets
$1/\beta =1+\epsilon$ and equivalently considers $\int_0^{\infty}p(x)\log{\frac{\beta p(x)}{p({x}/{\beta})}}dx \leq c (1-\frac{1}{\beta})^2$.
I don't think thus is true in general. 

Let's construct a case where the relative entropy is infinity. Let's consider a special case where $\beta = 2$. For each set $A$, let 

$\bar A = \{2x|x\in A\}$.  

Now, we can pick a sequence of segments on $R^+$, $A_1, A_2, ..., $ such that $A_1, \bar A_1, A_2, \bar A_2 ...$ do not intersect each other.

Now assign constant density on these sets, so that,

$P(\bar A_i) = \frac{1}{4^i}$ and

$P(A_i) =  \frac{1}{2}e^{-4^i}P(\bar A_i)$ 

Note that the density $p(x) = \frac{P(\bar A_i)}{\mu(\bar A_i)}$ and $p(x/2) = \frac{P(A_i)}{\mu(A_i)}$ for $x\in \bar A_i$, where $\mu(A)$ is the standard borel measure of $A$.

Also note that $\mu(\bar A_i) = 2\mu(A_i)$ by construction, so we have,

$log(\frac{p(x)}{\frac{1}{2}p(x/2)}) = 4^i$ 

on $\bar A_i$ and

$\int_{(\cup \bar A_i)}p(x)log(\frac{p(x)}{\frac{1}{2}p(x/2)})dx = \infty  $

On the remaining set, just assign, say, the exponential distribution, scaled by some constant to make $p$ a probability.  By renormalization, you can prove that 

$\int_{(\cup \bar A_i)^c}p(x)log(\frac{p(x)}{\frac{1}{2}p(x/2)})dx &gt;-C $

for some constant $C$.

So, overall, the relative entropy is $\infty$.
I don't think thus is true in general. 

Let's construct a case where the relative entropy is infinity. Let's consider a special case where $\beta = 2$. For each set $A$, let 

$\bar A = \{2x|x\in A\}$.  

Now, we can pick a sequence of segments on $R^+$, $A_1, A_2, ..., $ such that $A_1, \bar A_1, A_2, \bar A_2 ...$ do not intersect each other.

Now assign constant density on these sets, so that,

$P(\bar A_i) = \frac{1}{4^i}$ and

$P(A_i) =  \frac{1}{2}e^{-4^i}P(\bar A_i)$ 

Note that the density $p(x) = \frac{P(\bar A_i)}{\mu(\bar A_i)}$ and $p(x/2) = \frac{P(A_i)}{\mu(A_i)}$ for $x\in \bar A_i$, where $\mu(A)$ is the standard borel measure of $A$.

Also note that $\mu(\bar A_i) = 2\mu(A_i)$ by construction, so we have,

$log(\frac{p(x)}{\frac{1}{2}p(x/2)}) = 4^i$ 

on $\bar A_i$ and

$\int_{(\cup \bar A_i)}p(x)log(\frac{p(x)}{\frac{1}{2}p(x/2)})dx = \infty  $

On the remaining set, just assign, say, the exponential distribution, scaled by some constant to make $p$ a probability.  By renormalization, you can prove that 

$\int_{(\cup \bar A_i)^c}p(x)log(\frac{p(x)}{\frac{1}{2}p(x/2)})dx &gt;-C $

for some constant $C$.

So, overall, the relative entropy is $\infty$.