<jupyter_start><jupyter_text>Mixture of Discrete and Continuous over [0,1)This distribution is a mixture of a point masses at $0$ and a continuous distribution over the support $(0, 1)$.The distribution function looks like this:\begin{equation}f(x|\alpha, \beta) = \alpha \delta(x) + (1-\alpha) g(x|\beta)\end{equation}where $0 < \alpha < 1$ and $g(x|\beta)$ is a (properly normalised) density over $(0,1)$.<jupyter_code>from mixture import MixtureD0C01 as Mixture
from utils import mc_entropy, mc_kl
logits = torch.from_numpy(np.array([0.])).float()
logits
p = Mixture(logits0=logits, 
            cont=RightTruncatedExponential(rate=torch.ones(1), upper=1.))
print(p.p0, p.pc)
x = p.rsample(torch.Size([10]))
x
_= plt.hist(p.rsample(torch.Size([1000])).numpy(), bins=50)
p2 = Mixture(logits0=torch.ones(1), 
            cont=RightTruncatedExponential(rate=torch.ones(1) * 1., upper=1.))
print(p2.p0, p2.pc)
kl_mc(p, p2, 1000000)
torch.distributions.kl.kl_divergence(p, p2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16135458167330677, 'fraction_numerical': 0.043824701195219126, 'mean_word_length': 5.4423076923076925, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '205343', 'license': 'MIT', 'n_tokens_mistral': 402, 'n_tokens_neox': 386, 'n_words': 84, 'path': 'Mixture.ipynb', 'repo_name': 'akashrajkn/sparse-distributions', 'size': '1004'}