<jupyter_start><jupyter_text>1 ENTROPY- Measure of uncertainty of a random variable $X$. The higher the entropy, the more the uncertainty i.e. more privacy and vice-versa.- Discrete Entropy applies to Discrete Random Variables whilst Differential Entropy applies to  Continuous Random Variables.![256px-Entropy-mutual-information-relative-entropy-relation-diagram.svg.png](attachment:256px-Entropy-mutual-information-relative-entropy-relation-diagram.svg.png)Venn diagram showing additive and subtractive relationships of various information measures associated with correlated variables {\displaystyle X}X and {\displaystyle Y}Y. The area contained by both circles is the joint entropy {\displaystyle \mathrm {H} (X,Y)}{\displaystyle \mathrm {H} (X,Y)}. The circle on the left (red and violet) is the individual entropy {\displaystyle \mathrm {H} (X)}{\displaystyle \mathrm {H} (X)}, with the red being the conditional entropy {\displaystyle \mathrm {H} (X\mid Y)}{\displaystyle \mathrm {H} (X\mid Y)}. The circle on the right (blue and violet) is {\displaystyle \mathrm {H} (Y)}{\displaystyle \mathrm {H} (Y)}, with the blue being {\displaystyle \mathrm {H} (Y\mid X)}{\displaystyle \mathrm {H} (Y\mid X)}. The violet is the mutual information {\displaystyle \operatorname {I} (X;Y)}\operatorname {I} (X;Y).  1.1 Discrete Entropy (Discrete Random Variables)- Let $X$ be a discrete random variable with alphabet $X$ and probability mass function $p(x) = Pr({X = x})$, $x \epsilon X$.- The entropy $H(X)$ of a discrete random variable $X$ is defined by$$H(X) = -\sum_{x \epsilon X}^n p(x) log p(x)$$- NB: The log is to the base 2 and entropy is expressed in bits.- $H(X) \geqslant 0$- Conditioning reduces entropy (see section on conditional entropy) i.e. $H(X|Y) \leqslant H(X)$<jupyter_code>'''
Calculation of entropy from first principles

For example, we will show that the entropy of a fair coin toss is 1 bit.

Let X be the outcome of a coin toss - {H,T}
P(X) = 0.5 for each outcome since it is a fair coin

----|----|----|
X   |  H | T  |
----|----|----|
P(x)| 0.5|0.5 |
----|----|----|

'''
import math

#calculating discrete entropy from first principles (alternative would be to use
#scipy library
LOG_BASE_2 = 2
OUTCOMES = ['H', 'T']
DISTRIBUTION = [0.5, 0.5]
entropy_H_X = -( sum( (i * math.log(i, LOG_BASE_2)) for i in DISTRIBUTION ) )
print('Entropy of fair coin toss is %s bit' % entropy_H_X)

#manual calculation
entropy_H_X_test = -( (0.5 * math.log(0.5, 2)) + (0.5 * math.log(0.5, 2)) )
print('Manual check of Entropy of fair coin toss is %s bit' % entropy_H_X_test)
'''Using scipy to calculate the entropy of a fair coin toss which is 1 bit.

i.e. a Bernoulli trial with same p (0.5 Heads, 0.5 Tails). The outcome of a fair coin is the most uncertain unlike 
one in which the coin is not fair p (0.9 Heads, 0.1 Tails)
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html#:~:text=Calculate%20the%20entropy%20of%20a,)%2C%20axis%3Daxis)%20.
Calculate the entropy of a distribution for given probability values.

If only probabilities pk are given, the entropy is calculated as S = -sum(pk * log(pk), axis=axis).

If qk is not None, then compute the Kullback-Leibler divergence S = sum(pk * log(pk / qk), axis=axis)
Parameters
pk: sequence of prior probabilities
Defines the (discrete) distribution. pk[i] is the (possibly unnormalized) probability of event i.

qk: sequence of posterior probabilities, optional
Sequence against which the relative entropy is computed. Should be in the same format as pk.

basefloat, optional
The logarithmic base to use, defaults to e (natural logarithm).

axis: int, optional
The axis along which the entropy is calculated. Default is 0.

Returns
Sfloat
The calculated entropy.'''

from scipy.stats import entropy

#The outcome of a fair coin is very uncertain:
scipy_discrete_fair_entropy = entropy([1/2, 1/2], base=2)
print('scipy libary calculation of Entropy for a fair coin toss is %s bit' % scipy_discrete_fair_entropy)

#The outcome of a biased coin is less uncertain:
scipy_discrete_biased_entropy = entropy([9/10, 1/10], base=2)
print('scipy libary calculation of Entropy for a biased coin toss is %s bit' % scipy_discrete_biased_entropy)

#Relative entropy
scipy_relative_biased_entropy = entropy([1/2, 1/2], qk=[9/10, 1/10])
print('scipy libary calculation of relative entropy of a biased coin toss is %s bit' % scipy_relative_biased_entropy)<jupyter_output>scipy libary calculation of Entropy for a fair coin toss is 1.0 bit
scipy libary calculation of Entropy for a biased coin toss is 0.46899559358928117 bit
scipy libary calculation of relative entropy of a biased coin toss is 0.5108256237659907 bit<jupyter_text>1.1.1 Conditional Entropy and Mutual Information  (Discrete Random Variables)- Conditional entropy is the entropy of a random variable that is conditional on the knowledge of another.- Conditioning reduces entropy i.e. $H(X|Y) \leqslant H(X)$- Reduction in uncertainity due to another random variable is called Mutual Information(which is denoted as $I$)- For 2 random variables, $X$ & $Y$:$$ I(X;Y) = H(X) - H(X|Y) = \sum_{x, y}^n p(x,y) log \frac{p(x,y)}{p(x)p(y)} $$-Mutual information $I(X;Y)$ is a measure of dependence between 2 random variables, is symmetric, always non-negative and is zero if $X$ & $Y$ are independent. In the case of privacy applications, we want mutual information to be as low or close to zero as possible.  1.2 Differential Entropy (Continuous Random Variables)- Differential entropy differs from normal or absolute entropy in that the random variable need not be discrete.- Interpretation of $h(X)$ is the measure for amount of information we do not have about X- Let X be a continuous random variable with cumulative distribution function $F(x) = Pr(X \leqslant x)$ and probability density function $f(x)$ i.e. the PDF for $X$ is the first derivative of $F(x)$ which can be written as $F^\prime(x)$.- The differential entropy $h(X)$ of a continuous random variable $X$ with density $f(x)$ is defined as $$h(X) = -\int_S f(x) log f(x) \delta x$$where $S$ is the support set of the random variable (i.e. where $f(x) > 0)$- Example (Uniform distribution) - Consider a continuous random variable, $X$, distributed uniformly from $0$ to $a$, with $a$ > 0 and density  $f(x)$ is $1/a$ from 0 to $a$ and 0 elsewhere. Then its differential entropy of $X$ is$$h(X) = -\int_0^a \frac{1}{a} log \frac{1}{a} \delta x$$- Differential entropy can be negative  1.2.1 Joint Differential Entropy- The differential entropy of a set $X_{1}, X_{2},...X_{n}$ of continuous random variables with density $f(x_{1}, x_{2},...,x_{n})$ is$$h(X_{1}, X_{2},...X_{n}) = -\int f(x^n) log f(x^n) \delta x^n $$ 1.2.2 Conditional Differential Entropy- The differential entropy of a set $X, Y$ of continuous random variables with a joint density function $f(x, y)$ is$$h(X|Y) = -\int f(x,y) log f(x|y) \delta x \delta y $$which can also be written as $h(X|Y) = h(X,Y) - h(Y)$ since generally bayes suggests that $f(x|y) = \frac{f(x,y)}{f(y)}$ 1.2.3 Relative Entropy and Mutual Information- The relative entropy (Kullback-Leibler divergence) between two continuous densities $f$ and $g$ measures the difference between two probbility distributions and is given by$$ D(f||g) = \int f log \frac{f}{g}$$- The main interpretation of relative entropy is that the information we gained about $X$, if we originally thought $X ∼ g$ and now we learned X ∼ $p$ i.e. Expressed in the language of Bayesian inference, $D(f||g)$ is a measure of the information gained by revising one's beliefs from the prior probability distribution $g$ to the posterior probability distribution $p$.  In order to find a distribution $g$ that is closest to $f$, we can minimize KL divergence and compute an information projection. 1.2.4 Mutual Information (Information Gain)Mutual information is a distance between two probability distributions. Correlation is a linear distance between two random variables. Mutual Information is also known as information gain.In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as shannons (bits), nats or hartleys) obtained about one random variable through observing the other random variable. The concept of mutual information is intimately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.Intuitively, mutual information measures the information that {\displaystyle X}X and {\displaystyle Y}Y share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if {\displaystyle X}X and {\displaystyle Y}Y are independent, then knowing {\displaystyle X}X does not give any information about {\displaystyle Y}Y and vice versa, so their mutual information is zero. At the other extreme, if {\displaystyle X}X is a deterministic function of {\displaystyle Y}Y and {\displaystyle Y}Y is a deterministic function of {\displaystyle X}X then all information conveyed by {\displaystyle X}X is shared with {\displaystyle Y}Y: knowing {\displaystyle X}X determines the value of {\displaystyle Y}Y and vice versa. As a result, in this case the mutual information is the same as the uncertainty contained in {\displaystyle Y}Y (or {\displaystyle X}X) alone, namely the entropy of {\displaystyle Y}Y (or {\displaystyle X}X). Moreover, this mutual information is the same as the entropy of {\displaystyle X}X and as the entropy of {\displaystyle Y}Y. (A very special case of this is when {\displaystyle X}X and {\displaystyle Y}Y are the same random variable.)- Definition 1: The mutual information $I(X;Y)$ between two continuous random variables  $X$ & $Y$ is given by$$ I(X;Y) = h(X) - h(X|Y) = h(Y) - h(Y|X) $$-  Definition 2: The mutual information $I(X;Y)$ between two continuous random variables with joint density $f(x,y)$ is given by$$ I(X;Y) = \int f(x,y) log \frac{f(x,y)}{f(x)f(y)} \delta x \delta y = E\left[log\frac{f(X,Y)}{f(X)f(Y)}\right]$$**Mutual information versus correlation**A measure used to represent how strongly two random variables are related known as correlation. Measures of correlation include covariance, correlation coefficient (a correlation coefficient is a quantitative assessment that measures both the direction and the strength of this tendency to vary together. Values can range from -1 to +1. -1 is an extremely strong negative relationship, 0 is no relationship, +1 is a very strong positivie relationship) etc. Mutual information is a distance between two probability distributions. Correlation is a linear distance between two random variables.In these two plots the correlation coefficient is zero - Pearson’s correlation coefficients measure only linear relationships so if your data contain a curvilinear relationship, the correlation coefficient will not detect it. For example, the correlation for the data in the scatterplot below is zero. However, there is a relationship between the two variables—it’s just not linear. But we can get high shared mutual information even when the correlation is zero - just because the coefficient is near zero, it doesn’t necessarily indicate that there is no relationship.In the first, I see that if I have a high or low value of X then I'm likely to get a high value of Y. But if the value of X is moderate then I have a low value of Y. The first plot holds information about the mutual information shared by X and Y. In the second plot, X tells me nothing about Y.![mututal_information.png](attachment:mututal_information.png)Mutual information and correlation are not antagonistic—they are complementary, describing different aspects of the association between two random variables. Loosely, Mutual Information "is not concerned" whether the association is linear or not, while Covariance may be zero and the variables may still be stochastically dependent. On the other hand, Covariance can be calculated directly from a data sample without the need to actually know the probability distributions involved (since it is an expression involving moments of the distribution), while Mutual Information requires knowledge of the probability distributions.<jupyter_code>'''
Estimating differential entropy
Estimate entropy exercies, for each, for each, 
estimate the entropy based on samples of 4 different sizes i.e. N: 10,100,1000,10000.
1. Uniform distribution on the interval [0, 1].
2. Uniform distribution on the interval [0, 8].
3. Uniform distribution on the interval [0, 0.5].
4. Gaussian distribution with mean 0 and standard deviation 1.
5. Gaussian distribution with mean 0 and standard deviation 100.
6. Exponential distribution with mean 1.
7. Exponential distribution with mean 100.

Parametric statistics are based on assumptions about the distribution of population from 
which the sample was taken. Nonparametric statistics are not based on assumptions, that is, 
the data can be collected from a sample that does not follow a specific distribution.
'''

SAMPLE_SIZES = [10,100,1000,10000]

#list to store monte carlo simulation entropies
entropy_monte_carlo = []


#create distribution
xs = []

#create standard deviation for distribution 
sigma = 1

def samplingEntropyEst(xs, N, sigma):
    '''
    Since differential entropy is an expectation (of the negative log probability density), 
    we can estimate it by sampling from a distribution and forming an empirical average of the 
    negative log probability density. This empirical average will converge to the 
    true differential entropy by the law of large numbers.

    Params:
    xs is the sample points in the distribution
    N is the number of times to sample in the Monte Carlo procedure
    sigma is the standard deviation of the sample - need to find a way of generating an 
    optimal sigma
    '''
    pass

for N in SAMPLE_SIZES:
    entropy_monte_carlo.append(samplingEntropyEst(xs, N, sigma))

def mspacingEntropyEst(xs):
    '''
    m−spacings estimate of entropy.
    One way of choosing m  is to use the nearest integer value to the square root of N, the
    number of sample points.

    Params:
    xs is the sample points in the distribution
    '''
    pass

entropy2 = mspacingEntropyEst(xs)<jupyter_output><empty_output><jupyter_text>Synthetic Data GenerationSynthetic data is information that's artificially manufactured (created by an automated process which contains many of the statistical patterns of an original dataset) rather than generated by real-world events. Synthetic data is created algorithmically, and it is used as a stand-in for test datasets of production or operational data, to validate mathematical models and, increasingly, to train machine learning models. It is also sometimes used as a way to release data that has no personal information in it, even if the original did contain personally identifiable information. Links:https://stackoverflow.com/questions/38711541/how-to-compute-the-probability-of-a-value-given-a-list-of-samples-from-a-distribhttps://cran.r-project.org/web/packages/sdglinkage/vignettes/Synthetic_Data_Generation_and_Evaluation.htmlhttps://towardsdatascience.com/synthetic-data-generation-a-must-have-skill-for-new-data-scientists-915896c0c1aehttps://mostly.ai/2020/10/28/comparison-of-synthetic-data-types/https://pypi.org/project/sdgym/https://stackabuse.com/generating-synthetic-data-with-numpy-and-scikit-learn/https://www.kdnuggets.com/2019/09/scikit-learn-synthetic-dataset.htmlhttps://github.com/theodi/synthetic-data-tutorialhttps://towardsdatascience.com/synthetic-data-vault-sdv-a-python-library-for-dataset-modeling-b48c406e7398  Estimating Density for Continuous Data using Kernel Density Estimation A common task in statistics is to estimate the probability density function (PDF) of a random variable from a set of data samples. One method for non-parametric estimation of a probability density function (PDF) for continuous random variables is the histogram. The most well-known tool to do this is the histogram. A histogram is a useful tool for visualization (mainly because everyone understands it), but doesn’t use the available data very efficiently. Kernel density estimation (KDE) is a more efficient tool for the same task. Kernel density estimation (KDE) is a non-parametric method for estimating the probability density function of a given random variable. KDE works best when data distribution is not specified or the parameters (e.g. mean, mode. variance) are not specified. It is also referred to by its traditional name, the Parzen-Rosenblatt Window method, after its discoverers. The gaussian_kde estimator from scipy or Kernel Density from sklearn can be used to estimate the PDF of univariate as well as multivariate data. It works best if the data is unimodal. However, as you will see below, Kernel Density Estimation (KDE) can be better. Notes:- KDE assumes a sample of independent, identically distributed (i.i.d) observations - For a continuous distribution the probability of any exact point is supposed to be zero.- Useful link on KDE- https://stackabuse.com/kernel-density-estimation-in-python-using-scikit-learn/<jupyter_code>'''
Modelling data as a continuous distribution
'''

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

from sklearn.neighbors import KernelDensity

# Generate synthetic data (random samples) from a mixture of 2 Gaussian distributions
# with modes at 5 and 10
# np.random.randn returns a sample (or samples) from the “standard normal” distribution (mean 0 and variance 1).
# np.random.rand is for Uniform distribution (in the half-open interval [0.0, 1.0))
# Fix the seed to reproduce the results
seed = 17
rand = np.random.RandomState(seed)

# numpy.random.randn - random floats sampled from a univariate “normal” (Gaussian) distribution of mean 0 
# np.random.randn operates like np.random.normal with loc = 0 and scale = 1.
# and variance 1
# 1D Array with 5 values i.e. 1 row  rand.randn(5)
# 2D Array with 10 rows, 1 column i.e. 10x1 use rand.randn(10, 1)

# Draw random samples from a mixture of 2 Gaussians with modes at 5 and 10
rows1, cols1, mode1 = 10, 1, 5
rows2, cols2, mode2 = 30, 1, 10
data = np.concatenate((mode1 + rand.randn(rows1, cols1),
                       mode2 + rand.randn(rows2, cols2)))

# numpy.hstack() function is used to stack the sequence of input arrays horizontally (i.e. column wise) 
# to make a single array. Useful when printing to screen

print("\nGenerated data from 2 gaussian distributions with mean 0 and variance 1, and modes %s and %s:"
      "\n %s" % (mode1, mode2, np.hstack(data)))



# Plot the true distribution 
# numpy.linspace - Returns evenly spaced numbers over a specified interval i.e. uniformly distributed in linear space
start = 0  # Start of the range
end = 16    # End of the range
N = 1000    # Number of evaluation points
x = np.linspace(start, end, N)[:, np.newaxis] # make it as column vector by inserting an axis along second dimension

# norm.pdf- normalpdf( is the normal (Gaussian) probability density function. 
# Since the normal distribution is continuous, the value of normalpdf( doesn't represent an actual probability - 
# in fact, one of the only uses for this command is to draw a graph of the normal curve.
norm_vals = norm.pdf(x, mode1, 1) * 0.25 + norm.pdf(x, mode2, 1) * 0.75
plt.plot(x, norm_vals, label='True distribution', color="red")

# Plot the data using a normalized histogram
plt.hist(data, 50, density=True, label='Data using a normalized histogram', color="purple")

# Do kernel density estimation. The parameters you can tune in case of KDE are the kernel and the bandwidth
kd = KernelDensity(kernel='gaussian', bandwidth=0.75).fit(data)

print("\nIntegral of KDE PDF  using kd.score_samples i.e. Probability when x = 10:")
print(np.exp(kd.score_samples(np.array([10]).reshape(1, 1))))

print("\nIntegral of KDE PDF using kd.score_samples i.e. Probability when x = 12:")
print(np.exp(kd.score_samples(np.array([12]).reshape(1, 1))))

# Plot the estimated density
kd_vals = np.exp(kd.score_samples(x))
plt.plot(x, kd_vals, label='Estimated Density (KDE)', color="green")

# Add labels
plt.xlabel("x")
plt.ylabel("Density")
plt.title("Estimating Density using Histogram vs KDE")

# Position legend
# position legend outside the plot using bbox_to_anchor (i.e. bounding box)
legend = plt.legend(bbox_to_anchor=(1.05, 1.0), loc='upper left')
# legend.get_frame().set_facecolor('C0') # Put a nicer background color on the legend.

# Show the plots
plt.show()
'''
Alternative modelling of a continuous distribution. Instead of using a parametric approach where you assume that your 
data has some distribution and then fit that distribution to your data, you should take a non-parametric approach. 
One straightforward way is to use a kernel density estimate. You can simply think of this as a way of smoothing a 
histogram to give you a continuous probability density function. 
'''

import scipy.stats
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

from sklearn.neighbors import KernelDensity


'''
Representation of a kernel-density estimate using Gaussian kernels.
Kernel density estimation is a way to estimate the probability density function (PDF) of a 
random variable in a non-parametric way. gaussian_kde works for both uni-variate and multi-variate data. 
It includes automatic bandwidth determination. The estimation works best for a unimodal distribution; 
bimodal or multi-modal distributions tend to be oversmoothed.
'''

'''
Parameters
dataset: array_like Datapoints to estimate from. In case of univariate data this is a 1-D array, otherwise a 2-D array with shape (# of dims, # of data).

bw_methodstr: scalar or callable, optional
The method used to calculate the estimator bandwidth. This can be ‘scott’, ‘silverman’, a scalar constant or a callable. If a scalar, this will be used directly as kde.factor. If a callable, it should take a gaussian_kde instance as only parameter and return a scalar. If None (default), ‘scott’ is used. See Notes for more details.

weights: array_like, optional
weights of datapoints. This must be the same shape as dataset. If None (default), the samples are assumed to be equally weighted
'''
# Fix the seed to reproduce the results
seed = 17
rand = np.random.RandomState(seed)

# Draw random samples from a mixture of 2 Gaussians with modes at 5 and 10
data = np.concatenate((5 + rand.randn(10, 1),
                       10 + rand.randn(30, 1)))

#data.shape
#(40, 1) i.e. 40 rows, 1 column
data_reshaped = data.reshape(40,) #convert to 1-D array as univariate data 

kde2 = scipy.stats.gaussian_kde(data_reshaped)
print("\nIntegral of KDE PDF for scipy.stats.gaussian_kde i.e. Probability when x = 10:")
print(kde2.pdf(10))


# To get the probability of an observation in some interval:
print("\nProbability of an observation in interval 12 - 13:")
print(kde2.integrate_box_1d(12,13))<jupyter_output>Integral of KDE PDF for scipy.stats.gaussian_kde i.e. Probability when x = 10:
[0.19839732]

Probability of an observation in interval 12 - 13:
0.04823245382217381<jupyter_text>Estimating Density for Discrete Data using Kernel Density Estimation Kernel Density Estimation is a non-parametric technique which works best when data distribution is not specified or the parameters (e.g. mean, mode. variance) are not specified. For discrete data, one could also  simply count the occurrences and divide by the total number of events to get estimates for the probabilities, anywhere where there are have zero observations, our estimate for the probability is zero.<jupyter_code>'''
Modelling data as a discrete distribution given a discrete sample space and all positive integers. 
We simply count the occurrences and divide by the total number of events to get estimates for the probabilities. 
Anywhere we have zero observations, our estimate for the probability is zero.
'''

samples = [1,1,2,3,2,2,7,8,3,4,1,1,2,6,5,4,8,9,4,3]
from collections import Counter
counts = Counter(samples)
counts
#Counter({1: 4, 2: 4, 3: 3, 4: 3, 8: 2, 5: 1, 6: 1, 7: 1, 9: 1})
total = sum(counts.values())
total
#20
probability_mass = {k:v/total for k,v in counts.items()}
probability_mass
#{1: 0.2, 2: 0.2, 3: 0.15, 4: 0.15, 5: 0.05, 6: 0.05, 7: 0.05, 8: 0.1, 9: 0.05}
probability_mass.get(2,0)
#0.2
probability_mass.get(12,0)
#0

# alternative using scipy.stats guassian library
import scipy.stats
kde = scipy.stats.gaussian_kde(samples)
kde.pdf(2)
#array([ 0.15086911])<jupyter_output><empty_output><jupyter_text>Sythentic Data Generation of a Uniform DistributionA Uniform Distribution is used describe probability where every event has equal chances of occuring.E.g. Generation of random numbers.It has three parameters:a - lower bound - default 0 .0.b - upper bound - default 1.0.size - The shape of the returned array.<jupyter_code>'''
Create a 2x3 uniform distribution sample
'''

from numpy import random

# Fix the seed to reproduce the results
seed = 17
rand = np.random.RandomState(seed)

# Draw samples from a uniform distribution.
# Samples are uniformly distributed over the half-open interval [low, high) (includes low, but excludes high). 
# In other words, any value within the given interval is equally likely to be drawn by uniform.
x = rand.uniform(low=0.0, high=1.0, size=(2, 3))

print(x)<jupyter_output>[[0.294665   0.53058676 0.19152079]
 [0.06790036 0.78698546 0.65633352]]
{'fraction_non_alphanumeric': 0.08690564418613722, 'fraction_numerical': 0.02281858251745524, 'mean_word_length': 4.6112934996717, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '470391', 'license': 'MIT', 'n_tokens_mistral': 7739, 'n_tokens_neox': 7130, 'n_words': 3425, 'path': 'data-entropy.ipynb', 'repo_name': 'jajukajulz/getting-started-guides', 'size': '25637'}