<jupyter_start><jupyter_text>1. What makes something a probability?Answer:* It needs to be a number * The number needs to between 0 and 1 (including 0 and 1 themselves)* where 0 indicates impossibility and 1 indicates certainty (Note: this is supposed to be a dialogue between two people, the master and the learner. The learner's questions are *italicized*)How about this number?<jupyter_code>0.08<jupyter_output><empty_output><jupyter_text>Yes, that looks like a probability. Being close to zero, it means that it is nearly impossible.Question:* *Just what is nearly impossible here?*Answer:* Good question. You're right, this isn't very useful at all. To make the number useful, it needs to be attached to something. Let's attach that probability to a value.<jupyter_code># [value,probability]
('something', 0.02)<jupyter_output><empty_output><jupyter_text>*Okay, the value 'something' is nearly impossible because it has a probability of 0.02. I'm still not seeing where this is useful.*Let's take this a step further and attach **values** to **variables**. As a programmer, you know all about variables and how they can take on values. Here are some examples using `X` as the variable:<jupyter_code>X = 'something'
# or
X = 'something else'
# or 
X = 5
# or ...<jupyter_output><empty_output><jupyter_text>The way we attach probabilities to variable values is by writing and using functions. Let's write one called `P`:<jupyter_code>def P(X):
    pass<jupyter_output><empty_output><jupyter_text>The function doesn't do anything, but it's a start.  This probability function `P` is a generic placeholder for a function that YOU need to write. The contents of that function depend on a lot of different factors.  The function `P`, the variables that it takes, and the values that those variables can take are necessary for working in probability theory.  For example, you might be interested in tomorrow's weather. You decide that one variable that you call `X`, should do the trick here. X can take on the values of either `sunny` or `rainy`. You decide that it's more likely to be sunny than rainy, so you write your `P` function as follows:<jupyter_code>def P(X):
    if X == 'sunny': return 0.9
    if X == 'rainy': return 0.5<jupyter_output><empty_output><jupyter_text>There is a problem: * The individual values of variable `X` are both between 0 and 1--making them probabilities by definition--but there still isn't an established relationship between the variables. What does it mean that sunny is 0.9 and rain is 0.5? how do those values compare to each other?Remedy:* Add a new constraint: the **the probabilities of all the values of the variable must sum to one**. In fact, a more generally useful constraint would be that the probability function `P` should **distribute** probability mass among all values of all variables such that the sum of all of those values is one.Let's rewrite our function to meet the demands of this constraint:<jupyter_code>def P(X):
    if X == 'sunny': return 0.8
    if X == 'rainy': return 0.2
P('sunny') + P('rainy')<jupyter_output><empty_output><jupyter_text>That's better. The sum of the probabilities of all possible values sum to one. (Though, let's be honest, we have no clue what those values should actually be. More on that later.)Note from Dr. K:* Making everything sum to one is a way of scaling the probabilities to fit a defined space (i.e., between 0 and 1) and doing so forces the variables to be related to each other in that all variable values work together to fill all the space between 0 and 1.   There is another problem:* What if we're interested in if it's going to be windy tomorrow?<jupyter_code>P('windy')<jupyter_output><empty_output><jupyter_text>The above function doesn't return anything (i.e., a `NoneType`). **The remedy:*** All of the variable (in this case `X`) values that you are interested in (in this case, `sunny, rainy, windy`) need to be accounted for in your probability function. If you want a function that can handle all three, then all three need to be considered and some amount of probability mass needs to be distributed to all three values (and, as usual, everything must sum to one).Question:* *Can't we just `return 0` after everything is said and done? That way if someone attempts to use my probability function and they try to get a probability back for something I don't have defined, at least they can still use the result. A 0 is better than a NoneType, isn't it? Like this:*<jupyter_code>def P(X):
    if X == 'sunny': return 0.8
    if X == 'rainy': return 0.2
    return 0
P('windy')<jupyter_output><empty_output><jupyter_text>*See? At least that returned a value. Before it didn't do anything.*Dr. K's reply:* One could easily argue for that, and it makes mathematical and even programmatical sense. However, you have to ask yourself what that 0 means to anyone using your function. You are defining your function to know something about `sunny` and `rainy` and nothing else. If someone comes along and asks what your function knows about `windy` and your function returns a 0, then your function is telling that someone that you are completely uncertain as to what will happen with `windy`. That seems harmless so far. But someone who doesn't know that your function doesn't actually have any provisions for knowing anything about `windy`, would interpret 0 as "oh good, there's no wind" (in other words, it's almost as if your function is completely certain that it is not windy). That's of course a bad interpretation of the function (but a very common way that people interpret probabilities from probability functions like this). How you handle this is up to you. (We'll talk about more principled ways to handle this later in the semester. For now, the keyword you can hold onto is "smoothing"). For the rest of this tutorial, we'll refrain from returning 0 for those unseen events.  Checkpoint: * Probabilities are numbers between 0 and 1* Probability functions (also known as probability **distributions**), distributes probability mass of 1 throughout the variable values (also known as the **sample space**).   Example: coin tossesWhat would the sample space be for this scenario? (I.e., what would the variable values be, and what would their probabilities be)? And, with that, what would the probability function/distribution look like?<jupyter_code>def P(X):
    if X == 'heads': return 0.5
    if X == 'tails': return 0.5
P('tails')
P('heads')<jupyter_output><empty_output><jupyter_text>Nice. **Question**:* *Can we handle more than one variable in a probability function/distribution?***Answer**:* Yes. But remember that after you consider all the values that all variables can take on (and all of the combinations!!) everything still needs to sum to one. **Example**: * What is the probability that someone who walks past you on the street is wearing a hat AND is wearing something that is colored orange?<jupyter_code>def P(X,Y):
    if X == 'wearing-hat'     and Y == 'wearing-orange':     return 0.2
    if X == 'not-wearing-hat' and Y == 'wearing-orange':     return 0.3
    if X == 'wearing-hat'     and Y == 'not-wearing-orange': return 0.2
    if X == 'not-wearing-hat' and Y == 'not-wearing-orange': return 0.3
    
P('wearing-hat', 'wearing-orange')<jupyter_output><empty_output><jupyter_text>This is an example of a *joint distribution/function*. It's joint because it jointly considers all possible variable combinations of more than one variable.Just to be more conventional, let's write functions that look more similar to those that are actually used in probability theory. That is, instead of just writing a function `P(X)`, it is sometimes useful to write the actual value in the function call, e.g., `P(X='sunny')`. Python has a way to help us with that. We just have to change the function declaration for `P`:<jupyter_code>def P(X='',Y=''):
    if X == 'wearing-hat'     and Y == 'wearing-orange':     return 0.2
    if X == 'not-wearing-hat' and Y == 'wearing-orange':     return 0.3
    if X == 'wearing-hat'     and Y == 'not-wearing-orange': return 0.2
    if X == 'not-wearing-hat' and Y == 'not-wearing-orange': return 0.3<jupyter_output><empty_output><jupyter_text>Then we can call the function and actually write the variables that we are using. This is doable in Python:<jupyter_code>P(X='wearing-hat', Y='wearing-orange')<jupyter_output><empty_output><jupyter_text>Question:* *Why bother with this, though? Why is that notation more useful? Why not just call the function with the values like we've been doing so far?* Answer:* We know the variable names `X` and `Y` in the second call, whereas in the first call we only know the values themselves. But the values themselves don't tell us anything about which probability function we should be using. That's because lots of variables could take on values. So the way we determine a function's signature (i.e., which function we are applying) is the name of the variables `X` and `Y` AND the values). * Another way of looking at it is *overloading*. The function `P` is a super-duper overloaded function. So how do we know which version of `P` to apply? We can't really tell by the name of the function since ALL of the probability functions seemed to be called the same thing. We can tell which function to apply by looking at the variables. Knowing what variable we are applying is also useful so we often call the functions knowing which variable is in question. I.e., the probability function call P(X,Y) is different from P(X,Z). It would call a completely different function.    So it's at this point where we must depart slightly from how programming functions and probability functions are used. We can still write probabilistic functions using Python, but we're going to have to treat them differently from each other.  Question:     * *What if I have a joint distribution for P(X,Y), but I'm just interested in P(X)? In other words, what can I do in order to "get rid of" `Y`?* Answer:  *  You can step through all the values for `Y` and sum up the probabilities. That way you've accounted for all the possibilities for what can happen in `Y`, leaving you with a distribution over `X`.   Check it out:<jupyter_code>P(X='wearing-hat', Y='wearing-orange') + P(X='wearing-hat', Y='not-wearing-orange')<jupyter_output><empty_output><jupyter_text>So P(X='wearing-hat') is 0.4.<jupyter_code>P(X='not-wearing-hat', Y='wearing-orange') + P(X='not-wearing-hat', Y='not-wearing-orange')<jupyter_output><empty_output><jupyter_text>And P(X='not-wearing-hat') is 0.6. And everything still sums to one (i.e., 0.4 + 0.6 = 1.0)!This magical operation is called **marginalization** and `P(X)` here is a **marginal distribution** (a special kind of probability function/distribution) which is a subset of P(X,Y). The general formula to marginalize (for joint distributions) is:$$P(X) = \sum_{y \in Y} P(X,Y=y)$$Where we step through each value in $Y$. *So far so good.* Notes on multiple variables and independence:* When multiple variables are used in a probability function/distribution, then it is a special kind of function/distribution called a **joint** distribution, as noted above. The notation is what we've seen so far: `P(X,Y)` is read as "the joint distribution over X **and** Y." (The **and** here is very important: it's asking "what is the probability that these two things happen together?") One can define as many variables as needed (e.g., P(X,Y,Z)), but in a joint distribution like this, every possible combination (i.e., the cross product) of all of the values must be defined.  Joint distributions don't sound very fun, especially when there are many variables and each can take on many possible values. Question:* *Do we really have to come up with ALL possible combinations and still have everything sum to one? I mean, if we have five variables, each only having two possible values (i.e., binary), then that means we'd have to come up with 32 different possible outcomes and everything still has to sum to one. That's just outrageous.*I feel your pain. If it helps you feel better, there's a way to simplify by handling cases where variables don't really have any direct relation to each other. Consider the hat/orange example. Does the fact that someone is wearing a hat really have anything to do with their choice of wearing orange? Not really. (I mean, in Idaho it might, but let's just say that it doesn't actually matter.)In such a case case, the two variables are **independent** of each other. If that happens, then the following is true: $$P(X,Y) = P(X)P(Y)$$ There's a lot to gain from that, it turns out (though it doesn't seem to be the case for our little example with two binary variables--if we had 3 variables, each taking on 10 possible values, then we would only need to consider 30 different probabilities instead of 1,000). We can refactor the function from `P(X,Y)` into two functions, one for `P(X)` and one for `P(Y)`<jupyter_code>def Px(X=''):
    if X == 'wearing-hat': return 0.6
    if X == 'not-wearing-hat': return 0.4

def Py(Y=''):
    if Y == 'wearing-orange': return 0.4
    if Y == 'not-wearing-orange': return 0.6
Px(X='wearing-hat') * Py(Y='wearing-orange')<jupyter_output><empty_output><jupyter_text>*Hmm, that's not the same as before.*    That means that these two variables are not independent of each other (at least in the way we've defined the functions). But if we enumerate all outcomes, does it still really sum to one?See for yourself:<jupyter_code>Px(X='wearing-hat') * Py(Y='wearing-orange') + Px(X='not-wearing-hat') * Py(Y='wearing-orange') + Px(X='wearing-hat') * Py(Y='not-wearing-orange') + Px(X='not-wearing-hat') * Py(Y='not-wearing-orange')<jupyter_output><empty_output><jupyter_text>*Well then.*The same thing works if you want to link different **events** together, not just different functions. For example, What if we want to find the probability of tossing the coin three times with the sequence: `heads, heads, tails`?Answer:* The events themselves seem to be independent of each other. What can we do?* We can apply the same function `P` for all three events.* Since they are independent events, we can multiply all three events together!* Note: each 'event' is also known as a **trial**<jupyter_code>def P(X=''):
    if X == 'heads': return 0.5
    if X == 'tails': return 0.5
P(X='heads') * P(X='heads') * P(X='tails')<jupyter_output><empty_output><jupyter_text>Checkpoint* A **joint** probabiliy distribution is a way of expressing how multiple variables jointly affect a distribution. Just like in a single variable distribution, all combinations of the probabilities all values of all variables must sum to one. * We can factor the variables of joint functions/distributions into multiple easier-to-manage functions/distributions if we can determine that those variables are **independent** of each other.  Question:* *OK. We know how to handle one or more variables in a probability distribution. But these all seem to be cases when the variables aren't actually known. What if I **know** the value of a variable already? Would that not change how a probability function works?*Yes. Here's an example of that. You can write a joint probability function `P(X,Y)` where `X` can be `sunny/rainy` and `Y` can be `carry-umbrella/leave-umbrella`. Those obviously aren't independent of each other. But in this case, writing a joint probability distribution is somehow a disservice because you can look out the window and you can *know* with certainty what the value of `X` is: either's either sunny or it's rainy (at that particular time, at least). You can treat it as a different kind of variable where you know the value.  So the question you want to ask is: what is the probability that someone carries an umbrella when it's sunny? Put in other words, what is the probability that someone carries an umbrella **given that** it is raining? Put yet another way, what is the probability that we carry an umbrella **conditioned on** the *fact* that it's raining?Your function would look something like this:<jupyter_code>def P(X='',Y=''): 
    if X == 'sunny':
        if Y == 'carry-umbrella':return 0.08
        if Y == 'leave-umbrella':return 0.92
    if X == 'rainy':
        if Y == 'carry-umbrella':return 0.9
        if Y == 'leave-umbrella':return 0.1<jupyter_output><empty_output><jupyter_text>The structure of that function looks different from the single-variable and joint probability functions/distributions that we have seen so far. That's because you are taking information that you know about (i.e., whether `X` is sunny or rainy), and *conditioning* the results on that knowledge (hence, the outer-most `if` statement). As you might have guessed, this kind of probability distribution is a special case called a **conditional**distribution. Note: the way we wrote conditional probability functions in the form `P(X,Y)` is actually wrong. Conditional probabilities use the notation `P(X|Y)`, but we can't really do that with Python. So, from now on, we'll notate conditional probability functions with `Pcond`. It's also conventional for the items on the left to be conditioned on the variables on the right, so we actually wrote it backwards when we were calling it. Let's fix that:<jupyter_code>def Pcond(Y='', X=''): 
    if X == 'sunny':
        if Y == 'carry-umbrella':return 0.08
        if Y == 'leave-umbrella':return 0.92
    if X == 'rainy':
        if Y == 'carry-umbrella':return 0.9
        if Y == 'leave-umbrella':return 0.1
 # the probability that I carry an umbrella, given that it is sunny
Pcond(Y='carry-umbrella', X='sunny')
# the probability that I carry an umbrella, given that it is rainy
Pcond(Y='carry-umbrella', X='rainy')<jupyter_output><empty_output><jupyter_text>*Hold on, you said that probability distributions should sum to one, but `Pcond` doesn't sum to one!*Right. The way conditional probability functions/distributions work is this: since you know that something is the case (e.g. that it is sunny outside), then you can partition your function such that all of the sub partitions act as functions of their own--hence, each sum to one. Maybe rewriting the above function like this will help:<jupyter_code>def Psunny(Y=''):
    if Y == 'carry-umbrella':return 0.08
    if Y == 'leave-umbrella':return 0.92
    
def Prainy(Y=''):
    if Y == 'carry-umbrella':return 0.9
    if Y == 'leave-umbrella':return 0.1    
    
def Pcond(Y='', X=''): 
    if X == 'sunny': return Psunny(Y=Y)
    if X == 'rainy': return Prainy(Y=Y)<jupyter_output><empty_output><jupyter_text>Where `Pcond` calls functions that each properly sum to one on their own.<jupyter_code>Pcond(Y='carry-umbrella', X='sunny')<jupyter_output><empty_output><jupyter_text>More about Conditional Probabilities:* You may have noticed that for joint probabilities, it doesn't really matter how you set up your function. In other words, `P(X,Y)` and `P(Y,X)` are the same thing. * For conditional probabilities, this isn't necessarily the case. In other words, `P(X|Y)` isn't always the same as `P(Y|X)`. It *can* be the case, but it often is not the case. Our example above for `Pcond` doesn't illustrate this very well (i.e., if you changed the conditions to have `Y` be considered in the first `if` statements, it wouldn't really change the outcome because of the way the function is defined). But seriously, do you think that the rain depends on whether or not you carry your umbrella? Obviously, it doesn't. However, whether or not you carry your umbrella likely depends on the weather. Question:* *Can we marginalize over `Y` for `P(X|Y)` to get `P(X)` like we did for the joint distribution?*Answer:* Yes! But you have to know `P(Y)` somehow, because you have to do marginalizations for conditional probabilities a little differently than with joint distributions.Here's the formula for marginalizing over conditional distrbiutions:$$P(X) = \sum_{y \in Y} P(X|Y=y) P(Y=y)$$  Notes on Probability functions/distributions so far:* The functions that we wrote, `P`, always constrain their distributions to be one, no matter how many variables are involved, no matter how many values those variables can have, whether it's single-variable, joint, or conditional (for each partition). * That some variables are conditional upon each other and that others are not is a very important concept in probability theory. Variables that aren't related or conditional upon each other are also kown as **independent** variables. The coin toss events were indepenent of each other, but the decision to take an umbrella or not was dependent on the weather.     * *Why is independence so imporant?* The answer is: complexity. When variables are independent of each other, it makes the math to compute probabilities much easier: we can define extra `P` functions and just multiply things together! Easy! But if they aren't independent, we have to come up with the probability function/distribution that defines the conditions that the variables have on each other. That takes more effort and is more computationally expensive.  Checkpoint:* Functions are important* Probability functions/distributions constrain all combinations of variable probabilities to sum to one (except for conditional probabilities where each sub-function sums to one).* Conditional probability functions/distributions take the form `P(Y|X)` (read: "the probability of Y given X).* Joint probability functions/distrbutions take the form `P(X,Y)`.* Variables can be jointly related, conditionally related, or independent of each other.Question:* *Why do this? Why write probability functions that are constrained to have everything distributed such that they sum to one? Is there really no other way?*Dr. K's answer:* First, when we map everything to the probability scale, i.e., between 0 and 1, we get some nice tools that we can use. We get some nice equations that we can apply (we will see even more of them below!). We can make use of the entire field of statistics to do our bidding. Second, when you force the probabilities of all possible value combinations to sum to one, you are comparing and relating those variables. The amount of space/probability mass/probability distribution to play around with doesn't seem to be much, so the variable values that get more than others means something. Interpretability is really important.   2. Writing the Probability Functions / Estimating the Distributions So far, we've seen the difference between single-variable, joint, and conditional probability distributions. However, the examples that we wrote were completely made up (except maybe the coin toss with a 50/50 head/tails distribution). Obviously, this isn't how it's done. To write our own probability functions, we need to **estimate** the values. For that, we need data. We can either go find the data, or, for illustration purposes, we can just invent some data.<jupyter_code>#
# some data: a randomly generated list of 1s and 0s
# 

from random import *

n = 1597
data = [randint(0,1) for b in range(1,n+1)]
data[:10], len(data) # show the first 10 elements of the list<jupyter_output><empty_output><jupyter_text>Now we want to write the function and fill in the question marks:<jupyter_code>def P(X):
    if X == 1: return ?
    if X == 0: return ?<jupyter_output><empty_output><jupyter_text>Question:*How can we do that?*Answer:Remember that we are dealing with discrete distributions, so.... By COUNTING THINGS* There are lots of ways to count things. Since we are using Python, I'm going to use a Counter which is a very convenient library.  *In fields like NLP we often count things because we have things that are countable. There are many cases where we can't actually count the data to obtain a distribution. We'll talk about that later.<jupyter_code>from collections import Counter # the Counter library

counts = Counter(data) # just pass in the list and see what happens

counts
counts[0], counts[1]<jupyter_output><empty_output><jupyter_text>Well that was easy. But those are huge numbers. How do we get probabilities out of that? Answer:* Use the relative frequences as the probabilities<jupyter_code>def P(X=''):
    global counts # the global keyword tells this function to use the global variable called counts
    global data   # it's generally frowned upon to use globals in Python, but I'm doing this to keep the function
                  # calls looking like they should (i.e., like probability functions)
    return float(counts[X]) / float(len(data)) # I had to add these floats for some reason
P(X=1)
P(X=0)
P(X=1) + P(X=0) # sums to one?<jupyter_output><empty_output><jupyter_text>*That seems like a nice Probability function! We didn't have to explicitely write the actual probabilities of the two cases. That's super useful. What if we are dealing with more than just binary values?*Let's make some more data with 25 values:<jupyter_code>data = [randint(0,25) for b in range(1,n+1)]
counts = Counter(data)

counts
data[:10], len(data)<jupyter_output><empty_output><jupyter_text>Question:* *Okay, so we have a huge list of numbers between 0 and 25. Can we use our same probability function to calculate the probabilities?*Answer:* Let's see:<jupyter_code>P(X=12)<jupyter_output><empty_output><jupyter_text>*Does everything sum to one?*<jupyter_code>import numpy as np

np.sum([P(X=x) for x in set(data)]) # this function calls P for all possible values in data (hence the call to set)<jupyter_output><empty_output><jupyter_text>There might be a slight rounding error, but it's essentially one. Not bad, that. Still, that's a simple estimation of data for a single variable. Things get much more more complicated when estimating joint distributions with several variables, each of which can range over many possible values.   A lot of the work in machine learning (i.e., NLP) has to do with estimating values like this. This is what is often called "training" but we'll get to that in more detail later in the semester.  3. Joint and Conditional Probabilities: Mapping Between Them Question:* *What if I have the means to estimate a joint probability, but what I really want is a conditional probability? Or visa-versa?*Answer:* Vee haf vays. (We have ways.) It is the case that: $$P(A,B) = P(B) P(A|B) = P(A) P(B|A)$$This is called the **multiplication rule** and it's very, very useful.  (This explains why marginalizing over a conditional probability was kind of weird.) In fact, we can generalize it to probability functions/distributions that have an arbitrary number of variables: $$P(A_1, ..., A_n) = P(A_1) P(A_2|A_1) P(A_3|A_2,A_1)\ ...\ P(A_n|A_1, ..., A_{n-1})$$This is called the **chain rule**.  So we can use these equalities to go from joint to conditional probabilities and the other way around, if we need to (and we will need to).  4. Bayes' Theorem The multiplication rule and chain rule are useful for going between joint and conditional probabilities, but there's another thing that would be useful. Forget joint probabilities for the moment and let's focus on conditional probabilities. Recall that we can't just swap the variables; i.e., `P(A|B) != P(B|A)`.Question:* *Is there a way to swap the dependencies? I.e., is there a way to go from `P(A|B)` to `P(B|A)` without injuring myself?*Answer:* Yes. It's called Bayes' Theorem.  Bayes' Theorem: $$P(A|B) = \frac{P(A,B)}{P(B)} = \frac{P(B|A)P(A)}{P(B)} = \frac{P(B|A)P(A)}{\sum_i P(B|A_i)P(A_i)}  $$  5. Expected value, Variance, Covariance Expected value$$ E[X] = \sum_{x \in X} xP(X=x)$$<jupyter_code>mu = sum([x * P(X=x) for x in set(data)])

mu<jupyter_output><empty_output><jupyter_text>VarianceVariance is a measure of how varied the distribution function is. $$ Var(X) = \sum_{x \in X} (x - µ)^2 P(X=x) $$<jupyter_code>import math

var = sum([math.pow((x-mu),2) * P(X=x)])

var, math.sqrt(var) # <- standard deviation<jupyter_output><empty_output><jupyter_text>CovarianceCovariance is a measure of the variance between **two** distributions where the variables and variable values are the same. $$ cov(X,Y) = \frac{1}{n^2} \sum_{i=1}^{n} (x_i - E[X])(y_i - E[Y]) $$<jupyter_code># setup another distribution, get E[Y]
data2 = [randint(0,25) for b in range(1,n+1)]
counts2 = Counter(data2)
def P2(X=''):
    global counts2 
    global data2   
    return float(counts2[X]) / float(len(data2))
mu2 = sum([x * P2(X=x) for x in set(data2)]) # compute E[Y]

(1.0/float(len(data))) * sum([(x - mu)*(y - mu2) for x,y in zip(set(data),set(data2))])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08978490856862607, 'fraction_numerical': 0.00742660968261753, 'mean_word_length': 4.7036963036963035, 'pattern_counts': {'":': 0, '<': 126, '<?xml version=': 0, '>': 125, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '77501', 'license': 'MIT', 'n_tokens_mistral': 8269, 'n_tokens_neox': 7815, 'n_words': 4063, 'path': 'introds/classes/C4-Probability/probability_theory-discrete (1).ipynb', 'repo_name': 'gcaracas/bsu', 'size': '28546'}