<jupyter_start><jupyter_text>Extended Rant Version> This version needs significant refactoring. It is fairly complete tough.   What happenedI do not normally write blog posts.Today is not a normal day tough. I believe that I have found a missing link in my understanding of bayesian modelling theory.This is not so much about the bayes belief update equation. It's good. The problem is that the notation is confusing, making it difficult to distinguish and source(compute, obtain) the various terms in the equation, in connection to the real-world phenomena. And then, when solving a real world problem, this is not even where you put most of the time-effort in. You might think that bayes update rule:$p( A | B) = \frac{p(B|A)p(A)}{p(B)}$is all that it takes.This form is nice for proving that the bayes' theorem is correct. However, what do these terms mean? How does one apply this?   You might have heard that $A$ is a model parameter, and $B$ is observation. And that you can derive the model parameter $A$ from observations. Still, it is not at all obvious to someone that tries to figure it out for himself. Sources of confusion include:* `probability` and `belief` are not the same thing, yet they are denoted with the same letter* $p(B|A)$ does not need to sum to 1.0 while $p(A|B)$ does ? * values of $p(B|A)$ need to be between 0 and 1, while values of $p(A|B)$ do not ? * The name and function body for $p(A)$ and $p(B)$ is **completely** different, even tough they use the same symbols?* Some symbols evaluate to scalars, and others evaluate to vector? then, some places that used to be scalar can be vectors sometimes?* Data (observations) seems to have the same type as probability (`float`) , even tough the data can be `nominal`, like `T`/`H`, distribution, or vector what then?* `Discrete events`, `datapoints`, `sets` and `distributions` are very different things, yet they are denoted by the same symbol and *location** `latent model parameters` versus `observable data` -- and where they come from, and where they go into? these are conceptually very different, yet they are denoted by location in a composite symbol as if they were interchangeable. Granted, the interchangeability is proven and even exploited in deriving the Bayes update rule: $p(A|B) p(B) = p(B|A)p(A)$ for any meaning of $A$ and $B$. However, this neat algebraic trick does no favours to the conceptual understanding and practical applications of this equation. * Most books go straight to continuous distributions and multidimensional variables. These are beautiful exhibits of maths' notations' brevity and generalism, but also are absolutely redundant. Discrete events are fine. Scalars are fine. The additional hoop created by going straight to things like fractional dimensions is a serious barrier for the children AND their teachers.* `class of function`, `instance of function`, `function name`, `function body` and `value of function` are all separate concepts in computer programming, but they are used interchangeably in mathematics books. * The update equation looks simple, but all practical problems have more than 2 dimensions that we are interested in exploring.  Even for the most simple model $m$, we will want a (1) distribution of posterior beliefs for (2) possible $A$. That's already a 2D plot. but then, the $A$ can be 1,2,n dimensional, we can have different observation scenarios or multidimensional observations, and a variety of prior beliefs to worry about. And then, there is the variety of possible $models$. How do you visualize that? * It seems that, in paper-and-pencil maths, symbols are often aliased and shortened to save on hand movements. Modern practical computer science has long demonstrated that such savings are counter-productive.  And the most confusing of all -- how does one build the `likelihood function`? Where does it come from? This is not specified in the bayes update equation at all, yet when doing exercises, one is expected to just magically come up with a correct one. For the longest time I thought that this is something that needs to be derived from the Bayes' equation itself. This is how you solve all the other problems in the school, right? The teacher told you that $F=m*a$ so that's all that you should need. . . . actually, no. Bayes' theory tells you a true nothing, zero, nil, about the model $M$ of the world. This you have to invent separately. Only after you are done inventing, Bayes can tell you something about how good your model $M$ is. This is a true stunner for a high-school children who, up to this moment, were lead to believe that there exists only one correct solution to all problems. Amazingly, despite all the talk about probabilities, coin tossing and chances of getting cured by a new medicine -- the likelihood function, and the value of the likelihood function, as well as prior and posterior beliefs, have no random chance in them at all. In bayesian thinking, some things are hidden, but no things are random!  The only place where randomness is allowed is the plant. This randomness can be modelled in the model.  --- OK, so ranting over, let me try to clarify some of these things.> Warning: Over the course of this section, I will rewrite the classic equations a couple of times. Do not call me out on the blatant fact that I use different symbols for the same thing across the first part of this article. This is necessary to make my point. **1. Reality check.** To ground the concepts in some tangible reality, let's consider that we have a **plant**. Plant is just a name for some object, or process, that has inputs and outputs. It does not need to be a manufacturing plant, or like a greenhouse plant. All that it does is that it exists, has inputs, and outputs:>>> Drawing here <<<For now, we will only be concerned with the plant's outputs.This plant produces data $\{ D \}$, which we can observe. The process of producing instances of $D$ can be approximated and described by a $model$ $M$. The model $M$ describes the plant somehow, but at this point we should be clear that there can be more than one good model for that plant (even tough most models will be useless). Let's suppose that we have a model $M$ of the plant, that takes a hidden(latent) parameter $\phi$. It could be then said that the data $D = M(\phi)$. We'd like to know what the true $\phi$ is, but we cannot be sure. We cannot measure $\phi$ directly. The $M$ can have a probabilistic nature to it, in sense that depending on $\phi$ it can produce given $D$ more or less often. Still, we are allowed to have suspicions and beliefs about what the true $\phi$ is. We can observe the events $D$ that happen with $M$'s and $\phi$'s contribution.**2. Use the reasoning to figure out $\phi$**For a start, let's replace $A$ and $B$ with model parameter $\phi$ and data(observation) $D$:$$p( \phi | D) = \frac{p(D|\phi)p(\phi)}{p(D)}$$Next, let's rewrite the bayes' update equation *slightly* differently (this is not the end of the re-writing):$$p( \phi | D) = \frac{p(D|\phi)}{p(D)}\times p(\phi)$$Compared to the original form, this already gives you a better hint: you can get posterior belief, $p( \phi | D)$, from the prior $p(\phi)$, modified by an "updater term", $\frac{p(D|\phi)}{p(D)}$.Ok, so what about this updater term seems to be so difficult?Wait. Let me clarify the concepts one by one. And there is quite a bit to untangle, before the road becomes straightforward for us. **Untangle the concepts of Belief, Probability, and Sample**Let's take in a new concept: that `belief` is not the same as `probability`, and both are distinct from `sample`.Here are some hints as to how to separate the two:* `Belief` is something that you hold in your head. `Probability` is a property of the world. Since there exists an impenetrable epistemological barrier between your "inner" and "outer" world, these things are already distinct.* Things do not happen because of `belief`, nor because of their `probability`. * Things happen inside the `plant` according to the plant's model and parameters. The plant's model is $M$ and it's parameter is $\phi$. We can observe data $D=f(M, \phi)$.* `Belief` is something that we can change ourselves. `Probability` is not something that we can change.* `Probability` we can estimate from frequency of observation. It follows that for things that are never observed, we cannot talk about their probability. * `Belief` we can assume, virtually out of nothing. We can then either hold oto that belief, or update it. We can update it, for example, arbitrarily(without any reasoning), or using some rules. For example, using Bayes' update function (maximum amount of reason).* Estimate of the value of `probability` can be a function of assumptions, including `beliefs`. Updates of `belief` can be a function of `data`. * `Probability` and `belief` seem to have the same unit, "fraction of $\Omega$" -- where $\Omega$ is "all that there is, all possibilities". Maybe this is the reason why historically they both have been noted as $p()$. This is a poor reason. I propose that we give them a units. * I propose that, to make it easier to keep a distinction between `probability` and `belief`, they carry distinct units.Here, I propose the following units:I will be using unit of `Rey`, or R for belief, that certain statement is true.I will be using the symbol $\Omega$ for probability on that a certain parameter, event, or statement, or data point $E$ will happen in the future, or is happening while we do not see it happening. Note again, that this is very different from having a completely certain data set that shows that "$E$ has happened 70% of the time". While we are at it, we can also add an unit for that observed dataset, $\{ D \}$. The reason for a separate unit here is that the set of all possibilities, $\Omega$ is infinite, while the dataset $\{ D \}$ is a finite **sample** from $\Omega$. Hence, it is **not true** that if we have a dataset that shows "$D$ has, so far, happened 70% of the time" is equal to "$D$ is 70% of $\Omega$". Instead, let's define a new unit, [$S$] to describe the prevalence of $E$ in dataset $\{ D \}$.To summarize:Given that, for us, $E$ could mean either $\phi$ or $D$, we have:0.7[R]=700[mR] means "My belief is such that I am 70% sure that a specific value of $\phi$ is true. I leave the 30% to beliefs that some other value of $\phi$ can be true."0.8[$\Omega$]=800[m$\Omega$]  means "In this chance process, If I sample forever, I will get $D$ 80% of the time. I will get something else 30% of the time".0.9[$S$]=900[m$S$] means "In this dataset, which is a sample of $\Omega$, event $D$ occurred 90% of the time".Again, although [R], [$\Omega$], and [$S$] seem to be mathematically interchangeable and can be expressed in percent, or fraction of a whole -- semantically they are distinct, as they refer to different concepts. Having these insights, I propose a following notation.  The stage star $b(\phi)$Let's use function symbols, $f(something)$ : * $b$ for belief, both prior and posterior,* $l$ is for likelihood, * $m$ for marginal probability. Let's use the variable symbols, $something$: * $\phi$ for a certain model parameter that belongs to a set of considered model parameters $\{ \phi \}$ * $D$ for a datapoint that already occurred, and we have it in a dataset $\{ D \}$; the dataset $\{ D \}$ has been sampled from a population $\Omega$It follows that, The **new belief** about parameter $\phi$, that takes into account the **new information** from observing a datapoint $D$, is noted as $b(\phi|D)$, And that $b(\phi|D)$ **can be calculated** as coming from old(prior) belief about $\phi$, noted as $b(\phi)$ And that this calculation involves evaluating a Bayesian modifier term, $\frac{l(D|\phi)}{m(D)}$:$$b(\phi|D) \leftarrow \frac{l(D|\phi)}{m(D)} \times b(\phi)$$  The ugly duck $m(D)$The marginal probability $m(D)$ is also sometimes called "evidence". It is calculated by summing (or integrating) the likelihood over all considered values of $\phi$, weighted by the prior belief in $\phi$. $$m(D) = \sum_{\phi}{l(D|\phi)b(\phi)}$$But wait!!! In the above equation, the $\phi$ is not the same $\phi$ as in the previous equation! If you didn't know that, I do not blame you. Instead, here $\phi$ is to say "for all $\phi$s". Because of this, one is not allowed to substitute this expression for $m(D)$ into the bayes' update equation from the previous section -- at least, not using regular algebraic rules as learned in high school : that would create a notation conflict!In my school times, changing the meaning of notation between different classes was the single most confusing obstacle to quick assimilation of new concepts. If I learned something in the chemistry class, I had to forget about it before going to the Physics class, or else I was in for trouble!Be advised that for a child's mind, like for the early computers, all symbols and variables are "global". Have you seen that viral video where a 3-year old girl says that a black man ate the cookies? She did not learn to remap the concepts for political correctness when being filmed. She merely rehashed the concepts she heard from the people that surround her. For adults, switching the context is possible, but still taxing. Changing the meaning of notation in a middle of doing a school problem is a good recipe for catastrophe. This is in no small part due to that the children are plain not afforded the time and slowdowns that it takes to create correct concept remapping in their heads. They are expected to solve a problem in under 5 minutes, or fail. You tell me what happens to most of us. Let's make things easier to learn, by using a clearer, non ambiguous notation.In order to note that the $\phi$ in the following equation has a "set of $\phi$" meaning, rather than just "single scalar $\phi$" meaning, let's note it as $\{\phi\}$. The symbols $\{,  \}$ are used in secondary schools to denote closed sets. That hints the student that he must consider the entire set of $\phi$s and not just a single $\phi$. Moreover, we can prepare the same person for using list comprehension semantic from Python (and hence, make Python easier for this person), by using notation like "for $\alpha$ in $\{\phi\}$" or "for $\alpha \in  \left$":$$m(D) = \sum_{\alpha \ \in \ \{\phi\}}{l(D|\alpha)b(\alpha)}$$Now we are safe to write that$$b(\phi|D) \leftarrow b(\phi) \times \frac{l(D|\phi)}{  \sum_{\alpha \ \in \ \{\phi\}}{(l(D|\alpha)b(\alpha))} }$$The veterans will notice how now, no symbol aliasing occurs. Every symbol has unique semantic meaning.Moreover, we can be much more clear about where the specific numbers needed are to come from. Only now, having this unambiguous expression, one is equipped to attempt to solve practical problems. > Warning: In the following part of the article, I will not be changing the notation any more, so feel free to call me out on any errors. ---  Prior $b(\phi)$$b(\phi)$ is the initial, or prior **belief** about one of the possible $\phi$s. $b(\{\phi\})$ is the initial, or prior **belief** about all of the possible values for $\phi$s, and is a vector, or a set: that is, there is a new value $b(...)$ for each $\phi$ from the set of $\{ \phi \}$sValues for $b(\{\phi\})$ come truly from outside of the dataset and model. In other words, one has to start with some beliefs, something based on external knowledge of the problem at hand. At this point, most books go into depths about the challenges of holding an "informative" or "uninformative" prior, and give (often unclear) experimental examples on how do they affect the result.Here I hope to make the examples clearer, by the virtue of using the unambiguous notation developed in the previous section.For example, we can have, for $\{\phi\}$ = [0.1, 0.5, 0.9], $b(\{\phi\})$ = [0.1, 0.8, 0.1]. This means that our belief is that $\phi$ is most likely 0.5, but we also allow for a total doubt of 20% that it can give way to $\phi$ being 0.1 or 0.9. Importantly, this distribution does sum up to 1, that is $\sum_{\alpha \ \in \ \{\phi\}}b(\alpha) \ \ = 1$One can start either with an "uninformative" prior or an "informative" prior. "uninformative" priors are fine if there is enough experimental data to come by; however, if the data is scarce, and there is something that we already know about the problem (e.g., someone told us, or we made a related but not identical experiment), then would be a mistake not to incorporate it into our thought process. ---  Posterior $b(\phi | D)$$b(\phi | D)$ is the posterior **belief** after update. For simple problems, we often talk not about different $D$s, but rather just one discrete $D$. Obviously, data sets $\{ D\}$ will also enter the fray as we become more proficient.It can be important to note that there is a difference between:* $b(\phi | D)$ -> Scalar* $b(\{\phi\} | D)$  -> vector* $b(phi | \{ D\})$  -> vectorAs all of these involve a vector computation or a loop of some kind. Having that, it is even more clear to see that $b(\{ \phi \} | \{ D\})$ denotes and evaluates to a 2D array of numbers.Alternatively, we could talk about updating our belief for one specific $\phi$ depending on what $D$ we get. In other words, building a closed-form function. Although this view can be useful, this is not what most examples in the literature are about. $D$ is most often a given constant.$b(\{\phi\} | D)$ over a range of $\phi$, evaluates to a 1D numeric vector, and this is what you typically want to get to: a description of the final belief about what the latent $\phi$ could be. In other words, an indication of "what should you believe about the different possible $\phi$". This distribution does sum up to 1.For example, let's say, that after having our prior beliefs $b(\{\phi\})$ = [0.1, 0.8, 0.1] of what the latent value of $\phi$ could be, we observe new data point, $D$. Having this new data point, we update (it doesn't yet matter how) our belief for values of $\{\phi\}$=[0.1, 0.5, 0.9] to be new $b(\{\phi\} | D)$=[0.45, 0.45, 0.1]. Meaning, that we still believe that $\phi$ value of 0.9 is incredible, but now give equal credibility to values $\phi$=0.1 and $\phi$=0.5.To summarize, using the $"|D"$ part of the notation is there to symbolize that this is about the updated belief, given the data point $D$. Not that we have a sweep of datapoints, and not that we distribute our belief across different $\phi$.  ---  The Likelihood function $l(D|\phi)$$l(D|\phi)$ is the "likelihood function". "Likelihood" itself, is a word that typically doesn't really tell you much, because the meaning used here is quite different and distinct from the common-language synonyms of "likelihood". Here, by "likelihood" we do not mean any of "frequency", "chance", "odds", "feasibility", "plausibility" e.t.c. We mean something quite specific here.Let's attach concrete meaning to the word "likelihood".Here, "likelihood" it means "the function, and the values of probabilities that single point(or single batch) of data $D$ has been generated by the $model$ $M$ with a given parameter $\phi$".Likelihood values carry the unit of probability, $[\Omega]$ Notably, * A value array $l(D|\{\phi\})$ does not need to sum to 1.0 as in the case of belief distribution. * the individual values of $l(D|\phi)$ for any discrete $\phi$ must still be in the range of (0, 1), as it should be with probability. For a contrived example, we can have $l(D|\{ \phi \})$ = [1.0, 0.1, 0.5] for $\{ \phi \}$ = [0.1, 0.5, 0.9] and data $D$ = [0]. This means that: * The data value "0" is reliably always generated by the $model$ $M$ when the $model$'s $\phi$==0.1. * if the $\phi$==1, then this data would only be generated rarely, approx. 10% of cases. * if the $model$'s $\phi$=0.9, then the data value "0" is still generated approximately half of the time.When sampling infinitely from the model $M$ having the parameter $\phi$, we cannot get a $D$ population fraction bigger than 1.0$[\Omega]$ or smaller than 0.0$[\Omega]$.However, the sum of all $l(D|\{ \phi \})$, or $\sum_{\alpha \in \phi}{p(D|\alpha)}$ can be more or less than 1.00 Moreover, if we observe a single data point $D$=[0], then we still cannot be sure if the true value of model's $\phi$ was 0.1, 0.5, or 0.9. All possible values of $\phi$ are consistent with getting $D$=[0] sometimes.Note that I have used the word `probability that` rather than `belief that`. This is because for likelihood, there is no guessing of any kind involved, and there is nothing latent(hidden). Instead, given a datapoint, and given one (or a list) of model parameter values, we calculate the already-happened chance that the data, as seen, has happened. We can calculate this chance for all possible hidden parameter values, irrespectively of our belief in them. This is kind of like a grid search, or grid view. Us doing this calculation does not favour any $\phi$ out of the set of $\{ \phi \}$ (yet) and the result does not mean that any specific $\phi$ is the right one. Obviously, in order to perform this calculation, we need a model of the world, and a kind of that uses these hidden parameters and is conductive to our calculation. Here much of the trouble and effort of the user of bayes' rule comes in. Many treatises on how awesome the bayes rule is will not help you with this.It is up to the user to construct a model, make sure that the model is representative of the reality, and that it is conductive to the likelihood function value evaluation. Let's take a look at two most common, basic models, and how is the likelihood function constructed for them. The examples listed below are NOT to say that this is the only way the likelihood function can be constructed!   Example model of coin toss, and constructing it's likelihood function.The classic coin-toss toy model is a good one, and widely applicable and extensible, if explained correctly.  Model descriptionHere it is so that you don't have to go back to the book.Let's say that the result of the coin toss can be 0 or 1. Our model approximates the real coin by ignoring the possibility of the real coin landing on the edge.---Proposition: biased coin gives "1s" more often. ---Let's say that the coin could be biased(that is, unfair) and we describe this bias with a parameter $\phi$. We do not know, and cannot know directly what the true $\phi$ is. What we do know is that the $\phi$ can be anywhere from 0.0 to 1.0, with a 'fair' coin being at $\phi \equiv$ 0.50. * 0.10 means that the coin is very unfair towards only giving zeroes, * 0.90 would mean that it is very unfair towards only giving ones. * 0.50 means that it gives 0 in 50% times, and 1 in 50% of times, that is, the coin is fair  The Likelihood function for the modelTo compile this description into something computable, we can say that our model for probability of getting a 1 is $l(D  \equiv 1 | \phi)= \phi$.  Symmetrically, the model of probability of getting a zero, is $l(D \equiv 0 | \phi)=(1-\phi)$This likelihood function is not something that comes out of the world. It does not come from the bayes rule. It is a new construct that we have created to link our suspected coin bias $\phi$ with the probability of getting an observation $D$. We have created a $model M$. We are using the model $M$ to approximate and describe the real coin. Note that there is no belief involved here, only assumptions; we assume that $model M$ is an approximation of a real coin.  Prior beliefWe can give our prior belief that the coin is fair, and our disbelief that it is unfair. We can do it by setting, for a possible values of $\{\phi\}$ = [0.1, 0.5, 0.9],  values of initial belief as $b(\{\phi\})$ = b([0.1, 0.5, 0.9]) = [0.1, 0.8, 0.1]  CalculationsPutting in some concrete numbers, let's say that we believe that $\phi=0.5$ (perfectly fair) and then we toss the coin, and get $D\equiv1$. What was the likelihood to get such a result? $l(D \equiv1 | \phi  \equiv 0.5) = 0.5$. At this point, we can end our lame discussion -- the coin is fair, the chance of getting a one was 50%, so everything is fine, right? That's correct. However, what if we do not fully believe that the coin is fair? what if we suspect that the coin is actually biased towards zero?Let's compute what was the chance of getting a 1, if the latent parameter $\phi$ was 0.1: $l(D  \equiv 1 | \phi  \equiv 0.1) = 0.1$.So, **If** the coin was heavily biased towards zero, then we still could get a 1, 10% of the time, and getting a result of "1" in one coin toss is not very surprising. There is seemingly nothing to worry about, except for that we did not make any progress on discovering the true value of $\phi$,  even tough we have made an indirect observation of it, through observing the $D$.All that we know so far is that the likelihood of observing the result, depends on the parameter in a model $M$ of the plant $P$ that generated that result. To say this, is to say nearly quite nothing.  No result?What we really care to know, is what to believe about $\{ \phi \}$ -- is the real value closer to 0.5 or to 0.1? So far we have nothing from reason to go by to believe either of these things. Or do we?Here is the first time when the bayesian update comes in. And, this is what the books on bayesian reasoning fail the hardest at. They start with discussing prior belief, posterior belief, and worry about how the prior affects the posterior, and why having a prior belief is or is not a good idea. And how to convince other people to your priors. They droll about informative or uninformative priors. Prior this, prior that. However, all this fails because the Bayesian update is truly useless unless you explain and understand the likelihood function first.Again: Be aware that the likelihood function DOES NOT COME FROM THE BAYES EQUATION. It comes from the model of the plant!Before we get to the bayesian belief update, please muster your patience, and take a minute and think about other possible models of likelihood for coin toss-- even if they are implausible. Or for any other model of other world phenomena that you care about. Another classic problem in this category is looking for disease in population, using an imperfect test. I am sure you have heard about it.  Why bother?In the previous section, we have created and used a "forward model" of the world. The reason we do it this way,is that the "forward models" that is, casual models (where the principle of action-reaction is held) of this kind are relatively easy to make for a very wide variety of real-world phenomena.   Example model with linear regression, and constructing its likelihood function.I bet that you are anxious to hear if bayesian reasoning can be used to any more than coin tosses, or figuring out how if taking a coronavirus test makes sense or not.Yes, it can. Here is an example on how would you treat a linear regression problem, of the kind:Proposition: Taller people are heavier. Question: What is the proportionality coefficient?Clarification: In Bayesian belief terms, what should we believe about various propositions for the value of proportionality coefficient? Model description$\mapsto$ model input: height $h$$\mapsto$ model output: weight $w$$\mapsto$ model parameters: proportionality coefficient $\phi_{prop}$, and the uncertainty descriptor $\phi_{\sigma}$. In other words, we have two latent parameters, not one.$\mapsto$ Plant's model of uncertainty: random variable $N(0,\sigma )$$\mapsto$ The complete model for predicting the weight for a person of height $h$ is $\begin{equation}w = h * \phi_{prop} + N(0,\phi_{\sigma} )\end{equation}$$\mapsto$ nose-free version of the model -- that is, if we set noise to zero, we get weight a model $w = h * \phi_{prop}$   Likelihood function for linear regressionHere is the critical bit. We construct the equation for the likelihood function of data point $D_{weight}$ happening, given the pair of ($\phi_{prop}$, $\phi_{\sigma}$). Let's say that the likelihood, or probability of this data point happening is inversely proportional to the distance of the data point value from the model's predicted noiseless value. Hence we want something like$ l( D_{weight} | (\phi_{p}, \phi_{\sigma})) \ \ \propto \ \  1/distance$Here are some proposals on how this could look like.Distance, is simply the difference between the model's predicted, noiseless $w$ and the observed $D_{w}$. For clarity of notation, let's use symbol delta defined as $\Delta = D_{w} - h * \phi_{prop}$One possible distance metric is :$_{proposed}distance = abs ( \Delta ) $We can also use a square of difference, which has the nice property that it weighs large distance more. (It also has a couple of other nice properties):$_{proposed}distance = \Delta^{2} $We also want to weight the distance by the amount of uncertainty.* If the model certainty is high ($\sigma$ is small), then the "improbability" due to distance will be higher.* If the model certainty is low ($\sigma$ is large), then the probability is higher even at a distance.We account for this by "weighing" the distance by $\sigma$:$_{proposed}weightedDistance =  (\frac{\Delta}{\phi_{std}})^{2}$ One more property that we need, before we can get to the likelihood, is that the likelihood for any prospective parameter must evaluate to between 0..1. For that, we need to get a bit creative . . . or "inspired". Here's a proposed likelihood function that has all the properties required above, sourced from  https://en.wikipedia.org/wiki/Bayesian_linear_regressionModel_setup$ l( D_{weight} | (\phi_{prop}, \phi_{\sigma})) = \sigma * \exp(-\frac{1}{2\sigma^{2}} \Delta^{2}) $This function has this nice property that the maximum value is 1.0, quickly decays towards zero as the $\Delta$ increases, and works nice with $\Delta$ weighted by $\sigma$.This specific function also has the property that it integrates to 1.00 for $\Delta$ of $(-\infty , +\infty )$ -- but we really do not require this property to be there in the likelihood function. As long as you choose a function that outputs something between 0 and 1, it's fine. To see that this is the case, recall that we are talking about the probability of data given the model parameter. There can be more than one more model parameter that will often (or always) produce a given datapoint value. That's fine. We will make the probabilities in the likelihood compatible with beliefs using a normalizing term - "Evidence".Again, let me be clear that the above presented likelihood function, is merely an example function that happens to have desirable properties. It is by far not the only function that exhibits this properties! For example, you can have a process that does not exhibit gaussian uncertainty like $\propto \exp(-\frac{1}{x^{2}})$. That's fine for Bayesian reasoning!.Now, that you see that it is possible to build a likelihood function, suitable to describe the problem of "what should I believe..." in bayesian terms for a regression problem, you can begin to believe that it is possible to construct such a function for a very wide variety of problems!We are not done yet. There is one more hoop to clear before we can get to the posteriority: The Evidence function.  Evidence, $m()$The marginal probability $m(D)$ is also sometimes called "evidence".Why marginal? this word is not to be confused with things like "unimportant", "extreme", or "bad". The word "marginal" is just a code-word that came from the depths of history. One used to write the results of an experiment (when taking the data) in a table (on paper!). Then, came the time to analise the data. Since copying the numbers by hand is time consuming, one would use the margin of the page to squeeze in additional scribblings. Hence, these computed numbers are "marginal". Why "evidence"? I am not sure, but what I do understand is that again, the name-word, code-word "evidence" does not have anything to do with common-sense meaning of that word. It does not mean "certainty", "proof" , "confirmation", "verification", "display", "demonstration", e.t.c. Here we are only using this word as a name for a certain operation.As hinted above, the value of the marginal probability, $m(D)$ is calculated by summing (or integrating) the likelihood function of $D$ for all of the considered values of $\phi$, weighted by the prior belief in $\phi$. $$m(D) = \sum_{\alpha\in\{\phi\}}{l(D|\alpha)b(\alpha})$$What we get in effect, is a scaling, or normalization term that makes the units and scale of belief and likelihood match. In the original formula, the same thing is denoted as $p(D)$. Which is triple as confusing because (a) requires an integral or summation over all $\phi$, (b) it integrates to more than 1.0 and hence (c) it has none of the properties of the other $p()$'s. Uhh. (?!?!?). If you are confused, I do not blame you. Instead, I propose that it would make much more sense never to write $p(D)$ nor $p(B)$, but rather, teach what the marginal probability function really is, right away: that $m() = m(D,l(),\{\phi\}, b(\{\phi\}))$.  Sounds complicated? It's tedious to write, but it is not complicated. (That's why people `in the know` shorten it). Let's see an example: Here is an example how to perform this calculation. For our coin toss example, we have:$m(D \equiv 1, l(),\{\phi\}, b(\{\phi\}) ) = \ldots$$\ldots \sum_{\alpha\in\{\phi\}}{l(D \equiv 1|\alpha)b(\alpha}) = \ldots $$ \ldots l(D \equiv 1 | \phi \equiv 0.1)b(\phi \equiv 0.1) \ldots$$ \ldots + l(D \equiv 1 | \phi \equiv 0.5)b(\phi \equiv 0.5) \ldots$$ \ldots + l(D \equiv 1 | \phi \equiv 0.9)b(\phi \equiv 0.9) \ldots$$ = \ldots \\ $$ \ldots 0.1*0.1+0.5*0.8+0.9*0.1 = 0.01 + 0.4 + 0.09 \ldots \\ $$ \ldots  \large{= 0.5}$So, the marginal probability of the "1" happening, under current belief system, is 0.5.Kind of anticlimactic?Wait until you see what happens when the prior beliefs were different (unbalanced prior). For example, for $b(\{\phi\})$ = [0.8, 0.1, 0.1] we get $m(D \equiv 1, l(),\{\phi\}, b(\{\phi\}) )$ = 0.22 . In other words, if our prior belief was that the coin is biased towards zero, then the "marginal probability" (Evidence!) of getting $D \equiv 1$ is lower!And then, for $b(\{\phi\})$ = [0.1, 0.1, 0.8], $m(D \equiv 1, l(),\{\phi\}, b(\{\phi\}) )$ = 0.78For "uninformative" prior belief of $b(\{\phi\})$ = [0.33, 0.33, 0.33], $m(D \equiv 1, l(),\{\phi\}, b(\{\phi\}) )$ = 0.50 again. How come? If this surprises you, you are in a good company. The surprise comes from the historical fact that the "marginal probability function" has the word "probability" in it. It would seem that the probability of something happening depends on our belief about it???Alas, this is not the case. "Marginal Probability" or "evidence" is merely a scaling factor that we need to apply to the likelihood, in our full equation for update of the prior belief:$$b_{\phi, updated} \ = \ b(\phi|D) \leftarrow  b_{\phi, prior} \times \frac{l()}{m()} \ = \ b(\phi) \times \frac{l(D|\phi)}{  m(D,l(),\{\phi\}, b(\{\phi\})) } \ = \ b(\phi) \times \frac{l(D|\phi)}{  \sum_{\alpha \ \in \ \{\phi\}}{(l(D|\alpha)b(\alpha))} }$$Hence, it is much more enlightening to say that the "Bayes Factor" -- the ratio $\frac{l()}{m()}$ -- tells us how we should modify our prior belief, given the data point $D$. This factor can be less than one, or more than one. It could be close to zero if $D$ is unlikely, or it could be tending to infinity if our prior belief was very very low. The "Bayes Factor" is composed of the interplay between the likelihood function and our prior belief about all possible $\{ \phi \}$. Seeing it this way dispels any magic about it.   Finally, the Bayesian updatePhew.After all this introduction -- which is not really introduction, it is THE meat that should be taught in school in the first place -- we can get to the bayesian method for updating beliefs:$$b(\phi|D) \leftarrow b(\phi) \frac{l(D|\phi)}{  \sum_{\alpha \ \in \ \{\phi\}}{(l(D|\alpha)b(\alpha))} }$$See the next chapter for a demonstration of this method in action.<jupyter_code>#hide<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0927626286443452, 'fraction_numerical': 0.00854392632250548, 'mean_word_length': 4.154418072633686, 'pattern_counts': {'":': 1, '<': 8, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '335469', 'license': 'Apache-2.0', 'n_tokens_mistral': 9701, 'n_tokens_neox': 9175, 'n_words': 5806, 'path': '10-gr-extended-rant-version.ipynb', 'repo_name': 'jerzydziewierz/bayesian_reasoning_v0', 'size': '36049'}