<commit_before>The goal of probabilistic inference is to find model parameters that may explain
data you have observed. TFP performs probabilistic inference by evaluating the
model parameters using a `joint_log_prob` function. The idea behind the
`joint_log_prob` is simple: it is a function that takes in all of the values
that specify your model and data, and it returns the likelihood that the
parameterized model generated the data. TFP is built around defining this joint
log probability and then identifying the distribution of model parameters that
may have generated the data.

All `joint_log_prob` functions have a common structure:

1. The function takes a set of **inputs** to evaluate. Each input is either an
observed value or a random variate. (A random variate is a possible sample from
one of the model's distributions.) The purpose of the function is to evaluate
how likely the inputs are with respect to the model defined in the function’s
code.

1. The **model** used to evaluate the inputs is defined with a set of
distributions. These distributions measure the likelihood of the input values.
(By convention, the distribution that measures the likelihood of the variable
`foo` will be named `rv_foo` to note that it is a random variable.) We use two
types of distributions in `joint_log_prob` functions:

 a. **Prior distributions** measure the likelihood of input values individually.
 Prior distributions never depend on input values, and are used to measure the
 likelihood of a single value. Each unknown variable—one that has not been
 observed directly—needs a corresponding prior. Beliefs about which values could
 be reasonable determine the prior distribution. Choosing a prior can be tricky,
 so we will cover it in depth in Chapter 6.

 b. **Conditional distributions** measure the likelihood of an input value given
 other input values. The typical case is measuring the likelihood of observed
 data given the current guess of a parameter in the model. The `joint_log_prob`
 computes conditional distributions from input values. The conditional
 distributions can then return the likelihood of another input value, often the
 observed data.

1. Finally, we calculate and return the **joint log probability** of the inputs.
The joint log probability is the sum of the log probabilities from all of the
prior and conditional distributions. (We take the sum of log probabilities
instead of multiplying the probabilities directly for reasons of numerical
stability: floating point numbers in computers cannot represent the very small
values necessary to calculate the joint log probability unless they are in the
log domain.) The sum is actually an unnormalized density; although the total
joint probability over all possible inputs might not sum to one, it is
proportional to the true probability density. This proportional distribution is
sufficient to estimate the distribution of likely inputs.

Let's look at an example:

```python
def joint_log_prob(total_count, num_heads, heads_prob):
  rv_heads_prob = tfd.Uniform(low=0., high=1.)
  rv_num_heads = tfd.Binomial(total_count=total_count, probs=heads_prob)
  return (rv_heads_prob.log_prob(heads_prob) +
          rv_num_heads.log_prob(num_heads))
```

We can map these terms onto the code above. In this example, the input values
are the observed values in `num_heads` and `total_count` and the unknown value
for `heads_prob`. The `joint_log_prob` takes the current guess for `heads_prob`
and answers, what is the likelihood of the data, `num_heads` and `total_count`,
if `heads_prob` is the probability of a coin landing on heads. The answer
depends on two distributions. The prior distribution, `rv_heads_prob`, indicates
how likely the current value of `heads_prob` is by itself. The conditional
distribution, `rv_num_heads`, indicates the likelihood of `num_heads` if
`heads_prob` were the  probability for the Binomial distribution. The sum of
these log probabilities is the joint log probability.

The `joint_log_prob` is particularly useful in conjunction with the `tfp.mcmc`
package. MCMC algorithms proceed by making educated guesses about the unknown
test values (we’ll talk about how it makes those guesses in Chapter 3) and
computing what the likelihood of this set of arguments is. By repeating this
many times, MCMC builds a distribution of likely parameters. Constructing this
distribution is the goal of probabilistic inference.<commit_msg>update text and example for joint_log_prob description<commit_after>
The goal of probabilistic inference is to find a probability distribution
consistent with one or more observed values. TensorFlow Probability (TFP)
offers different tools for probabilistic inference, which are all based on a
user-defined Python callable. This callable returns a single
`Tensor` representing the log of the joint probability of a given set of
concretized values of some model. The concretized values must themselves also
be `Tensor`s and are specified by the callable's function signature.

All `joint_log_prob` functions have the following structure:

1. The function takes a set of **inputs** representing the values of the random
   variables in the model. These inputs must be convertible to `Tensor`s.
   Typically the function would support computing batches of inputs, i.e., a
   vector of inputs. (For more information on batches see [Tensorflow
   Distributions](https://arxiv.org/abs/1711.10604]), section 3.3.)

2. Internally the `joint_log_prob` function uses probability distributions to
   **measure** the plausibility of the given set of values. By convention, we
   recommend that the distribution which measures the value `foo` be named
   `rv_foo`. Distributions may or may not be composed from values associated
   with other random variables; we use the following names to distinguish these
   two cases:

   a. A **prior distribution** is an unconditional measure of the likelihood of
      an input value. It is not parameterized by other input values.
      Mathematically, a prior probability is written as `p(X=x)`.

   b. A **conditional distribution** measures the likelihood of an input value
      but is parameterized by other input values. Mathematically, a conditional
      probability is written as `p(Y=y|X=x)`.

3. The function returns the **total log probability** of the inputs. The total
   log probability is the sum of the log probabilities as measured by the
   model's constituent prior and conditional distributions. As a convention, we
   measure model values using log probabilities rather than "straight"
   probabilities. Since computers represent continuous numbers in log domain,
   computing log probabilities is often more numerically stable.

Consider the following example.

```python
def joint_log_prob(heads, coin_bias):
    """Joint log probability of coin flips under a non-informative prior.

    Args:
      heads: `Tensor` of coin flips; a `0` or `1`.
      coin_bias: `Tensor` of possible coin biases.

    Returns:
      joint_log_prob: The joint log probability measure of given coin flips and
        coin bias.
    """
    rv_coin_bias = tfp.distributions.Uniform(low=0., high=1.)  # Prior
    rv_heads = tfp.distributions.Bernoulli(probs=coin_bias)    # Conditional
    return (rv_coin_bias.log_prob(coin_bias) +
           tf.reduce_sum(rv_heads.log_prob(heads)))
```

In this example, the input values are observed outcomes from a coin flipping
experiment and the distribution for heads and tails for that coin—also known as
coin bias. . The `joint_log_prob` takes the provided values of `heads` and
`coin_bias` and computes the overall log probability of these values.

1. The prior distribution, `rv_coin_bias`, measures the plausibility of the
   provided `coin_bias`.

2. The conditional distribution, `rv_heads`, measures the plausibility of
 `heads`, assuming `coin_bias` is the probability of heads.

The resulting sum of the log of these probabilities is the joint log
probability.

The `joint_log_prob` is particularly useful in concert with the
[`tfp.mcmc`](https://www.tensorflow.org/probability/api_docs/python/tfp/mcmc)
and [`tfp.vi`](https://www.tensorflow.org/probability/api_docs/python/tfp/vi)
modules. Markov chain Monte Carlo (MCMC) algorithms are useful for sampling from
implicitly normalized probability measures, i.e., they need only an unnormalized
log-probability function to draw samples from the corresponding normalized log-
probability.

Although we generically refer to this callable as the `joint_log_prob`, it is
not presumed normalized. In other words, it doesn't necessarily integrate to
`1`. This relaxation is useful, for example, when using MCMC to generate samples
from the posterior distribution, i.e., `p(x|y) = p(y|x)p(x) / p(y)` where `p(y)
= integral{ p(y|x)p(x) : x in X }`. If `p(y)` is intractable (and it often is),
the [`tfp.mcmc`](
https://www.tensorflow.org/probability/api_docs/python/tfp/mcmc) module can
generate samples from the normalized posterior log-probability using only the
unnormalized posterior log-probability. For example, returning to our coin
flipping example, the unnormalized (`heads`) posterior log-probability is:

```python
unnormalized_posterior_log_prob = lambda coin_bias: joint_log_prob(heads,
                                                                   coin_bias)
```

Notice that this closure "pins" the `joint_log_prob` at some value of `heads`
and is only a function of `coin_bias`. If we had normalized this "pinned"
function, we'd obtain the normalized posterior log-probability. To normalize
this pinned function we could calculate the sum of all possible joint
probabilities over all possible combinations of `heads`.
{'fraction_non_alphanumeric': 0.055647155812036275, 'fraction_numerical': 0.0028854080791426216, 'mean_word_length': 4.56160458452722, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2201206', 'n_tokens_mistral': 2497, 'n_tokens_neox': 2394, 'n_words': 1303}