<commit_before><!DOCTYPE html>
<html>
  <head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <title>Probability Review</title>
  <meta name="description" content="Lecture notes for Stanford cs228.">


  <link rel="stylesheet" href="/cs228-notes/css/tufte.css">	
  

  <!-- Google Fonts loaded here depending on setting in _data/options.yml true loads font, blank does not-->
  
    <link href='//fonts.googleapis.com/css?family=Lato:400,400italic' rel='stylesheet' type='text/css'>
  
  <!-- Load up MathJax script if needed ... specify in /_data/options.yml file-->
  
    <script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
  

  <script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-75587219-1', 'auto');
  ga('send', 'pageview');

  </script>

  <link rel="canonical" href="http://localhost:4000/cs228-notes/preliminaries/probabilityreview/">
  <link rel="alternate" type="application/rss+xml" title="Probabilistic graphical modeling course" href="http://localhost:4000/cs228-notes/feed.xml" />
</head>

  <body>
    <!--- Header and nav template site-wide -->
<header>
    <nav class="group">
        <a href="/cs228-notes/">Contents</a>
	<a href="http://cs.stanford.edu/~ermon/cs228/index.html">Class</a>
	<a href="http://github.com/ermongroup/cs228-notes">Github</a>
	</nav>
</header>

    <article class="group">
      <h1>Probability review</h1>
<p class="subtitle"></p>


<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    TeX: {
      Macros: {
        e: "\\epsilon",
        xti: "x^{(i)}",
        yti: "y^{(i)}",
        bfy: "{\\bf y}",
        bfx: "{\\bf x}",
        bfg: "{\\bf g}",
        bfbeta: "{\\bf \\beta}",
        tp: "\\tilde p",
        pt: "p_\\theta",
        Exp: "{\\mathbb{E}}",
        Ind: "{\\mathbb{I}}",
        KL: "{\\mathbb{KL}}",
        Dc: "{\\mathcal{D}}",
        Tc: "{\\mathcal{T}}",
        Xc: "{\\mathcal{X}}",
        note: ["\\textcolor{blue}{[NOTE: #1]}",1]
      }
    }
  });
</script>


<p>We will go through a review of probability concepts over here, all of review the material have been adapted from <a href="http://cs229.stanford.edu/section/cs229-prob.pdf">CS229 Probability Notes</a>.</p>

<h1 id="1-elements-of-probability">1. Elements of probability</h1>
<p>In order to define a probability on a set we need a few basic elements,</p>

<p><strong>Sample space</strong> Ω: The set of all the outcomes of a random experiment. Here, each outcome ω ∈ Ω can be thought of as a complete description of the state of the real world at the end of the experiment.</p>

<p><strong>Set of events (or event space) F</strong>: A set whose elements A ∈ F (called events) are subsets of Ω (i.e., A ⊆ Ω is a collection of possible outcomes of an experiment)</p>

<p><strong>Probability measure</strong>: A function <script type="math/tex">P : F \rightarrow I\!R</script> that satisfies the following <strong>properties</strong></p>
<ul>
  <li><script type="math/tex">P(A) \geq 0</script>, for all <script type="math/tex">A \in F</script></li>
  <li>
    <script type="math/tex; mode=display">P(\Omega) = 1</script>
  </li>
  <li>If <script type="math/tex">A_1</script>, <script type="math/tex">A_2</script>, . . . are disjoint events <script type="math/tex">(i.e., A_i ∩ A_j = ∅  \text{ whenever }  i \neq j)</script>, then <script type="math/tex">P(∪_iA_i) = \sum_i P(A_i)</script></li>
</ul>

<p>These three <strong>properties</strong> are called the <strong>Axioms of Probability</strong>.</p>

<p><strong>Example</strong>: Consider the event of tossing a six-sided die. The sample space is Ω = {1, 2, 3, 4, 5, 6}. We can define different event spaces on this sample space. For example, the simplest event space is the trivial event space F = {∅, Ω}. Another event space is the set of all subsets of Ω. For the first event space, the unique probability measure satisfying the requirements above is given by P(∅) = 0, P(Ω) = 1. For the second event space, one valid probability measure is to assign the probability of each set in the event space to be <script type="math/tex">\frac{i}{6}</script> where <script type="math/tex">i</script> is the number of elements of that set; for example, <script type="math/tex">P({1, 2, 3, 4}) = \frac{4}{6}</script> and <script type="math/tex">P({1, 2, 3}) = \frac{3}{6}</script>.</p>

<h3 id="properties"><strong>Properties</strong>:</h3>
<ul>
  <li>If A ⊆ B ⇒ P(A) ≤ P(B).</li>
  <li><script type="math/tex">P(A \cap B) \leq min(P(A), P(B))</script>.</li>
  <li><strong>Union Bound</strong> P(A ∪ B) ≤ P(A) + P(B).</li>
  <li>P(Ω \ A) = 1 − P(A).</li>
  <li><strong>Law of Total Probability</strong> If <script type="math/tex">A_1, . . . , A_k</script> are a set of disjoint events such that <script type="math/tex">\bigcup^k_{i=1} A_i = \Omega</script> then <script type="math/tex">\sum^k_{i=1} P(A_k) = 1.</script></li>
</ul><commit_msg>formatting issues 9<commit_after><!DOCTYPE html>
<html>
  <head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <title>Probability Review</title>
  <meta name="description" content="Lecture notes for Stanford cs228.">


  <link rel="stylesheet" href="/cs228-notes/css/tufte.css">	
  

  <!-- Google Fonts loaded here depending on setting in _data/options.yml true loads font, blank does not-->
  
    <link href='//fonts.googleapis.com/css?family=Lato:400,400italic' rel='stylesheet' type='text/css'>
  
  <!-- Load up MathJax script if needed ... specify in /_data/options.yml file-->
  
    <script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
  

  <script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-75587219-1', 'auto');
  ga('send', 'pageview');

  </script>

  <link rel="canonical" href="http://localhost:4000/cs228-notes/preliminaries/probabilityreview/">
  <link rel="alternate" type="application/rss+xml" title="Probabilistic graphical modeling course" href="http://localhost:4000/cs228-notes/feed.xml" />
</head>

  <body>
    <!--- Header and nav template site-wide -->
<header>
    <nav class="group">
        <a href="/cs228-notes/">Contents</a>
	<a href="http://cs.stanford.edu/~ermon/cs228/index.html">Class</a>
	<a href="http://github.com/ermongroup/cs228-notes">Github</a>
	</nav>
</header>

    <article class="group">
      <h1>Probability review</h1>
<p class="subtitle"></p>


<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    TeX: {
      Macros: {
        e: "\\epsilon",
        xti: "x^{(i)}",
        yti: "y^{(i)}",
        bfy: "{\\bf y}",
        bfx: "{\\bf x}",
        bfg: "{\\bf g}",
        bfbeta: "{\\bf \\beta}",
        tp: "\\tilde p",
        pt: "p_\\theta",
        Exp: "{\\mathbb{E}}",
        Ind: "{\\mathbb{I}}",
        KL: "{\\mathbb{KL}}",
        Dc: "{\\mathcal{D}}",
        Tc: "{\\mathcal{T}}",
        Xc: "{\\mathcal{X}}",
        note: ["\\textcolor{blue}{[NOTE: #1]}",1]
      }
    }
  });
</script>


<p>We will go through a review of probability concepts over here, all of review the material have been adapted from <a href="http://cs229.stanford.edu/section/cs229-prob.pdf">CS229 Probability Notes</a>.</p>

<h1 id="1-elements-of-probability">1. Elements of probability</h1>
<p>In order to define a probability on a set we need a few basic elements,</p>

<p><strong>Sample space</strong> Ω: The set of all the outcomes of a random experiment. Here, each outcome ω ∈ Ω can be thought of as a complete description of the state of the real world at the end of the experiment.</p>

<p><strong>Set of events (or event space) F</strong>: A set whose elements A ∈ F (called events) are subsets of Ω (i.e., A ⊆ Ω is a collection of possible outcomes of an experiment)</p>

<p><strong>Probability measure</strong>: A function <script type="math/tex">P : F \rightarrow I\!R</script> that satisfies the following <strong>properties</strong></p>
<ul>
  <li><script type="math/tex">P(A) \geq 0</script>, for all <script type="math/tex">A \in F</script></li>
  <li>If <script type="math/tex">A_1</script>, <script type="math/tex">A_2</script>, . . . are disjoint events <script type="math/tex">(i.e., A_i ∩ A_j = ∅  \text{ whenever }  i \neq j)</script>, then <script type="math/tex">P(∪_iA_i) = \sum_i P(A_i)</script></li>
  <li>
    <script type="math/tex; mode=display">P(\Omega) = 1</script>
  </li>
</ul>

<p>These three <strong>properties</strong> are called the <strong>Axioms of Probability</strong>.</p>

<p><strong>Example</strong>: Consider the event of tossing a six-sided die. The sample space is Ω = {1, 2, 3, 4, 5, 6}. We can define different event spaces on this sample space. For example, the simplest event space is the trivial event space F = {∅, Ω}. Another event space is the set of all subsets of Ω. For the first event space, the unique probability measure satisfying the requirements above is given by P(∅) = 0, P(Ω) = 1. For the second event space, one valid probability measure is to assign the probability of each set in the event space to be <script type="math/tex">\frac{i}{6}</script> where <script type="math/tex">i</script> is the number of elements of that set; for example, <script type="math/tex">P({1, 2, 3, 4}) = \frac{4}{6}</script> and <script type="math/tex">P({1, 2, 3}) = \frac{3}{6}</script>.</p>

<h3 id="properties"><strong>Properties</strong>:</h3>
<ul>
  <li>If A ⊆ B ⇒ P(A) ≤ P(B).</li>
  <li><script type="math/tex">P(A \cap B) \leq min(P(A), P(B))</script>.</li>
  <li><strong>Union Bound</strong> P(A ∪ B) ≤ P(A) + P(B).</li>
  <li>P(Ω \ A) = 1 − P(A).</li>
  <li><strong>Law of Total Probability</strong> If <script type="math/tex">A_1, . . . , A_k</script> are a set of disjoint events such that <script type="math/tex">\bigcup^k_{i=1} A_i = \Omega</script> then <script type="math/tex">\sum^k_{i=1} P(A_k) = 1.</script></li>
</ul>
{'fraction_non_alphanumeric': 0.19977447848148844, 'fraction_numerical': 0.018323623379064086, 'mean_word_length': 3.822383325781604, 'pattern_counts': {'":': 0, '<': 265, '<?xml version=': 0, '>': 265, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5518658', 'n_tokens_mistral': 4058, 'n_tokens_neox': 3798, 'n_words': 1025}