<jupyter_start><jupyter_text>Homogeneity Metrics© 2018 <NAME>  1. Definitions Entropy***Entropy*** is a measure of ***uncertainty*** associated with a given ***distribution q(y)***.From Wikipedia:    ...is the average rate at which information is produced by a stochastic source of data.         ...when a low-probability event occurs, the event carries more "information" ("surprisal")...    $$H(q) = -\sum_{c=1}^{C}{q(y_c) \cdot log(q(y_c))}$$where: - ***q*** is the ***distribution*** (as in the distribution of red and green balls) - ***y*** are the ***labels*** (the respective colors of each ball) - ***C*** is the number of ***classes*** (as in ***red*** and ***green*** - 2 classes) - ***q(yc) represents the proportion of balls having the same color c*** Gini Impurity***Gini Impurity*** is a measure of ***heterogeneity*** associated with a given ***distribution q(y)***.$$G(q) = \sum_{c=1}^{C}{q(y_c) \cdot (1 - q(y_c))}$$From Wikipedia:    ...is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.<jupyter_code>from intuitiveml.Information import *
X, y = data(10)
myinfo = plotInfo(X, y)
vb = VBox(build_figure(myinfo), layout={'align_items': 'center'})<jupyter_output><empty_output><jupyter_text>2. ExperimentThere are 10 balls (data points) of two possible colors (***classes***). Each ball has its own color (***label***), red or green.The slider control at the bottom allows you to change the number of red balls and, consequently, the number of green balls (the total stays the same) - so, you are changing the ***distribution***.This change will have an impact on both ***entropy*** and ***gini impurity*** measures.Use the slider to play with different configurations and answer the ***questions*** below.<jupyter_code>vb<jupyter_output><empty_output><jupyter_text>Questions:1. How to maximize (minimize) Entropy?2. How to maximize (minimize) Gini Impurity?3. What's the entropy when all balls have the same color?4. What kind of distribution yields the maximum Entropy?5. Using the formula, compute the ***entropy*** if you had 3 red balls6. Using the formula, compute the ***gini impurity*** if you had 7 red balls 3.) Zero4.) What kind of distribution yields the maximum Entropy? Uniform  This material is copyright <NAME> and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)).  Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT)).<jupyter_code>from IPython.display import HTML
HTML('''<script>
  function code_toggle() {
    if (code_shown){
      $('div.input').hide('500');
      $('#toggleButton').val('Show Code')
    } else {
      $('div.input').show('500');
      $('#toggleButton').val('Hide Code')
    }
    code_shown = !code_shown
  }

  $( document ).ready(function(){
    code_shown=false;
    $('div.input').hide()
  });
</script>
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>''')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15125516364791866, 'fraction_numerical': 0.010168414362885288, 'mean_word_length': 4.996190476190476, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '6593', 'license': 'MIT', 'n_tokens_mistral': 938, 'n_tokens_neox': 890, 'n_words': 354, 'path': '7. Homogeneity Metrics.ipynb', 'repo_name': 'csoehnel/DSR-FundamentalsOfML', 'size': '3171'}