<jupyter_start><jupyter_text>Entropy for Data ScienceNotes from [Introduction to Entropy for Data Science](https://www.youtube.com/watch?v=IPkRVpXtbdY).* Concept borrowed from physics.* Always a calculation of a vector of categorical variables.* Formula: $\text{Entropy} (E(\overrightarrow{d}))$ $E(\overrightarrow{d}) = -\sum\limits_{i=1}^{k} p_i \log_2(p_i)$ * Sum across the probability of some class occuring. Eg if it is gender and there are 60/100 male recorded, the probability would be (60/100). ``-( (60/100) * log2(60/100) + (40/100) * log2(40/100) ) = 0.9709505944546686`` * So, a series where it's very hard to predict what you will get if you took an object at random would be high, and if easy, low.* If it was mostly woman at some event, let's say 95% of the participants were woman, the entropy would be low:<jupyter_code>woman_prob = 95/100
man_prob = 5/100

-(woman_prob * log2(woman_prob) + man_prob * log2(man_prob))<jupyter_output><empty_output><jupyter_text>* Note: you take the negative to inverse the negative result returned from the log:<jupyter_code>log2(99/100)<jupyter_output><empty_output><jupyter_text>Whereas, if it was a 50/50 split, the entropy would be high: your predictions are uncertain.<jupyter_code>woman_prob = 50/100
man_prob = 50/100

-(woman_prob * log2(woman_prob) + man_prob * log2(man_prob))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11695906432748537, 'fraction_numerical': 0.06286549707602339, 'mean_word_length': 5.613526570048309, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '186630', 'license': 'MIT', 'n_tokens_mistral': 512, 'n_tokens_neox': 450, 'n_words': 151, 'path': 'notes/reference/talks/entropy-for-data-science.ipynb', 'repo_name': 'aav789/study-notes', 'size': '1368'}