<jupyter_start><jupyter_text>Unit 4 - Noisy data---1. [Histograms](section1)2. [Standard deviation](section2)3. [Boxplots](section3)<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>1. Histograms  Gym exampleTaken from: https://data36.com/plot-histogram-python-pandas/<jupyter_code>mu = 168 #mean
sigma = 5 #stddev
sample = 250
np.random.seed(0)
height_f = np.random.normal(mu, sigma, sample).astype(int)
mu = 176 #mean
sigma = 6 #stddev
sample = 250
np.random.seed(1)
height_m = np.random.normal(mu, sigma, sample).astype(int)
gym = pd.DataFrame({'height_f': height_f, 'height_m': height_m})
gym.head(7)<jupyter_output><empty_output><jupyter_text>To begin with - why a histogram?This is what happens if we just plot lines:<jupyter_code>gym.plot()<jupyter_output><empty_output><jupyter_text>Ummm.. it's very messy.   Different histograms:<jupyter_code>gym.hist('height_f')
gym.hist(bins=5)
gym.plot.hist(bins=20)
gym.plot.hist(bins=20, alpha=0.7)<jupyter_output><empty_output><jupyter_text>--- Back to our vaccinations data:<jupyter_code>url = 'https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.csv'
vacc_df = pd.read_csv(url) 
vacc_df.head(2)<jupyter_output><empty_output><jupyter_text>Look at data for different countries Use `groupby()` to group according to location  Attempt 1 - what is wrong?<jupyter_code>vacc_df.groupby('location')[['total_vaccinations','daily_vaccinations','people_fully_vaccinated','people_fully_vaccinated_per_hundred']].max()<jupyter_output><empty_output><jupyter_text>Attempt 2  Is this better?<jupyter_code>vacc_df.fillna('0').groupby('location')[['total_vaccinations','daily_vaccinations','people_fully_vaccinated','people_fully_vaccinated_per_hundred']].max()<jupyter_output><empty_output><jupyter_text>Attempt 3 - change the 0 from string to int, and finally, it works :-)<jupyter_code>vacc_df.fillna(0).groupby('location')[['total_vaccinations','daily_vaccinations','people_fully_vaccinated','people_fully_vaccinated_per_hundred']].max()<jupyter_output><empty_output><jupyter_text>The `world` row shouldn't be there. Remove it using `.drop()` and `.index() :<jupyter_code>vacc_df.drop(vacc_df.loc[vacc_df.location == 'World'].index, inplace = True)<jupyter_output><empty_output><jupyter_text>--- Your turn:>What do you think `.index` does? Why is it there? How can you find out?--- Before we continue, just assign this long row to a new dataframe, will be easier<jupyter_code>grouped_df = vacc_df.fillna(0).groupby('location')[['total_vaccinations','daily_vaccinations','people_fully_vaccinated','people_fully_vaccinated_per_hundred']].max().copy()
grouped_df.tail()<jupyter_output><empty_output><jupyter_text>sort the values using 'sort_values()`<jupyter_code>grouped_df.sort_values('people_fully_vaccinated_per_hundred', ascending = False).head(10)<jupyter_output><empty_output><jupyter_text>Histogram according to all values:<jupyter_code>grouped_df.hist(bins=50)<jupyter_output><empty_output><jupyter_text>Histogram according to people_fully_vaccinated_per_hundred<jupyter_code>grouped_df.hist('people_fully_vaccinated_per_hundred',bins=50)<jupyter_output><empty_output><jupyter_text>Remove rows with 0'sNote that this is different than changing values to 0's<jupyter_code>grouped_df.drop(grouped_df[grouped_df.people_fully_vaccinated_per_hundred == 0.0].index, inplace=True)
#grouped_df
grouped_df.hist('people_fully_vaccinated_per_hundred',bins=50)<jupyter_output><empty_output><jupyter_text>---> Your turn:>>Do the same, but for another column--- ---> Functions covered in this unit:>> `std()` - standard deviation>> `.hist(data, num_bins)` - the data and the number of bins >> `.plot()` - simple line plot>> `.hist(data, num_bins, alpha)` - the data, the number of bins and the transparency (default is 10 bins, all data and not transparent)>> `.plot.hist()` - histograms on same chart>> `.drop()` - removes unwanted rows or columns>> `.index()` - index of the selected rows>> `sort_values()` - self explantory.. it just sorts values>---  2. Standard deviation An small example showing that the more the data is spread, the higher the std:<jupyter_code>df = pd.DataFrame({'height' : [161, 156, 172], 
                   'weight': [67, 65, 89],
                   'age': [20,20,20]})
df
df.mean()
df.std()<jupyter_output><empty_output><jupyter_text>3. Boxplots<jupyter_code>np.random.seed(2345)
df = pd.DataFrame(np.random.randn(20,4),
                  columns=['C1', 'C2', 'C3', 'C4'])
df.head()
boxplot = df.boxplot(column=['C1', 'C2', 'C3'])
grouped_df.boxplot('people_fully_vaccinated_per_hundred')
grouped_df.sort_values('people_fully_vaccinated_per_hundred')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14331008243500318, 'fraction_numerical': 0.020080321285140562, 'mean_word_length': 7.588021778584392, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 87, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '569615', 'license': 'MIT', 'n_tokens_mistral': 1811, 'n_tokens_neox': 1650, 'n_words': 312, 'path': 'unit4.ipynb', 'repo_name': 'NirNagary/DataS', 'size': '4731'}