<jupyter_start><jupyter_text>EDA for the first time Literatúry- [Pandas documentation](https://pandas.pydata.org/docs/)- [Pandas tutorials](https://pandas.pydata.org/pandas-docs/version/0.15/tutorials.html)- https://github.com/FIIT-IAU- https://github.com/FIIT-IAU/IAU-2019-2020 Dnes sa budeme venovať tomu, ako...- načítať dáta- analyzovať jednotlivé atribúty- analyzovať vzťahy medzi atribútmi- vizualizovať dáta (vhodné typy vizualizácií, vlastnosti dobrých vizualizácií, ako neklamať vizualizáciou) Predtým, ako začneme analyzovať dáta, by sme si mali ujasniť...- Aké otázky máme analýzou zodpovedať- Akú úlohu máme riešiť  Country dataset - Reading from a file using Pandas<jupyter_code>import matplotlib
import matplotlib.pylab as pylab
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
import scipy.stats as stats
filename = "data/countries.csv"
df = pd.read_csv(filename)
df.head()<jupyter_output><empty_output><jupyter_text>**Reading the file again with the proper `decimal` for numbers for this file**<jupyter_code>df = pd.read_csv(filename, decimal=',')
df.head()
df.info()
df.Region.unique()<jupyter_output><empty_output><jupyter_text>**String cleaning - remove blank spaces**<jupyter_code>df.Region = df.Region.str.strip()
df.Country = df.Country.str.strip()
df.Region.unique()<jupyter_output><empty_output><jupyter_text>Missing value?**In how many row?**<jupyter_code>df.shape[0] - df.dropna().shape[0]
df.isnull().sum()
df.isnull().sum().sum()
df[df.isnull().any(axis=1)]<jupyter_output><empty_output><jupyter_text>Visualization<jupyter_code>df.info()
plt.rcParams["figure.figsize"] = (10,5)
sns.boxplot('Region', 
            'GDP ($ per capita)', 
            data=df)
pylab.xticks(rotation=90)
df.Region.value_counts().plot(kind='bar')
sns.pairplot(df.dropna()[['Pop. Density (per sq. mi.)', 
                          'GDP ($ per capita)', 
                          'Birthrate', 
                          'Net migration', 
                          'Literacy (%)', 
                          'Phones (per 1000)', 
                          'Deathrate']])
df['GDP ($ per capita)'].corr(df['Birthrate'])
sns.scatterplot('GDP ($ per capita)', 
                'Birthrate', 
                data=df)
df['Phones (per 1000)'].corr(df['Birthrate'])
sns.scatterplot('Phones (per 1000)', 
                'Birthrate', 
                data=df)<jupyter_output><empty_output><jupyter_text>Na tomto predmete sa budeme zaoberať len supervised ML (učenia s učiteľom)- Regresie  𝑌∈𝑅- Klasifikácie  𝑌∈{𝐶1,𝐶2,…,𝐶𝑁}**Snažíme sa nájsť funkciu $f$ atribútov $X$, ktorá bude predikovať hodnotu závislej premennej $Y$**  Obese dataset<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
import scipy.stats as stats
df = pd.read_csv("data/obese.csv")
df.info()<jupyter_output><empty_output><jupyter_text>I don't like one column's name. It is too long, I'm going to shorten it<jupyter_code>df.rename(columns={'Share of adults who are obese (%)' : 'obesity_rate'}, inplace=True)
df.Entity.unique()<jupyter_output><empty_output><jupyter_text>**Visualization**<jupyter_code>af = df[df.Entity == 'Africa']
sns.regplot(af.Year, af.obesity_rate)

usa = df[df.Entity == 'United States']
sns.regplot(usa.Year, usa.obesity_rate)

plt.legend(labels=['Afrika', 'United States'])<jupyter_output><empty_output><jupyter_text>**Simple Y=f(X)**<jupyter_code>slope, intercept, r_value, p_value, std_err = stats.linregress(af.Year, af.obesity_rate)
line = slope * af.Year + intercept
plt.plot(af.Year, af.obesity_rate, 'o', af.Year, line)

slope, intercept, r_value, p_value, std_err = stats.linregress(usa.Year, usa.obesity_rate)
line = slope * usa.Year + intercept
plt.plot(usa.Year, usa.obesity_rate, 'o', usa.Year, line)
x = 2300
y = slope * x + intercept
y<jupyter_output><empty_output><jupyter_text>Diamond dataset<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
import scipy.stats as stats

df = pd.read_csv('data/diamonds.csv')
df.describe()
df.color.value_counts()
df.color.value_counts().plot(kind='bar')
df.color.value_counts().plot(kind='pie')<jupyter_output><empty_output><jupyter_text>**Your code:**  Monitoring dataset<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
import scipy.stats as stats

filename = 'data/monitoring.csv'
df = pd.read_csv(filename)
df.describe()
%%bash
head -n 10 data/monitoring.csv
data = pd.read_csv(filename, 
                   sep='\t', 
                   header=None, 
                   na_values=[-999, -9999], 
                   index_col=0)
data.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13690224291155312, 'fraction_numerical': 0.009944985188319933, 'mean_word_length': 4.211686879823595, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '624104', 'license': 'Apache-2.0', 'n_tokens_mistral': 1872, 'n_tokens_neox': 1775, 'n_words': 361, 'path': 'cvicenia/notebooks/tyzden-03/IAU_uvod_EDA.ipynb', 'repo_name': 'xmajzel/2020-2021', 'size': '4726'}