<jupyter_start><jupyter_text>Data CleaningIn this lesson we will learn the basics of Data Cleaning and the Exploratory Data Analysis Pipeline.  IntroductionThis is a **comprehensive EDA technique with python**.It is clear that everyone in this community is familiar with Meta Kaggle and kaggle survey 2018 datasets but if you need to review your information about the datasets please visit  [meta-kaggle](https://www.kaggle.com/kaggle/meta-kaggle) and [kaggle survey 2018](https://www.kaggle.com/kaggle/kaggle-survey-2018).  Loading PackagesIn this kernel we are using the following packages:   Now we import all of them<jupyter_code># Now import the libraries
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from wordcloud import WordCloud as wc
from nltk.corpus import stopwords
import matplotlib.pylab as pylab
import matplotlib.pyplot as plt
from pandas import get_dummies
import matplotlib as mpl
import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib
import warnings
import sklearn
import string
import scipy
import numpy
import nltk
import json
import sys
import csv
import os

import warnings
warnings.filterwarnings('ignore')
print('matplotlib: {}'.format(matplotlib.__version__))
print('sklearn: {}'.format(sklearn.__version__))
print('scipy: {}'.format(scipy.__version__))
print('seaborn: {}'.format(sns.__version__))
print('pandas: {}'.format(pd.__version__))
print('numpy: {}'.format(np.__version__))
print('Python: {}'.format(sys.version))
#print('wordcloud: {}'.format(wordcloud.version))<jupyter_output>matplotlib: 3.0.3
sklearn: 0.20.3
scipy: 1.2.1
seaborn: 0.9.0
pandas: 0.24.2
numpy: 1.16.2
Python: 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)]<jupyter_text>A few tiny adjustments for better **code readability**<jupyter_code>sns.set(style='white', context='notebook', palette='deep')
pylab.rcParams['figure.figsize'] = 12,8
warnings.filterwarnings('ignore')
mpl.style.use('ggplot')
sns.set_style('white')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>3- Exploratory Data Analysis (EDA)In this section, you'll learn how to use graphical and numerical techniques to begin uncovering the structure of your data.  * Which variables suggest interesting relationships?* Which observations are unusual?By the end of the section, you'll be able to answer these questions and more, while generating graphics that are both insightful and beautiful.  then We will review analytical and statistical operations:1. Data Collection1. Visualization1. Data Cleaning1. Data Preprocessing  Data Collection**Data collection** is the process of gathering and measuring data.I start Collection Data by the Users and Kernels datasets into **Pandas DataFrames**<jupyter_code># import kernels and users to play with it
users = pd.read_csv("data/kaggle_Users.csv")
kernels = pd.read_csv("data/kaggle_Kernels.csv")
messages = pd.read_csv("data/kaggle_ForumMessages.csv")
freeFormResponses=pd.read_csv("data/kaggle_freeFormResponses.csv")
multipleChoiceResponses=pd.read_csv("data/kaggle_multipleChoiceResponses.csv")<jupyter_output><empty_output><jupyter_text>**Note 1*** Each row is an observation (also known as : sample, example, instance, record)* Each column is a feature (also known as: Predictor, attribute, Independent Variable, input, regressor, Covariate)<jupyter_code>users.sample(1) 
kernels.sample(1) 
freeFormResponses.sample(1) 
multipleChoiceResponses.sample(1)<jupyter_output><empty_output><jupyter_text>Select a random userid from the dataset to use for the experiment.<jupyter_code>username="mjbahmani"
userid=int(users[users['UserName']=="mjbahmani"].Id)
userid<jupyter_output><empty_output><jupyter_text>FeaturesFeatures can be from following types:1. numeric1. categorical1. ordinal1. datetime1. coordinatesFind the type of features in **Meta Kaggle**?!for getting some information about the dataset you can use **info()** command<jupyter_code>print(users.info())
print(freeFormResponses.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 23860 entries, 0 to 23859
Data columns (total 35 columns):
Q11_OTHER_TEXT     545 non-null object
Q12_OTHER_TEXT     1765 non-null object
Q12_Part_1_TEXT    3928 non-null object
Q12_Part_2_TEXT    1448 non-null object
Q12_Part_3_TEXT    1134 non-null object
Q12_Part_4_TEXT    9553 non-null object
Q12_Part_5_TEXT    1353 non-null object
Q13_OTHER_TEXT     1193 non-null object
Q14_OTHER_TEXT     502 non-null object
Q15_OTHER_TEXT     588 non-null object
Q16_OTHER_TEXT     597 non-null object
Q17_OTHER_TEXT     114 non-null object
Q18_OTHER_TEXT     162 non-null object
Q19_OTHER_TEXT     451 non-null object
Q1_OTHER_TEXT      80 non-null object
Q20_OTHER_TEXT     94 non-null object
Q21_OTHER_TEXT     493 non-null object
Q22_OTHER_TEXT     95 non-null object
Q27_OTHER_TEXT     308 non-null object
Q28_OTHER_TEXT     187 non-null object
Q29_OTHER_TEXT     532 non-null object
Q30_OTHER_TEXT     203 non-null object
Q31_OTHER_TEXT     235 non-nu[...]<jupyter_text>Explore the Dataset1- Dimensions of the dataset.2- Peek at the data itself.3- Statistical summary of all attributes.4- Breakdown of the data by the class variable.Don’t worry, each look at the data is **one command**. These are useful commands that you can use again and again on future projects.<jupyter_code># shape
print(users.shape)
# shape
print(kernels.shape)
print(freeFormResponses.shape)
#columns*rows
users.size
#columns*rows
kernels.size<jupyter_output><empty_output><jupyter_text>We can get a quick idea of how many instances (rows) and how many attributes (columns) the data contains with the shape property.To grab information about the dataset you can use **info()** command<jupyter_code>print(users.info())
print(kernels.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 241956 entries, 0 to 241955
Data columns (total 16 columns):
Id                           241956 non-null int64
AuthorUserId                 241956 non-null int64
CurrentKernelVersionId       204413 non-null float64
ForkParentKernelVersionId    71347 non-null float64
ForumTopicId                 22255 non-null float64
FirstKernelVersionId         240318 non-null float64
CreationDate                 241764 non-null object
EvaluationDate               208703 non-null object
MadePublicDate               227063 non-null object
IsProjectLanguageTemplate    241956 non-null bool
CurrentUrlSlug               241954 non-null object
Medal                        10444 non-null float64
MedalAwardDate               10444 non-null object
TotalViews                   241956 non-null int64
TotalComments                241956 non-null int64
TotalVotes                   241956 non-null int64
dtypes: bool(1), float64(5), int64(5), object(5)
memory usage: [...]<jupyter_text>Check the number of unique items for Species:<jupyter_code>kernels['Medal'].unique()
kernels["Medal"].value_counts()<jupyter_output><empty_output><jupyter_text>Check the first 5 rows of the data set:<jupyter_code>kernels.head(5)<jupyter_output><empty_output><jupyter_text>Check the last 5 rows of the data set:<jupyter_code>users.tail()<jupyter_output><empty_output><jupyter_text>Check 5 random rows from the data set:<jupyter_code>kernels.sample(5)<jupyter_output><empty_output><jupyter_text>Statistical summary about the dataset:<jupyter_code>kernels.describe()<jupyter_output><empty_output><jupyter_text>Data CleaningWhen dealing with real-world data, dirty data is the norm rather than the exception. We continuously need to predict correct values, inspite of missing ones, and find links between various data artefacts such as schemas and records. We need to stop treating data cleaning as a piecemeal exercise (resolving different types of errors in isolation), and instead leverage all signals and resources (such as constraints, available statistics, and dictionaries) to accurately predict corrective actions.The primary goal of data cleaning is to detect and remove errors and **anomalies** to increase the value of data in analytics and decision making. While it has been the focus of many researchers for several years, individual problems have been addressed separately. These include missing value correction, outliers detection, transformations, integrity constraints violations detection and repair, consistent query answering, deduplication, and many other related problems such as profiling and constraints mining. Check how many nulls are on the dataset:<jupyter_code>#How many NA elements in every column
users.isnull().sum()
kernels.isnull().sum()
kernels.groupby('Medal').count()<jupyter_output><empty_output><jupyter_text>Print dataset **columns**<jupyter_code>kernels.columns
users.columns<jupyter_output><empty_output><jupyter_text>**Note**in Pandas you can perform queries like "where"  Find yourself in Users dataset<jupyter_code>users[users['Id']==userid]<jupyter_output><empty_output><jupyter_text>Find your kernels in Kernels dataset<jupyter_code>yourkernels=kernels[kernels['AuthorUserId']==userid]
yourkernels<jupyter_output><empty_output><jupyter_text>Data Preprocessing**Data preprocessing** refers to the transformations applied to our data before feeding it to the algorithm. Data Preprocessing is a technique that is used to convert the raw data into a clean data set. In other words, whenever the data is gathered from different sources it is collected in raw format which is not feasible for the analysis.there are plenty of steps for data preprocessing and we just listed some of them in general (not just for Quora) :* removing Target column (id)* Sampling (without replacement)* Making part of iris unbalanced and balancing (with undersampling and SMOTE)* Introducing missing values and treating them (replacing by average values)* Noise filtering* Data discretization* Normalization and standardization* PCA analysis* Feature selection (filter, embedded, wrapper) **Note**Preprocessing and generation pipelines depend on a model type. Visualization**Data visualization**  is the presentation of data in a graphical format. It enables decision makers to "see" analytics presented visually, so they can grasp difficult concepts or identify new patterns.With interactive visualization, you can take the concept a step further by using technology to drill down into charts and graphs for more detail, interactively changing what data you see and how it’s processed.In this section we will look at  **11 plots** with **matplotlib** and **seaborn**   Scatter plotScatter plot Purpose To identify the type of relationship (if any) between two quantitative variables<jupyter_code>yourkernels.columns
# Modify the graph above by assigning each species an individual color.
x=yourkernels["TotalVotes"]
y=yourkernels["TotalViews"]
plt.scatter(x, y)
plt.legend()
plt.show()

f,ax=plt.subplots(1,2,figsize=(18,8))
yourkernels['Medal'].value_counts().plot.bar(color=['#CD7F32','#FFDF00','#D3D3D3'],ax=ax[0])
ax[0].set_title('Number Of Medal')
ax[0].set_ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>BoxIn descriptive statistics, a **box plot** or boxplot is a method for graphically depicting groups of numerical data through their quartiles. Box plots may also have lines extending vertically from the boxes (whiskers) indicating variability outside the upper and lower quartiles, hence the terms box-and-whisker plot and box-and-whisker diagram.<jupyter_code>yourkernels["TotalVotes"].plot(kind='box')
plt.figure()
#This gives us a much clearer idea of the distribution of the input attributes:


yourkernels["TotalComments"].plot(kind='box')
plt.figure()

# To plot the species data using a box plot:

sns.boxplot(x="TotalComments", y="TotalVotes", data=yourkernels )
plt.show()
# Use Seaborn's striplot to add data points on top of the box plot 
# Insert jitter=True so that the data points remain scattered and not piled into a verticle line.
# Assign ax to each axis, so that each plot is ontop of the previous axis. 

ax= sns.boxplot(x="TotalViews", y="TotalVotes", data=yourkernels)
ax= sns.stripplot(x="TotalViews", y="TotalVotes", data=yourkernels, jitter=True, edgecolor="gray")
plt.show()
# Tweek the plot above to change fill and border color color using ax.artists.
# Assing ax.artists a variable name, and insert the box number into the corresponding brackets

ax= sns.boxplot(x="TotalViews", y="TotalVotes", data=yourkernels)
ax= sns.stripplot(x="TotalViews", y="TotalVotes", data=yourkernels, jitter=True, edgecolor="gray")

boxtwo = ax.artists[2]
boxtwo.set_facecolor('red')
boxtwo.set_edgecolor('black')
boxthree=ax.artists[1]
boxthree.set_facecolor('yellow')
boxthree.set_edgecolor('black')

plt.show()
sns.factorplot('TotalViews','TotalVotes',hue='Medal',data=yourkernels)
plt.show()
sns.factorplot('TotalComments','TotalVotes',hue='Medal',data=yourkernels)
plt.show()<jupyter_output><empty_output><jupyter_text>HistogramWe can also create a **histogram** of each input variable to get an idea of the distribution.<jupyter_code># histograms
yourkernels.hist(figsize=(15,20))
plt.figure()<jupyter_output><empty_output><jupyter_text>It looks like perhaps two of the input variables have a Gaussian distribution. This is useful to note as we can use algorithms that can exploit this assumption.<jupyter_code>yourkernels["TotalViews"].hist();
yourkernels["TotalComments"].hist();
sns.factorplot('TotalViews','TotalVotes',data=yourkernels)
plt.show()
sns.factorplot('TotalComments','TotalVotes',data=yourkernels)
plt.show()<jupyter_output><empty_output><jupyter_text>violinplots<jupyter_code># violinplots on petal-length for each species
sns.violinplot(data=yourkernels,x="TotalViews", y="TotalVotes")
# violinplots on petal-length for each species
sns.violinplot(data=yourkernels,x="TotalComments", y="TotalVotes")
sns.violinplot(data=yourkernels,x="Medal", y="TotalVotes")
sns.violinplot(data=yourkernels,x="Medal", y="TotalComments")<jupyter_output><empty_output><jupyter_text>how many NA elements in every column   kdeplot<jupyter_code># seaborn's kdeplot, plots univariate or bivariate density estimates.
#Size can be changed by tweeking the value used
sns.FacetGrid(yourkernels, hue="Medal", size=5).map(sns.kdeplot, "TotalComments").add_legend()
plt.show()
sns.FacetGrid(yourkernels, hue="Medal", size=5).map(sns.kdeplot, "TotalVotes").add_legend()
plt.show()
f,ax=plt.subplots(1,3,figsize=(20,8))
sns.distplot(yourkernels[yourkernels['Medal']==1].TotalVotes,ax=ax[0])
ax[0].set_title('TotalVotes in Medal 1')
sns.distplot(yourkernels[yourkernels['Medal']==2].TotalVotes,ax=ax[1])
ax[1].set_title('TotalVotes in Medal 2')
sns.distplot(yourkernels[yourkernels['Medal']==3].TotalVotes,ax=ax[2])
ax[2].set_title('TotalVotes in Medal 3')
plt.show()<jupyter_output><empty_output><jupyter_text>jointplot<jupyter_code># Use seaborn's jointplot to make a hexagonal bin plot
#Set desired size and ratio and choose a color.
sns.jointplot(x="TotalVotes", y="TotalViews", data=yourkernels, size=10,ratio=10, kind='hex',color='green')
plt.show()<jupyter_output><empty_output><jupyter_text>andrews_curves<jupyter_code># we will use seaborn jointplot shows bivariate scatterplots and univariate histograms with Kernel density 
# estimation in the same figure
sns.jointplot(x="TotalVotes", y="TotalViews", data=yourkernels, size=6, kind='kde', color='#800000', space=0)<jupyter_output><empty_output><jupyter_text>Heatmap<jupyter_code>plt.figure(figsize=(10,7)) 
sns.heatmap(yourkernels.corr(),annot=True,cmap='cubehelix_r') #draws  heatmap with input as the correlation matrix calculted by(iris.corr())
plt.show()<jupyter_output><empty_output><jupyter_text>WordCloudIt is possible that you have seen a cloud filled with lots of words in different sizes, which represent the frequency or the importance of each word. This is called Tag Cloud or WordCloud.<jupyter_code>import nltk
nltk.download('stopwords')
#nltk.data.LazyLoader('data/nltk_dataset/corpora/stopwords.zip')
#nltk.open('data/nltk_dataset/corpora/stopwords')

from wordcloud import WordCloud as wc
from nltk.corpus import stopwords
eng_stopwords = set(stopwords.words("english"))
messages.head(1)
def generate_wordcloud(text): 
    wordcloud = wc(relative_scaling = 1.0,stopwords = eng_stopwords).generate(text)
    fig,ax = plt.subplots(1,1,figsize=(10,10))
    ax.imshow(wordcloud, interpolation='bilinear')
    ax.axis("off")
    ax.margins(x=0, y=0)
    plt.show()
text=','.join(str(v) for v in messages['Message'])
#text =" ".join(messages['Message'])
generate_wordcloud(text)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09662125667646883, 'fraction_numerical': 0.025385584828662305, 'mean_word_length': 6.292778993435449, 'pattern_counts': {'":': 0, '<': 108, '<?xml version=': 0, '>': 108, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '292911', 'license': 'MIT', 'n_tokens_mistral': 5492, 'n_tokens_neox': 5085, 'n_words': 1573, 'path': 'notebooks/Python-in-2-days/D2_L1_Exploratory_Data_Analysis/02-Data_Cleaning.ipynb', 'repo_name': 'alexzzlin/ml_training', 'size': '16663'}