<jupyter_start><jupyter_text>Unit 3 - A Data Science campaign with pandas and PCAThis unit covers:* Essential data wrangling with `pandas`;* Working with different data types;* Discerning categorical from numerical features;* Spotting and interpreting the PCA 'variance bug';* Working with a public data set.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Essential Python tools and concepts – `pandas`In this practical we will predominantly be working with the `pandas` library.<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Pandas SeriesA Series is a one-dimensional list of values. Note the `NaN` value - it means "Not a Number". Originally this represents an undefined numerical, like the result of a division by zero. It is often used to denote missing values.<jupyter_code>values =[1, 3, 5, np.nan, 6, 8,9] # a list of values
pd.Series(values)<jupyter_output><empty_output><jupyter_text>It can have an index associated with each value.<jupyter_code>s = pd.Series(values, index=['a', 'b', 'c', 'd', 'e', 'f','g'])
s<jupyter_output><empty_output><jupyter_text>TaskAdd a letter to the index. What happens? How do you need to change the values list to fix the error?  Pandas Data FramesA **data frame** is like a two-dimensional series.<jupyter_code>df = pd.DataFrame(np.random.randn(6,4)) # note the size: 6 rows, 4 columns
df<jupyter_output><empty_output><jupyter_text>You can pass it an index during construction:<jupyter_code>df = pd.DataFrame(np.random.randn(6,5), 
                  index=["{}".format(i) for i in range(6)]) # note the size: 6 rows, 5 columns
df<jupyter_output><empty_output><jupyter_text>Likewise, you can pass column names:<jupyter_code>df = pd.DataFrame(np.random.randn(6,5), 
                  index=["row {}".format(i) for i in range(6)],
                  columns=['col {}'.format(i) for i in range(5)]) 
df<jupyter_output><empty_output><jupyter_text>You can also construct it from a dictionary:<jupyter_code>columns = {'beep': np.random.randn(1000),
           'bop': np.random.randn(1000),
           'bup': np.random.randn(1000),
           'bap': np.random.randn(1000)}
df = pd.DataFrame(columns, index=["xxx{}".format(i) for i in range(1000)])
df<jupyter_output><empty_output><jupyter_text>Data frame addressingWe can now address parts of the data by their row and column names. Columns are addressed like this:<jupyter_code>df['bop']<jupyter_output><empty_output><jupyter_text>Note that the return value type is a Series, not like a Data Frame! This is because it's one-dimensional.Extract multiple columns at once by passing a list of column names. Does not need to be unique!<jupyter_code>df[['bop', 'bap', 'bap']]<jupyter_output><empty_output><jupyter_text>Here, the return value is a DataFrame because it's two-dimensional. Rows use the `.loc` attribute:<jupyter_code>df.loc[['xxx0', 'xxx525']]<jupyter_output><empty_output><jupyter_text>The `.iloc` attribute will allow you to specify the number of the row, instead of its label:<jupyter_code>df.iloc[619]<jupyter_output><empty_output><jupyter_text>It can also use numpy-style addressing for the value matrix:<jupyter_code>df.iloc[27,:3]
df.iloc[67:99,0:3]<jupyter_output><empty_output><jupyter_text>Advanced data framesPandas supports all kinds of data types. A data frame can hold all sorts of data types at the same time. This is the principal difference to a `numpy.array`, where all elements need to be of the same data type.<jupyter_code>df2 = pd.DataFrame({'A': 1.,
                    'B': pd.Timestamp('20130102'),
                    'C': pd.Series(6, index=list(range(8)), dtype='float32'),
                    'D': np.array([9] * 8, dtype='int64'),
                    'E': pd.Categorical(["test", "train", "test1", "train","test", "train1", "test", "train"]),
                    'F': 'ool'})
df2<jupyter_output><empty_output><jupyter_text>The `info()` function gives you an overview of the data types:<jupyter_code>df2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 8 entries, 0 to 7
Data columns (total 6 columns):
 #   Column  Non-Null Count  Dtype         
---  ------  --------------  -----         
 0   A       8 non-null      float64       
 1   B       8 non-null      datetime64[ns]
 2   C       8 non-null      float32       
 3   D       8 non-null      int64         
 4   E       8 non-null      category      
 5   F       8 non-null      object        
dtypes: category(1), datetime64[ns](1), float32(1), float64(1), int64(1), object(1)
memory usage: 564.0+ bytes<jupyter_text>This is all the `pandas` you'll need in this practical. Feel free to make yourself familiar with what else `pandas` has to offer. A good starting point is [10 minutes to pandas](https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html) in the official `pandas` documentation. Data campaign: Cars You now know (almost) all you need to know to start your first data science campaign. We will analyse the "cars" data set. It contains data on historic car models. You will be guided through the first steps, then it's up to you to apply PCA to explore the data. Let's load the data from the internet and make a data frame:<jupyter_code># use read_csv to read data from url 
# use set_index to set index 'model" 
# show dataframe.

cars = pd.read_csv('https://gist.githubusercontent.com/seankross/a412dfbd88b3db70b74b/raw/5f23f993cd87c283ce766e7ac6b329ee7cc2e1d1/mtcars.csv')
cars = cars.set_index('model')
cars<jupyter_output><empty_output><jupyter_text>Descriptive statistics Let's have a quick look at some descriptive statistics.<jupyter_code>cars.mean()
cars.var()<jupyter_output><empty_output><jupyter_text>TaskWhat do you notice about the variance? There's also a command that gives you a few common statistical descriptors, all in one data frame. Up to you to decide which you like better!<jupyter_code>cars.describe()
cars.drop(["vs","am"], axis=1, inplace=True)
cars
from sklearn.decomposition import PCA
pca = PCA()
pca.fit(cars)
cars_pca = pca.transform(cars)

ax = plt.gca()
ax.scatter(cars_pca[:,6], cars_pca[:,8])
ax.axis('scaled')
t = ax.set_xlabel('x')
t = ax.set_ylabel('y')
ax = plt.gca()
expl_var = pca.explained_variance_ratio_
ax.plot(expl_var, marker='x',c='red')
ax.set_ylim(0,1.)
ax.set_xticks([0,1,2,3,4,5,6,8])
ax.set_xticklabels(["PC{}".format(i) for i in range(8)])
ax = plt.gca()
covmat = pca.get_covariance()
img = ax.imshow(covmat, cmap='seismic', vmin=-10, vmax=10)
plt.colorbar(img, label='COVARIANCE')
# we have observed that feature of 2 is directly corelated with feaature 1. 
# from below graph you can observe feature is 3 also is directly corelated with feaature 1.
# feature 8 is directly corelated with feaature 3 and also same as with feature 3.
# feature 5 is directly corelated with feaature 2& feature 3.
from sklearn import preprocessing as pp
ss = pp.StandardScaler()
ss.fit(cars)
pca_norm = PCA()
pca_norm.fit(ss.transform(cars))
cars_pca_norm = pca_norm.transform(ss.transform(cars))
cars_pca_norm

# plotting scatter plot after Normalise the data to zero mean and unit variance
ax = plt.gca()
ax.scatter(cars_pca_norm[:,0], cars_pca_norm[:,1])
ax.axis('scaled')
t = ax.set_xlabel('x')
t = ax.set_ylabel('y')

ax.scatter(cars_pca[:,0], cars_pca[:,1],marker="x",c="red")# this is before normalization
ax.axis('scaled')
t = ax.set_xlabel('x')
t = ax.set_ylabel('y')

ax.legend(['o is NORMALIZED DATA','x is RAW DATA'])
ax = plt.gca()
expl_var = pca.explained_variance_ratio_
ax.plot(expl_var, marker='x',c='red')
ax.set_ylim(0,1.)
ax.set_xticks([0,1,2,3,4,5,6,8])
ax.set_xticklabels(["PC{}".format(i) for i in range(8)])

ax.plot(pca_norm.explained_variance_ratio_,marker='o',c='blue'),
ax.legend(['x is RAW DATA','0 is NORMALIZED DATA'])

# As we can observe PC0,PC1,PC2 has declined.

f,(ax1,ax2)=plt.subplots(1,2)
covmat = pca.get_covariance()
img = ax1.imshow(covmat, cmap='seismic', vmin=-10, vmax=10)
plt.colorbar(img, label='COVARIANCE')
ax1.set_title('RAW DATA')

img1 = ax2.imshow(pca_norm.get_covariance(), cmap='seismic', vmin=-5, vmax=5)
ax2.set_title('NORMALIZED DATA')

# we can clearly observe that after normalization, covariance is reduced.
# after normalization 
ax = plt.gca()
ax.scatter(cars_pca_norm[:,0], cars_pca_norm[:,1])
ax.axis('scaled')
t = ax.set_xlabel('x')
t = ax.set_ylabel('y')


cars_pca_norm[:,0:2]
from sklearn.cluster import KMeans 
import numpy as np
data=np.array(cars_pca_norm[:,0:2])
kk=KMeans(n_clusters=7).fit(data)
kk1=kk.predict(cars_pca_norm[:,0:2])
kk1
data[kk1==1]


import matplotlib.pyplot as plt
ax = plt.gca()
ax.scatter(data[kk1==1][:,0],data[kk1==1][:,1],c="red")
ax.scatter(data[kk1==0][:,0],data[kk1==0][:,1],c="blue")
ax.scatter(data[kk1==2][:,0],data[kk1==2][:,1],c="green")
ax.scatter(data[kk1==3][:,0],data[kk1==3][:,1],c="yellow")
ax.scatter(data[kk1==4][:,0],data[kk1==4][:,1],c="brown")
ax.scatter(data[kk1==5][:,0],data[kk1==5][:,1],c="black")
ax.axis('scaled')
t = ax.set_xlabel('x')
t = ax.set_ylabel('y')

cars[kk1==0]
# the common features in data set after clustering.<jupyter_output><empty_output><jupyter_text>Numerical vs. categorical features**Numerical features** express a quantitative relationship between an instance and a feature. For example, 'height' is a numerical feature of a human. **Categorical features** express whether an instance belongs in a certain category. 'Male', 'female' are two categories that apply to humans (alongside others).Most interesting data sets contain numerical **and** categorical features.For PCA, only numerical features are useful (most of the time). These are the features of the cars dataset:* mpg: Miles per (US) gallon* cyl: Number of cylinders* disp: Displacement (cubic inches)* hp: Gross horsepower* drat: Rear axle ratio* wt: Weight (1000 lbs)* qsec: 1/4 mile time* vs: V-engine (0) or straight engine (1)* am: Transmission (0 = automatic, 1 = manual)* gear: Number of forward gears* carb: Number of carburators **Task: remove categorical features** 1. Decide which features are categorical, which are numerical. Search the internet if you don't know what a certain feature means.  2. Delete categorical features from the data frame. Use the `.drop()` function. Documentation is available [online](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html) or with the built-in documentation accessed by typing `pd.DataFrame.drop?` in a cell and executing it.   Task: Analyse the cleaned data set using PCA1. Do a pca on the raw data.2. Produce a scatter plot of the PCA'd data.2. Produce a scree plot and analyse how much variance is captured in the first 2 components.3. Plot the covariance matrix of the dataset. What stands out? 3. Look at the components of the dataset. Which features are highly represented in the first two components? Task: Normalise and observe the effect (the *variance bug*)1. Normalise the data to zero mean and unit variance and repeat the steps above. 5. How does the scatter plot of the first two PCs compare to the PCA on the raw data before normalisation? 6. What's the difference in the scree plot?7. How is the covariance matrix different?7. How do components differ? 8. *(Advanced)* Spot a cluster in the plotted data, find the corresponding data points, figure out what they have in common.  Coronavirus epidemic dynamicsHere's a task for advanced students. As you all are aware, last spring we saw the outbreak of Covid-19, aka coronavirus. Here, we're going to analyse a dataset from the beginning of the outbreak, when it was just about to spread around the world.  Task: Explore data on kaggle.com1. Go to the website on the kaggle site: https://www.kaggle.com/sudalairajkumar/novel-corona-virus-2019-dataset 2. Inspect the data on the website and the various ways the site lets you explore it. Registration on the website is not necessary. Working with the Coronavirus outbreak data1. The coronavirus dataset is provided on canvas. Download and save it in the same folder as this notebook.2. Unzip the data.3. Verify that the folder in which this notebook resides should now contain a sub-folder named `novel-corona-virus-2019-dataset`First we load the main dataset into a DataFrame:<jupyter_code>df = pd.read_csv('/content/2019_nCoV_data.csv')
df<jupyter_output><empty_output><jupyter_text>If this fails then please check again whether the dataset folder is unzipped and resides in the same folder as this notebook. Check that the filename in the command matches the filename of the data set on disk.The `Sno` column contains the serial number and is identical with the aut-generated index column. let's set the Index to track the `Sno` column:<jupyter_code>df = df.set_index('Sno')
df 
# Numeric: confirmed, deaths, recovered
# continuous: date, last update
# categorical: province state/country<jupyter_output><empty_output><jupyter_text>Let's explore the data types.  Task: Which features are numeric? Which are continuous? Which are categorical? Initial explorationLet's plot the number of number of confirmed cases for the whole data set.<jupyter_code>df['Confirmed'].plot()<jupyter_output><empty_output><jupyter_text>TaskIs this the plot you expected? Why not? Solution: The above command naively plots the whole column, but ignores the structure of the data set, where each line applies to a different province/state, or even country.We need to filter by country! Let's look only at the Hubei province, the root of the outbreak:<jupyter_code>df.loc[df['Province/State'] == 'Hubei']<jupyter_output><empty_output><jupyter_text>That looks better! Let's plot those values:<jupyter_code>ax=df.loc[df['Province/State'] == 'Hubei'].plot()
ax.set_xlabel('DATES')
ax.set_ylabel('CASES')
array=[]
from datetime import datetime
for i in ["Date"]:
 array.append(datetime.strptime(i,'%m/%d/%Y %H:%M:%S'))
df['Date']=array
df

df1=df.loc[df['Province/State'] == 'Hubei']
import matplotlib.pyplot as plt
ax=plt.gca()
ax.figure.set_size_inches(15,8.0)
ax.plot(df1['Date'],df1['Confirmed'],c='blue')
ax.plot(df1['Date'],df1['Deaths'],c='red')
ax.plot(df1['Date'],df1['Recovered'],c='green')
ax.set_xlabel('DATES')
ax.set_ylabel('TOTAL CASES')
# ax.legend(["No. of cases])
# ax.legend(['x is RAW DATA','0 is NORMALIZED DATA'])<jupyter_output><empty_output><jupyter_text>Note how the `.plot()` function of the `DataFrame` object already gives us a plot of all numerical features, complete with a legend!The plot is still lackiong though; It needs for example:* Dates on the x-axis* proper labels on the y-axis Task (advanced)* Read the documentation of the `DataFrame.plot` command to learn how to make it plot the date on the x-axis.* use the `ax = gca()` method the get an axis object, and call the `set_ylabel()` method to set an appropriate y-label. Task (advanced)* Plot the data for all of China.* Aggregate the data for the rest of the world and plot it.<jupyter_code>df2=df.loc[df['Country'] == 'China']

import matplotlib.pyplot as plt
ax=plt.gca()
ax.figure.set_size_inches(15,8)
ax.plot(df2['Date'],df2['Confirmed'],c='blue')
ax.plot(df2['Date'],df2['Deaths'],c='red')
ax.plot(df2['Date'],df2['Recovered'],c='green')
ax.set_xlabel('DATES')
ax.set_ylabel('TOTAL CASES')

df3=df.loc[df['Country'] != 'China']

import matplotlib.pyplot as plt
ax=plt.gca()
ax.figure.set_size_inches(15,8)
ax.plot(df3['Date'],df3['Confirmed'],c='blue')
ax.plot(df3['Date'],df3['Deaths'],c='red')
ax.plot(df3['Date'],df3['Recovered'],c='green')
ax.set_xlabel('DATES')
ax.set_ylabel('TOTAL CASES')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1273950656840756, 'fraction_numerical': 0.02223646267222044, 'mean_word_length': 5.212579617834395, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 97, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '517713', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 5465, 'n_tokens_neox': 5185, 'n_words': 1692, 'path': '03_A_Data_Science_campaign_with_pandas_and_PCA.ipynb', 'repo_name': 'Haarrsshhiitt/datapractice1', 'size': '15605'}