<jupyter_start><jupyter_text>Exercise 2: Convert data type for Ames Housing datasetIn this exercise, we will practise how to prepare a dataset by converting its variables to different data types.The dataset used for this exercise is the Ames Housing dataset compiled by <NAME>: http://www.amstat.org/publications/jse/v19n3/decock.pdf 1. Open on a new Colab notebook and import the pandas package<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>2. Assign the link to the AMES dataset to a variable called 'file_url':<jupyter_code>file_url = 'https://raw.githubusercontent.com/TrainingByPackt/The-Data-Science-Workshop/master/Chapter10/dataset/ames_iowa_housing.csv'<jupyter_output><empty_output><jupyter_text>3. Using the read_csv method from the package pandas, load the dataset into a new variable called 'df':<jupyter_code>df = pd.read_csv(file_url)<jupyter_output><empty_output><jupyter_text>4. Print the data type of each column using the attributes 'dtypes':<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>From chapter 10, we know the columns 'Id', 'MSSubClass', 'OverallQual' and 'OverallCond' are wrongly classified as numerical variables. They have a finite number of unique values and we can't perform any mathematical operations on them. For example, it doesn't make sence to add, remove, multiply or divide 2 different values from the 'Id' column. So we are going to convert them into categorical variables. 5. Using the method astype(), convert the column 'Id' into a categorical variable:<jupyter_code>df['Id'] = df['Id'].astype('category')<jupyter_output><empty_output><jupyter_text>6. Convert the columns 'MSSubClass', 'OverallQual' and 'OverallCond' into categorical variables as in step 5:<jupyter_code>df['MSSubClass'] = df['MSSubClass'].astype('category')
df['OverallQual'] = df['OverallQual'].astype('category')
df['OverallCond'] = df['OverallCond'].astype('category')<jupyter_output><empty_output><jupyter_text>7. Create a for loop that will iterate through the 4 categorical columns ('Id', 'MSSubClass', 'OverallQual' and 'OverallCond') and print their names and categories using the attribute '.cat.categories':<jupyter_code>for col_name in ['Id', 'MSSubClass', 'OverallQual', 'OverallCond']:
  print(col_name)
  print(df[col_name].cat.categories)<jupyter_output>Id
Int64Index([   1,    2,    3,    4,    5,    6,    7,    8,    9,   10,
            ...
            1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459, 1460],
           dtype='int64', length=1460)
MSSubClass
Int64Index([20, 30, 40, 45, 50, 60, 70, 75, 80, 85, 90, 120, 160, 180, 190], dtype='int64')
OverallQual
Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype='int64')
OverallCond
Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')<jupyter_text>Great! Now these 4 columns are converted into categorical variables. From the output of step 4, we also saw a lot of variables of type 'object'. Let's have a look at them and see of they need to be converted as well. 8. Create a new dataframe called 'obj_df' that will only contain variables of type 'object' using the method 'select_dtypes' and the parameter include='object':<jupyter_code>obj_df = df.select_dtypes(include='object')<jupyter_output><empty_output><jupyter_text>9. Create a new variable called obj_cols containing the list of column names from the dataframe obj_df using the attributes '.columns' and display its content:<jupyter_code>obj_cols = obj_df.columns
obj_cols<jupyter_output><empty_output><jupyter_text>10. Similarly to step 7, create a for loop that will iterate through the columns names contained in 'obj_cols' and print their names and unique values using the method 'unique()':<jupyter_code>for col_name in obj_cols:
  print(col_name)
  print(df[col_name].unique())<jupyter_output>MSZoning
['RL' 'RM' 'C (all)' 'FV' 'RH']
Street
['Pave' 'Grvl']
Alley
[nan 'Grvl' 'Pave']
LotShape
['Reg' 'IR1' 'IR2' 'IR3']
LandContour
['Lvl' 'Bnk' 'Low' 'HLS']
Utilities
['AllPub' 'NoSeWa']
LotConfig
['Inside' 'FR2' 'Corner' 'CulDSac' 'FR3']
LandSlope
['Gtl' 'Mod' 'Sev']
Neighborhood
['CollgCr' 'Veenker' 'Crawfor' 'NoRidge' 'Mitchel' 'Somerst' 'NWAmes'
 'OldTown' 'BrkSide' 'Sawyer' 'NridgHt' 'NAmes' 'SawyerW' 'IDOTRR'
 'MeadowV' 'Edwards' 'Timber' 'Gilbert' 'StoneBr' 'ClearCr' 'NPkVill'
 'Blmngtn' 'BrDale' 'SWISU' 'Blueste']
Condition1
['Norm' 'Feedr' 'PosN' 'Artery' 'RRAe' 'RRNn' 'RRAn' 'PosA' 'RRNe']
Condition2
['Norm' 'Artery' 'RRNn' 'Feedr' 'PosN' 'PosA' 'RRAn' 'RRAe']
BldgType
['1Fam' '2fmCon' 'Duplex' 'TwnhsE' 'Twnhs']
HouseStyle
['2Story' '1Story' '1.5Fin' '1.5Unf' 'SFoyer' 'SLvl' '2.5Unf' '2.5Fin']
RoofStyle
['Gable' 'Hip' 'Gambrel' 'Mansard' 'Flat' 'Shed']
RoofMatl
['CompShg' 'WdShngl' 'Metal' 'WdShake' 'Membran' 'Tar&Grv' 'Roll'
 'ClyTile']
Exterior1st
['VinylSd' 'MetalSd'[...]<jupyter_text>We can see all these columns have a finite number of unique values composed of text so we are certain they are categorical variables. 11. Create a for loop that will iterate through the columns names contained in 'obj_cols' and convert each of them into a categorical variable using the method 'astype()':<jupyter_code>for col_name in obj_cols:
  df[col_name] = df[col_name].astype('category')<jupyter_output><empty_output><jupyter_text>12. Print the data type of each column using the attributes 'dtypes':<jupyter_code>df.dtypes<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12713754646840147, 'fraction_numerical': 0.03234200743494424, 'mean_word_length': 6.117724867724868, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '568107', 'license': 'MIT', 'n_tokens_mistral': 2052, 'n_tokens_neox': 1876, 'n_words': 599, 'path': 'Chapter11/chapter11_exercise2.ipynb', 'repo_name': 'TrainingByPackt/Data-Science-in-Python-the-Simple-Way', 'size': '5386'}