<jupyter_start><jupyter_text>Exercise 10.02: Analyse the categorical variables from the Ames Housing datasetIn this exercise, we will continue our dataset exploration by analysing the categorical variables of this dataset. To do so, we will implement our own describe functions.The dataset used for this exercise is the Ames Housing dataset compiled by <NAME>: http://www.amstat.org/publications/jse/v19n3/decock.pdf 1. Open on a new Colab notebook and import the pandas package<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>2. Assign the link to the AMES dataset to a variable called 'file_url':<jupyter_code>file_url = 'https://raw.githubusercontent.com/TrainingByPackt/The-Data-Science-Workshop/master/Chapter10/dataset/ames_iowa_housing.csv'<jupyter_output><empty_output><jupyter_text>3. Using the read_csv method from the package pandas, load the dataset into a new variable called 'df':<jupyter_code>df = pd.read_csv(file_url)<jupyter_output><empty_output><jupyter_text>4. Create a new dataframe called 'obj_df' with only the columns that are of numerical types using the method select_dtypes from pandas packages and pass in the value 'number' to the parameter 'include:<jupyter_code>obj_df = df.select_dtypes(include='object')<jupyter_output><empty_output><jupyter_text>5. Using the attribute **columns** from pandas, extract the list of columns of this dataframe 'obj_df' and assign it to a new variable called 'num_cols' and print its content: Expected Output:![Figure 27 - List of categorical variables](https://docs.google.com/uc?export=download&id=1n5gkiNhA0bPeZFUTdrXEssgqJtqwuBW0)<jupyter_code>obj_cols = obj_df.columns
obj_cols<jupyter_output><empty_output><jupyter_text>6. Create a function called 'describe_object' that takes a **pandas** dataframe and a column name as input parameters. Then inside the function, print out the name of the given column, its number of unique values using the method **nunique()** and the list of values and their occurence using the method **value_counts()**:<jupyter_code>def describe_object(df, col_name):
  print(f"\nCOLUMN: {col_name}")
  print(f"{df[col_name].nunique()} different values")
  print(f"List of values:")
  print(df[col_name].value_counts(dropna=False, normalize=True))<jupyter_output><empty_output><jupyter_text>7. Test this function by providing the 'df' dataframe and the column 'MSZoning': Expected Output:![Figure 28 - Display of the created function for the column ‘MSZoning’](https://docs.google.com/uc?export=download&id=1PE8Ive1mpHZeeC0GN5wDrjxvkaufT5Pv)<jupyter_code>describe_object(df, 'MSZoning')<jupyter_output>COLUMN: MSZoning
5 different values
List of values:
RL         0.788356
RM         0.149315
FV         0.044521
RH         0.010959
C (all)    0.006849
Name: MSZoning, dtype: float64<jupyter_text>For the column ‘MSZoning’, the value ‘RL’ represents almost 79% of the values while ‘C (all) is only present less than 1% of the rows. 8. Create a for loop that will call the created function for every element from the list 'obj_cols': Expected Output:![Figure 29 - Display of the created function for the first columns contained in ‘obj_cols’](https://docs.google.com/uc?export=download&id=1cou8u_aD4Z8ec1EqDKRPFXJGGyEQUzh_)<jupyter_code>for col_name in obj_cols:
  describe_object(df, col_name)<jupyter_output>COLUMN: MSZoning
5 different values
List of values:
RL         0.788356
RM         0.149315
FV         0.044521
RH         0.010959
C (all)    0.006849
Name: MSZoning, dtype: float64

COLUMN: Street
2 different values
List of values:
Pave    0.99589
Grvl    0.00411
Name: Street, dtype: float64

COLUMN: Alley
2 different values
List of values:
NaN     0.937671
Grvl    0.034247
Pave    0.028082
Name: Alley, dtype: float64

COLUMN: LotShape
4 different values
List of values:
Reg    0.633562
IR1    0.331507
IR2    0.028082
IR3    0.006849
Name: LotShape, dtype: float64

COLUMN: LandContour
4 different values
List of values:
Lvl    0.897945
Bnk    0.043151
HLS    0.034247
Low    0.024658
Name: LandContour, dtype: float64

COLUMN: Utilities
2 different values
List of values:
AllPub    0.999315
NoSeWa    0.000685
Name: Utilities, dtype: float64

COLUMN: LotConfig
5 different values
List of values:
Inside     0.720548
Corner     0.180137
CulDSac    0.064384
FR2        0.032192
FR3        0.00[...]
{'fraction_non_alphanumeric': 0.08350634371395617, 'fraction_numerical': 0.06274509803921569, 'mean_word_length': 4.980689655172414, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '499263', 'license': 'MIT', 'n_tokens_mistral': 1601, 'n_tokens_neox': 1452, 'n_words': 468, 'path': 'Chapter10/chapter10_exercise2.ipynb', 'repo_name': 'TrainingByPackt/Data-Science-in-Python-the-Simple-Way', 'size': '4341'}