<jupyter_start><jupyter_text>Task Structured Tabular Data: Dataset Link:Dataset can be found at " /data/structured_data/data.csv " in the respective challenge's repo. Description:Tabular data is usually given in csv format (comma-separated-value). CSV files can be read and manipulated using pandas and numpy library in python. Most common datatypes in structured data are 'numerical' and 'categorical' data. Data processing is required to handle missing values, inconsistent string formats, missing commas, categorical variables and other different kinds of data inadequacies that you will get to experience in this course.  Objective:How to process and manipulate basic structured data for machine learning (Check out helpful links section to get hints) Tasks:- Load the csv file (pandas.read_csv function)- Classify columns into two groups - numerical and categorical. Print column names for each group.- Print first 10 rows after handling missing values- One-Hot encode the categorical data- Standarize or normalize the numerical columns Ask yourself:- Why do we need feature encoding and scaling techniques?- What is ordinal data and should we one-hot encode ordinal data? Are any better ways to encode it?- What's the difference between normalization and standardization? Which technique is most suitable for this sample dataset?- Can you solve the level-up challenge: Complete all the above tasks without using scikit-learn library ? Helpful Links:- Nice introduction to handle missing values: https://analyticsindiamag.com/5-ways-handle-missing-values-machine-learning-datasets/- Scikit-learn documentation for one hot encoding: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html- Difference between normalization and standardization: https://medium.com/towards-artificial-intelligence/how-when-and-why-should-you-normalize-standardize-rescale-your-data-3f083def38ff<jupyter_code># Import the required libraries
# Use terminal commands like "pip install numpy" to install packages
import numpy as np
import pandas as pd
# import sklearn if and when required
!wget https://raw.githubusercontent.com/DeepConnectAI/challenge-week-1/master/data/structured_data/data.csv
df = pd.read_csv('data.csv')
df.head(10)
df.fillna(axis = 0, method = 'ffill')
numerical = ['Age','Salary','Price Category Of Purchase']
categorical = ['Country','Purchased']
c = df[categorical]
n = df[numerical]
df.isna()
df["Age"].fillna(0, inplace = True)
df["Salary"].fillna(0, inplace = True)  
df.isna()
from sklearn.preprocessing import OneHotEncoder 
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import LabelBinarizer
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder() 
  
df['Country']= le.fit_transform(df['Country']) 
df['Purchased']= le.fit_transform(df['Purchased']) 
df
normalized_df=(df-n.min())/(n.max()-n.min())
df[numerical] = normalized_df[numerical].values
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08796764408493428, 'fraction_numerical': 0.005055611729019211, 'mean_word_length': 5.760820045558087, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '122873', 'license': 'MIT', 'n_tokens_mistral': 830, 'n_tokens_neox': 789, 'n_words': 305, 'path': 'Week1/Week1_task_1_tabular.ipynb', 'repo_name': 'Kinshukg04/DeepConnectAI', 'size': '2967'}