<jupyter_start><jupyter_text>1) Getting the data ready Three of the main steps you'll often have to take are:1. splitting the data into features (usually X) and labels (usually y)2. Filling data (also called imputing) or disregarding missing values - (Cleaning up the data)3. Converting non-numerical values to numerical values (also call feature encoding) - (Cleaning up the data)  We will learn how to put it all together laterfor now we will dive into separate sections  and focus on them individually<jupyter_code>heart_df = pd.read_csv("data/heart-disease.csv")
heart_df.head()
# axis = 1 --> column axis, axis = 0 -> row axis

# X --> every single column except target (data or features)
X = heart_df.drop("target",axis=1)
X.head()
# y --> target column(labels)
y = heart_df["target"]
y.head()
# most imp principle: - In ML, never evaluate or test ur models on data that it has learned from 
# thats why, we split data.
X.shape,len(heart_df) # number of data (rows) in data frame<jupyter_output><empty_output><jupyter_text>Split data into training and test sets  after converting it to numberical and cleaning data<jupyter_code>from sklearn.model_selection import train_test_split

# it will return 4 different values
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)

# test_size=0.2 => means test data set will be 20% of overall data
# so, X.shape[0]*0.8 --> 242 data in train set, rest in test set
# same done for y
# data is in the form of matrices or numpy nd array in a data frame

X_train.shape, X_test.shape, y_train.shape, y_test.shape
# Clean data --> Transform (manipulate) data --> Reduce data

# Clean data --> remove and replace data (when data or labels is missing)
# remove empty row/column or fill it with average values

# Transform data --> convert information into numbers the computer can understand
# e.g. convert colors into numbers - transform data between 0 and 1

# Reduce data --> The more data we have, the more CPU or energy needed to run our computation, costs more money
# So, sometimes having more data is not necessarily good 
# This process of data reduction is known as dimensionality reduction or column reduction
# remove irrelevant data. Change it into a form that is useful for you.

# Remember(IMP): - Don't assume that all data will be perfect or needed.<jupyter_output><empty_output><jupyter_text>1.1) Make sure all the data is numerical - conversion<jupyter_code>car_df = pd.read_csv("data/car-sales-extended.csv")
car_df.head()
len(car_df) # number of data (rows) in data frame
car_df.dtypes # check out their data types - for int and string columns
# use the first 4 columns (X) to predict prices (y) 
# --> although we need more information in real life

# build a ml model that will train on the training data, and predict on the test data

# But a ml model cannot deal with strings, only numerical data
# Split into X & y and train/test set
X = car_df.drop("Price", axis=1)
y = car_df["Price"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
car_df["Doors"].value_counts()
# Door is also based on categories
# Turn the categories (Make and Colour) into numbers, as well as Door
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer 

categorical_features = ["Make","Colour","Doors"]
one_hot = OneHotEncoder()
transformer = ColumnTransformer([
    ("one_hot",one_hot,categorical_features)],
    remainder="passthrough")

transformed_X = transformer.fit_transform(X)
transformed_X

# remainder="passthrough" --> ignore all other columns other than the ones mentioned above
X.head() # Data in Training set of X
pd.DataFrame(transformed_X) # all data is now numerical
# The categorical categories are now either 1 or 0
# Another way to transform data 
# usign pandas and pd.get_dummies()

# Have to convert doors to object for dummies to work on it
car_df["Doors"] = car_df["Doors"].astype(object)
dummies = pd.get_dummies(car_df[["Make","Colour","Doors"]])
dummies # u can then assign dummies to transformed X
X["Make"].value_counts()
# Finally, refit the model
# by refitting the dataset
X_train, X_test, y_train, y_test = train_test_split(transformed_X,y,test_size=0.2)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
# -----------------------------------------------------------<jupyter_output><empty_output><jupyter_text>Feature Scaling Once your data is all in numerical format, making sure all of your numerical data is on the same scale.For example, say you were trying to predict the sale price of cars and the number of kilometres on their odometers varies from 6,000 to 345,000 but the median previous repair cost varies from 100 to 1,700. A machine learning algorithm may have trouble finding patterns in these wide-ranging variables.To fix this, there are two main types of feature scaling.* Normalization (also called min-max scaling) - This rescales all the numerical values to between 0 and 1, with the lowest value being close to 0 and the highest previous value being close to 1. Scikit-Learn provides functionality for this in the [ MinMaxScalar class](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html)* Standardization - This subtracts the mean value from all of the features (so the resulting features have 0 mean). It then  scales the features to unit variance (by dividing the feature by the standard deviation). Scikit-Learn provides functionality for this in the [StandardScalar class](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) A couple of things to note.* Feature scaling usually isn't required for your target variable.* Feature scaling is usually not required with tree-based models (e.g. Random Forest) since they can handle varying features. Extra reading* [Feature Scaling with Scikit-Learn](https://benalexkeen.com/feature-scaling-with-scikit-learn/) by <NAME>* [Feature Scaling for Machine Learning: Understanding the Difference Between Normalization vs. Standardization](https://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/) by <NAME><jupyter_code>transformed_pd = pd.DataFrame(transformed_X) # transformed_X contains numerical data for training set of X
transformed_pd<jupyter_output><empty_output><jupyter_text>Min-Max Scaler The MinMaxScaler is the probably the most famous scaling algorithmIf the distribution is not Gaussian or the standard deviation is very small, the min-max scaler works better.However, it is sensitive to outliers  Methods to know* fit(X,y) - Compute the minimum and maximum to be used for later scaling.* fit_transform(X,y) - Fit to data, then transform it.* transform(X) - Scale features of X according to feature_range<jupyter_code>from sklearn import preprocessing

scaler = preprocessing.MinMaxScaler()
scaled_df = scaler.fit_transform(transformed_pd)
scaled_df = pd.DataFrame(scaled_df)
scaled_df
# -----------------------------------------------------------------------------------------------<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09998599635905335, 'fraction_numerical': 0.008402184567987676, 'mean_word_length': 4.510802469135802, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 36, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '178571', 'license': 'MIT', 'n_tokens_mistral': 2064, 'n_tokens_neox': 1980, 'n_words': 851, 'path': 'Scikit-Learn/1) Get Data Ready/Sklearn_GetDataReady1_SplittingData_and_NumericalConversion.ipynb', 'repo_name': 'Jawwad-Fida/Jupyter-Notebooks', 'size': '7160'}