<jupyter_start><jupyter_text>Importsnumpy import to handle arrayspandas to handle dataset as dataframe in pandas and all other columns and row level modifications of dataseaborn for plotting graphs of high levelmatplotlib is basic graphs also to support seaborn<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Reading the DatasetHere we are using iris dataset which is a famous dataset from Fisher R.A botanist The dataset consits of fifty instances , four features and three classes<jupyter_code>dataset = pd.read_csv('..//data//iris_dataset//iris.csv')
x_labels = ['sepal length', 'sepal width', 'petal length', 'petal width']
y_labels = ['iris']<jupyter_output><empty_output><jupyter_text>Understanding the data Checking for null'sAs part of machine learing , in the section of pre processing of our data we start with checking if there are any null's in our data.This step is required because while passing the data to the machine learning model, the model would not be able to process null's and would raise an exception.`what to do with these nulls then..?``` Depends on your data preposseing skills`` You can either ...* if catorgical data is missing , you can remove those rows if they are not a large number of rows compared to whole data* if its numericl data,you can replace it by mean of the column.* if its a time series data you can interpolate base on the previous timestamps data and next timestamps data> Note : In the Iris dataset there are no nulls so the else part of the code is not written<jupyter_code>if dataset.isnull().values.any():
    print('There are null in the dataset')
else:
    print('There are no nulls in the dataset') # In case nulls are there much more preprocessing is required by replacing nulls with appropriate values`<jupyter_output>There are no nulls in the dataset<jupyter_text>Knowing the columns and datatypesHere we are exploring the dataset to know what are numerical columns and what are catogorical columns. Knowing the columns would give us a better understanding of the data. Knowing the count of the data would tell us if all the columns have equal count or one  column is longer than other> Note : In iris dataset all the columns are uniform and no columns is catogorical so no pre processing have been done. But in case of catogorical columns there is One hot encoding of catogorical columns required<jupyter_code>dataset.info() # To Know the columns in the dataset and types of values and number of values<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 150 entries, 0 to 149
Data columns (total 5 columns):
sepal length    150 non-null float64
sepal width     150 non-null float64
petal length    150 non-null float64
petal width     150 non-null float64
iris            150 non-null object
dtypes: float64(4), object(1)
memory usage: 5.9+ KB<jupyter_text>Checking the scale of the data `dataset.describe()` would tell us min ,max,count standard diviation ,varience of each column idividually. This insight is required to know if feature scaling is required or not.All columns are sappose to under similar scale to be comaprible with each other.For example .... an age column would be of scale 0-100 where as salary would be around 1000-100000 so in the process of training the salary factor would have 1000 times more effect than age column on models learning.so we need to get down the scale of the all the columns to a uniform level and this process is called `feature scaleng`General Procedures of Feature Scaling * Normalization* Standardization``Note:``> * Here the process of feature scaling is not done since the sckit learn internally does feature scaling> * There is no need for feature scaleing because there is no significant difference between min,max,mean of columns> * There is not hard and fast rule that feature scaling need to be done. If its not done training the modle would be slower comparatively.<jupyter_code>print(dataset.describe()) # To know min,max,count standard diviation ,varience in each column which would tell us if there is any outliers,normalization or standadization required.<jupyter_output>sepal length  sepal width  petal length  petal width
count    150.000000   150.000000    150.000000   150.000000
mean       5.843333     3.054000      3.758667     1.198667
std        0.828066     0.433594      1.764420     0.763161
min        4.300000     2.000000      1.000000     0.100000
25%        5.100000     2.800000      1.600000     0.300000
50%        5.800000     3.000000      4.350000     1.300000
75%        6.400000     3.300000      5.100000     1.800000
max        7.900000     4.400000      6.900000     2.500000<jupyter_text>Viewing the dataThis just to see the data to get an overview.`dataset.head()` would give the first five columns of the data<jupyter_code>print(dataset.head()) # To view first five columns of the dataset<jupyter_output>sepal length  sepal width  petal length  petal width         iris
0           5.1          3.5           1.4          0.2  Iris-setosa
1           4.9          3.0           1.4          0.2  Iris-setosa
2           4.7          3.2           1.3          0.2  Iris-setosa
3           4.6          3.1           1.5          0.2  Iris-setosa
4           5.0          3.6           1.4          0.2  Iris-setosa<jupyter_text>Outlier check with the help of histogram`Outlier`: stands for abnormally having a value which is way beyound the usual domain of values Why do we need to take care of outliers...?`Ans`: Some machine learning models are very much sensitive to these outliers.Sometimes in dataset instead of null they can also be replaced with some number as `99999` these numbers can appear as outliers which need to be taken care of Here we are drawing histogram so that if a bar appears way far from the the data we can identify visually.`dataset.hist()`: This function would draw histograms for all numerical columns.> Note: In iris dataset there are no outlier found so outlier processing have not been done.<jupyter_code>dataset.hist(column = x_labels,bins=20, figsize=(10,5))<jupyter_output><empty_output><jupyter_text>Visualizing the datasetIn this we are drawing pair plots to check relation of one feature with every other feature, to ge a overview of how each feature is realated with every other feature<jupyter_code>sns.pairplot(dataset, hue=y_labels[0], size=2.5,markers=["o", "s", "D"])<jupyter_output><empty_output><jupyter_text>Extracting dependent and Independent variblesWe are extracting the dependent varibles and Independent varibles. Based on the values of dependent varibles the model would try to predict the appropriate Independent varible.`y.ravel()` in line 3 is just to convert 2-D vector into 1-D vector. In further code framework model would be expecting a 1-D vector<jupyter_code>X = dataset[x_labels].values
y = dataset[y_labels].values
y = y.ravel() # we would be requiring 1-D array for processing in further code<jupyter_output><empty_output><jupyter_text>Test-Train SplitWe are Spliting the dataset into two sets ,One is train set on which the model would be trained and another is test set on which the model would try to make predictions.Based on the knowledge acquired on train set the model would try to make well educated predictions on test set data.`test_size`: this attribute of the function would tell in what ratio the dataset need to be split.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05422396856581532, 'fraction_numerical': 0.04702030124426981, 'mean_word_length': 3.8390367553865654, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '473256', 'license': 'MIT', 'n_tokens_mistral': 2306, 'n_tokens_neox': 2048, 'n_words': 1023, 'path': 'src/multinomial_logistic_regerssion_using_scikit_learn.ipynb', 'repo_name': 'phanimadhusudhanthontepu/logistic_regression', 'size': '7635'}