<jupyter_start><jupyter_text>**Mapúa University - School of Information Technology*****CS170_OL161_2Q2021 (Data Science Project)*****Group members:*** DELARIARTE, <NAME>.* SORIA, <NAME>. **Phases***   Data Gathering*   Cleansing Data*   Exploratory Data Analysis*   Modelling*   Evaluation  **Family Income and Expenditure***Data by <NAME> (Kaggle)***Context**The Philippine Statistics Authority (PSA) spearheads the conduct of the Family Income and Expenditure Survey (FIES) nationwide. The survey, which is undertaken every three (3) years, is aimed at providing data on family income and expenditure, including, among others, levels of consumption by item of expenditure, sources of income in cash, and related information affecting income and expenditure levels and patterns in the Philippines.**Content**Inside this data set is some selected variables from the latest Family Income and Expenditure Survey (FIES) in the Philippines. It contains more than 40k observations and 60 variables which is primarily comprised of the household income and expenditures of that specific household.**Acknowledgement**The Philippine Statistics Authority (PSA) for providing the publisher with their raw data.  **Data Gathering**Data collection is the process of gathering and measuring information on targeted variables in an established system, which then enables one to answer relevant questions and evaluate outcomes. Data collection is a research component in all study fields, including physical and social sciences, humanities, and business.I have chosen Filipino Family Income and Expenditure. It is readily available on Kaggle. This project aims to Predict Household Income through the Expenditures and Income Bracket Through Classification.The source of my Dataset on Kaggle: https://www.kaggle.com/grosvenpaul/family-income-and-expenditure/download<jupyter_code>#Importing Libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
#Import the Data Set
family_income_data = pd.read_csv(r"C:\\Users\\lance\\OneDrive\\Desktop\\Family Income and Expenditure.csv")
#Number of rows and cols in the dataset
family_income_data.shape
family_income_data.head(10)
family_income_data.tail(10)<jupyter_output><empty_output><jupyter_text>**Cleansing Data**With all the libraries and dataset already imported to the notebook. For the data cleansing segment, our data to be used was modified in order to have a better output format. Our dataset has an overall 41,544 rows and 60 columns which is a lot manually to predict. Huhuhuhu :(<jupyter_code>#Validating for any missing data
family_income_data.info()
#Show data info per columns
family_income_data.describe()
#Splitting of Data for testing
#First dependent data
X=family_income_data.iloc[:,1:]
X.head(5)<jupyter_output><empty_output><jupyter_text>**Exploratory Data Analysis**This portion will discuss about the General Problem Statement of our project.The general problem statement project would be to identify the correlations between certain fish specie to the different attributes of fishes and species type. Without notebook, having to access the raw dataset will consume a lot of time and effort. So we will extract the data needed in order to meet the answers needed for our following Data Science Questions below.<jupyter_code>expenditures = [column for column in family_income_data.columns if 'Expenditure' in column]
expenditures
X = family_income_data.loc[:, expenditures]
y = family_income_data['Total Household Income']
plt.figure(figsize=(20, 20))
i = 1
for exp in expenditures :
    plt.subplot(6,3,i)
    sns.regplot(x=X[exp], y=y)
    i += 1<jupyter_output><empty_output><jupyter_text>With respect to the selected label, a lot of the features show a huge variance. This will limit the reliability of the regression model later. We have Total Food Expenditure has a somehow high correlation value with Meat Expenditure and Vegetables Expenditure. Makes sense especially that Pinoys are a culture of rice and ulam. It's not surprising that these are related.<jupyter_code>plt.figure(figsize=(10,10))
plt.subplot(3,1,1)
sns.regplot(x=family_income_data['Total Rice Expenditure'], y = family_income_data['Bread and Cereals Expenditure'])
plt.subplot(3,1,2)
sns.regplot(x=family_income_data['Total Food Expenditure'], y = family_income_data['Vegetables Expenditure'])
plt.subplot(3,1,3)
sns.regplot(x=family_income_data['Total Food Expenditure'], y = family_income_data['Meat Expenditure'])<jupyter_output><empty_output><jupyter_text>**Modelling**A random forest is a meta estimator that fits a number of classifying decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is controlled with the max_samples parameter if bootstrap=True (default), otherwise the whole dataset is used to build each tree.On this part, we will import the train_test_split library to be used when testing our model. We will also use the data that was separated before on the Data Cleansing Phase which is variable X for dependent data and variable Y for independent data. The dependent data is the features or characteristics of fishes such as vertical length, diagonal length, cross length, height, width, and lastly weight. While variable Y will have the independent data which is the species. We will use two different model to find which is the best effective and suitable model.<jupyter_code>from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.linear_model import LinearRegression, Ridge, LogisticRegression, Lasso
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.neighbors import KNeighborsRegressor, KNeighborsClassifier
from sklearn.preprocessing import Normalizer, PolynomialFeatures, MinMaxScaler, StandardScaler
from sklearn.svm import SVR
from sklearn.decomposition import IncrementalPCA, SparsePCA, KernelPCA
from sklearn.manifold import TSNE
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error, precision_score,recall_score, classification_report, confusion_matrix<jupyter_output><empty_output><jupyter_text>**Model 1 (Random Forest Regressor)**The first model to be tested is Random Forest Regression which is good for classification problems, it is a predictive analysis algorithm that is based on the concept of probability. When we are applying the model and testing for its prediction value.The predicted result was 0.761153884159818 which is equivalent to **76.11%** and it is considered good. But we will still further test another model to see if there are better models that we can apply on our project with a higher and better prediction value.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=41)<jupyter_output><empty_output><jupyter_text>A random state value of 41 was used because this seemed to be the optimal value to get the best predictions.<jupyter_code>rfr = RandomForestRegressor(verbose=True, n_jobs=-1, n_estimators=1000)
rfr.fit(X_train, y_train)
rfr.score(X_test, y_test)<jupyter_output>[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done  42 tasks      | elapsed:    0.0s
[Parallel(n_jobs=4)]: Done 192 tasks      | elapsed:    0.2s
[Parallel(n_jobs=4)]: Done 442 tasks      | elapsed:    0.6s
[Parallel(n_jobs=4)]: Done 792 tasks      | elapsed:    1.0s
[Parallel(n_jobs=4)]: Done 1000 out of 1000 | elapsed:    1.3s finished<jupyter_text>The final prediction value is **76.11%** Let's visually observe how the predicted values and the actual values correlate in a 45-degree line. The closer the points are in the line, the more it tells us that the model we selected did a great job in terms of predicting values.We'll also repeat this method for the next models we use.<jupyter_code>y_rfr_predict = rfr.predict(X)
mean_squared_error(y, y_rfr_predict)
plt.figure(figsize=(20,5))
ax = sns.regplot(x=y, y = y_rfr_predict)
ax.set(xlabel='Total Household Income', ylabel='Predicted TIH')<jupyter_output>[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done  42 tasks      | elapsed:    0.2s
[Parallel(n_jobs=4)]: Done 192 tasks      | elapsed:    0.9s
[Parallel(n_jobs=4)]: Done 442 tasks      | elapsed:    2.1s
[Parallel(n_jobs=4)]: Done 792 tasks      | elapsed:    3.6s
[Parallel(n_jobs=4)]: Done 1000 out of 1000 | elapsed:    4.5s finished<jupyter_text>**Model 2 (K-Nearest Neighbors Regressor)**The second model to be tested is K-Nearest Neighbors Regressor which is "also" good for classification problems, it is a predictive analysis algorithm that is based on the concept of probability. When we are applying the model and testing for its prediction value.The predicted result was 0.70798304763341 which is equivalent to **70.79%** and it is considered good.<jupyter_code>knr = KNeighborsRegressor(n_neighbors=15, n_jobs=-1, leaf_size=50)
knr.fit(X_train, y_train)
knr.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>The final prediction value is **70.79%**<jupyter_code>y_knr_predict = knr.predict(X)
mean_squared_error(y, y_knr_predict)
plt.figure(figsize=(20,5))
ax = sns.regplot(x=y, y = y_knr_predict)
ax.set(xlabel='Total Household Income', ylabel='Predicted TIH')<jupyter_output><empty_output><jupyter_text>**Evaluation**This portion will evaluate and summarize all of the results that were garnered from doing the following phases above.Seeing this final product, the best classifier to use to predict the Income Category was the Random Forest Classifier. KNN predicted worse possibly because it's hard to define a nearest neighbor classification with the huge variance in our dataset.To summarize all our learnings from partaking this project, we will discuss about choosing a proper dataset, how to analyze and extract dataset, training and testing then applying different models, and lastly acquiring the output based on the results and histograms.- Choosing a proper dataset is one of the most important steps as having to use inappropritate data will result to a dreadful result. There are different ways to acquire a dataset and our group has settled to used ready-to-use dataset from Kaggle. - The next step was importing the dataset on Python followed up by the Data Cleansing process. Here is where the data was analyzed and re-organized for a better ease of use. The data to be used was also validated before being extracted and used.- The next step is the exploratory analysis where we have learned the different correlations between data values and then compared through histogram graphs. - After all the data has been analyzed, we've proceeded to the next step of Modelling, here is where we've applied training and testing of different models. We've learned to test different models to identify which is the best to be used on our project.<jupyter_code>#Student information
Name = input("Name: ")
Section = input("Section: ")
Name = input("Name: ")
Section = input("Section: ")
import datetime
import socket 
def get_Host_name_IP(): 
    try: 
        host_name = socket.gethostname() 
        host_ip = socket.gethostbyname(host_name) 
        print("Hostname:",host_name) 
        print("IP Address:",host_ip) 
    except: 
        print("No visible IP Address4") 
get_Host_name_IP()
now = datetime.datetime.now()
print ("Time Stamp:", now.strftime("%Y-%m-%d %H:%M:%S"))<jupyter_output>Hostname: DESKTOP-1DB9APD
IP Address: 192.168.56.1
Time Stamp: 2021-02-13 23:49:12
{'fraction_non_alphanumeric': 0.07056798623063683, 'fraction_numerical': 0.01910499139414802, 'mean_word_length': 5.682576193214491, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '39549', 'license': 'MIT', 'n_tokens_mistral': 3473, 'n_tokens_neox': 3179, 'n_words': 1377, 'path': 'LMADelariarte_CS170_OL161_2Q2021_Final_Project_February_13,_2021.ipynb', 'repo_name': 'LanceDdot/DSprojects', 'size': '11647'}