<jupyter_start><jupyter_text>**Financial Inclusion Report** **SECTION 1:** Defining the data analytical question a) Specifying the Data Analytic QuestionFinding out the state of Financial inclusion in Kenya, Rwanda, Tanzania and Uganda between 2016-2018  b) Defining the Metric for SuccessThe research problem is to figure out how we can predict which individuals are most likely to have or use a bank account.  c.) Understanding the contextFinancial Inclusion remains one of the main obstacles to economic and human development in Africa. For example, across Kenya, Rwanda, Tanzania, and Uganda only 9.1 million adults (or 13.9% of the adult population) have access to or use a commercial bank account.Traditionally, access to bank accounts has been regarded as an indicator of financial inclusion. Despite the proliferation of mobile money in Africa and the growth of innovative fintech solutions, banks still play a pivotal role in facilitating access to financial services. Access to bank accounts enables households to save and facilitate payments while also helping businesses build up their credit-worthiness and improve their access to other financial services. Therefore, access to bank accounts is an essential contributor to long-term economic growth.  d.) Recording the Experimental Design  > Here is the breakdown of the process:   1. Started with loading data, and previewing the top and bottom rows   2. Explored the data a bit to understand the kind of dataset we have   3. Validated the data from outside source   4. Cleaned the data   5. Perfomed Univariate Analysis   6. Perfomed Bivariate Analysis   7. Performed Multivariate Analysis   9. Challenged the Solution  e.) Data RelevanceYes the data was relevant to the research question asked as it evaluates the data set to identify whether the information within is useful to solve the identfied problemThe data source was from Finscope surveys ranging from 2016 to 2018, from Kenya, Rwanda, Tanzania and uganda which are our countries of focus. The surveys were conducted recently, hence the data is current and has individual demograpgics that will support us in assessing factors that drive financial inclusion.  f.) Data validationChecking data against external sources to see if they match.This dataset was validated with an external source that is reliable.   **SECTION 2:** Reading and checking the dataset<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Dataset files :Variable Definitions: http://bit.ly/VariableDefinitions Dataset: http://bit.ly/FinancialDataset<jupyter_code>dataset_df = pd.read_csv('http://bit.ly/FinancialDataset')
var_def_df = pd.read_csv('http://bit.ly/VariableDefinitions')<jupyter_output><empty_output><jupyter_text>Determining the number of records<jupyter_code>dataset_df.shape #Determining the number of records in the dataset
print('This dataset has ' + str(dataset_df.shape[0]) + ' rows, and ' + str(dataset_df.shape[1]) + ' columns')
var_def_df.shape #Determining the number of records in the dataset description
print('This dataset description has ' + str(var_def_df.shape[0]) + ' rows, and ' + str(var_def_df.shape[1]) + ' columns')<jupyter_output>This dataset description has 12 rows, and 2 columns<jupyter_text>Previewing our data<jupyter_code>dataset_df.head()<jupyter_output><empty_output><jupyter_text>3. Checking the Data<jupyter_code>dataset_df.tail() #Previewing the bottom of our dataset
dataset_df.describe() # Checking the statistical description of our dataset<jupyter_output><empty_output><jupyter_text>We found that the mean of the household sizes was 3.68 and our respondent age was 38.8 indicating that the respondents were neither too young nor old<jupyter_code>dataset_df.columns #checking the column data that is in our dataset
dataset_df.dtypes #Checking each column's datatypes and finding out wheteher they are appropriate<jupyter_output><empty_output><jupyter_text>Our dataset has objects ,integers and float datatypes<jupyter_code>dataset_df.nunique() #checking for unique values in our dataset<jupyter_output><empty_output><jupyter_text>4. Tidying the Dataset<jupyter_code>dataset_df.isnull().sum() #checking for null values<jupyter_output><empty_output><jupyter_text>Most of our records have null values but are negligible compared to the total records of the dataset (23,524 records)<jupyter_code>dataset_df.dropna (inplace =True) #Dealing with missing values by dropping them
dataset_df.isnull().sum() #Checking if missing values have been dropped and find out that they have.
# Aligning the columns in our dataset by ensuring uniformity

dataset_df.columns = dataset_df.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
dataset_df.rename({'has_a_bank_account': 'bank_account', 'type_of_location': 'location','cell_phone_access' : 'phone','the_relathip_with_head' : 'relationship_with_head', 'level_of_educuation' : 'education_level'}, axis=1, inplace=True)
dataset_df.head()
# checking and dealing with outliers 

col_names= ['year','household_size', 'respondent_age',]

fig, ax = plt.subplots(len(col_names), figsize=(10,15))

for i, col_val in enumerate(col_names):

    sns.boxplot(y=dataset_df[col_val], ax=ax[i])
    ax[i].set_title('Box plot for numerical variables - {}'.format(col_val), fontsize=12)
    ax[i].set_xlabel(col_val, fontsize=12)

plt.show()<jupyter_output><empty_output><jupyter_text>The box plot indicates that the data seems to be skewed to the right with the above variables (year,household size and the respondent age) having outliers. These were 2029, 2056 and 2039 which were incorrect. For household size, we had those outside the range from 10 and above 20 and for respondent age we had outliers of respondent aged above 85<jupyter_code># Dealing with Outliers by removing them
Q1 = dataset_df.quantile(0.25)
Q3 = dataset_df.quantile(0.75)
IQR = Q3 - Q1
dataset_df_out = dataset_df[~((dataset_df < (Q1 - 1.5 * IQR)) | (dataset_df > (Q3 + 1.5 * IQR))).any(axis=1)]
print(dataset_df)
print(dataset_df_out)<jupyter_output>country  year  ...                  education_level                type_of_job
0       Kenya  2018  ...              Secondary education              Self employed
1       Kenya  2018  ...              No formal education       Government Dependent
2       Kenya  2018  ...  Vocational/Specialised training              Self employed
3       Kenya  2018  ...                Primary education  Formally employed Private
4       Kenya  2018  ...                Primary education        Informally employed
...       ...   ...  ...                              ...                        ...
23519  Uganda  2018  ...              No formal education               Other Income
23520  Uganda  2018  ...              Secondary education               Other Income
23521  Uganda  2018  ...                Primary education               Other Income
23522  Uganda  2018  ...              Secondary education              Self employed
23523  Uganda  2018  ...              Secondary education        [...]<jupyter_text>5. Exploratory Analysisa)Univariate analysis<jupyter_code># Plotting our numerical variables on histogram to assess the relationship
dataset_df_out.drop(['country','uniqueid', 'bank_account','location' , 'phone',	'gender_of_respondent','relationship_with_head','marital_status',	'education_level',	'type_of_job'] ,axis=1).hist(bins=10, figsize=(10,10))
plt.suptitle("Histogram of numeric variables in financial dataset")
plt.savefig('Financial access Histogram')
plt.show()<jupyter_output><empty_output><jupyter_text>Based on the histograms above we can see the kinds of distributions that seem to follow a positive skew.<jupyter_code># Plotting the count of those with a bank account
sns.countplot(dataset_df_out['bank_account'])

# Plotting respondents with bank accounts by percentage
(dataset_df_out['bank_account'].value_counts()/len(dataset_df_out)).plot.bar()

# Those without bank accounts were more than 80&
# Plotting the count of gender of respondents

sns.countplot(dataset_df_out['gender_of_respondent'])
# We had more female respondents that male
# Plotting the count of respondents that had access to a cell phone

sns.countplot(dataset_df_out['phone'])
# Most respondents did not have access to a cell phone
# Plotting the count of respondents based on location

sns.countplot(dataset_df_out['location'])

# Most respondents were from rural area
# Plotting the count of respondents based on education level
sns.countplot(dataset_df_out['education_level'])

#  Most repondents had a primary level education<jupyter_output><empty_output><jupyter_text>b.) Bivariate Analysis<jupyter_code>dataset_df_out['country'].value_counts().plot.bar(title='Frequency of repondents by country')<jupyter_output><empty_output><jupyter_text>We saw a bulk of respondents from Rwanda, followed by Tanzania,kenya and Uganda<jupyter_code># Ditribution plot for respondent age
sns.distplot(dataset_df_out['respondent_age'], bins=10)<jupyter_output><empty_output><jupyter_text>It shows that a huge population of respondents was aged between 20-40 years<jupyter_code>(dataset_df_out['country'].value_counts().head(10)/len(dataset_df_out)).plot.line()
# Plotting repondents by marital status
dataset_df_out['marital_status'].value_counts().plot.bar()<jupyter_output><empty_output><jupyter_text>It shows that highest number of residents were married<jupyter_code># Plotting percentage of respondents by marital status

(dataset_df_out['marital_status'].value_counts()/len(dataset_df_out)).plot.bar()<jupyter_output><empty_output><jupyter_text>A bulk of our respondents who are married were more than 40%<jupyter_code>#Creating subplots for our dataset.

sns.barplot(x='country', y='household_size', hue ='bank_account', data = dataset_df_out)<jupyter_output><empty_output><jupyter_text>Tanzania has households with smaller size, followed by Kenya where Uganda and Rwanda seem to be at the same level.<jupyter_code>#Plotting subplots for our dataset

sns.barplot(x='country', y='respondent_age', hue ='bank_account', data = dataset_df_out)<jupyter_output><empty_output><jupyter_text>There is an equal numbers of those with and without bank accounts under the age of 40 accross the highlighted countries<jupyter_code>#Creating frequency tables to observe the Variable distribution

dataset_df_out['bank_account'].value_counts()<jupyter_output><empty_output><jupyter_text>There are 3297 people with bank account and 20046 without them<jupyter_code>#Calculating range between minimum and maximum age of respondents
dataset_df_out_max = dataset_df_out["respondent_age"].max()
dataset_df_out_min = dataset_df_out["respondent_age"].min()

# Calculating the range
dataset_df_out_max - dataset_df_out_min

# Finding the quantiles of age of respondents variable
# 
dataset_df_out["respondent_age"].quantile([0.25, 0.5, 0.75])

#Checking skewness
dataset_df_out["respondent_age"].skew()<jupyter_output><empty_output><jupyter_text>The respondent age is posively skewed<jupyter_code>#checking Kurtosis
dataset_df_out["respondent_age"].kurt()<jupyter_output><empty_output><jupyter_text>The -0.23 indicates a Platykurtic distribution<jupyter_code>#Checking summary statistics for respondent's age
dataset_df_out["respondent_age"].describe()<jupyter_output><empty_output><jupyter_text>b)Bivariate analysis<jupyter_code># Checking how our varibales are corelated using a matrix
corrMatrix = dataset_df_out.corr()
corrMatrix<jupyter_output><empty_output><jupyter_text>From the matrix we observe that correlation amongst our variables is not very strongThe highest correation is a negative correlation of -0.119709 between household size and the age of repondent. c.)<jupyter_code>corelation = dataset_df_out.corr()
# using a heatmap to display the results
sns.heatmap(corelation,xticklabels = corelation.columns, yticklabels = corelation.columns, annot=True)<jupyter_output><empty_output><jupyter_text>From the heat map below, we observe that correlation amongst our variables is not very strong, the highest being negative correlation of -0.119709 between household size and the age of respondent.This therefore means that there are key variables/determinants that can influence whether an individual will open or use a bank account or not. From our dataset we will consider the following when implementing the solution:1. Location; individuals in the urban areas are likey to open accounts more than those in the rural areas as they have access to banking halls, transport etc. 2. Level of education: Individuals with higher literacy levels are likely to open and use bank accounts as they have a better understanding3. Type of job: Those formally employed are likely to open and use bank accounts compared to those in the informal sector and farming in rural areas4. Gender of respondent; we are likely to have more male open and use bank accounts as they have access to resources<jupyter_code># Plotting the bivariate summaries and recording our observations
sns.pairplot(dataset_df_out)
plt.show()
sns.relplot( x = 'household_size', y = 'respondent_age', hue = 'phone',data= dataset_df_out)
# Plotting staked plots
sns.barplot(x= 'country', y = 'household_size', data = dataset_df_out, hue='bank_account')<jupyter_output><empty_output><jupyter_text>We see Tanzania had smaller household sizes, followed by Kenya, Uganda and rwanda seems to be at same level<jupyter_code># Plotting staked plots
sns.barplot(x= 'country', y = 'respondent_age', data = dataset_df_out, hue='bank_account')
# This shows that there is 50/50 distribution of repondents with or without bank accounts below age 40 in the 4 countries<jupyter_output><empty_output><jupyter_text>This shows that there is a 50/50 distribution of repondents with or without bank accounts below age 40<jupyter_code># Plotting relational plots
sns.relplot( x = 'household_size', y = 'respondent_age', hue = 'location',data= dataset_df_out)<jupyter_output><empty_output><jupyter_text>We see that Most respondents are from rural area<jupyter_code># Plotting relational plots
sns.relplot( x = 'household_size', y = 'respondent_age', hue = 'bank_account',data= dataset_df_out)<jupyter_output><empty_output><jupyter_text>Most respondents had no bank accounts SECTION 6: Multivariate Analysis<jupyter_code># Calculating the correlation between the variables

correlation = dataset_df_out.corr()

sns.heatmap(correlation,xticklabels = correlation.columns, yticklabels = correlation.columns, annot=True)<jupyter_output><empty_output><jupyter_text>From the heat map below, we observe that correlation amongst our variables is not very strong, the highest being negative correlation of -0.119709 between household size and the age of respondent. This therefore means that there are key variables/determinants that can influence whether an individual will open or use a bank account or not. From our dataset we will consider the following when implementing the solution:Location; - Individuals in the urban areas are likey to open accounts more than those in the rural areas as they have access to banking halls, transport etc.Level of education: - Individuals with higher literacy levels are likely to open and use bank accounts as they have a better understandingType of job: - Those formally employed are likely to open and use bank accounts compared to those in the informal sector and farming in rural areasGender of respondent: - We are likely to have more male open and use bank accounts as they have access to resources This indicates that Tanzania had smaller household sizes, followed by Kenya, Uganda and rwanda seems to be at same levelMost of the repondents in Kenya did not have bank accounts to the other 3 countries, this could be attributed to the fact that mobile money penetration in Kenya is well adopted, hence individuals may not find the need to open bank accounts  7. Implementing the Solution For us to effectively implement our solution, we need to determine which factors have significant influence on whether an individual will open or use a bank account or not<jupyter_code>#Preparing our dataset for implementation, we will change datatypes in form of string to numeric for ease of comparison

dataset_df_out["gender_of_respondent"] = np.where(dataset_df_out["gender_of_respondent"].str.contains("Male", "Female"), 1, 0)
dataset_df_out.head()
dataset_df_out["bank_account"] = np.where(dataset_df_out["bank_account"].str.contains("Yes", "No"), 1, 0)
dataset_df_out.head()
dataset_df_out["location"] = np.where(dataset_df_out["location"].str.contains("Rural", "Urban"), 1, 0)
dataset_df_out.head()
dataset_df_out["phone"] = np.where(dataset_df_out["phone"].str.contains("Yes", "No"), 1, 0)
dataset_df_out.head()
#calculating corelation
corrMatrix = dataset_df_out.corr()
corrMatrix<jupyter_output><empty_output><jupyter_text>This indicates that correlation between some variables has improved though not very high where the Highest being 0.27 between household size and location<jupyter_code># Dropping unnecessary columns that are in string format
# 
dataset_df_out.drop(['country','year','uniqueid', 'marital_status', 'relationship_with_head','education_level','type_of_job'],axis=1,inplace=True)

dataset_df_out.head()<jupyter_output>/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py:3997: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,<jupyter_text>Principal Component Analysis<jupyter_code># Preprocessing

# We we will divide our dataset into features and labels 

#X = dependent variable 
#y = dependent variable which is having a bank account or not
# 
X = dataset_df_out.iloc[:,1:6].values
y = dataset_df_out.iloc[:, 1].values 
# We split the dataset into the Training set and Test set
#
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
# We will then perform Normalization Testing

# PCA performs best with a normalized feature set. 
# We will perform standard scalar normalization to normalize our feature set. 

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
#Applying PCA
# Initialize the PCA class by passing the number of components to the constructor.
# Call the fit and then transform methods by passing the feature set to these methods. 
# The transform method returns the specified number of principal components. 
# We did not specify the number of components in the constructor. 
# All the entries in the feature set will be returned for both the training and test sets.
# 
from sklearn.decomposition import PCA

pca = PCA()
X_train = pca.fit_transform(X_train)
X_test = pca.transform(X_test)
# Explained Variance Ratio
# The PCA class contains explained_variance_ratio_ which returns the variance caused by each of the principal components. 

explained_variance = pca.explained_variance_ratio_

# The explained_variance variable is now a float type array which contains variance ratios for each principal component. 
# The values for the explained_variance variable looks like what is shown in the output. 
# It can be seen that first principal component is responsible for 26.99% variance. .
explained_variance
#Using 1 Principal Component
# We will first try by using 1 principal component to train our algorithm.
# 
from sklearn.decomposition import PCA

pca = PCA(n_components=1)
X_train = pca.fit_transform(X_train)
X_test = pca.transform(X_test)
# Training and Making Predictions
# In this case we'll use random forest classification for making the predictions.
# 
from sklearn.ensemble import RandomForestClassifier

classifier = RandomForestClassifier(max_depth=2, random_state=0)
classifier.fit(X_train, y_train)

# Predicting the Test set results
y_pred = classifier.predict(X_test)
# Performance Evaluation
# 
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

cm = confusion_matrix(y_test, y_pred)
print(cm)
print('Accuracy' , accuracy_score(y_test, y_pred)) 

# Noting that from the output that with only one feature, 
# the random forest algorithm is able to correctly predict with an accuracy of 83.1% 

#We will perform LDA to compare the Accuracy<jupyter_output>[[1439  329]
 [ 389 2388]]
Accuracy 0.842024202420242<jupyter_text>Linear Disciminant Analysis<jupyter_code>#We will divide our dataset into features and labels 
#X will represent our dependent variable and y our dependent variable which is having a bank account or not

X = dataset_df_out.iloc[:,1:6].values
y = dataset_df_out.iloc[:, 1].values 
# Then we will divide our data into training and test sets

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
# Then we will perform Feature scaling
# We do this by performing the following 
# 
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
# We the peform LinearDiscriminantAnalysis (LDA)
# We will perform LDA with Scikit-Learn. 
# The LinearDiscriminantAnalysis class of the sklearn.discriminant_analysis 
# library can be used to Perform LDA in Python. 
#

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
lda = LDA(n_components=1)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)

# In the script above the LinearDiscriminantAnalysis class is imported as LDA. 
# We have to pass the value for the n_components parameter of the LDA, 
# which refers to the number of linear discriminates that we want to retrieve. 
# In this case we set the n_components to 1, since we first want to check the performance 
# of our classifier with a single linear discriminant. 
# Finally we execute the fit and transform methods to actually retrieve the linear discriminants.
# In case of LDA, the transform method takes two parameters: the X_train and the y_train. 
# This reflects the fact that LDA takes the output class labels into account while selecting the linear discriminants.
# Training and Making Predictions
# We will use the random forest classifier to evaluate the performance of a PCA-reduced algorithms as shown
# 

from sklearn.ensemble import RandomForestClassifier

classifier = RandomForestClassifier(max_depth=2, random_state=0)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
# Evaluating the Performance
# Last step is  to evaluate performance of the algorithm 
# with the help of a confusion matrix and find the accuracy of the prediction.
# 

from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

cm = confusion_matrix(y_test, y_pred)
print(cm)
print('Accuracy' + str(accuracy_score(y_test, y_pred)))

# We can see that with one linear discriminant, the algorithm achieved an accuracy of 100%, 
# which is greater than the accuracy achieved with one principal component, which was 83.1%.<jupyter_output>[[1768    0]
 [   0 2777]]
Accuracy1.0
{'fraction_non_alphanumeric': 0.07255885758394442, 'fraction_numerical': 0.01466615206483983, 'mean_word_length': 5.4903979961035345, 'pattern_counts': {'":': 0, '<': 129, '<?xml version=': 0, '>': 130, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '550018', 'license': 'MIT', 'n_tokens_mistral': 6966, 'n_tokens_neox': 6501, 'n_words': 2512, 'path': 'Financial_Inclusion_Report_in_Kenya,_Rwanda,_Tanzania_and_Uganda_2016_2018.ipynb', 'repo_name': 'Askoye/IP-Core', 'size': '23319'}