<jupyter_start><jupyter_text>IntroductionThis notebook aims to analyze insurance data, communicate usefuls insights derived from the analysis to policy makers, generate a predictive model and accurately predict insurance charges given the modelled features.  The dataset can be obtained from the [kaggle_link](https://www.kaggle.com/mirichoi0218/insurance).  The Data has the following column descriptions- `age`: age of primary beneficiary- `sex`: insurance contractor gender, female, male- `bmi`: Body mass index, providing an understanding of body, weights that are relatively high or low relative to height,objective index of body weight (kg / m ^ 2) using the ratio of height to weight, ideally 18.5 to 24.9- `children`: Number of children covered by health insurance / Number of dependents- `smoker`: Smoking- `region`: the beneficiary's residential area in the US, northeast, southeast, southwest, northwest.- `charges`: Individual medical costs billed by health insurance EDA First off we begin with performing some exploratory data analysis and visualization, after which we'll answer some questions which will help us derive some intuition about the dataset.   Specifically, we'll answer the following questions:  1. Does insurance charge differ across gender?2. Do smokers incur more insurance charges than non-smokers?3. Do people with more children incur higher insurance charges?4. what region has the highest and least insurance charge?5. Do older people incur more insurance charges than younger adults?Let's get started!<jupyter_code>#import required python libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# load the dataset
insurance = pd.read_csv('./Dataset/insurance.csv')
# inspect the dataset
print(insurance.shape)
insurance.head()<jupyter_output>(1338, 7)<jupyter_text>The above output shows that there are 1,338 individual respondents in the dataset<jupyter_code># show descriptives of the dataset
insurance.describe(include='all')<jupyter_output><empty_output><jupyter_text>The above output shows the descriptive statistics for all columns in the dataset.  The mean age is of individuals is 39, the maximum age is 64, while the minimum is 18.  Males have a greater representation (676) in the dataset than females.  Majority of the respondents are non-smokers (1064).  Southeast region has the most respondents in the dataset (364).<jupyter_code># show column details
insurance.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1338 entries, 0 to 1337
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   age       1338 non-null   int64  
 1   sex       1338 non-null   object 
 2   bmi       1338 non-null   float64
 3   children  1338 non-null   int64  
 4   smoker    1338 non-null   object 
 5   region    1338 non-null   object 
 6   charges   1338 non-null   float64
dtypes: float64(2), int64(2), object(3)
memory usage: 73.3+ KB<jupyter_text>The above output shows that there are no null values in the dataset columns.  The output also shows that the data types for the columns are 2 float colums, 2 integer columns and 3 string clomns    Visual EDA<jupyter_code># visualize the sex distribution
sns.countplot(data=insurance, x= 'sex')

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above chart shows that there are slightly more males than females in the dataset.<jupyter_code># visualize smokers distribution
sns.countplot(data = insurance, x = 'smoker')

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above chart shows that there are more non smokers in the dataset than there are smokers.<jupyter_code># show the  of charges for each sex
sns.countplot(data = insurance, x='sex', hue='smoker')

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above plot shows amongst the smokers, that there are more male smokers than females smokers.<jupyter_code># show the distribution of respondents region
sns.countplot(data=insurance , x = 'region')
plt.show()<jupyter_output><empty_output><jupyter_text>The above plot shows that majority of the respondents are from the southeast region, while the other regions have an almost equal representation of respondents.<jupyter_code># show the distribution respondents age
insurance.boxplot(column='age', grid=False)

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above chart shows that 25% of respondents are below 25 years, 50% are below 40 years and 75% are below 52 years.<jupyter_code># show the distribution of ages of respondents
insurance.hist(column='age', grid=False, by= 'sex', figsize= (10, 4))

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above histogram plots show age is right skewed for both sexes that majority of the respondents are above 30 years old for both genders.<jupyter_code># show the distribution of BMI
insurance.hist(column='bmi', grid=False, figsize= (10, 4))

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The BMI of respondents is slightly skewed to the right, indicating that most respondents have BMI greater than 30<jupyter_code># plot the distribution of insurance charges
plt.hist(insurance['charges'])

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above plot shows that the charges for insurance is right skewed. This invariably means that majority of the respondents are charged greater than mean insurance charge (13270.422265)<jupyter_code># plot the distribution of charges between gender 
insurance.hist(column='charges', by='sex', grid=False, figsize=(10, 4))

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above plot shows that the charges for insurance is right skewed for both sexes. This implies that majority of both male and female respondents are charged greater than mean insurance charge (13270.422265) Does insurance charge differ across gender?<jupyter_code># get a dataframe for just males
male_insurance = insurance[insurance['sex'] == 'male']

# get a dataframe of just females
female_insurance = insurance[insurance['sex'] == 'female']

# get the descriptives for just males
print('Males Insurance charge Stats'.upper(), '\n',male_insurance['charges'].describe(), '\n')
print('Females Insurance charge Stats'.upper(), '\n', female_insurance['charges'].describe())<jupyter_output>MALES INSURANCE CHARGE STATS 
 count      676.000000
mean     13956.751178
std      12971.025915
min       1121.873900
25%       4619.134000
50%       9369.615750
75%      18989.590250
max      62592.873090
Name: charges, dtype: float64 

FEMALES INSURANCE CHARGE STATS 
 count      662.000000
mean     12569.578844
std      11128.703801
min       1607.510100
25%       4885.158700
50%       9412.962500
75%      14454.691825
max      63770.428010
Name: charges, dtype: float64<jupyter_text>The above output shows the descrptive analysis of insurance charges for males and females respectively.  On the average, we can see that males (13956.75) incur higher isurance charges than females (12,569.70), though the difference in charges is not too large. Looking at the counts for males (676) and females (662), we see that there are 14 more males than females, with this observation, it is expected that the mean charges for males be lower than that of females, but since that is not the case, we conclude that this difference in means is as a result of a bias in insurance charges, favouring females more than males. Do smokers incur more insurance charges than non-smokers?<jupyter_code>fig = plt.figure()
ax = fig.add_axes([0,0,1,1])

# get the dataframe and mean charges for smokers
smoker = insurance[insurance['smoker'] == 'yes']
mean_smoker = smoker['charges'].mean()

# get the dataframe and mean charges for non-smokers
non_smoker = insurance[insurance['smoker'] == 'no']
mean_non_smoker = non_smoker['charges'].mean()

# plot the bar charts
ax.bar(smoker['smoker'], mean_smoker)
ax.bar(non_smoker['smoker'], mean_non_smoker)
plt.title('Average insurance charges for smokers and non-smokers')

# show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The above chart depicts that, on the average, smokers incur higher insurance charges than non, smokers. Let's see th counts of both smokers and non-smokers in order to get a better intuition.<jupyter_code># show the number of smokers and non-smokers
print(f'number of smokers is: {smoker.shape[0]}')
print(f'The number of non-smokers is: {non_smoker.shape[0]}', '\n')

# mean difference between smokers and non-smokers
print(f' The mean for smokers is {mean_smoker}', '\n', f'while the mean for non-smokers is: {mean_non_smoker}' '\n',
      f'The mean difference is {mean_smoker - mean_non_smoker}')<jupyter_output>number of smokers is: 274
The number of non-smokers is: 1064 

 The mean for smokers is 32050.23183153285 
 while the mean for non-smokers is: 8434.268297856199
 The mean difference is 23615.96353367665<jupyter_text>The number of non-smokers is 4 times the number of smokers, this could be the reason for its lower mean insurance charge, since the division is done across multiple values. But the mean difference shows that the average charge for smokers is about 5 times the charge for non-smokers.Thus, we conclude that insurance charges differs according to individuals smoking habits, people who have smoking habits pay heavier insurance charges than those who don't.  Do people with more children incur higher insurance charges?<jupyter_code># lets see the descriptives for the children column
print(insurance['children'].describe())

# show the children counts
insurance['children'].value_counts()<jupyter_output>count    1338.000000
mean        1.094918
std         1.205493
min         0.000000
25%         0.000000
50%         1.000000
75%         2.000000
max         5.000000
Name: children, dtype: float64<jupyter_text>The above output shows us that majority of the individuals in the dataset had no children at all, while the maximum number of chidren for a single respondent is 5.Next we'll see the average insurance charges for individuals with the different number of children.<jupyter_code># get a pivot table of the average charges for the different number of children
pv_children = insurance.pivot_table(values = 'charges', index = 'children', aggfunc= np.mean)

# plot a line chart of the pivot table
pv_children.plot(title= 'Average insurance charges for number of children',
                 legend= None,
                 figsize= (10, 5))

# show the pivot table
print(pv_children)

# show the pivot line plot
plt.show()<jupyter_output>charges
children              
0         12365.975602
1         12731.171832
2         15073.563734
3         15355.318367
4         13850.656311
5          8786.035247<jupyter_text>The above line chart shows the trend of insurance charges with the number of children. From the chart we can observe that those with 3 children incured the highest charges while those with 5 children incured the lowest charges.  Those without kids incured greater charges than those with 5 kids.Although it is expected that insurance charges be greater for those with more children than those with fewer children, an observation of the above line chart shows that the charges increases steadily for those with 0-3 kids where it got to its peak and declined significantly for those with 5 kids. The reason for this sharp decline could be attributed to the very few number of individuals in the dataset with 5 children (18).Thus we conclude that number of children affects insurance charges as individuals with more children incur more charges than individuals with fewer children. what region has the highest and least insurance charge?<jupyter_code># show the region counts
insurance['region'].value_counts()<jupyter_output><empty_output><jupyter_text>The above output shows that there are four regions in the dataset with most of the individuals coming from the southeast region.Let's see the minimum and maximum charges for each region The above output shows that the southeast region has both the maximum and minimum<jupyter_code># get the region with the highest insurance charge
print('region with highest insurance charge is:'.upper(), '\n', insurance[insurance['charges'] == insurance['charges'].max()]['region'], '\n')

# get the region with the least insurance charge
print('region with the least insurance charge is:'.upper(), '\n', insurance[insurance['charges'] == insurance['charges'].min()]['region'])<jupyter_output>REGION WITH HIGHEST INSURANCE CHARGE IS: 
 543    southeast
Name: region, dtype: object 

REGION WITH THE LEAST INSURANCE CHARGE IS: 
 940    southeast
Name: region, dtype: object<jupyter_text>Southeast region has both the highest and least insurance charges. Let's plot the average charges for each region.<jupyter_code># get a pivot table of the average charges for each region
pv_region = insurance.pivot_table(values = 'charges', index = 'region', aggfunc= np.mean)

# plot a bar chart of the pivot table
pv_region.plot(kind= 'barh',
              title= 'Average insurance charges for regions',
              legend= None,
              figsize= (10, 5))

# show the pivot table
print(pv_region)

# show the pivot line plot
plt.show()<jupyter_output>charges
region                 
northeast  13406.384516
northwest  12417.575374
southeast  14735.411438
southwest  12346.937377<jupyter_text>From the above chart we see that on the average, southeast charges the highest insurance fees, while southwest has the least insurance charges. Do older adults incur more insurance charges than younger adults?<jupyter_code># check the unique age counts
insurance['age'].value_counts().sort_index()<jupyter_output><empty_output><jupyter_text>From the above chart, we'll group those who are 35 years and above as older adults and those below 35 as younger adults.<jupyter_code># dataframe of respndents above 35 years
above_35_yrs = insurance[insurance['age'] >= 35]

# dataframe of respondents below 35 years
below_35_yrs = insurance[insurance['age'] < 35]

# Lets get the minimum, maximum and average insurance charges for older adults with respect to their number of children
pv_age_above_35 = above_35_yrs.pivot_table(values = 'charges', index = 'children', aggfunc= [np.min, np.max, np.mean])

# # Lets get the minimum, maximum and average insurance charges for younger adults with respect to their number of children
pv_age_below_35 = below_35_yrs.pivot_table(values = 'charges', index = 'children', aggfunc= [np.min, np.max, np.mean])

# plot a bar chart of the pivot table for older adults
pv_age_above_35.plot(kind= 'bar',
              title= 'Average insurance charges for older adults with respect to their number of kids'.upper(),
              legend= True,
              figsize= (7, 4))

# plot a bar chart of the pivot table for younger adults
pv_age_below_35.plot(kind= 'bar',
              title= 'Average insurance charges for younger adults with respect to their number of kids'.upper(),
              legend= True,
              figsize= (7, 4))

# show the pivot line plot
plt.show()<jupyter_output><empty_output><jupyter_text>From the above charts we can deduce the following:- older adults have more children than younger adults- Generally on the average, older adults incur more insurance charges than younger adults.- Generally, Judging from the maximum charges, older adults incur more insurance charges than younger adults. Convert the categorical columns to Numerical values<jupyter_code>model_data = pd.get_dummies(insurance, columns= ['sex', 'smoker', 'region'], drop_first = True)

# check
model_data.head()<jupyter_output><empty_output><jupyter_text>We'll only care if a person has a child or not and not bother about the number of children.  Therefore we'll binarize the children column:<jupyter_code># create a new children column filled with zeros
model_data['kids'] = 0

# Replace all the kids values with a 1 where the corresponding children column is greater than 0.
model_data.loc[model_data['children'] > 0, 'kids'] = 1

# check
model_data.head()
# lets delete the children column
model_data.drop('children', axis=1, inplace = True)

# check
model_data.head()<jupyter_output><empty_output><jupyter_text>Modelling<jupyter_code># next we get the dependent and independent variavles separately
y = model_data['charges']
x = model_data.drop(['charges'], axis = 1)

# check
print(x.shape)
y.shape
# Import Lasso from sklearn.linear_model.
from sklearn.linear_model import Lasso

# instantiate lasso
lasso = Lasso()
# first we get the hyper parameters
# import relevant modules
from sklearn.model_selection import GridSearchCV

# Setup the hyperparameter grid by using alpha_space as the grid of values to tune alpha over
alpha_space = np.logspace(-1, 1, 15)
param_grid = {"alpha": alpha_space}

# Use GridSearchCV with 5-fold cross-validation to tune C:
# Inside GridSearchCV(), specify the classifier, parameter grid, and number of folds to use.
lasso_cv = GridSearchCV(lasso, param_grid = param_grid, cv = 5)

# Use the .fit() method on the GridSearchCV object to fit it to the data x and y.
lasso_cv.fit(x, y)

# Print the tuned parameters and score
print("Tuned Lasso Regression Parameters: {}".format(lasso_cv.best_params_)) 
print("Best score is {}".format(lasso_cv.best_score_))
# split the data set into training and test set
# import package for splitting training and testing data
from sklearn.model_selection import train_test_split

# create training and test sets such that 30% is used for testing
# and 70% for training. Use a random state of 42.
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.3, random_state = 42)

# instantiate lasso again now with the tuned hyperparameter
lasso_tuned = Lasso(alpha = 0.1, normalize = True)

# fit lasso
lasso_tuned.fit(x_train, y_train)

# get lasso efficients
lasso_coef = lasso_tuned.fit(x_train, y_train).coef_

# predict on the test set
y_pred = lasso_tuned.predict(x_test)

# get the lasso score
print(f' Lasso regression score is: {lasso_tuned.score(x_test, y_test)}, \n')

# get the names of our feature variables
names = model_data.drop('charges', axis = 1).columns

# plot the lasso coefficients
plt.plot(range(len(names)), lasso_coef)

# Plot the coefficients
plt.plot(range(len(names)), lasso_coef)

# set the x-axis names
plt.xticks(range(len(names)), names, rotation= 60)

# set the y-axis label
plt.ylabel('coefficients')

plt.show()<jupyter_output>Lasso regression score is: 0.7689248397829056,
{'fraction_non_alphanumeric': 0.0780286373143133, 'fraction_numerical': 0.037646806456802705, 'mean_word_length': 4.776951672862453, 'pattern_counts': {'":': 1, '<': 94, '<?xml version=': 0, '>': 95, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '583954', 'license': 'MIT', 'n_tokens_mistral': 5914, 'n_tokens_neox': 5212, 'n_words': 2233, 'path': 'Insurance_cost_prediction.ipynb', 'repo_name': 'Akawi85/Stutern_Projects', 'size': '18647'}