<jupyter_start><jupyter_text>Linear Regression  Now we'll discuss a case study solution using multiple linear regression, and regularised linear regresison [Ridge and Lasso] . We'll also look at hyper parameter tuning for regularised regression.<jupyter_code>import pandas as pd 
import numpy as np<jupyter_output><empty_output><jupyter_text>A little background on the case study. This data belongs to a loan aggregator agency which connects loan applications to different financial institutions in attempt to get the best interest rate. They want to now utilise past data to predict interest rate given by any financial institute just by looking at loan application characteristics.To achieve that , they have decided to do a POC with a data from a particular financial institution. The data is given in the file "loans data.csv". Lets begin:<jupyter_code>train_file='../data/loan_data_train.csv'
test_file='../data/loan_data_test.csv'

ld_train=pd.read_csv(train_file)
ld_test=pd.read_csv(test_file)               
print(ld_train.shape)
ld_train.head()
#test data does not have interest rate
print(ld_test.shape)
ld_test.head()
# lets combine the data for data prep (cleaning)
ld_test['Interest.Rate']=np.nan
ld_train['data']='train'
ld_test['data']='test'
ld_test=ld_test[ld_train.columns] #to ensure same order of columns
ld_all=pd.concat([ld_train,ld_test],axis=0)
ld_all.head(5)
ld_all.dtypes
# 1. ID,Amount.Funded.By.Investors : drop 
# 2. Interest Rate , Debt to income ratio : remove % and then to numeric
# 3. Amount.Requested , 'Open.CREDIT.Lines','Revolving.CREDIT.Balance': convert it to numeric 
# 4. FICO.Range : replace it by a numeric column which is average of the range
# 5. Employment Length : convert to number
# 6. Loan Lenth, Loan Purpose , State , Home ownership: dummies for categories with good occurence rate
ld_all.drop(['ID','Amount.Funded.By.Investors'],axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>variable `Interest.Rate` and `Debt.To.Income.Ratio` contain "%" sign in their values and because of which they have come as character columns in the data. Lets remove these percentages first.<jupyter_code>for col in ['Interest.Rate','Debt.To.Income.Ratio']:
    ld_all[col]=ld_all[col].str.replace("%","")
ld_all.head(5)<jupyter_output><empty_output><jupyter_text>We can see that many columns which should have really been numbers have been imported as character columns , probably because some characters values in those columns in the files. We'll convert all such columns to numbers .<jupyter_code>for col in ['Amount.Requested', 'Interest.Rate','Debt.To.Income.Ratio',
            'Open.CREDIT.Lines','Revolving.CREDIT.Balance']:
    ld_all[col]=pd.to_numeric(ld_all[col],errors='coerce')<jupyter_output><empty_output><jupyter_text>If we look at first few values of variable FICO.Range , we can see that we can convert it to numeric by taking average of the range given. To do that first we need to split the column with "-", so that we can have both end of ranges in separate columns and then we can simply average them.<jupyter_code>k=ld_all['FICO.Range'].str.split("-",expand=True).astype(float)
k.dtypes
ld_all['fico']=0.5*(k[0]+k[1])
del ld_all['FICO.Range']
ld_all["Employment.Length"]
ld_all['Employment.Length'].value_counts()
ld_all['Employment.Length']=ld_all['Employment.Length'].str.replace('years',"")
ld_all['Employment.Length']=ld_all['Employment.Length'].str.replace('year',"")
#np.where(condition, value_if_True, value_if_False)
ld_all['Employment.Length']=np.where(ld_all['Employment.Length'].str[:2]=="10",10,ld_all['Employment.Length'])
ld_all['Employment.Length']=np.where(ld_all['Employment.Length'].str[0]=="<",0,ld_all['Employment.Length'])
ld_all['Employment.Length']=pd.to_numeric(ld_all['Employment.Length'],errors='coerce')
ld_all.dtypes
# Notice that to apply string function on pandas data frame columns you need to str attribute
cat_cols=ld_all.select_dtypes(['object']).columns
cat_cols
cat_cols=cat_cols[:-1]
cat_cols
ld_all["State"].unique()
# you can use following method if you want to ignore categories with too low frequencies ,
# in next section for logistic regression we will be using  pandas' get dummies function. 
# you can work with either of these . 
# ignoring categories with low frequencies however will result in fewer columns without 
# affecting model performance too much .

for col in cat_cols:
    freqs=ld_all[col].value_counts()
    k=freqs.index[freqs>20][:-1]
    for cat in k:
        name=col+'_'+cat
        ld_all[name]=(ld_all[col]==cat).astype(int)
    del ld_all[col]
    print(col)  
ld_all.columns
ld_all.isnull().sum()
#result = ld.loc[condition,desired_cols]
for col in ld_all.columns:
    if (col not in ['Interest.Rate','data'])& (ld_all[col].isnull().sum()>0):
        ld_all.loc[ld_all[col].isnull(),col]=ld_all.loc[ld_all['data']=='train',col].mean()
ld_all.isnull().sum()
ld_train=ld_all[ld_all['data']=='train']
del ld_train['data']
ld_test=ld_all[ld_all['data']=='test']
ld_test.drop(['Interest.Rate','data'],axis=1,inplace=True)
del ld_all
from sklearn.model_selection import train_test_split
ld_train1,ld_train2=train_test_split(ld_train,test_size=0.2,
                                     random_state=2)
# Notice that only train data is used for imputing missing values in both train and test 
x_train1=ld_train1.drop('Interest.Rate',axis=1)
y_train1=ld_train1['Interest.Rate']
x_train1.shape, y_train1.shape
from sklearn.linear_model import LinearRegression
lm=LinearRegression()
lm.fit(x_train1,y_train1)
x_train1.shape
lm.intercept_
list(zip(x_train1.columns,lm.coef_))
x_train2=ld_train2.drop('Interest.Rate',axis=1)
predicted_ir=lm.predict(x_train2)
from sklearn.metrics import mean_absolute_error
mean_absolute_error(ld_train2['Interest.Rate'],predicted_ir)<jupyter_output><empty_output><jupyter_text>We know the tentative performance now, lets build the model on entire training to make prediction on test/production<jupyter_code>x_train=ld_train.drop('Interest.Rate',axis=1)
y_train=ld_train['Interest.Rate']
lm.fit(x_train,y_train)
test_pred=lm.predict(ld_test)<jupyter_output><empty_output><jupyter_text>We can write these to a csv file for submission like this :<jupyter_code>pd.DataFrame(test_pred).to_csv("mysubmission.csv",index=False)<jupyter_output><empty_output><jupyter_text>Ridge  Regression<jupyter_code>from sklearn.linear_model import Ridge,Lasso
from sklearn.model_selection import GridSearchCV
lambdas=np.linspace(1,100,100)
params={'alpha':lambdas}
model=Ridge(fit_intercept=True)
grid_search=GridSearchCV(model,param_grid=params,cv=10,scoring='neg_mean_absolute_error')
grid_search.fit(x_train,y_train)
grid_search.best_estimator_
grid_search.cv_results_<jupyter_output><empty_output><jupyter_text>if you want you can now fit a ridge regression model with obtained value of alpha , although there is no need, grid search automatically fits the best estimator on the entire data, you can directly use this to make predictions on test_data. But if you want to look at coefficients , its much more convenient to fit the model with direct function Using the report function given below you can see the cv performance of top few models as well, that will the tentative performance<jupyter_code>def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidates = np.flatnonzero(results['rank_test_score'] == i)
        for candidate in candidates:
            print("Model with rank: {0}".format(i))
            print("Mean validation score: {0:.3f} (std: {1:.3f})".format(
                  results['mean_test_score'][candidate],
                  results['std_test_score'][candidate]))
            print("Parameters: {0}".format(results['params'][candidate]))
            print("")
report(grid_search.cv_results_,100)
test_pred=grid_search.predict(ld_test)
pd.DataFrame(test_pred).to_csv("mysubmission.csv",index=False)<jupyter_output><empty_output><jupyter_text>For looking at coefficients<jupyter_code>ridge_model=grid_search.best_estimator_
ridge_model.fit(x_train,y_train)
list(zip(x_train1.columns,ridge_model.coef_))<jupyter_output><empty_output><jupyter_text>Lasso Regression<jupyter_code>lambdas=np.linspace(1,10,100)
model=Lasso(fit_intercept=True)
params={'alpha':lambdas}
grid_search=GridSearchCV(model,param_grid=params,cv=10,scoring='neg_mean_absolute_error')
grid_search.fit(x_train,y_train)
grid_search.best_estimator_<jupyter_output><empty_output><jupyter_text>you can see that, the best value of alpha comes at the edge of the range that we tried , we should expand the trial range on that side and run this again<jupyter_code>lambdas=np.linspace(.001,2,100)
params={'alpha':lambdas}
grid_search=GridSearchCV(model,param_grid=params,cv=10,scoring='neg_mean_absolute_error')
grid_search.fit(x_train,y_train)
grid_search.best_estimator_
report(grid_search.cv_results_,5)
lasso_model=grid_search.best_estimator_
lasso_model.fit(x_train,y_train)
list(zip(x_train.columns,lasso_model.coef_))<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>import numpy as np
import pandas as pd
train_file='../data/rg_train.csv'
test_file='../data/rg_test.csv'
bd_train=pd.read_csv(train_file)

bd_test=pd.read_csv(test_file)


bd_test['Revenue.Grid']=np.nan
bd_train['data']='train'
bd_test['data']='test'
bd_test=bd_test[bd_train.columns]
bd_all=pd.concat([bd_train,bd_test],axis=0)
bd_all['Revenue.Grid'].value_counts()
list(zip(bd_all.columns,bd_all.dtypes,bd_all.nunique()))
# REF_NO,post_code , post_area  : drop 
# children : Zero : 0 , 4+ : 4 and then convert to numeric
# age_band : dummies 
# status , occupation , occupation_partner , home_status,family_income : dummies
# self_employed, ` : dummies
# TVArea , Region , gender : dummies
# Revenue Grid : 1,2 : 1,0
bd_all.drop(['REF_NO','post_code','post_area'],axis=1,inplace=True)
bd_all['children']=np.where(bd_all['children']=='Zero',0,bd_all['children'])
bd_all['children']=np.where(bd_all['children'][:1]=='4',4,bd_all['children'])
bd_all['children']=pd.to_numeric(bd_all['children'],errors='coerce')
bd_all['Revenue.Grid']=(bd_all['Revenue.Grid']==1).astype(int)
cat_vars=bd_all.select_dtypes(['object']).columns

cat_vars
for col in cat_vars[:-1]:
    dummy=pd.get_dummies(bd_all[col],drop_first=True,prefix=col)
    bd_all=pd.concat([bd_all,dummy],axis=1)
    del bd_all[col]
    print(col)
del dummy
bd_all.shape
bd_all.isnull().sum()
bd_all.loc[bd_all['children'].isnull(),'children']=bd_all.loc[bd_all['data']=='train','children'].mean()

bd_train=bd_all[bd_all['data']=='train']
del bd_train['data']
bd_test=bd_all[bd_all['data']=='test']
bd_test.drop(['Revenue.Grid','data'],axis=1,inplace=True)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
params={'class_weight':['balanced',None],
        'penalty':['l1','l2'],
        'C':np.linspace(0.01,1000,10)}
model=LogisticRegression(fit_intercept=True, solver="saga", max_iter=1000)
grid_search=GridSearchCV(model,param_grid=params,cv=5,
                         scoring="roc_auc", n_jobs=5)
x_train=bd_train.drop('Revenue.Grid',axis=1)
y_train=bd_train['Revenue.Grid']
x_train.shape
grid_search.fit(x_train,y_train)
grid_search.best_estimator_
logr=grid_search.best_estimator_
report(grid_search.cv_results_,5)
logr.fit(x_train,y_train)
cutoffs=np.linspace(0.01,0.99,99)

cutoffs
train_score=logr.predict_proba(x_train)[:,1]

real=y_train
train_score>0.2
KS_all=[]

for cutoff in cutoffs:
    
    predicted=(train_score>cutoff).astype(int)

    TP=((predicted==1) & (real==1)).sum()
    TN=((predicted==0) & (real==0)).sum()
    FP=((predicted==1) & (real==0)).sum()
    FN=((predicted==0) & (real==1)).sum()
    
    P=TP+FN
    N=TN+FP
      
    KS=(TP/P)-(FP/N)
    
    
    KS_all.append(KS)

# try out what cutoffs you get when you use F_beta scores with different values of betas [0.5 , 5]
# beta < 1 : you will get cutoff , which is high ( favours precision)
# beta > 1 : you will get cutoff , which is low (favours precision )
mycutoff=cutoffs[KS_all==max(KS_all)][0]
mycutoff
logr.intercept_
list(zip(x_train.columns,logr.coef_[0]))<jupyter_output><empty_output><jupyter_text>if you simply had to submit probability scores , you could do this<jupyter_code>test_score=logr.predict_proba(bd_test)[:,1]
pd.DataFrame(test_score).to_csv("mysubmission.csv",index=False)<jupyter_output><empty_output><jupyter_text>if you had to submit hardclasses , you can apply the cutoff obtained above and then submit<jupyter_code>test_classes=(test_score>mycutoff).astype(int)
pd.DataFrame(test_classes).to_csv("mysubmission.csv",index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13086849813949805, 'fraction_numerical': 0.012033884886390626, 'mean_word_length': 6.651120533010297, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '425018', 'license': 'MIT', 'n_tokens_mistral': 4530, 'n_tokens_neox': 4375, 'n_words': 917, 'path': 'intro-to-python-ml/week3/pratical_exercise_linear_models.ipynb', 'repo_name': 'FaizSOA/Faiz', 'size': '12631'}