<jupyter_start><jupyter_text>FRAUD DETECTIONIn this assignment, we have a data set for fraud detection. While being designed as an end-to-end machine learning project, the following questions will be answered respectively.1-) How is one hot encoding applied to data that the model does not see?2-) How to scale for related column in Labelencoding?(Digital conversion of data with the right effect)3-) What should we consider when doing a train test split on an Imbalance dataset?4-) How to create a validation dataset (that the model does not see) and make it ready to predict?5-) How to calculate ratio and change tHreshold with predict_proba method?<jupyter_code>pd.set_option('display.max_columns', None)
df = pd.read_csv("auto_insurance_csv.csv")
df<jupyter_output><empty_output><jupyter_text>Determining Null Values ​​and Filling These Values<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 41 columns):
 #   Column                       Non-Null Count  Dtype  
---  ------                       --------------  -----  
 0   Unnamed: 0                   1000 non-null   int64  
 1   months_as_customer           1000 non-null   int64  
 2   age                          865 non-null    float64
 3   policy_number                1000 non-null   int64  
 4   policy_bind_date             1000 non-null   object 
 5   policy_state                 1000 non-null   object 
 6   policy_csl                   127 non-null    object 
 7   policy_deductable            1000 non-null   int64  
 8   policy_annual_premium        1000 non-null   float64
 9   umbrella_limit               802 non-null    float64
 10  insured_zip                  1000 non-null   int64  
 11  insured_sex                  1000 non-null   object 
 12  insured_education_level      1000 non-null   object 
 13  insured_occupation [...]<jupyter_text>As seen above, there are null values ​​in the columns. These values ​​will be populated with the same values ​​for comparison with the models in the shared Auto Insurence Fraud notebook.<jupyter_code>#umbrella_limit kolonundaki null değerler 9999 sayısıyla doldurulmaktadır ve 
#sayısal değerler other isimli bir değişkende toplanmaktadır
df['umbrella_limit'] = df.umbrella_limit.fillna("9999")
df['umbrella_limit'] = df.umbrella_limit.astype(str)
umbrealla = df['umbrella_limit'].unique()
for umb in umbrealla:
    if (umb != '0.0') & (umb != '9999'):
        df['umbrella_limit'] = df['umbrella_limit'].str.replace(umb, 'other')

#police_report_available kolonundaki null değerler MISSING olarak değiştirilmektedir.
df['police_report_available'] = df.police_report_available.fillna("MISSING")

#policy_csl kolonundaki null değerler MISSING olarak değiştirilmektedir.
df['policy_csl'] = df.policy_csl.fillna("MISSING")

#policy_bind_date ve incident_date kolonundaki değerler tarih değerlerine dönüştürülür ve 
#birbirlerinden çıkarılarak claim_day_of_policy adında bir kolon oluşturulur.
df['policy_bind_date'] = pd.to_datetime(df['policy_bind_date'])
df['incident_date'] = pd.to_datetime(df['incident_date'])
df['claim_day_of_policy'] = (df.incident_date -  df.policy_bind_date).dt.days

#location_check adlı bir kolon oluşturulur. Bu kolon poliçenin hazırlandığı yer ve kazanın yaşandığı yerin birbiriyle 
#aynı olup olmadığını kontrol eden bool bir kolon oluşturur.
df['location_check'] = np.nan
df['location_check'] = np.where(df['policy_state'] == df['incident_state'], True, False)

#Buradan itibaren gruplamalar yapılmıştır.

#age kolonunun null değerleri 9999 değeriyle doldurulur ve age_group adında yeni bir kolon oluşturularak
#yaş değerleri 5'er yıllık gruplara bölünmüştür.
df['age'] = df.age.fillna(9999)
bin_labels = ['15-20', '21-25', '26-30', '31-35', '36-40', '41-45', '46-50', '51-55', '56-60', '61-65','9999']
bins = [15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 9999]
df['age_group'] = pd.cut(df['age'], bins = bins, labels = bin_labels, include_lowest = True)

#months_as_customer kolonu, months_as_customer_groups adında kişinin kaç aydır müşteri olduğunun
#gruplandığı bir kolona dönüştürülmüştür.
bins = [0, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500]
bin_labels = ['0-50','51-100','101-150','151-200','201-250','251-300','301-350','351-400','401-450','451-500']
df['months_as_customer_groups'] = pd.cut(df['months_as_customer'], bins = 10, labels = bin_labels, include_lowest= True)

#policy_annual_premium kolonu, policy_annual_premium_groups adında poliçenin kalitesinin gruplandığı bir kolona dönüşmüştür
bins = list(np.linspace(0,2500, 6, dtype = int))
bin_labels = ['very low', 'low', 'medium', 'high', 'very high']
df['policy_annual_premium_groups'] = pd.cut(df['policy_annual_premium'], bins = bins, labels=bin_labels)

bins = list(np.linspace(0,2000, 5, dtype = int))
bin_labels = ['0-500', '501-1000', '1001-1500', '1501-2000']
df['policy_deductable_group'] = pd.cut(df['policy_deductable'], bins = bins, labels = bin_labels)

#fraud_reported kolonuna elle bir label encoding işlemi uygulanmıştır.
df['fraud_reported'] = df['fraud_reported'].str.replace('Y', '1')
df['fraud_reported'] = df['fraud_reported'].str.replace('N', '0')
df['fraud_reported'] = df['fraud_reported'].astype(int)<jupyter_output><empty_output><jupyter_text>Visualization Here, the relationship of the columns that we have reconstructed by grouping their values ​​with the ***fraud_reported*** column is examined.<jupyter_code>plt.figure(figsize=(12,5))
sns.countplot(x=df['fraud_reported'],hue='age_group',data=df)
plt.figure(figsize=(12,5))
sns.countplot(x=df['fraud_reported'],hue='policy_annual_premium_groups',data=df)
plt.figure(figsize=(12,5))
sns.countplot(x=df['fraud_reported'],hue='months_as_customer_groups',data=df)
plt.figure(figsize=(12,5))
sns.countplot(x=df['fraud_reported'],hue='policy_deductable_group',data=df)<jupyter_output><empty_output><jupyter_text>- Considering the age groups, although the number of frauds among young people is high, it is seen that people aged 46 and above are more involved in fraud than expected.- Looking at the insurance package information, it is seen that those who buy medium-level packages are more likely to commit fraud. From the information in the same chart, it is also revealed that the majority of them prefer the mid-level package more.- The fraud rate is higher for those who have been a customer for 8 and 25 years during the month spent as a customer. The reason for this is the large number of people who became customers during this period.- Insurance deductible means the fee received before taking out insurance. The higher this fee, the more the insurance company pays after an accident. In this chart, the group that pays this fee at the lower limit is more involved in fraud. Multicollinearity problem is seen in the heatmap printed after the data is filled.<jupyter_code>plt.figure(figsize = (18, 12))

corr = df.corr()

sns.heatmap(data = corr, annot = True, fmt = '.2g', linewidth = 1)
plt.show()<jupyter_output><empty_output><jupyter_text>Multicollinearity In short, multicollinearity is the situation where there is a very high correlation between at least two variables that predict a variable. Since two variables with a very high relationship between them will be partially similar to each other, it would be appropriate to discard either of them.<jupyter_code>Image(url= "images/multicollinearity.png", width=600, height=600)<jupyter_output><empty_output><jupyter_text>In order to determine this numerically, Variance Inflation Factor (variance increase factor) should be calculated.  Variance Inflation Factor(VIF) VIF is an index that measures how much the variance of a variable to be estimated has increased due to ***collinearity***.- Collinearity defines the linear relationship between two variables. Unlike multicollinearity, it does not show multicollinearity.After calculating the VIF, the values ​​are analyzed as follows:- VIF=1 NO multicollinearity.- VIF=<5 low multicollinearity or partial correlation- VIF=>5 high multicollinearity or high korelasyon<jupyter_code>from statsmodels.stats.outliers_influence import variance_inflation_factor

#nümerik kolonların seçilmesi
num_cols0 = df[list(df.select_dtypes(include=['int64']))]

#nümerik kolonların VIF değerlerini göstermek için farklı bir dataFrame oluşturulması ve değerlerin yazdırılması
vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(num_cols0.values, i) for i in range(num_cols0.shape[1])]
vif_info['Column'] = num_cols0.columns
vif_info.sort_values('VIF', ascending=False)<jupyter_output><empty_output><jupyter_text>According to these results, there are ***infinite**** values ​​in 4 columns. This is called **Perfect Correlation**. The reason is that these 4 columns are highly related to each other. We can solve this problem by discarding the columns. First, let's show that the vehicle_claim, property_claim, and property_claim columns are equal to the total_claim_amount column.<jupyter_code>df["Total"] = df["vehicle_claim"] + df["property_claim"] + df["injury_claim"]
df.Total.sum()
df.total_claim_amount.sum()<jupyter_output><empty_output><jupyter_text>As can be seen here, discarding these 3 columns and leaving only the total_claim_amount column will solve the infinite problem. Any other columns deemed unnecessary will also be discarded.<jupyter_code>#multicollinearity düzenlemesi için atıldı
df = df.drop(['vehicle_claim', 'property_claim', 'injury_claim', 'Total', 'auto_year', 'insured_zip'], axis = 1)

#claim_day_of_policy kolonu oluşturulduğu için atıldı
df = df.drop(['policy_bind_date','incident_date'],axis=1)

#sadece null değerler var
df = df.drop(['_c39','Unnamed: 0'], axis = 1)

#bu kolonlarda gruplama yapıldığı için atıldılar.
df = df.drop(['age', 'months_as_customer', 'policy_deductable', 'policy_annual_premium'], axis = 1)

#policy_number kolonu modeli eğitirken önemsiz olduğu için atıldı
df = df.drop(['policy_number'], axis = 1)<jupyter_output><empty_output><jupyter_text>- When we print the VIF values ​​again, we see that the infinite problem in the total_claim_amount column is fixed and the columns with high VIF values ​​are discarded.- We also check again that the null values ​​are cleared.<jupyter_code>#nümerik kolonların VIF değerlerini göstermek için farklı bir dataFrame oluşturulması ve değerlerin yazdırılması
num_cols0 = df[list(df.select_dtypes(include=['int64']))]
vif_info = pd.DataFrame()
vif_info['VIF'] = [variance_inflation_factor(num_cols0.values, i) for i in range(num_cols0.shape[1])]
vif_info['Column'] = num_cols0.columns
vif_info.sort_values('VIF', ascending=False)
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>QUESTION 1 **How is one hot encoding applied to data that the model does not see?** In order to prepare the data for encoding, it is necessary to convert the data type ***object*** into ***category*** first. There are several reasons for this:- Category data type is much more suitable for memory optimization. This saves time when training the model.- The word order of a variable is not the same as the logical order. By converting to category and specifying an order on the categories, it will use the logical order instead of the dictionary order.<jupyter_code>#Seçilen kolonların category veri tipine dönüştürülmesi
for col in [ 'policy_state','umbrella_limit', 'policy_csl', 'insured_sex', 'insured_education_level', 'insured_occupation', 'insured_hobbies', 'insured_relationship',
            'incident_type', 'collision_type', 'incident_severity', 'authorities_contacted', 'incident_state', 'incident_city', 'incident_location',
             'property_damage',  'police_report_available', 'auto_make', 'auto_model', 'fraud_reported']:
    df[col] = df[col].astype('category')
#Kolon veri tipleri dönüştürüldü
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 33 columns):
 #   Column                        Non-Null Count  Dtype   
---  ------                        --------------  -----   
 0   policy_state                  1000 non-null   category
 1   policy_csl                    1000 non-null   category
 2   umbrella_limit                1000 non-null   category
 3   insured_sex                   1000 non-null   category
 4   insured_education_level       1000 non-null   category
 5   insured_occupation            1000 non-null   category
 6   insured_hobbies               1000 non-null   category
 7   insured_relationship          1000 non-null   category
 8   capital-gains                 1000 non-null   int64   
 9   capital-loss                  1000 non-null   int64   
 10  incident_type                 1000 non-null   category
 11  collision_type                1000 non-null   category
 12  incident_severity             1000 non-null   cat[...]<jupyter_text>When one hot encoding is applied on categorical data, a numerical irregularity occurs in the data in the train and test data sets. In addition, it can cause multicollinearity again. To overcome these situations, ***Pipeline*** will be used.  Pipeline - Pipeline is not actually a data science concept but a very basic computer architecture and systems programming concept. It allows multiple jobs to be run simultaneously in parallel.- The concept we define as work here is called ***process(task)*** with the system programming term and it is actually aimed to use the CPU as much as possible.- Different CPU sharing methods used depending on the characteristics of the processes are called job sequencing algorithms.- ***In the picture below, a pipeline structure that processes 5 different processes in parallel at t time is seen.***<jupyter_code>Image(url= "images/pipeline.jpg", width=600, height=600)<jupyter_output><empty_output><jupyter_text>- The Scikit Learn library has also created a Pipeline structure by taking this structure as an example. It allows the desired operations (StandardScaler, OneHotEncoder, LabelEncoder) to run parallel to each other by giving an object named ***column_transformer*** inside. It also prevents possible errors.- **Pipeline structure will be shown with answering question 2.**  QUESTION 2 **How to scale for the relevant column in Label Encoding? (Digital conversion of data with the right effect)** - Label Encoding is an encoding process that numbers categorical data from 0 to ***n*** since the model training process is performed on numerical data.- The biggest problem with this type of encoding is that each categorical data does not have an order or weight.- For this, ordinal data must be used. We will use the ***policy_annual_premium_groups*** column for this process.- In addition, while encoding, it does this according to the alphabetical order of the values ​​in the column. What we want is to be able to renumber in a logical order.- For this, we can do ***Ordinal Encoding*** and assign our values ​​manually.<jupyter_code>#Bu kolon ordinal değerler içermekte ve veri setinde ordinal encoding'e en uygun kolondur.
df[["policy_annual_premium_groups"]]<jupyter_output><empty_output><jupyter_text>The 5 values ​​in this column are also shown. If label encoding was done, numeric values ​​from 0 to 4 would be assigned to the first printed list, but since the data type is converted to categorical, the logical relationship between them is shown in the line below.<jupyter_code>df.policy_annual_premium_groups.unique()<jupyter_output><empty_output><jupyter_text>Manually ordinal encoding is done simply with the map() function. The same can be done from the scikit learn library.<jupyter_code>df["policy_annual_premium_groups_encoded"] = df["policy_annual_premium_groups"].map(
    {'very low':1, 'low':2, 'medium':3, 'high':4, 'very high':5})
#Veriler istediğimiz şekilde encode edildi
df[["policy_annual_premium_groups", "policy_annual_premium_groups_encoded"]]
#kategorik verileri içeren verilerin düşürülmesi
df = df.drop(['policy_annual_premium_groups'], axis = 1)<jupyter_output><empty_output><jupyter_text>The pipeline structure mentioned in the first question can now be created. First, a column_transformer object will be defined and printed into the StandardScaler and OneHotEncoder processes. For this, categorical and numeric columns must be printed first.<jupyter_code>#kategorik kolonların yazdırılması. burada policy_annual_premium_groups_encoded kolonu gözükmekte ama 
#zaten ordinal encoding yapıldığı için column_transformer'da işleme katılmayacaktır.
cat_cols = df.select_dtypes(include=['category'])
cat_cols.columns
#numerik kolonların yazdırılması.
num_cols = df.select_dtypes(include=['int64'])
num_cols.columns
#column_transformer nesnesinin oluşturulması. burada OneHotEncoder içerisindeki handle_unknown parametresi
#verilerin tekrar ederek shape error vermesi engellenecektir.
column_trans = make_column_transformer(
               (StandardScaler(), ['capital-gains', 'capital-loss',
                                   'incident_hour_of_the_day', 'number_of_vehicles_involved',
                                   'bodily_injuries', 'witnesses', 'total_claim_amount',
                                   'claim_day_of_policy'
                                   ]),
               (OneHotEncoder(handle_unknown="ignore"), ['policy_state', 'policy_csl', 'insured_sex', 'insured_education_level',
                                   'insured_occupation', 'insured_hobbies', 'insured_relationship',
                                   'incident_type', 'collision_type', 'incident_severity',
                                   'authorities_contacted', 'incident_state', 'incident_city',
                                   'incident_location', 'property_damage', 'police_report_available',
                                   'auto_make', 'auto_model', 'age_group',
                                   'months_as_customer_groups',
                                   'policy_deductable_group', 'umbrella_limit']),
               remainder="passthrough")<jupyter_output><empty_output><jupyter_text>Now we can move on to the model training phase. Logistic Regression was chosen for the first training phase.<jupyter_code>X = df.drop("fraud_reported", axis="columns")
y = df.fraud_reported<jupyter_output><empty_output><jupyter_text>When the column_trans object is fit_transformed, it prints the information that it holds a matrix with 31009 values ​​of (1000,1184).<jupyter_code>column_trans.fit_transform(X, y)
#train ve test veri setlerinin ayrılması
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 42)
#Mantıksal regresyon modelinin çağırılması
log_reg = LogisticRegression(random_state=42)
#pipeline yapısı kurulumu. bu şekilde column_trans içerisindeki çok sayıda veri hızlı bir şekilde eğitilecektir.
pipe_log_reg = make_pipeline(column_trans, log_reg)
#artık model yerine pipe değişkenini verebiliriz
pipe_log_reg.fit(X_train, y_train)
preds = pipe_log_reg.predict(X_test)

#Mantıksal regresyon skorunun yazdırılması
score = pipe_log_reg.score(X_test, y_test)
print(score*100)<jupyter_output>76.8<jupyter_text>The first model was trained by printing the last information.<jupyter_code>#evaluation metriklerinin yazdırılması ve heatmap
print(classification_report(y_test, preds))
cm = confusion_matrix(y_test, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.81      0.90      0.85       183
           1       0.60      0.42      0.49        67

    accuracy                           0.77       250
   macro avg       0.70      0.66      0.67       250
weighted avg       0.75      0.77      0.75       250<jupyter_text>QUESTION 3 **What should we consider when doing a train test split on an Imbalance dataset?** First of all, it should be checked whether there is an imbalance in the data to be examined.<jupyter_code>sns.countplot(df['fraud_reported'])<jupyter_output><empty_output><jupyter_text>Imbalance data are very common in ***fraud detection, cancer cell prediction, or intrusion detection in networks*** datasets. But most machine learning algorithms do not work correctly on imbalanced datasets. Therefore, an editing process is required. There are several methods for editing Imbalance datasets:- **Under sampling majority class**: It is the use of less of the values ​​of the majority class. But it is not recommended as there will be a lot of data loss.- **Over sampling minorty class by duplication**: It means copying the data of the minor cluster to be equal to the number of data of the major cluster.- **Over sampling minorty class using SMOTE**: It is over-sampling using the K-Nearest Neighbors algorithm. Here, instead of copying the same data, the minor cluster data is filled by taking their neighborhood.- **Ensemble Method**: The best of these models is selected by dividing the clusters into equal parts that can be divided (for example, in the data set we have, we can divide the major cluster 3 times to equal the minor cluster) and train the models.  UndersamplingUndersampling will be attempted first. For this, functions from the ***imblearn*** library will be called and how the results change will be examined.<jupyter_code>#undersample yapmak için ayırdığımız X_train ve y_train ögelerini tekrar ayırırız.
rus = RandomUnderSampler(random_state=42, replacement=True)
X_under, y_under = rus.fit_resample(X_train, y_train)

X_train_u, X_test_u, y_train_u, y_test_u = train_test_split(X_under, y_under, test_size = 0.3, random_state = 42)
#kontrol sonucu majör ve minör kümelerin düşük bir değerde eşitlendiği görülmektedir.
chck2 = pd.DataFrame()
chck2['fraud_reported'] = y_train_u

sns.countplot(chck2['fraud_reported'])<jupyter_output><empty_output><jupyter_text>For comparison, the logistic regression model was trained again.<jupyter_code>#artık model yerine pipe değişkenini verebiliriz
pipe_log_reg.fit(X_train_u, y_train_u)
preds = pipe_log_reg.predict(X_test_u)

#mantıksal regresyon skorunun yazdırılması
score = pipe_log_reg.score(X_test_u, y_test_u)
print(score*100)<jupyter_output>79.62962962962963<jupyter_text>As a result of training the model, the fraud prediction rate and overall score increased, but the reliability rate decreased as the number of data of the model decreased. So oversampling should also be tried.<jupyter_code>#evaluation metriklerinin yazdırılması ve heatmap
print(classification_report(y_test_u, preds))
cm = confusion_matrix(y_test_u, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.81      0.78      0.80        55
           1       0.78      0.81      0.80        53

    accuracy                           0.80       108
   macro avg       0.80      0.80      0.80       108
weighted avg       0.80      0.80      0.80       108<jupyter_text>Oversampling RandomOverSampler from the imblearn library will be used here.<jupyter_code>ros = RandomOverSampler(random_state=42)
X_over, y_over = ros.fit_resample(X_train, y_train)
X_train_o, X_test_o, y_train_o, y_test_o = train_test_split(X_over, y_over, test_size = 0.3, random_state = 42)
chck2 = pd.DataFrame()
chck2['fraud_reported'] = y_train_o

sns.countplot(chck2['fraud_reported'])
#artık model yerine pipe değişkenini verebiliriz
pipe_log_reg.fit(X_train_o, y_train_o)
preds = pipe_log_reg.predict(X_test_o)

#Mantıksal regresyon skorunun yazdırılması
score = pipe_log_reg.score(X_test_o, y_test_o)
print(score*100)
#evaluation metriklerinin yazdırılması ve heatmap
print(classification_report(y_test_o, preds))
cm = confusion_matrix(y_test_o, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.94      0.86      0.90       176
           1       0.86      0.95      0.90       166

    accuracy                           0.90       342
   macro avg       0.90      0.90      0.90       342
weighted avg       0.90      0.90      0.90       342<jupyter_text>QUESTION 4 **How ​​to create a validation dataset (which the model does not see) and how to make it ready to predict?** Creating a validation dataset means dividing the dataset into three as train, validation and test. First, the train dataset is arranged and the model is selected on the validation dataset. Then the most suitable model is tried again on the test dataset and the results are compared.- It is sufficient to call train_test_split 2 times to create a validation dataset. Splitting datasets was determined according to the image.<jupyter_code>Image(url= "images/val.png", width=600, height=600)
#verisetinin 3 farklı gruba bölünmesi
X_train_val, X_test, y_train_val, y_test = train_test_split(X_over, y_over, test_size = 0.4)
relative_train_size = 0.6 / ((0.4/0.5) + 0.6)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val,
                                                  train_size = relative_train_size, test_size = 1-relative_train_size)
X_val
y_val<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>#artık model yerine pipe değişkenini verebiliriz
pipe_log_reg.fit(X_train, y_train)
preds = pipe_log_reg.predict(X_val)

#Mantıksal regresyon skorunun yazdırılması
score_logreg = pipe_log_reg.score(X_val, y_val)
print(score_logreg)
print(classification_report(y_val, preds))
cm = confusion_matrix(y_val, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.77      0.84      0.80       190
           1       0.83      0.76      0.79       201

    accuracy                           0.80       391
   macro avg       0.80      0.80      0.80       391
weighted avg       0.80      0.80      0.80       391<jupyter_text>Random Forest<jupyter_code>random_forest = RandomForestClassifier(criterion= 'entropy', 
                                  max_depth= 10, 
                                  max_features= 'sqrt', 
                                  min_samples_leaf= 1, 
                                  min_samples_split= 3, 
                                  n_estimators= 140,
                                  random_state=42)
pipe_random_forest = make_pipeline(column_trans, random_forest)
pipe_random_forest.fit(X_train, y_train)
preds = pipe_random_forest.predict(X_val)

score_ranfor = pipe_random_forest.score(X_val, y_val)
print(score_ranfor)
print(classification_report(y_val, preds))
cm = confusion_matrix(y_val, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.78      0.88      0.83       190
           1       0.88      0.77      0.82       201

    accuracy                           0.82       391
   macro avg       0.83      0.83      0.82       391
weighted avg       0.83      0.82      0.82       391<jupyter_text>Decision Tree<jupyter_code>dtc = DecisionTreeClassifier(random_state=42)
pipe_dtc = make_pipeline(column_trans, dtc)
pipe_dtc.fit(X_train, y_train)
preds = pipe_dtc.predict(X_val)

score_dtc = pipe_dtc.score(X_val, y_val)
print(score_dtc)
print(classification_report(y_val, preds))
cm = confusion_matrix(y_val, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.91      0.89      0.90       190
           1       0.90      0.92      0.91       201

    accuracy                           0.91       391
   macro avg       0.91      0.91      0.91       391
weighted avg       0.91      0.91      0.91       391<jupyter_text>K-Neighbours Classifier<jupyter_code>knn = KNeighborsClassifier()
pipe_knn = make_pipeline(column_trans, knn)
pipe_knn.fit(X_train, y_train)
preds = pipe_knn.predict(X_val)

score_knn = pipe_knn.score(X_val, y_val)
print(score_knn)
print(classification_report(y_val, preds))
cm = confusion_matrix(y_val, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.67      0.71      0.69       190
           1       0.71      0.67      0.69       201

    accuracy                           0.69       391
   macro avg       0.69      0.69      0.69       391
weighted avg       0.69      0.69      0.69       391<jupyter_text>Support Vector Machine<jupyter_code>svc = SVC(random_state=42)
pipe_svc = make_pipeline(column_trans, svc)
pipe_svc.fit(X_train, y_train)
preds = pipe_svc.predict(X_val)

score_svm = pipe_svc.score(X_val, y_val)
print(score_svm)
print(classification_report(y_val, preds))
cm = confusion_matrix(y_val, preds)
sns
sns.heatmap(cm, annot=True)<jupyter_output>precision    recall  f1-score   support

           0       0.80      0.89      0.84       190
           1       0.88      0.79      0.83       201

    accuracy                           0.84       391
   macro avg       0.84      0.84      0.84       391
weighted avg       0.84      0.84      0.84       391<jupyter_text>After training 5 different models, it seems that Decision Tree gives the best results.<jupyter_code>models = pd.DataFrame({
    'Model' : ['Logistic Regression', 'KNN', 'Decision Tree', 'Random Forest','SVM'],
    'Score' : [score_logreg, score_knn, score_dtc, score_ranfor, score_svm]
})

models.sort_values(by = 'Score', ascending = False)<jupyter_output><empty_output><jupyter_text>QUESTION 5 **predict_proba metoduyla oran nasıl hesaplanır ve treshold nasıl değiştirilir?** In a classification problem, we may decide to directly estimate the class values.Alternatively, it may be more flexible to estimate probabilities for each class instead. This is because it provides the ability to choose and even calibrate the threshold for how to interpret the predicted probabilities.The difference between predict() and predict_proba() functions; In your studies using methods such as regression, classification, clustering, the predicted label information can be obtained with the predict function, if you want to obtain the probability of the observations belonging to the classes in classification problems, the predict_proba function should be used.- The default threshold for interpreting probabilities into class labels is 0.5, and setting this hyperparameter is called ***threshold moving***.- The Threshold value can be determined using the ROC curve.Here, the logical regression model is chosen as an example.<jupyter_code>#threshold değeri rastgele seçilerek predict_proba metodunun yazdırdığı olasılık değerleriyle karşılaştırılır.
threshold = 0.25
predictions = (pipe_log_reg.predict_proba(X_test)[:, 1] >= threshold).astype(int)
#mantıksal regresyon için test dataseti değerleri ölçülür. yukarıda train işlemleri tamamlanmıştı
preds = pipe_log_reg.predict(X_test)
score_logreg = pipe_log_reg.score(X_test, y_test)
print(score_logreg)
#threshold belirlenmeden önceki değerler ve threshold belirlendikten sonraki değerler yazdırılır.
print('Valuation for test data only:')
print(classification_report(y_test, preds))
print("----------------------------------------------------------------------")
print('Valuation for test data only  (new_threshold):')
print(classification_report(y_test, predictions))<jupyter_output>Valuation for test data only:
              precision    recall  f1-score   support

           0       0.80      0.83      0.81       226
           1       0.82      0.79      0.81       230

    accuracy                           0.81       456
   macro avg       0.81      0.81      0.81       456
weighted avg       0.81      0.81      0.81       456

----------------------------------------------------------------------
Valuation for test data only  (new_threshold):
              precision    recall  f1-score   support

           0       0.85      0.68      0.76       226
           1       0.74      0.88      0.80       230

    accuracy                           0.78       456
   macro avg       0.79      0.78      0.78       456
weighted avg       0.79      0.78      0.78       456
{'fraction_non_alphanumeric': 0.08659366862234494, 'fraction_numerical': 0.03633169140024472, 'mean_word_length': 3.567129961312509, 'pattern_counts': {'":': 0, '<': 119, '<?xml version=': 0, '>': 120, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 64, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '212806', 'license': 'MIT', 'n_tokens_mistral': 11034, 'n_tokens_neox': 9937, 'n_words': 3079, 'path': 'enerjisa_bootcamp_homework_3.ipynb', 'repo_name': 'kagan-gungor/auto-insurance-fraud-detection', 'size': '31873'}