<jupyter_start><jupyter_text>This project is about business. the idea in this project is to predict the probability of a click being fraud or not in the chiness company by looking  features like ip address operating system, device type, time of click...<jupyter_code>import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
from scipy.stats import zscore
%matplotlib inline
data=pd.read_csv("C:\\Users\\marce\\OneDrive\\Ambiente de Trabalho\\DataScience\\Tree\\talking_data.csv")<jupyter_output><empty_output><jupyter_text>Explore the dataset for anomalies and missing values and take corrective actions if necessary.*<jupyter_code>print(data.shape)
print (data.columns)
data.info() 
data.describe()<jupyter_output>(100000, 8)
Index(['ip', 'app', 'device', 'os', 'channel', 'click_time', 'attributed_time',
       'is_attributed'],
      dtype='object')
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100000 entries, 0 to 99999
Data columns (total 8 columns):
 #   Column           Non-Null Count   Dtype 
---  ------           --------------   ----- 
 0   ip               100000 non-null  int64 
 1   app              100000 non-null  int64 
 2   device           100000 non-null  int64 
 3   os               100000 non-null  int64 
 4   channel          100000 non-null  int64 
 5   click_time       100000 non-null  object
 6   attributed_time  227 non-null     object
 7   is_attributed    100000 non-null  int64 
dtypes: int64(6), object(2)
memory usage: 6.1+ MB<jupyter_text>As we can see in the information, we have 6 numerical variable and 2 categorical. Also100000 row and  8 column<jupyter_code>print (data.isnull().sum())<jupyter_output>ip                     0
app                    0
device                 0
os                     0
channel                0
click_time             0
attributed_time    99773
is_attributed          0
dtype: int64<jupyter_text>we have  99773 missing value in attributed_time .<jupyter_code># les't drop the column attributed_time by creating a new variable called data1

data1=data.drop('attributed_time',axis=1)
print (data1.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 100000 entries, 0 to 99999
Data columns (total 7 columns):
 #   Column         Non-Null Count   Dtype 
---  ------         --------------   ----- 
 0   ip             100000 non-null  int64 
 1   app            100000 non-null  int64 
 2   device         100000 non-null  int64 
 3   os             100000 non-null  int64 
 4   channel        100000 non-null  int64 
 5   click_time     100000 non-null  object
 6   is_attributed  100000 non-null  int64 
dtypes: int64(6), object(1)
memory usage: 5.3+ MB
None<jupyter_text>Now we can seewe do not have the colum attributed_time<jupyter_code>data1 # by looking the data, we can see that, we have no NaN value anymore.
# let examine the column click_time because it was the only one object type that we have as you can see in the data1.info()

print (data1['click_time'].unique())<jupyter_output>['2017-11-07 09:30:38' '2017-11-07 13:40:27' '2017-11-07 18:05:24' ...
 '2017-11-09 04:21:28' '2017-11-07 09:33:14' '2017-11-09 13:25:41']<jupyter_text>vizualize data<jupyter_code># so, now let's do some plot and see the relationship between variables
sns.scatterplot(x='ip',y='device',hue='is_attributed',data=data1)
sns.pairplot(data1, hue="is_attributed", diag_kind = 'hist', palette="husl")<jupyter_output><empty_output><jupyter_text>Graphs show us that, almost 90% of our data is 0, mean that we do not have fraud. (but we cannot afirm that . we do not have fraud because the data can be wrong)Also we can see that, we do not have much relationship between variable because the data have the same result. (zero)this kind of observation can tell us that, probabily our data is unbalanced   Do the correlation between my variable in my data<jupyter_code>corr = data1.corr()
corr<jupyter_output><empty_output><jupyter_text>As we can see in the correlation table, we do not have a very good correlationship between variables, the only good corelactionship that we have is corelationship between DEVICE and OS. we can say that, these two varible have a strong correlationship  Heatmap to get more elivible result about corelation between variable<jupyter_code>plt.figure(figsize=(20,10))
sns.heatmap(corr,annot=True)<jupyter_output><empty_output><jupyter_text>As we can see in the map, we have some corelation between variable almost zero. That's mean does variable do not have a  important impact in my analizesfor example correlaction between ip and os is almost zero, but I will keep this variable in my analizesbecause it has a good corelaction with others variable  See if the data is balance or not by ploting the countplot<jupyter_code>sns.countplot(x='is_attributed', data=data1, palette='Set3')<jupyter_output><empty_output><jupyter_text>As we can see, the data is unbalanced because we have more than 90% of data as 0we can say that The vast majority of the transactions will be in the “Not-Fraud” class and a verysmall minority will be in the “Fraud” class.  2. Which column has maximum number of unique values present among all the available columns<jupyter_code>categorical_columns = ['ip','app','device','os','channel','click_time','is_attributed']
#Print frequency of categories
for col in categorical_columns:
    print (data1[col].value_counts())<jupyter_output>5348      669
5314      616
73487     439
73516     399
53454     280
         ... 
125982      1
5153        1
177205      1
60478       1
92122       1
Name: ip, Length: 34857, dtype: int64
3      18279
12     13198
2      11737
9       8992
15      8595
       ...  
407        1
145        1
271        1
398        1
92         1
Name: app, Length: 161, dtype: int64
1       94338
2        4345
0         541
3032      371
3543      151
        ...  
420         1
37          1
549         1
2980        1
347         1
Name: device, Length: 100, dtype: int64
19     23870
13     21223
17      5232
18      4830
22      4039
       ...  
116        1
142        1
137        1
185        1
127        1
Name: os, Length: 130, dtype: int64
280    8114
245    4802
107    4543
477    3960
134    3224
       ... 
261       1
498       1
465       1
474       1
451       1
Name: channel, Length: 161, dtype: int64
2017-11-08 12:01:02    7
2017-11-07 05:00:11    6
2017-11-07 04:36:16    6
2017-11[...]<jupyter_text>The column that has maximum number of unique values present  is the click_time column. It has 80350 UV  Use an appropriate technique to get rid of all the apps that are very rare (say which comprise of less than 20% clicks) and plot the rest..*<jupyter_code>data1['app'].value_counts() 
app_count=data1['app'].value_counts()
# create a variable app_add to have all app that have > 20 of click.
#app_count[app_count<50].index
app_add=app_count[app_count>20].index
#app_count[app_count>50]
app_add
# create a variable app_remove to remove all app that have < 20 of click.
app_remove=app_count[app_count<20].index
app_remove
# create a list inicially empty and write a loop for to add the new value (app_add) into it. 
clean_data=[]
for app in app_add:
    clean_data.append (data1.loc[data1['app']==app])
    
clean_data
#  create a final_data to have my clean_data. (data which has >20% of click app)


final_data=pd.concat(clean_data,axis=0)
print (final_data.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 99546 entries, 5 to 98446
Data columns (total 7 columns):
 #   Column         Non-Null Count  Dtype 
---  ------         --------------  ----- 
 0   ip             99546 non-null  int64 
 1   app            99546 non-null  int64 
 2   device         99546 non-null  int64 
 3   os             99546 non-null  int64 
 4   channel        99546 non-null  int64 
 5   click_time     99546 non-null  object
 6   is_attributed  99546 non-null  int64 
dtypes: int64(6), object(1)
memory usage: 6.1+ MB
None<jupyter_text>4. By using Pandas derive new features such as - 'day_of_week' , 'day_of_year' , 'month' , and 'hour' as float/int datatypes using the 'click_time' column . Add the newly derived columns in original dataset<jupyter_code>data['click_time'] # see what we have in the column click_time. and as we can see the dtpy is a object.
pd.to_datetime(final_data['click_time']) # using the datetime from pandas labrary, convert the object type into datetime 
# create a variable call convert_final_data as datetime and add it into my final_data
final_data['convert_final_data'] = pd.to_datetime(final_data['click_time'])
print (final_data.info())
# now we have the final_data_year let's use the lambda and apply function from pandas labrary to extrate the day, month and
# year in the numeical dtype.

final_data['final_data_month'] = final_data['convert_final_data'].apply(lambda x: x.month)
final_data['final_data_day'] = final_data['convert_final_data'].apply(lambda x: x.day)
final_data['final_data_year'] = final_data['convert_final_data'].apply(lambda x: x.year)
print(final_data)
final_data.info()
# As we can see, we have all data that we will use to do prediction the numerical type, so let's remove others columns
# that is not usifull in our analyses.

final_data = final_data.drop(['click_time','convert_final_data'],axis=1)
final_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 99546 entries, 5 to 98446
Data columns (total 9 columns):
 #   Column            Non-Null Count  Dtype
---  ------            --------------  -----
 0   ip                99546 non-null  int64
 1   app               99546 non-null  int64
 2   device            99546 non-null  int64
 3   os                99546 non-null  int64
 4   channel           99546 non-null  int64
 5   is_attributed     99546 non-null  int64
 6   final_data_month  99546 non-null  int64
 7   final_data_day    99546 non-null  int64
 8   final_data_year   99546 non-null  int64
dtypes: int64(9)
memory usage: 7.6 MB<jupyter_text>5. Divide the data into training and testing subsets into 80:20 ratio(Train_data = 80% , Testing_data = 20%) and check the average download rates('is_attributed') for train and test data, scores should be comparable.**<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn import preprocessing 

# let's create variable x and y where the y will be the dependent variable and x independent variable
x=final_data.drop(['is_attributed'], axis=1)
y=final_data['is_attributed']
X_train,x_test,y_train,y_test=train_test_split(x,y,train_size=0.8,test_size=0.2,random_state=10)
from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier()
tree.fit(X_train, y_train)
y_pred_tree = tree.predict(x_test)
print(X_train.shape)
print (y_train.shape)
print (x_test.shape)
print (y_test.shape)
conda install -c conda-forge imbalanced-learn
from sklearn.metrics import balanced_accuracy_score
from imblearn.metrics import geometric_mean_score

print("Decision tree classifier performance:")
print(
    f"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_tree):.2f} - "
    f"Geometric mean {geometric_mean_score(y_test, y_pred_tree):.2f}")
import seaborn as sns
from sklearn.metrics import plot_confusion_matrix

sns.set_context("poster")

disp = plot_confusion_matrix(tree, x_test, y_test, colorbar=False)
_ = disp.ax_.set_title("Decision tree")<jupyter_output><empty_output><jupyter_text>Apply XGBoostClassifier with default parameters on training data and make first 10 prediction for Test data. NOTE: Use y_pred = model.predict_proba(X_test) since we need probabilities to compute AUC.**<jupyter_code>from sklearn.tree import DecisionTreeClassifier
import xgboost as xgb
from sklearn.metrics import mean_squared_error
from sklearn.metrics import classification_report,confusion_matrix


xgcl = xgb.XGBClassifier(n_estimators=10 ,criterion = 'gini', class_weight=None, max_depth = None, min_samples_leaf=1)
xgcl.fit(X_train, y_train)
y_pred = xgcl.predict(x_test)

print("XGBClassifier classifier performance:")
print(
    f"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred):.2f} - "
    
)
print("XGBClassifier classifier performance:")
print(
    f"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred):.2f} - "
)
fig, axs = plt.subplots(ncols=2, figsize=(10, 5))
plot_confusion_matrix(xgcl, x_test, y_test, ax=axs[0], colorbar=False)
axs[0].set_title("XGBoost")

plot_confusion_matrix(balanced_bagging, x_test, y_test, ax=axs[1], colorbar=False)
axs[1].set_title("Balanced XGBoost")

fig.tight_layout()
# Random forest is another popular ensemble method and it is usually
# outperforming bagging. Here, we used a vanilla random forest and its balanced
# counterpart in which each bootstrap sample is balanced.

# %%
from sklearn.ensemble import RandomForestClassifier
from imblearn.ensemble import BalancedRandomForestClassifier

rf = RandomForestClassifier(n_estimators=10, random_state=0)
brf = BalancedRandomForestClassifier(n_estimators=10, random_state=0)

rf.fit(X_train, y_train)
brf.fit(X_train, y_train)

y_pred_rf = rf.predict(x_test)
y_pred_brf = brf.predict(x_test)
print("Random Forest classifier performance:")
print(
    f"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_rf):.2f} - "
    f"Geometric mean {geometric_mean_score(y_test, y_pred_rf):.2f}"
)
print("Balanced Random Forest classifier performance:")
print(
    f"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_brf):.2f} - "
    f"Geometric mean {geometric_mean_score(y_test, y_pred_brf):.2f}")<jupyter_output>Random Forest classifier performance:
Balanced accuracy: 0.56 - Geometric mean 0.34
Balanced Random Forest classifier performance:
Balanced accuracy: 0.89 - Geometric mean 0.89<jupyter_text><jupyter_code>xgcl = xgb.XGBClassifier(n_estimators=10 ,criterion = 'gini', class_weight=None, max_depth = None, min_samples_leaf=1)
xgcl.fit(X_train, y_train)
y_pred = xgcl.predict_proba(x_test)
from sklearn.datasets import make_blobs
X, y = make_blobs(n_samples=10, centers=3, n_features=2,random_state=0)

# ===================

xgcl.fit(X, y) # traing data

#=================

for i in range(10):
    print(X[i], y_pred[i]) # prediction for the 10 first element<jupyter_output>[12:52:24] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:541: 
Parameters: { criterion, min_samples_leaf, scale_pos_weight } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.  Or some parameters are not used but slip through this
  verification. Please open an issue if you find above cases.


[12:52:24] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
[1.12031365 5.75806083] [0.97740114 0.02259888]
[1.7373078  4.42546234] [0.97740114 0.02259888]
[2.36833522 0.04356792] [0.97740114 0.02259888]
[0.87305123 4.71438583] [0.97740114 0.02259888]
[-0.66246781  2.17571724] [0.44066435 0.55933565]
[0.74285061 1.46351659] [...]<jupyter_text>On evaluating the predictions made by the model what is the AUC/ROC score with default hyperparameters<jupyter_code>from sklearn.model_selection import cross_val_score, KFold
scores = cross_val_score(xgcl, X_train, y_train, cv=5)
print("Mean cross-validation score: %.2f" % scores.mean())

print("Accuracy: %.2f%% (%.2f%%)" % (scores.mean()*100, scores.std()*100))<jupyter_output>[11:47:32] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:541: 
Parameters: { criterion, min_samples_leaf } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.  Or some parameters are not used but slip through this
  verification. Please open an issue if you find above cases.


[11:47:32] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
[11:47:32] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:541: 
Parameters: { criterion, min_samples_leaf } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.[...]<jupyter_text>8. Compute feature importance score and name the top 5 features/columns<jupyter_code>import xgboost
# Plot the top 7 features
xgboost.plot_importance(xgcl, max_num_features=5)
# Show the plot
plt.show()<jupyter_output><empty_output><jupyter_text>The XGBoost  model tells us that the ip is the most important feature of the others  Apply BaggingClassifier with base_estimator LogisticRegression and compute AUC/ROC score.<jupyter_code>from sklearn.ensemble import BaggingClassifier
from imblearn.ensemble import BalancedBaggingClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
#====================================

pipeline = make_pipeline(StandardScaler(), 
                        LogisticRegression(random_state=1))
#=========================================

bagging = BaggingClassifier(base_estimator=pipeline, random_state=0)
balanced_bagging = BalancedBaggingClassifier(base_estimator=pipeline, random_state=0)

bagging.fit(X_train, y_train)
balanced_bagging.fit(X_train, y_train)

y_pred_bc = bagging.predict(x_test)
y_pred_bbc = balanced_bagging.predict(x_test)
print("Bagging classifier performance:")
print(
    f"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_bc):.2f} - "
    f"Geometric mean {geometric_mean_score(y_test, y_pred_bc):.2f}"
)
print("Balanced Bagging classifier performance:")
print(
    f"Balanced accuracy: {balanced_accuracy_score(y_test, y_pred_bbc):.2f} - "
    f"Geometric mean {geometric_mean_score(y_test, y_pred_bbc):.2f}"
)<jupyter_output>Bagging classifier performance:
Balanced accuracy: 0.50 - Geometric mean 0.00
Balanced Bagging classifier performance:
Balanced accuracy: 0.80 - Geometric mean 0.80<jupyter_text>On the basis of AUC/ROC score which one will you choose from BaggingClassifier and XGBoostClassifier and why?What does AUC/ROC score signifies?<jupyter_code>from sklearn.model_selection import cross_val_score, KFold
scores = cross_val_score(xgcl, X_train, y_train, cv=5)
print("Mean cross-validation score: %.2f" % scores.mean())

print("Accuracy for XGBoostClassifier: %.2f%% (%.2f%%)" % (scores.mean()*100, scores.std()*100))


#==========================================
scores1 = cross_val_score(bagging, X_train, y_train, cv=5)
print("Mean cross-validation score: %.2f" % scores1.mean())

print("Accuracy for BaggingClassifier : %.2f%% (%.2f%%)" % (scores1.mean()*100, scores1.std()*100))<jupyter_output>[12:00:02] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:541: 
Parameters: { criterion, min_samples_leaf } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.  Or some parameters are not used but slip through this
  verification. Please open an issue if you find above cases.


[12:00:02] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
[12:00:02] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:541: 
Parameters: { criterion, min_samples_leaf } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.[...]
{'fraction_non_alphanumeric': 0.1002424422344268, 'fraction_numerical': 0.06417297511256247, 'mean_word_length': 4.166666666666667, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '301368', 'license': 'Unlicense', 'n_tokens_mistral': 7484, 'n_tokens_neox': 6571, 'n_words': 2051, 'path': 'projects/Bagging and Boosting ensemble_project.ipynb', 'repo_name': 'MarceloD02/Marcelo-Goncalves-Portfolio', 'size': '20211'}