<jupyter_start><jupyter_text>Preprocessing and data cleaning<jupyter_code>#for fininding missing values
#this headmap is illustrating the missing values with yellow bars

plt.figure(figsize=(20,12))
sns.heatmap(training_data.isnull(),yticklabels=False,cbar=False,cmap='viridis')<jupyter_output><empty_output><jupyter_text>Missing value handling We are going to use different approches with missing values:1. Removing the column having more than 10,000 missing values (**Ref)2. Removing the column having 80% missing values (**Self intuition)3. Keeping all the features4. Later, we will try to implement some feature engineering **For the rest of the missing values, we are replacing them with their mean() for now (**Ref)   First Approach<jupyter_code>missing_data = training_data.isna().sum().to_frame()

plt.figure(figsize=(10,7))
ax= sns.distplot(missing_data[0],kde = False,bins = 20,color='b')
ax.set(xlabel='Missing values', ylabel='Number of features')
plt.show()
missing_data[missing_data[0]>10000]<jupyter_output><empty_output><jupyter_text>we are dropping the columns that have more than 10,000 missing values<jupyter_code>#sample_data varible holds the data for approach one
temp = "ab_000 ad_000 bk_000 bl_000 bm_000 bn_000 bo_000 bp_000 bq_000 br_000 cf_000 cg_000 ch_000 co_000 cr_000 ct_000 cu_000 cv_000 cx_000 cy_000 cz_000 da_000 db_000 dc_000 ec_00"
temp = temp.split()

sample_data = training_data.drop(temp,axis=1)
sample_data.head()
#after dropping columns having more than 10,000 missing values
plt.figure(figsize=(20,12))
sns.heatmap(sample_data.isnull(),yticklabels=False,cbar=False,cmap='viridis')<jupyter_output><empty_output><jupyter_text>Here we can see much improvement in the heatmapFor the rest of the values we are going to replace them with their mean() -- (**Ref)<jupyter_code>sample_data.fillna(sample_data.mean(),inplace=True)<jupyter_output><empty_output><jupyter_text>After mean(), the heatmap for missing values will look like<jupyter_code>plt.figure(figsize=(20,12))
sns.heatmap(sample_data.isnull(),yticklabels=False,cbar=False,cmap='viridis')<jupyter_output><empty_output><jupyter_text>So we have handled the missing value problem in this approachNow We need to handle the classification variables(class) by taking a dummy value instead of a string<jupyter_code>#as all the other values are numerical except Class column so we can replace them with 1 and 0

sample_data = sample_data.replace('neg',0)
sample_data = sample_data.replace('pos',1)

sample_data.head(15)<jupyter_output><empty_output><jupyter_text>Testing data preprocessing and implementation<jupyter_code>testing_data = pd.read_csv("../Data/aps_failure_test_set.csv",na_values="na")
testing_data.head()
plt.figure(figsize=(20,12))
sns.heatmap(testing_data.isnull(),yticklabels=False,cbar=False,cmap='viridis')<jupyter_output><empty_output><jupyter_text>Here we are using the same preprocessing method like the training data<jupyter_code>missing_testing_data = testing_data.isna().sum().to_frame()

plt.figure(figsize=(10,7))
ax= sns.distplot(missing_testing_data[0],kde = False,bins = 20,color='b')
ax.set(xlabel='Missing values', ylabel='Number of features')
plt.show()
missing_testing_data[missing_testing_data[0]>10000]<jupyter_output><empty_output><jupyter_text>Here the main issue is , if missing data over 10,000 is removed from testing data , then columns remain is 163, but in training it is 145So, we removed the same columns as we have removed in the training data<jupyter_code>temp =missing_data[missing_data[0]>10000]
temp = temp.index

sample_testing_data = testing_data.drop(temp,axis=1)
sample_testing_data.head()
#after dropping columns having more than 10,000 missing values
plt.figure(figsize=(20,12))
sns.heatmap(sample_testing_data.isnull(),yticklabels=False,cbar=False,cmap='viridis')
sample_testing_data.fillna(sample_testing_data.mean(),inplace=True)
plt.figure(figsize=(20,12))
sns.heatmap(sample_testing_data.isnull(),yticklabels=False,cbar=False,cmap='viridis')
#as all the other values are numerical except Class column so we can replace them with 1 and 0

sample_testing_data = sample_testing_data.replace('neg',0)
sample_testing_data = sample_testing_data.replace('pos',1)

sample_testing_data.head(15)<jupyter_output><empty_output><jupyter_text>Model Implementation<jupyter_code>#Training data
X_train = sample_data.drop('class',axis=1)
y_train = sample_data['class']
#testing data

X_test = sample_testing_data.drop('class',axis=1)
y_test = sample_testing_data['class']
from sklearn.linear_model import LogisticRegression
logmodel = LogisticRegression()
#fitting the data
logmodel.fit(X_train,y_train)
prediction = logmodel.predict(X_test)
from sklearn.metrics import classification_report
print(classification_report(y_test,prediction))<jupyter_output>precision    recall  f1-score   support

           0       0.99      1.00      0.99     15625
           1       0.82      0.63      0.71       375

   micro avg       0.99      0.99      0.99     16000
   macro avg       0.90      0.82      0.85     16000
weighted avg       0.99      0.99      0.99     16000<jupyter_text>Here we can see, Negative class prediction is about 99% correct , where Positive class prediction guarantees upto 82%Now, for the cost<jupyter_code>from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_test,prediction).ravel()
cost = 10*fp+500*fn
values = {'Score':[cost],'Number of Type 1 faults':[fp],'Number of Type 2 faults':[fn]}
pd.DataFrame(values)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10349194861588565, 'fraction_numerical': 0.04758458476569567, 'mean_word_length': 6.709902370990237, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '408565', 'license': 'MIT', 'n_tokens_mistral': 1996, 'n_tokens_neox': 1793, 'n_words': 452, 'path': 'Codes/.ipynb_checkpoints/Approach 1 (Logistic Regression) - With testing data-checkpoint.ipynb', 'repo_name': 'masudurHimel/ML---APS-Failure-at-Scania-Trucks-Data-Set', 'size': '5527'}