<jupyter_start><jupyter_text>remove outliers using the inter-quartile range<jupyter_code>df = pd.read_csv('diabetes.csv')
df.describe()<jupyter_output><empty_output><jupyter_text>insulin looks like it might have some outliers - mean, quartlies a lot smaller than the max value - let's try to remove these outliers to check if they help with model performance  calculate the inter-quartile range- In descriptive statistics, the interquartile range (IQR), also called the midspread or middle 50%, or technically H-spread, is a measure of statistical dispersion, being equal to the difference between 75th and 25th percentiles, or between upper and lower quartiles, IQR = Q3 −  Q1. In other words, the IQR is the first quartile subtracted from the third quartile; these quartiles can be clearly seen on a box plot on the data. (https://en.wikipedia.org/wiki/Interquartile_range)<jupyter_code># calculate interquartile range
q25, q75 = np.percentile(df['Insulin'], 25), np.percentile(df['Insulin'], 75)
iqr = q75 - q25

iqr<jupyter_output><empty_output><jupyter_text>let's remove all observations that have insulin higher than 127.25<jupyter_code>df2 = df[df['Insulin'] <= 127.25]
print('original df size:', df.shape)
print('new df size:', df2.shape)<jupyter_output>original df size: (768, 9)
new df size: (576, 9)<jupyter_text>let's compare models  baseline logistic regression model<jupyter_code>X, y = df.loc[:, df.columns != 'Outcome'], df['Outcome']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))
pd.crosstab(y_test, y_pred, rownames=['Actual Result'], colnames=['Predicted Result'])<jupyter_output>Accuracy of logistic regression classifier on test set: 0.75<jupyter_text>logistic regression on data without insulin outliers<jupyter_code>X, y = df2.loc[:, df2.columns != 'Outcome'], df2['Outcome']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_pred = logreg.predict(X_test)
print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))
pd.crosstab(y_test, y_pred, rownames=['Actual Result'], colnames=['Predicted Result'])<jupyter_output>Accuracy of logistic regression classifier on test set: 0.80
{'fraction_non_alphanumeric': 0.11377488825680618, 'fraction_numerical': 0.022348638764729784, 'mean_word_length': 6.766561514195583, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '552196', 'license': 'MIT', 'n_tokens_mistral': 897, 'n_tokens_neox': 803, 'n_words': 250, 'path': 'Section 3/3.4.ipynb', 'repo_name': 'PacktPublishing/Hands-On-Feature-Engineering-with-Python', 'size': '2461'}