<jupyter_start><jupyter_text>Assignment- Learn about the mathematics of Logistic Regression by watching <NAME>'s [video 1](https://www.youtube.com/watch?v=pREaWFli-5I) (12 minutes) & [video 2](https://www.youtube.com/watch?v=bDQgVt4hFgY) (9 minutes).- Start a clean notebook.- Do train/validate/test split with the Tanzania Waterpumps data.- Begin to explore and clean the data. For ideas, refer to [The Quartz guide to bad data](https://github.com/Quartz/bad-data-guide),  a "reference to problems seen in real-world data along with suggestions on how to resolve them." One of the issues is ["Zeros replace missing values."](https://github.com/Quartz/bad-data-guidezeros-replace-missing-values)- Select different numeric and categorical features. - Do one-hot encoding. (Remember it may not work with high cardinality categoricals.)- Scale features.- Use scikit-learn for logistic regression.- Get your validation accuracy score.- Get and plot your coefficients.- Submit your predictions to our Kaggle competition.- Commit your notebook to your fork of the GitHub repo. Stretch Goals- Begin to visualize the data.- Try different [scikit-learn scalers](https://scikit-learn.org/stable/modules/preprocessing.html)- Try [scikit-learn pipelines](https://scikit-learn.org/stable/modules/compose.html):> Pipeline can be used to chain multiple estimators into one. This is useful as there is often a fixed sequence of steps in processing the data, for example feature selection, normalization and classification. Pipeline serves multiple purposes here:> - **Convenience and encapsulation.** You only have to call fit and predict once on your data to fit a whole sequence of estimators.> - **Joint parameter selection.** You can grid search over parameters of all estimators in the pipeline at once.> - **Safety.** Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors.  Load and split data<jupyter_code>!pip install category_encoders
import numpy as np
import pandas as pd
from math import sqrt
import pandas_profiling
import category_encoders as ce
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
LOCAL = '../data/tanzania/'
WEB = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Linear-Models/master/data/tanzania/'

train_features = pd.read_csv(WEB + 'train_features.csv')
train_labels = pd.read_csv(WEB + 'train_labels.csv')
test_features = pd.read_csv(WEB + 'test_features.csv')
sample_submission = pd.read_csv(WEB + 'sample_submission.csv')

assert train_features.shape == (59400, 40)
assert train_labels.shape == (59400, 2)
assert test_features.shape == (14358, 40)
assert sample_submission.shape == (14358, 2)
train_features.sample(1)<jupyter_output><empty_output><jupyter_text>Best Features for Logistic Regression<jupyter_code>train_features.describe()
# len(train_features['region_code'].value_counts().index)
# .query('unique <= 21')
# 11,17,12,3,5,18,19
train_features.describe(exclude='number').sort_values(by='unique', axis=1)
train_features['quantity'].value_counts()
# feature engineer
# train_features['enough_source'] = (train_features['quantity_group']=='enough')|(train_features['quantity']=='enough')
# train_features['shallow_well_source'] = (train_features['source']=='shallow well')|(train_features['source_type']=='shallow well')
# train_features['spring_source'] = (train_features['source']=='spring')|(train_features['source_type']=='spring')
# train_features['communal_standpipe_waterpoint'] = (train_features['waterpoint_type']=='communal standpipe')&(train_features['waterpoint_type_group']=='communal standpipe')
# train_features['handpump'] = (train_features['extraction_type_class']=='handpump')|(train_features['waterpoint_type']=='hand pump')|(train_features['waterpoint_type_group']=='hand pump')
# train_features['never_pay'] = (train_features['payment']=='never pay')|(train_features['payment_type']=='never pay')
# train_features['soft_good_quality'] = (train_features['water_quality']=='soft')|(train_features['quality_group']=='good')
# train_features['Victoria_basin'] = train_features['basin']=='Lake Victoria'
# train_features['2_source'] = (train_features['source']=='spring')&(train_features['source']=='shallow well')
# train_features['vwc_management'] = (train_features['management']=='vwc')|(train_features['scheme_management']=='VWC')
# train_features['gravity_extraction'] = (train_features['extraction_type']=='gravity')|(train_features['extraction_type_group']=='gravity')|(train_features['extraction_type_class']=='gravity')
# train_features['3_region'] = (train_features['region']=='Iringa')&(train_features['region']=='Shinyanga')&(train_features['region']=='Mbeya')
# train_features['region_code_11'] = train_features['region_code']==11
# train_features['region_code_17'] = train_features['region_code']==17
# train_features['region_code_12'] = train_features['region_code']==12
# train_features['Njombe_lga'] = train_features['lga']=='Njombe'
train_features['gov_funder'] = train_features['funder']=='Government Of Tanzania'
train_features['Igosi_ward'] = train_features['ward']=='Igosi'
train_features['Imalinyi_ward'] = train_features['ward']=='Imalinyi'
train_features['Siha_ward'] = train_features['ward']=='Siha'
train_features['gov_installer'] = train_features['installer']=='Government'
train_features['dwe_installer'] = train_features['installer']=='DWE'
train_features['none_scheme_name'] = train_features['scheme_name']=='None'
train_features['gov_scheme_name'] = train_features['scheme_name']=='Government'
train_features['Madukani_subvillage'] = train_features['subvillage']=='Madukani'
train_features['Shuleni_subvillage'] = train_features['subvillage']=='Shuleni'
train_features['Majengo_subvillage'] = train_features['subvillage']=='Majengo'   
train_features['none_wpt'] = train_features['wpt_name']=='none'
train_features['Shuleni_wpt'] = train_features['wpt_name']=='Shuleni'
train_features['dwe'] = (train_features['installer']=='DWE')&(train_features['funder']=='Dwe')
train_features['gov_install_fund'] = (train_features['installer']=='Government')&(train_features['funder']=='Government Of Tanzania')
train_features['gov'] = (train_features['installer']=='Government')&(train_features['funder']=='Government Of Tanzania')&(train_features['scheme_name']=='Government')
test_features['gov_funder'] = test_features['funder']=='Government Of Tanzania'
test_features['Igosi_ward'] = test_features['ward']=='Igosi'
test_features['Imalinyi_ward'] = test_features['ward']=='Imalinyi'
test_features['Siha_ward'] = test_features['ward']=='Siha'
test_features['gov_installer'] = test_features['installer']=='Government'
test_features['dwe_installer'] = test_features['installer']=='DWE'
test_features['none_scheme_name'] = test_features['scheme_name']=='None'
test_features['gov_scheme_name'] = test_features['scheme_name']=='Government'
test_features['Madukani_subvillage'] = test_features['subvillage']=='Madukani'
test_features['Shuleni_subvillage'] = test_features['subvillage']=='Shuleni'
test_features['Majengo_subvillage'] = test_features['subvillage']=='Majengo'   
test_features['none_wpt'] = test_features['wpt_name']=='none'
test_features['Shuleni_wpt'] = test_features['wpt_name']=='Shuleni'
test_features['dwe'] = (test_features['installer']=='DWE')&(test_features['funder']=='Dwe')
test_features['gov_install_fund'] = (test_features['installer']=='Government')&(test_features['funder']=='Government Of Tanzania')
test_features['gov'] = (test_features['installer']=='Government')&(test_features['funder']=='Government Of Tanzania')&(test_features['scheme_name']=='Government')
# train_test_split
X_train = train_features
y_train = train_labels['status_group']

X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, train_size=0.8, test_size=.2,
    stratify=y_train, random_state=42)

X_train.shape, X_val.shape, y_train.shape, y_val.shape
X_val.columns
# categorical codes
# df.interest_level = pd.Categorical(df.interest_level)
# df['interest_code'] = df.interest_level.cat.codes
categorical_features = X_train.describe(exclude='number').T.query('unique <= 400').index.drop(['recorded_by']).tolist()
numeric_features = X_train.select_dtypes('number').columns.drop(['id']).tolist()
features = categorical_features + numeric_features

X_train_subset = X_train[features]
X_val_subset = X_val[features]

encoder = ce.OneHotEncoder(use_cat_names=True)
X_train_encoded = encoder.fit_transform(X_train_subset)
X_val_encoded = encoder.transform(X_val_subset)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_encoded)
X_val_scaled = scaler.transform(X_val_encoded)

model = LogisticRegression(solver='lbfgs', multi_class='auto', max_iter=1000)
model.fit(X_train_scaled, y_train)
score = model.score(X_val_scaled, y_val)
print('Validation Accuracy: {:5.4f}%'.format(score*100))
# Validation Accuracy: 76.6751% 'unique <= 2000'
# Validation Accuracy: 75.5724% 'unique <= 400'
# Validation Accuracy: 75.0673% 'unique <= 200'
# Validation Accuracy: 75.5724% 'unique <= 400'
categorical_features = X_train.describe(exclude='number').T.query('unique <= 400').index.drop(['recorded_by']).tolist()
numeric_features = X_train.select_dtypes('number').columns.drop(['id']).tolist()
features = categorical_features + numeric_features

# longitude, district_code, water_quality
# 0: 73.7710
# 1: longitude - 
# 2: district_code - 
# 3: water_quality - 
new_f_score = []
for feature in features:
  f_drop = features.copy()
  f_drop.remove(feature)
  X_train_subset = X_train[f_drop]
  X_val_subset = X_val[f_drop]

  encoder = ce.OneHotEncoder(use_cat_names=True)
  X_train_encoded = encoder.fit_transform(X_train_subset)
  X_val_encoded = encoder.transform(X_val_subset)

  scaler = StandardScaler()
  X_train_scaled = scaler.fit_transform(X_train_encoded)
  X_val_scaled = scaler.transform(X_val_encoded)

  model_drop = LogisticRegression(solver='lbfgs', multi_class='auto', max_iter=1000)
  model_drop.fit(X_train_scaled, y_train)
  print(feature, model_drop.score(X_val_scaled, y_val)*100)
  if model_drop.score(X_val_scaled, y_val) > score:
    score = model_drop.score(X_val_scaled, y_val)
    new_f_score = [feature, score]
new_f_score
categorical_features = X_train.describe(exclude='number').T.query('unique <= 3000').index.drop(['recorded_by']).tolist()
numeric_features = X_train.select_dtypes('number').columns.drop(['id']).tolist()
features = categorical_features + numeric_features

X_train_subset = X_train[features]
X_val_subset = X_val[features]

encoder = ce.OneHotEncoder(use_cat_names=True)
X_train_encoded = encoder.fit_transform(X_train_subset)
X_val_encoded = encoder.transform(X_val_subset)
# scaler = StandardScaler()
# X_train_scaled = scaler.fit_transform(X_train_encoded)
# X_val_scaled = scaler.transform(X_val_encoded)
model = LogisticRegression(solver='lbfgs', multi_class='auto', max_iter=1000)
model.fit(X_train_encoded, y_train)<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.
  "of iterations.", ConvergenceWarning)<jupyter_text>Get & plot coefficients<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
plt.figure(figsize=(30, 30))

coefficients = pd.Series(model.coef_[0], X_train_encoded.columns)
coefficients.sort_values().plot.barh();<jupyter_output><empty_output><jupyter_text>Submit to predictive modeling competition Write submission CSV fileThe format for the submission file is simply the row id and the predicted label (for an example, see `sample_submission.csv` on the data download page.For example, if you just predicted that all the waterpoints were functional you would have the following predictions:id,status_group50785,functional51630,functional17168,functional45559,functional49871,functionalYour code to generate a submission file may look like this:  estimator is your scikit-learn estimator, which you've fit on X_train X_test is your pandas dataframe or numpy array,  with the same number of rows, in the same order, as test_features.csv,  and the same number of columns, in the same order, as X_trainy_pred = estimator.predict(X_test) Makes a dataframe with two columns, id and status_group,  and writes to a csv file, without the indexsample_submission = pd.read_csv('sample_submission.csv')submission = sample_submission.copy()submission['status_group'] = y_predsubmission.to_csv('your-submission-filename.csv', index=False)<jupyter_code>X_test_subset = test_features[features]
X_test_encoded = encoder.transform(X_test_subset)
# X_test_scaled = scaler.transform(X_test_encoded)
all(X_test_encoded.columns == X_train_encoded.columns)
# y_pred = model.predict(X_test_scaled)
y_pred = model.predict(X_test_encoded)
submission = sample_submission.copy()
submission['status_group'] = y_pred
submission.to_csv('submission-01.csv', index=False)
!head submission-01.csv<jupyter_output>id,status_group
50785,non functional
51630,non functional
17168,non functional
45559,non functional
49871,functional
52449,functional
24806,functional
28965,non functional
36301,functional
{'fraction_non_alphanumeric': 0.13710701217235458, 'fraction_numerical': 0.017175715032484506, 'mean_word_length': 9.061607813673929, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 21, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '245537', 'license': 'MIT', 'n_tokens_mistral': 4609, 'n_tokens_neox': 4220, 'n_words': 813, 'path': 'module4-logistic-regression/gh_214_kaggle_challenge.ipynb', 'repo_name': 'gyhou/DS-Unit-2-Linear-Models', 'size': '13398'}