<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 2, Module 1*---<jupyter_code># see if changes are saved (12)
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    !pip install category_encoders==2.*
    !pip install pandas-profiling==2.*

DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Kaggle-Challenge/master/data/'<jupyter_output>Requirement already satisfied: category_encoders==2.* in /usr/local/lib/python3.6/dist-packages (2.2.2)
Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders==2.*) (1.4.1)
Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders==2.*) (0.22.2.post1)
Requirement already satisfied: pandas>=0.21.1 in /usr/local/lib/python3.6/dist-packages (from category_encoders==2.*) (1.1.5)
Requirement already satisfied: patsy>=0.5.1 in /usr/local/lib/python3.6/dist-packages (from category_encoders==2.*) (0.5.1)
Requirement already satisfied: statsmodels>=0.9.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders==2.*) (0.10.2)
Requirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from category_encoders==2.*) (1.19.5)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20.0->category_enc[...]<jupyter_text>Decision Trees  Kaggle**Task 1:** [Sign up for a Kaggle account](https://www.kaggle.com/), if you don’t already have one. Go to our Kaggle InClass competition website (the URL is in Slack). Go to the Rules page. Accept the rules of the competition. Notice that the Rules page also has instructions for the Submission process. The Data page has feature definitions.  Wrangle Data**Task 2:** Add to the code below so that `id` is set as the index for `df`.<jupyter_code>import pandas as pd

df = pd.merge(pd.read_csv(DATA_PATH + 'waterpumps/train_features.csv'),
              pd.read_csv(DATA_PATH + 'waterpumps/train_labels.csv'))
# YOUR CODE HERE
df = df.set_index('id')

df.head()
df.shape<jupyter_output><empty_output><jupyter_text>**Test 2**<jupyter_code>'''Task 2 Testing'''

assert isinstance(df, pd.DataFrame), 'Have you created the DataFrame `df`?'
assert df.shape == (59400, 40), '`df` is the wrong shape. Did you set the index to `id`?'
assert 69572 in df.index, 'The index for `df` is has the wrong values. Did you set the index to `id`?'<jupyter_output><empty_output><jupyter_text>Split Data**Task 3:** Create your target vector `y` and feature matrix `X`.<jupyter_code>target = 'status_group'

# YOUR CODE HERE
df_features= df.drop(columns= 'status_group')
X= df_features
y= df[target]
X.shape, y.shape<jupyter_output><empty_output><jupyter_text>**Test 3**<jupyter_code>'''Task 3 Testing'''

assert isinstance(X, pd.DataFrame), '`X` is the wrong data type.'
assert isinstance(y, pd.Series), '`y` is the wrong data type.'
assert y.shape == (59400,), '`y` is the wrong shape.'
assert X.shape == (59400,39), '`X` is the wrong shape.'<jupyter_output><empty_output><jupyter_text>We already have a test set for this model, the `test.csv` that you'll use to make the predictions you'll upload to Kaggle. However, since our competition only allows for 2 submissions per day, we need a way to estimate our training error so that we can quickly iterate and improve our model. We can achieve this by creating a validation set from the data we have.**Task 4:** Split `X` and `y` into training and validation sets. Your validation set should be 20% of the data you have. You should have four variables: `X_train`, `X_val`, `y_train`, and `y_val`.<jupyter_code>from sklearn.model_selection import train_test_split

# YOUR CODE HERE
train, val= train_test_split(df, train_size= 0.8, test_size= 0.2,
                             stratify= df['status_group'], random_state= 41)
train.shape, val.shape
X_train= train.drop(columns= 'status_group')
X_val= val.drop(columns= 'status_group')
y_train= train['status_group']
y_val= val['status_group']

X_train.shape, X_val.shape, y_train.shape, y_val.shape<jupyter_output><empty_output><jupyter_text>**Test 4**<jupyter_code>'''Task 4 Testing'''

assert X_train.shape == (47520, 39), '`X_train` is the wrong shape.'
assert X_val.shape == (11880, 39), '`X_val` is the wrong shape.'
assert y_train.shape == (47520,), '`y_train` is the wrong shape.'
assert y_val.shape == (11880,), '`y_val` is the wrong shape.'<jupyter_output><empty_output><jupyter_text>Establish Baseline**Task 5:** This is a **classification** problem, so you need to establish the baseline accuracy for your training set. Find the majority class for `y_train` and calculate the percentage of labels in `y_train` belonging to that class. Assign your answer to the variable name `baseline_acc`.<jupyter_code># YOUR CODE HERE
baseline_acc= train['status_group'].value_counts(normalize= True).functional
baseline_acc
#print('Baseline Accuracy:', baseline_acc)<jupyter_output><empty_output><jupyter_text>**Task 5**<jupyter_code>'''Task 5 Testing'''

assert isinstance(baseline_acc, float), '`baseline_acc` should be a `float`.'
assert 0.0 <= baseline_acc <= 1.0, '`baseline_acc` is a score that should be between 0 and 1.'<jupyter_output><empty_output><jupyter_text>Build Model **Task 6:** Create a model named `model` and train it with your training data. Your model should be a pipeline with (a) transformers that you think are appropriate to this dataset and (b) a `DecisionTreeClassifier` as your predictor. **Tip:** How can you transform categorical features and missing values in order to train your model?<jupyter_code>X_train.head()
X_train.columns
X_val.columns
# clean the data
import numpy as np

def wrangle(X):
  X= X.copy()
  X['latitude']= X['latitude'].replace(-2e-08, 0)
  cols_with_zeros= ['longitude', 'latitude']
  for col in cols_with_zeros:
    X[col]= X[col].replace(0, np.nan)

  X= X.drop(columns= 'quantity_group')

  return X


X_train= wrangle(X_train)
X_val= wrangle(X_val)
X_train.shape, X_val.shape
# further clean up the data
train_features = X_train.copy()

numeric_features= train_features.select_dtypes(include= 'number').columns.tolist()
cardinality= train_features.select_dtypes(exclude= 'number').nunique()
categorical_features= cardinality[cardinality <= 50].index.tolist()
features= numeric_features + categorical_features
print(features)

# need to make this a function!
X_train= X_train[features]
X_val= X_val[features]
X_train.shape, X_val.shape
from category_encoders import OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier
import category_encoders as ce

# YOUR CODE HERE
model = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True), 
    SimpleImputer(strategy='mean'), 
    DecisionTreeClassifier(min_samples_leaf= 20, random_state=34)
)

model.fit(X_train, y_train)

print('Train Accuracy', model.score(X_train, y_train))
print('Validation Accuracy', model.score(X_val, y_val))<jupyter_output>/usr/local/lib/python3.6/dist-packages/category_encoders/utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>**Test 6**<jupyter_code>'''Task 6 Testing'''

assert len(model) > 1, 'Your model pipeline should have multiple steps.'
assert isinstance(model[-1], DecisionTreeClassifier), 'Your pipline should end in a `DecisionTreeClassifier`.'
assert hasattr(model, 'classes_'), 'Have you fit your model?'<jupyter_output><empty_output><jupyter_text>Check Metrics**Task 7:** Calculate the training and validation accuracy of your model, and assign them to the valiables `training_acc` and `validation_acc`, respectively. Your validation accuracy should be greater than your baseline accuracy.<jupyter_code># YOUR CODE HERE
training_acc= model.score(X_train, y_train)
validation_acc= model.score(X_val, y_val)

print('Training Accuracy:', training_acc)
print('Validation Accuracy:', validation_acc)<jupyter_output>Training Accuracy: 0.8061868686868687
Validation Accuracy: 0.7686868686868686<jupyter_text>**Test 7**<jupyter_code>'''Task 7 Testing'''

assert isinstance(training_acc, float)
assert isinstance(validation_acc, float)
assert 0.0 <= training_acc <= 1.0
assert 0.0 <= validation_acc <= 1.0<jupyter_output><empty_output><jupyter_text>Create Kaggle Submission**Task 8:** Load `'waterpumps/test.csv'` into a DataFrame named `X_test`. Generate a list of predictions, and then put them into a DataFrame `submission`. Be sure that `submission` has the same index as `X_test` and that the column name for your predictions is `'status_group'`.<jupyter_code>X_test = pd.read_csv(DATA_PATH + 'waterpumps/test_features.csv', index_col='id')
# YOUR CODE HERE
X_test = pd.read_csv(DATA_PATH + 'waterpumps/test_features.csv', index_col='id')
# YOUR CODE HERE
X_test= wrangle(X_test)
test_features = X_test.copy()

numeric_features= test_features.select_dtypes(include= 'number').columns.tolist()
cardinality= test_features.select_dtypes(exclude= 'number').nunique()
categorical_features= cardinality[cardinality <= 50].index.tolist()
features2= numeric_features + categorical_features
print(features2)

#submission.head()
X_test
X_test= X_test[features2]
X_test.shape
model = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True), 
    SimpleImputer(strategy='mean'), 
    DecisionTreeClassifier(min_samples_leaf= 20, random_state=34)
)

model.fit(X_train, y_train)

print('Train Accuracy', model.score(X_train, y_train))
print('Validation Accuracy', model.score(X_val, y_val))

y_pred = model.predict(X_test)
submission= pd.DataFrame(y_pred, columns= ['status_group'],  index= X_test.index)
submission.reset_index(inplace= True)
submission.rename(columns= {'index':'id'})
submission.shape
submission.columns
X_test
X_test.columns<jupyter_output><empty_output><jupyter_text>**Test 8**<jupyter_code>'''Task 8 Testing'''

assert isinstance(submission, pd.DataFrame), '`submission` should be a DataFrame.'
assert len(submission) == 14358, '`submission` should have 14358 rows.'<jupyter_output><empty_output><jupyter_text>**Task 9 (`stretch goal`):** Save `submission` as a csv file using [`.to_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) and submit it to our Kaggle competition.<jupyter_code># YOUR CODE HERE
submission.to_csv('submission.csv', index=False)<jupyter_output><empty_output><jupyter_text>Explain**Task 10 (`stretch goal`):** Make a horizontal barchart of the 10 most important features for your model.<jupyter_code>import matplotlib.pyplot as plt

# YOUR CODE HERE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12426696453504607, 'fraction_numerical': 0.023457134878525552, 'mean_word_length': 6.225285810356422, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 76, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '359632', 'license': 'MIT', 'n_tokens_mistral': 3792, 'n_tokens_neox': 3573, 'n_words': 1001, 'path': 'DS_module_project_221_Kevin_Manning_6.ipynb', 'repo_name': 'kvnmnnng/DS-Unit-2-Kaggle-Challenge', 'size': '10743'}