<jupyter_start><jupyter_text>[Career Exploration Kaggle Project](https://www.kaggle.com/t/cf06cb988cbc45969caf84ab4d03affb) For your final project in Career Exploration, you will be participating in a **Kaggle competition**, a data science and machine learning competition where you use *real* data and develop models to solve *real* problems.View the competition description / FAQ [here](https://docs.google.com/document/d/1mo0RIUJd1MqOxnaXJHvmWhaO69doDJR9lKfvG-CRYu8/edit?ts=5caadad4) for more info.**Note**: There is not much guidance provided in this project (on purpose). You'll be doing a lot of going through [previous lectures](https://github.com/SUSA-org/Spring-2019-Career-Exploration) to try to adapt the code provided there to this dataset, and reading documentation that's been linked in most of the problems. We are pushing you, fledgling data scientists, out of the nest and letting you spread your wings and fly.  About the CompetitionRead about it [here](https://www.kaggle.com/c/petfinder-adoption-prediction).You will be predicting how long it will take for a stray animal (dog or cat) to be adopted based on a variety of characteristics, including categorical data like **breed**, **color**, numerical data like **age**, a **description** of the pet from its rescuer, and an **image** of the pet itself. Note that this is a **classification**, not **regression** task: you will be predicting a pet's **Adoption Speed**, or placing it into one of [five](https://www.kaggle.com/c/petfinder-adoption-prediction/data) categories.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import json

import os
import seaborn as sns

np.random.seed(42)
% matplotlib inline<jupyter_output><empty_output><jupyter_text>Loading DataThere are **three** different data sources for the competition at your disposal.  DataFrameThere are 23 columns of interest, you can read about what each feature means [here](https://www.kaggle.com/c/petfinder-adoption-prediction/data).<jupyter_code>train = pd.read_csv('data/train/train.csv')
test = pd.read_csv('data/test/test.csv')<jupyter_output><empty_output><jupyter_text>Images The competition provides an image of each pet. Thankfully, so your model does not need to handle images (which are large and hard to deal with), each image has been run through Google's Vision API, and metadata about each image is given.<jupyter_code>!head data/train/metadata/86e1089a3-1.json<jupyter_output><empty_output><jupyter_text>To use this data, you can load this data into a **dictionary** in Python as follows:<jupyter_code># JSON Loading in Python

pet_image_metadata = json.load(open('data/train/metadata/86e1089a3-1.json'))
pet_image_metadata['labelAnnotations'][0]<jupyter_output><empty_output><jupyter_text>DescriptionsThe rescuer of the pet also provides a description (in the DataFrame).<jupyter_code>pet = train.iloc[0]
pet['PetID'], pet['Description']<jupyter_output><empty_output><jupyter_text>Each of these descriptions has also been preprocessed, run through Google's Natural Language API. The analysis of each description is used as follows:<jupyter_code>pet_sentiment_metadata = json.load(open('data/train/sentiment/86e1089a3.json'))
pet_sentiment_metadata['sentences'][0]<jupyter_output><empty_output><jupyter_text>1. Exploratory Data AnalysisProvide two plots that demonstrate the correlation between a specific features of the dataset. Here are a two examples:  Gender vs. Type<jupyter_code>train['dataset_type'] = 'train'
test['dataset_type'] = 'test'
all_data = pd.concat([train, test])


sns.factorplot('Type', col='Gender', data=all_data, kind='count', hue='dataset_type');
plt.subplots_adjust(top=0.8)
plt.suptitle('Count of cats and dogs in train and test set by gender');<jupyter_output><empty_output><jupyter_text>AdoptionSpeed by Type and Fee<jupyter_code>train.drop('dataset_type', inplace=True, axis=1)
test.drop('dataset_type', inplace=True, axis=1)

sns.violinplot(x="AdoptionSpeed", y="Fee", hue="Type", data=all_data);
plt.title('AdoptionSpeed by Type and Fee');<jupyter_output><empty_output><jupyter_text>2. Feature EngineeringTransform your training (and test!) data into two cleaned DataFrames with the features (from the original DataFrame or reading from the images or from the metadata JSON files) you believe will be the most helpful towards creating a model for AdoptionSpeed. In order to use the models below, you will need to make every feature **numerical**, not categorical.Note that this is probably the **most** important part of the project! People underestimate the power of good features: a linear classifier with descriptive features is often more powerful than a more complex model (XGBoost, for example) with bad features. And, since everyone will be using the same models for this competition (aside from tuning the hyperparameters), this step will most likely determine your leaderboard score!DataHub is a little restrictive in terms of memory space, so feel free to spin up a [Jupyter Notebook locally](https://jupyter.readthedocs.io/en/latest/install.html) to get around this! This is definitely not required, but it could help you do well in the competition, especially in this step! Please ask us for help with this if you have trouble.<jupyter_code>animals_transformed = ... # DataFrame with numerical data where each row is each training point
animals_labels = ... # AdoptionSpeed of each training point in the order in animals_transformed (Series object)

animals_test_transformed = ... # DataFrame with the same features as animals_transformed but each row is each test point
# IMPORTANT: make sure this has the same number of features (columns) as animals_transformed!

animals_test_index = ... # the PetIDs associated with the test points in the same order as the rows in animals_test_transformed
# IMPORTANT: make sure these are the PetIDs IN THE CORRECT ORDER associated with the test points. 
# PetID is how we check if your prediction is correct, so if this is wrong, you will get bad scores on the leaderboard.
animals_transformed.head() # Sanity check to see if your data is formatted nicely<jupyter_output><empty_output><jupyter_text>Run the cells below to test if your DataFrames are formatted correctly<jupyter_code># test rows are the same
assert animals_test_transformed.shape[0] == test.shape[0]

# number of train columns is same as number of test columns
assert animals_transformed.shape[1] == animals_test_transformed.shape[1]
# tests if the columns are numeric

from pandas.api.types import is_numeric_dtype

for column in animals_transformed.columns:
    assert is_numeric_dtype(animals_transformed[column])<jupyter_output><empty_output><jupyter_text>3. ModelThis competition uses the **Quadratic Weighted Kappa (QWK)**, or Cohen's kappa for evaluating a model. Read about it [here](https://en.wikipedia.org/wiki/Cohen%27s_kappaWeighted_kappa). We will use a few different models to try to optimize for this metric.<jupyter_code>from sklearn.metrics import cohen_kappa_score
from sklearn.metrics import confusion_matrix<jupyter_output><empty_output><jupyter_text>3.0 Validation SetMake an 80-20 split of the training data to create the validation set. Use [sklearn.model_seletion.train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html).<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_valid, y_train, y_valid = ...<jupyter_output><empty_output><jupyter_text>3.1 Linear Regression  3.1.1 ModelUse linear regression to classify the points in the validation set (don't expect an amazing score). Note: linear regression will give you a **continuous** prediction—it won't fit into the 0, 1, 2, 3, 4 categories. You need to find some way to map the output of the predicted values to the correct classes.<jupyter_code>from sklearn.linear_model import LinearRegression

# YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>3.1.2 Model Evaluation Calculate the QWK for these predictions on the validation set. We have already imported it for you.<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Visualize the [confusion matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) for your predictions. Use [sklearn.metrics.confusion_matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) (already imported) to compute it, and use [sns.heatmap](https://seaborn.pydata.org/generated/seaborn.heatmap.html) to visualize. What do each of cells mean?<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>3.2 KNN  3.2.1 ModelFit a [k-nearest-neighbors classifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.htmlsklearn.neighbors.KNeighborsClassifier) to the training data with **k = 3**. Report the QWK for the predictions on the validation set.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
# YOUR CODE HERE
# YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>3.2.2. Hyperparameter TuningPerform [3-fold cross-validation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html) on the parameter **k**. Find the best parameter of **k = [1, 10, 20]** and report the **QWK** on the validation set if you use this parameter.<jupyter_code>from sklearn.model_selection import KFold

kf = KFold(n_splits=3)

ks = [1, 10, 20]

cv_scores = np.zeros(len(ks))

for ki, k in enumerate(ks):
    print('Training k =', k, end='\r')
    scores = np.zeros(3)
    for i, (train_index, test_index) in enumerate(kf.split(X_train)):
        # YOUR CODE HERE
    cv_scores[ki] = scores.mean()
best_k = ks[cv_scores.argmax()]
best_k
# Fit the classifier with this parameter<jupyter_output><empty_output><jupyter_text>3.3. Decision Trees and Random Forest Fit a [Decision Tree](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) to the training data. Report the QWK for the predictions on the validation set.<jupyter_code>from sklearn.tree import DecisionTreeClassifier

# YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>You'll notice that your score is pretty low (maybe even close to your score using Linear Regression). To investigate why, calculate the **QWK** when your classifier predicts the training data.<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>This number is very **high** (why do you think that is the case?). Fit another Decision Tree and tune the **max_depth** parameter (using cross-validation or just by trying numbers). Report your best **QWK** on the validation set.<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Now, fit a [Random Forest classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) to your data and report your best **QWK** on the validation set.<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>3.4 Neural NetworksTrain a neural network on the data. Report your QWK on the validation set.<jupyter_code>import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout

model = Sequential()
# YOUR CODE HERE: add some layers to your neural network<jupyter_output><empty_output><jupyter_text>3.5 XGBoost Now that we've tried many different types of classifiers, it's time to bring out the big guns.Below are hyperparameters for an XGBoost model: tinker around with these (or use cross-validation) to achieve the best validation score (below). Learn about what some of the hyperparameters mean [here](https://xgboost.readthedocs.io/en/latest/python/python_api.htmlxgboost.train).<jupyter_code>params = {
    'n_splits': ...,
    'verbose_eval': ...,
    'num_boost_rounds': ...,
    'early_stop': ...
}
from model import train_and_test
xgb_preds, model, qwk = train_and_test(params, animals_transformed, animals_labels, animals_test_transformed, verbose=True)
print('Validation QWK:', qwk)<jupyter_output><empty_output><jupyter_text>4. Submitting to KaggleRun the cells below, and then download the **submission.csv** file that is created by selecting the checkbox next to the file name and hitting the 'Download' button (IMPORTANT: do not hit the button that looks like a download button, it will download your entire DataHub files). Upload this file [here](https://www.kaggle.com/c/susa-cx-spring-19/submit).<jupyter_code>preds = pd.DataFrame({
    'PetID': animals_test_index,
    'AdoptionSpeed': xgb_preds # or choose another classifier's predictions, if you'd like
}).set_index('PetID')

preds.to_csv('submission.csv')<jupyter_output><empty_output><jupyter_text>Before try to submit to the competition, run the cell below to look at the first few lines of your submission file as a sanity check.<jupyter_code>!head submission.csv<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09061388588217102, 'fraction_numerical': 0.009355187876913562, 'mean_word_length': 6.041371801850843, 'pattern_counts': {'":': 0, '<': 101, '<?xml version=': 0, '>': 101, 'https://': 18, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '46370', 'license': 'MIT', 'n_tokens_mistral': 4000, 'n_tokens_neox': 3784, 'n_words': 1349, 'path': 'CX-Final-Project/CX-Final-Project-Starter.ipynb', 'repo_name': 'alaymodi/Spring-2019-Career-Exploration-master', 'size': '12934'}