<jupyter_start><jupyter_text>Loosely follows Kaggle's tutorial on predicting house prices in Melbourne, Australia: https://www.kaggle.com/learn/machine-learning (sections 1 to 6)- Can we find Hawaii's housing information?<jupyter_code>import pandas as pd

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load data into a Pandas dataframe  Pandas is a powerful python data analysis library "providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language"https://pandas.pydata.org/ It totally new to pandas, here is a 10 minute intro to basic, common functionality: http://pandas.pydata.org/pandas-docs/stable/10min.html<jupyter_code>## Load data as data frame
melbourne_file_path = '../input/melbourne-housing-snapshot/melb_data.csv'
melbourne_data = pd.read_csv(melbourne_file_path)<jupyter_output><empty_output><jupyter_text>Dataframes are big objects, sometimes you just want a snapshot, for example the top<jupyter_code>## show first 5 entries
melbourne_data.head()<jupyter_output><empty_output><jupyter_text>or you want a summary<jupyter_code>## print a summary of the data in Melbourne data
melbourne_data.describe()<jupyter_output><empty_output><jupyter_text>Pandas dataframes have a lot of methods, we recommend googling around and exploring functionality. In the jupyter notebook, putting a ? after a method and "shift+enter" will pop up info about that method, including his sinature call<jupyter_code>melbourne_data.Rooms.hist?
## Price's histogram
melbourne_data.Price.hist(bins=100)<jupyter_output><empty_output><jupyter_text>For simplicity, make Price into a binary variable This is not for students. We should binarize other variables as well. TODO - Choose which variables we want to be binary and set the split value<jupyter_code>melbourne_data.loc?
## use 1 million as the cutoff point
price_mean = 1e+06

## assign categories
category_name = 'PriceBinary'
melbourne_data.loc[melbourne_data.Price >= price_mean, category_name] = '>=1m'
melbourne_data.loc[melbourne_data.Price < price_mean, category_name] = '<1m'
melbourne_data.PriceBinary.describe()
## drop Price so there's no information leak/ so the true prices don't predict if it will be greater than or less than a million dollars
melbourne_data = melbourne_data.drop('Price', 1)
melbourne_data.head()<jupyter_output><empty_output><jupyter_text>Prediction  Y of integer values, size [n_samples], holding the class labels for the training samples<jupyter_code>## prediction target
y = melbourne_data.PriceBinary<jupyter_output><empty_output><jupyter_text>X, sparse or dense, of size [n_samples, n_features] holding the training samples<jupyter_code>## predictors
melbourne_predictors = ['Rooms', 'Bathroom', 'Landsize', 'BuildingArea', 
                        'YearBuilt', 'Lattitude', 'Longtitude']
X = melbourne_data[melbourne_predictors]
X.head()
## Notice that there are missing values in the data.
## For now we fill them with 0s.

X = X.fillna(0)<jupyter_output><empty_output><jupyter_text>So we've set up our data set, X is in the N labels we want to use to do the prediction, and y is 1 label we want to predict We will use a prebuilt decision tree classifier from sci kit learn to build a decision tree Scikit Learn is a library for doing Machine Learning in Python. It is a simple and efficient tools for data mining and data analysis, and reusable in various contexts. http://scikit-learn.org/stable/http://scikit-learn.org/stable/modules/tree.html Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. DecisionTreeClassifier is a class capable of performing multi-class classification on a dataset. As with other classifiers, DecisionTreeClassifier takes as input two arrays: an array X, sparse or dense, of size [n_samples, n_features] holding the training samples, and an array Y of integer values, size [n_samples], holding the class labels for the training samples:<jupyter_code>## Call decision tree classifier from library
from sklearn.tree import DecisionTreeClassifier

# Define model
melbourne_model = DecisionTreeClassifier(criterion='entropy')

# Fit model
melbourne_model.fit(X.fillna(0), y)<jupyter_output><empty_output><jupyter_text>After being fitted, the model can then be used to predict the class of samples:<jupyter_code>print("Making predictions for the following 5 houses:")
print(X.head())
print("The predictions are")
print(melbourne_model.predict(X.head()))<jupyter_output>Making predictions for the following 5 houses:
   Rooms  Bathroom  Landsize  BuildingArea  YearBuilt  Lattitude  Longtitude
0      2       1.0     202.0           0.0        0.0   -37.7996    144.9984
1      2       1.0     156.0          79.0     1900.0   -37.8079    144.9934
2      3       2.0     134.0         150.0     1900.0   -37.8093    144.9944
3      3       2.0      94.0           0.0        0.0   -37.7969    144.9969
4      4       1.0     120.0         142.0     2014.0   -37.8072    144.9941
The predictions are
['>=1m' '>=1m' '>=1m' '<1m' '>=1m']<jupyter_text>http://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.metrics We want to see how accurate our prediction is given the default values for the classifier, though this is a bit of a cheat because we are using the same data to both build the tree, and then running that data through the tree to predict the y values What does it mean that this accuracy is not 100%?<jupyter_code>## Calculate accuracy
from sklearn.metrics import accuracy_score

predicted_home_prices = melbourne_model.predict(X)
accuracy_score(y, predicted_home_prices)<jupyter_output><empty_output><jupyter_text>Let's try to be a bit more principled and not use the same data to generate the tree and then try to predict the prices for We will split the dataset into a training set to build the decision tree on  and a test set, where we'll pretend we don't know the price, predict the price using the decision tree built on the training set, the compare the predicted price to the true price splitting data sets into a training set and a test set in a very common operation in machine learning, so sci kit learn has nice funcationality for that http://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.model_selection why did the accuracy go down? The default value for train_size is 0.25, is that low? Play with values? Make training set larger, smaller?<jupyter_code>## split train/validate to help with generalization
from sklearn.model_selection import train_test_split

## setting random state allows reproduciability
train_X, val_X, train_y, val_y = train_test_split(X.fillna(0), y, random_state = 42)

# Define model
melbourne_model = DecisionTreeClassifier()

# Fit model
melbourne_model.fit(train_X, train_y)

# get predicted prices on validation data
val_predictions = melbourne_model.predict(val_X)

print(accuracy_score(val_y, val_predictions))<jupyter_output>0.8236573168080017<jupyter_text>We mostly used the default values for the decision tree classifier, but there are many parameters to tweak, sci kit learn just naturally sets them to common values. Here are some of the parameters, but more can be found in the documentation criterion : string, optional (default=”gini”)The function to measure the quality of a split. Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain.splitter : string, optional (default=”best”)The strategy used to choose the split at each node. Supported strategies are “best” to choose the best split and “random” to choose the best random split.max_depth : int or None, optional (default=None)The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.min_samples_split : int, float, optional (default=2)The minimum number of samples required to split an internal node:If int, then consider min_samples_split as the minimum number.If float, then min_samples_split is a percentage and ceil(min_samples_split * n_samples) are the minimum number of samples for each split.Changed in version 0.18: Added float values for percentages.  Test the accuracy of your tree as a function of different parameters, for example the maximum depth of your tree  Connect depth of tree to how much fitting the tree does<jupyter_code>## experiment with different models to find the best performing model

## define a function so it's easier to loop
def get_accuracy_score(max_depth, predictors_train, predictors_val, targ_train, targ_val):
    model = DecisionTreeClassifier(max_depth=max_depth, random_state=42)
    model.fit(predictors_train, targ_train)
    preds_val = model.predict(predictors_val)
    accuracy = accuracy_score(targ_val, preds_val)
    return accuracy, model
## compare accuracy with differing values of max_depth

depth_range = range(1, 50, 2) # or use a list such as [1,2,3,4,5,10,50,100,1000]
accuracy_list = []

for max_depth in depth_range:
    my_accuracy, my_model = get_accuracy_score(max_depth, train_X, val_X, train_y, val_y)
    accuracy_list.append(my_accuracy)
    print('max depth:', max_depth, '\t', 'accuracy:', my_accuracy)
len(depth_range), len(accuracy_list)<jupyter_output><empty_output><jupyter_text>Plot the accuracy as a function of the parameters you vary<jupyter_code>## tradeoff plots
plt.plot(depth_range, accuracy_list, '.-')
plt.xlabel('Complexity (Max Depth)')
plt.ylabel('Accuracy')
plt.title('PriceBinary Accuracy vs. Complexity')
## best accuracy
best_accuracy = max(accuracy_list)
## find corresponding index for best depth
index = accuracy_list.index(max(accuracy_list))
best_max_depth = depth_range[index]
best_accuracy, best_max_depth<jupyter_output><empty_output><jupyter_text>The best model has an accuracy of ~0.84 with maximum tree depth of 13 levels  Visualization of the treeOnce trained, we can export the tree in Graphviz format using the export_graphviz exporterThe export_graphviz exporter also supports a variety of aesthetic options, including coloring nodes by their class (or value for regression) and using explicit variable and class names if desired. Jupyter notebooks also render these plots inline automatically:<jupyter_code>## Install pydotplus
!pip install pydotplus
from sklearn.externals.six import StringIO
from sklearn.tree import export_graphviz
import pydotplus
## Create final model with best depth
best_accuracy, best_model = get_accuracy_score(best_max_depth, train_X, val_X, train_y, val_y)
%%time

## creates a png file of the tree
## should take less than 1 min

dot_data = StringIO()
export_graphviz(best_model, out_file=dot_data,  
                filled=True, rounded=True,
                special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
graph.write_png('tree.png')<jupyter_output>CPU times: user 15.3 s, sys: 405 ms, total: 15.7 s
Wall time: 23.6 s
{'fraction_non_alphanumeric': 0.07176947321026565, 'fraction_numerical': 0.02125168842863575, 'mean_word_length': 5.16315205327414, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 64, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '159676', 'license': 'MIT', 'n_tokens_mistral': 3406, 'n_tokens_neox': 3205, 'n_words': 1260, 'path': 'notebooks/tutorial.ipynb', 'repo_name': 'andersdot/PDP2018', 'size': '11105'}