<jupyter_start><jupyter_text>To work with this notebook you must have Anaconda installed on your computer. To be sure you have all the packages installed and running normally you can run the cells with import statements and look for any error messages. If you get an error download the missing package and pip install from the command line First Import Pandas<jupyter_code>from sklearn import cross_validation
from sklearn.ensemble import RandomForestClassifier
import pandas as pd<jupyter_output><empty_output><jupyter_text>Next import your data file Here I am using data from the World Bank. The primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates. The topics covered are: Agriculture & Rural Development, Aid Effectiveness, Climate Change, Economy & Growth, Education, Energy & Mining, Environment, External Debt, Financial Sector, Gender, Health, Infrastructure, Labor & Social Protection, Poverty, Private Sector, Public Sector, Science & Technology, Social Development, Trade, Urban Development. From the economies of the countries of WLD, EAP, ECA, LAC, MNA, SAS, SSA, HIC, LMY, IBRD, IDA gathered from 1960 - 2015, a time series covering 217 economieshttp://data.worldbank.org/data-catalog/world-development-indicators<jupyter_code>Wdata = pd.read_csv("WorldData.csv")<jupyter_output><empty_output><jupyter_text>Next look at a snapshot of your data file. We will see the statistics for each year as a sample of the years in the dataset<jupyter_code>print (Wdata.describe())<jupyter_output>1960          1961          1962          1963          1964  \
count  2.587000e+04  2.993500e+04  3.190800e+04  3.217500e+04  3.272300e+04   
mean   2.668505e+11  2.459610e+11  2.434990e+11  2.434428e+11  2.538379e+11   
std    7.405405e+12  7.346839e+12  7.488231e+12  7.398136e+12  7.627142e+12   
min   -8.181071e+13 -9.465063e+13 -1.071675e+14 -1.181558e+14 -1.319199e+14   
25%    6.099798e+00  6.521122e+00  6.641570e+00  6.905873e+00  7.254888e+00   
50%    7.055984e+01  6.567180e+01  6.565022e+01  6.345300e+01  6.501782e+01   
75%    1.148247e+07  3.293826e+06  2.280636e+06  2.429516e+06  2.799999e+06   
max    4.645807e+14  4.921465e+14  5.037593e+14  4.898643e+14  5.048614e+14   

               1965          1966          1967          1968          1969  \
count  3.563500e+04  3.521800e+04  3.680100e+04  3.658100e+04  3.728600e+04   
mean   2.509145e+11  2.703316e+11  2.763226e+11  3.136419e+11  3.387058e+11   
std    7.720802e+12  8.182320e+12  8.518432e+12  9.[...]<jupyter_text>Next do you need to clean the data? If so try this short line of code to clean the data remove any NULL or Nan or blank spaces and replace with anything you want here we are replacing all NULL with zero, or you can replace with the means etc depending upon your data and how/if it will affect your statistical output<jupyter_code>WData2 = Wdata.fillna("0")<jupyter_output><empty_output><jupyter_text>Next give your new file a name- here I called the file WDataClean- and extract as a csv file to work with moving forward. When you run the code below it will download the file to your computer<jupyter_code>WData2.to_csv("WDataClean.csv", index=False)<jupyter_output><empty_output><jupyter_text>Next import Random Forest  Classifier from sklearn and a cross validation package<jupyter_code>from sklearn import cross_validation
from sklearn.ensemble import RandomForestClassifier<jupyter_output><empty_output><jupyter_text>Next ask pandas to read in your new cleaned file<jupyter_code>WC = pd.read_csv("WDataClean.csv")<jupyter_output><empty_output><jupyter_text>Next tell Random Forest what columns you want to use as your predictor. choose from your data describe call<jupyter_code>from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(categorical_features='all', dtype= np.float,
       handle_unknown='error', n_values='auto', sparse=True)
def fit(self, X, y=None, **kwargs):
 enc.fit( X,y)  

enc.transform([[0,1,20]]).toarray()


import pandas as pd
import copy
import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
import random

# Let's peak at some of the data. You can change the number to reflect how many headers you want to see
print (WC.head(15))
#Name what you want to use as your predictors.
predictors = ["Row Labels", "Sum of 2014" ]<jupyter_output><empty_output><jupyter_text>Make a prediction of accuracy based upon the data and the predictors you choose and choose what you want to predict. The data will be broken into cross validation pieces to improve accuracy using 3 folds. Assign the result to scores and Print scores as a mean of all three scores you will obtain from the algorithmn<jupyter_code>alg = RandomForestClassifier(random_state=1, n_estimators=10, min_samples_split=2, min_samples_leaf=1)
scores = cross_validation.cross_val_score(alg, NameOfYourDataSetHere[predictors], NameOfYourDataSetHere["Sum of 2013"], cv=3)
print (scores.mean())<jupyter_output><empty_output><jupyter_text>Wow years are a terrible predictor of the country, as we expected. A few things might be wrong. Maybe we need to further clean and transform the data to get a better predictor and also maybe we are terrible at choosing the right features that make a good prediction.Next let's see if the computer can help with choosing the best features to select as predictors Let's import a new series of packages to help with this<jupyter_code>import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif<jupyter_output><empty_output><jupyter_text>..and let's work with a portion of the dataset using just the world data and the data has been transformed. Most of the work in machine learning comes from cleaning and working with the best dataset. Let's import only data taken from the original dataset on one country- here Afghanistan. Let's look at the data If the code throws errors you need to clean and transform your data more remember sklearn does not take strings only numbers and arrays. Try the code below to change all column to floats if you through floats errors predictions = alg.predict_proba(YourDataSetNameHere[predictors].astype(float))[:,1]    full_predictions.append(predictions)<jupyter_code>print(WC.head(15))
from sklearn import cross_validation
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Add all your possible column choices of features for the computer to choose from<jupyter_code>predictors = ["Predictor1", "Predictor2","Predictor3","Predictor4"]
alg = RandomForestClassifier(random_state=1, n_estimators=150, min_samples_split=8, min_samples_leaf=4)
scores = cross_validation.cross_val_score(alg, NameOfYourDataSetHere[predictors], NameOfYourDataSetHere["WhatYouWantToPredict"], cv=3)
print (scores.mean())


# Ask the computer to perform feature selection based upon the predictors and choosing your column to predict
selector = SelectKBest(f_classif, k='5')
selector.fit(NameOfYourDataSet[predictors], NameOfYourDataSet["WhatYouWantToPredict"])
# Plot the scores.So you can visually see and choose you best predictor
plt.bar(range(len(predictors)), scores)
plt.xticks(range(len(predictors)), predictors, rotation='vertical')
plt.show()<jupyter_output><empty_output><jupyter_text>Now pick only the best features chosen by the computer's algorithmn above-you can choose as many as you like and re-run the analysis of prediction for accuracy and print the resulting new accuracy scores. Was it better or worse than what you choose as the predictors of what you wanted to predict?<jupyter_code>predictors = ["Predictor1", "Predictor2", "Predictor3", "predictor4"]

alg = RandomForestClassifier(random_state=1, n_estimators=150, min_samples_split=8, min_samples_leaf=4)
scores = cross_validation.cross_val_score(alg, NameOfYourDataSet[predictors], NameOfYourDataSet["WhatYouWantToPredict"], cv=3)
print (scores.mean())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07295980511571254, 'fraction_numerical': 0.0712545676004872, 'mean_word_length': 5.41484375, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '95373', 'license': 'Apache-2.0', 'n_tokens_mistral': 2838, 'n_tokens_neox': 2472, 'n_words': 908, 'path': 'World Impact Data Analysis.ipynb', 'repo_name': 'acourtney2015/ImpactData', 'size': '8210'}