<jupyter_start><jupyter_text>Lesson 1: Download and Install Python and SciPyYou cannot get started with machine learning in Python until you have access to the platform.Today’s lesson is easy, you must download and install the Python 2.7 platform on your computer.Visit the Python homepage and download Python for your operating system (Linux, OS X or Windows). Install Python on your computer. You may need to use a platform specific package manager such as macports on OS X or yum on RedHat Linux.You also need to install the SciPy platform and the scikit-learn library. I recommend using the same approach that you used to install Python.You can install everything at once (much easier) with Anaconda. Recommended for beginners.Start Python for the first time by typing “python” at the command line.Check the versions of everything you are going to need using the code below:<jupyter_code># Python version
import sys
print('Python: {}'.format(sys.version))
# scipy
import scipy
print('scipy: {}'.format(scipy.__version__))
# numpy
import numpy
print('numpy: {}'.format(numpy.__version__))
# matplotlib
import matplotlib
print('matplotlib: {}'.format(matplotlib.__version__))
# pandas
import pandas
print('pandas: {}'.format(pandas.__version__))
# scikit-learn
import sklearn
print('sklearn: {}'.format(sklearn.__version__))<jupyter_output>Python: 2.7.13 |Continuum Analytics, Inc.| (default, Dec 19 2016, 13:29:36) [MSC v.1500 64 bit (AMD64)]
scipy: 0.19.0
numpy: 1.14.0
matplotlib: 2.0.0
pandas: 0.22.0
sklearn: 0.19.1<jupyter_text>Lesson 2: Get Around In Python, NumPy, Matplotlib and Pandas.You need to be able to read and write basic Python scripts.As a developer, you can pick-up new programming languages pretty quickly. Python is case sensitive, uses hash () for comments and uses whitespace to indicate code blocks (whitespace matters).Today’s task is to practice the basic syntax of the Python programming language and important SciPy data structures in the Python interactive environment.Practice assignment, working with lists and flow control in Python.Practice working with NumPy arrays.Practice creating simple plots in Matplotlib.Practice working with Pandas Series and DataFrames.For example, below is a simple example of creating a Pandas DataFrame.<jupyter_code># dataframe
import numpy
import pandas
myarray = numpy.array([[1, 2, 3], [4, 5, 6]])
rownames = ['a', 'b']
colnames = ['one', 'two', 'three']
mydataframe = pandas.DataFrame(myarray, index=rownames, columns=colnames)
print(mydataframe)<jupyter_output>one  two  three
a    1    2      3
b    4    5      6<jupyter_text>Lesson 3: Load Data From CSVMachine learning algorithms need data. You can load your own data from CSV files but when you are getting started with machine learning in Python you should practice on standard machine learning datasets.Your task for today’s lesson is to get comfortable loading data into Python and to find and load standard machine learning datasets.There are many excellent standard machine learning datasets in CSV format that you can download and practice with on the UCI machine learning repository.Practice loading CSV files into Python using the CSV.reader() in the standard library.Practice loading CSV files using NumPy and the numpy.loadtxt() function.Practice loading CSV files using Pandas and the pandas.read_csv() function.To get you started, below is a snippet that will load the Pima Indians onset of diabetes dataset using Pandas directly from the UCI Machine Learning Repository.<jupyter_code># Load CSV using Pandas from URL
import pandas as pd
url = "https://goo.gl/vhm1eU"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
data = pandas.read_csv(url, names=names)
print(data.shape)
print(data.head())<jupyter_output>(768, 9)
   preg  plas  pres  skin  test  mass   pedi  age  class
0     6   148    72    35     0  33.6  0.627   50      1
1     1    85    66    29     0  26.6  0.351   31      0
2     8   183    64     0     0  23.3  0.672   32      1
3     1    89    66    23    94  28.1  0.167   21      0
4     0   137    40    35   168  43.1  2.288   33      1<jupyter_text>Lesson 4: Understand Data with Descriptive StatisticsOnce you have loaded your data into Python you need to be able to understand it.The better you can understand your data, the better and more accurate the models that you can build. The first step to understanding your data is to use descriptive statistics.Today your lesson is to learn how to use descriptive statistics to understand your data. I recommend using the helper functions provided on the Pandas DataFrame.Understand your data using the head() function to look at the first few rows.Review the dimensions of your data with the shape property.Look at the data types for each attribute with the dtypes property.Review the distribution of your data with the describe() function.Calculate pairwise correlation between your variables using the corr() function.The below example loads the Pima Indians onset of diabetes dataset and summarizes the distribution of each attribute.Attribute Information:1. Number of times pregnant 2. Plasma glucose concentration a 2 hours in an oral glucose tolerance test 3. Diastolic blood pressure (mm Hg) 4. Triceps skin fold thickness (mm) 5. 2-Hour serum insulin (mu U/ml) 6. Body mass index (weight in kg/(height in m)^2) 7. Diabetes pedigree function 8. Age (years) 9. Class variable (0 or 1="tested positive for diabetes")<jupyter_code># Statistical Summary
import pandas
url = "https://goo.gl/vhm1eU"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
data = pandas.read_csv(url, names=names)
description = data.describe()
print(description)<jupyter_output>preg     plas     pres     skin     test     mass     pedi      age    class
count  768.000  768.000  768.000  768.000  768.000  768.000  768.000  768.000  768.000
mean     3.845  120.895   69.105   20.536   79.799   31.993    0.472   33.241    0.349
std      3.370   31.973   19.356   15.952  115.244    7.884    0.331   11.760    0.477
min      0.000    0.000    0.000    0.000    0.000    0.000    0.078   21.000    0.000
25%      1.000   99.000   62.000    0.000    0.000   27.300    0.244   24.000    0.000
50%      3.000  117.000   72.000   23.000   30.500   32.000    0.372   29.000    0.000
75%      6.000  140.250   80.000   32.000  127.250   36.600    0.626   41.000    1.000
max     17.000  199.000  122.000   99.000  846.000   67.100    2.420   81.000    1.000<jupyter_text>Lesson 5: Understand Data with VisualizationContinuing on from yesterday’s lesson, you must spend time to better understand your data.A second way to improve your understanding of your data is by using data visualization techniques (e.g. plotting).Today, your lesson is to learn how to use plotting in Python to understand attributes alone and their interactions. Again, I recommend using the helper functions provided on the Pandas DataFrame.- Use the hist() function to create a histogram of each attribute.- Use the plot(kind=’box’) function to create box-and-whisker plots of each attribute.- Use the pandas.scatter_matrix() function to create pairwise scatterplots of all attributes.For example, the snippet below will load the diabetes dataset and create a scatterplot matrix of the dataset<jupyter_code># Scatter Plot Matrix
import pylab
from matplotlib import pyplot as plt
import pandas
from pandas.plotting import scatter_matrix
url = "https://goo.gl/vhm1eU"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
data = pandas.read_csv(url, names=names)
scatter_matrix(data, figsize=(15,15))
plt.show()<jupyter_output><empty_output><jupyter_text>Lesson 6: Prepare For Modeling by Pre-Processing DataYour raw data may not be setup to be in the best shape for modeling.Sometimes you need to preprocess your data in order to best present the inherent structure of the problem in your data to the modeling algorithms. In today’s lesson, you will use the pre-processing capabilities provided by the scikit-learn.The scikit-learn library provides two standard idioms for transforming data. Each transform is useful in different circumstances: Fit and Multiple Transform and Combined Fit-And-Transform.There are many techniques that you can use to prepare your data for modeling. For example, try out some of the following- Standardize numerical data (e.g. mean of 0 and standard deviation of 1) using the scale and center options.- Normalize numerical data (e.g. to a range of 0-1) using the range option.- Explore more advanced feature engineering such as Binarizing.For example, the snippet below loads the Pima Indians onset of diabetes dataset, calculates the parameters needed to standardize the data, then creates a standardized copy of the input data.<jupyter_code># Standardize data (0 mean, 1 stdev)
import pandas
import numpy
from pandas.plotting import scatter_matrix

url = "https://goo.gl/vhm1eU"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = pandas.read_csv(url, names=names)
array = dataframe.values
# separate array into input and output components
# X = array[:,0:8]
X = dataframe.drop(labels='class', axis=1)
# Y = array[:,8]
Y = dataframe['class']<jupyter_output><empty_output><jupyter_text>Standard scaler<jupyter_code>from sklearn.preprocessing import StandardScaler, MinMaxScaler

scaler = StandardScaler().fit(dataframe.drop(labels='class', axis=1))
rescaledX = pd.DataFrame(scaler.transform(X), columns=X.columns)
# summarize transformed data
numpy.set_printoptions(precision=3)
# print(rescaledX[0:5,:])
# scatter_matrix(rescaledX, figsize=(10,10))
# plt.show()
print(rescaledX.describe())<jupyter_output>preg       plas       pres       skin       test       mass       pedi        age
count  7.680e+02  7.680e+02  7.680e+02  7.680e+02  7.680e+02  7.680e+02  7.680e+02  7.680e+02
mean  -6.476e-17 -9.252e-18  1.503e-17  1.006e-16 -3.007e-17  2.591e-16  2.452e-16  1.931e-16
std    1.001e+00  1.001e+00  1.001e+00  1.001e+00  1.001e+00  1.001e+00  1.001e+00  1.001e+00
min   -1.142e+00 -3.784e+00 -3.573e+00 -1.288e+00 -6.929e-01 -4.060e+00 -1.190e+00 -1.042e+00
25%   -8.449e-01 -6.852e-01 -3.673e-01 -1.288e+00 -6.929e-01 -5.956e-01 -6.890e-01 -7.863e-01
50%   -2.510e-01 -1.219e-01  1.496e-01  1.545e-01 -4.281e-01  9.420e-04 -3.001e-01 -3.608e-01
75%    6.399e-01  6.058e-01  5.632e-01  7.191e-01  4.120e-01  5.848e-01  4.662e-01  6.602e-01
max    3.907e+00  2.444e+00  2.735e+00  4.922e+00  6.653e+00  4.456e+00  5.884e+00  4.064e+00<jupyter_text>Scaling to rangeThis transformation is often used as an alternative to zero mean, unit variance scaling - StandardScaler.<jupyter_code>scaler =  MinMaxScaler(feature_range=(0, 1), copy=True).fit(dataframe.drop(labels='class', axis=1))
rescaledX = pd.DataFrame(scaler.transform(X), columns=X.columns)
# summarize transformed data
numpy.set_printoptions(precision=3)
# print(rescaledX[0:5,:])
# scatter_matrix(pd.DataFrame(rescaledX), figsize=(10,10))
# plt.show()
print(rescaledX.describe())<jupyter_output>preg     plas     pres     skin     test     mass     pedi      age
count  768.000  768.000  768.000  768.000  768.000  768.000  768.000  768.000
mean     0.226    0.608    0.566    0.207    0.094    0.477    0.168    0.204
std      0.198    0.161    0.159    0.161    0.136    0.117    0.141    0.196
min      0.000    0.000    0.000    0.000    0.000    0.000    0.000    0.000
25%      0.059    0.497    0.508    0.000    0.000    0.407    0.071    0.050
50%      0.176    0.588    0.590    0.232    0.036    0.477    0.126    0.133
75%      0.353    0.705    0.656    0.323    0.150    0.545    0.234    0.333
max      1.000    1.000    1.000    1.000    1.000    1.000    1.000    1.000<jupyter_text>Binarizing<jupyter_code>from sklearn.preprocessing import Binarizer


scaler = Binarizer(threshold=0.0, copy=True).fit(X)
rescaledX = pd.DataFrame(scaler.transform(X), columns=X.columns)
# summarize transformed data
numpy.set_printoptions(precision=3)

# scatter_matrix(pd.DataFrame(rescaledX), figsize=(10,10))
# plt.show()
print(rescaledX.describe())
rescaledX.shape<jupyter_output><empty_output><jupyter_text>Lesson 7: Algorithm Evaluation With Resampling MethodsThe dataset used to train a machine learning algorithm is called a training dataset. The dataset used to train an algorithm cannot be used to give you reliable estimates of the accuracy of the model on new data. This is a big problem because the whole idea of creating the model is to make predictions on new data.You can use statistical methods called resampling methods to split your training dataset up into subsets, some are used to train the model and others are held back and used to estimate the accuracy of the model on unseen data.Your goal with today’s lesson is to practice using the different resampling methods available in scikit-learn, for example:- Split a dataset into training and test sets.- Estimate the accuracy of an algorithm using k-fold cross validation.- Estimate the accuracy of an algorithm using leave one out cross validation.The snippet below uses scikit-learn to estimate the accuracy of the Logistic Regression algorithm on the Pima Indians onset of diabetes dataset using 10-fold cross validation.<jupyter_code># Evaluate using Cross Validation
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler, MinMaxScaler

url = "https://goo.gl/vhm1eU"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(url, names=names)
array = dataframe.values
# X = array[:,0:8]
# Y = array[:,8]

X = dataframe.drop(labels='class', axis=1)
# Y = array[:,8]
Y = dataframe['class']

# scaler = MinMaxScaler().fit(dataframe.drop(labels='class', axis=1))
scaler = StandardScaler().fit(dataframe.drop(labels='class', axis=1))
rescaledX = pd.DataFrame(scaler.transform(X), columns=X.columns)

kfold = KFold(n_splits=10, random_state=7)
model = LogisticRegression()
# results = cross_val_score(model, X, Y, cv=kfold)
results = cross_val_score(model, rescaledX, Y, cv=kfold)
print("Accuracy: %.3f%% (%.3f%%)") % (results.mean()*100.0, results.std()*100.0)<jupyter_output>Accuracy: 76.174% (5.218%)<jupyter_text>Accuracy results:- normal data accuracy 76.951% (4.841%)- Standard scaled data accuracy: 77.996% (5.009%) - MinMax scaled (0, 1) data accuracy: 76.174% (5.218%)  Lesson 8: Algorithm Evaluation MetricsThere are many different metrics that you can use to evaluate the skill of a machine learning algorithm on a dataset.You can specify the metric used for your test harness in scikit-learn via the cross_validation.cross_val_score() function and defaults can be used for regression and classification problems. Your goal with today’s lesson is to practice using the different algorithm performance metrics available in the scikit-learn package.- Practice using the Accuracy and LogLoss metrics on a classification problem.- Practice generating a confusion matrix and a classification report.- Practice using RMSE and RSquared metrics on a regression problem.The snippet below demonstrates calculating the LogLoss metric on the Pima Indians onset of diabetes dataset.<jupyter_code># Evaluate using Cross Validation
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB, BernoulliNB
from sklearn.neighbors import KNeighborsRegressor, KNeighborsClassifier
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.metrics import confusion_matrix, classification_report

url = "https://goo.gl/vhm1eU"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(url, names=names)
array = dataframe.values
# X = array[:,0:8]
# Y = array[:,8]

X = dataframe.drop(labels='class', axis=1)
# Y = array[:,8]
Y = dataframe['class']

# scaler = MinMaxScaler().fit(dataframe.drop(labels='class', axis=1))
scaler = StandardScaler().fit(dataframe.drop(labels='class', axis=1))
rescaledX = pd.DataFrame(scaler.transform(X), columns=X.columns)

kfold = KFold(n_splits=10, random_state=7)
model = LogisticRegression()
# model = KNeighborsClassifier()
scorings = [
            'neg_log_loss', 
            'accuracy',
            'f1',
            'neg_mean_squared_error'
           ]

for scoring in scorings:
    results = cross_val_score(model, rescaledX, Y, cv=kfold, scoring=scoring)
    print("%s: %.4f (%.4f)") % (scoring, results.mean(), results.std())<jupyter_output>neg_log_loss: -0.4843 (0.0612)
accuracy: 0.7800 (0.0501)
f1: 0.6475 (0.0487)
neg_mean_squared_error: -0.2200 (0.0501)<jupyter_text>LogisticRegression:- neg_log_loss: -0.4843 (0.0612)- accuracy: 0.7800 (0.0501)- f1: 0.6475 (0.0487)- neg_mean_squared_error: -0.2200 (0.0501)KNeighborsClassifier:- neg_log_loss: -1.7540 (0.7222)- accuracy: 0.7421 (0.0715)- f1: 0.6053 (0.1119)- neg_mean_squared_error: -0.2579 (0.0715)  Confusion Matrix<jupyter_code>print(__doc__)

import itertools
import numpy as np
import matplotlib.pyplot as plt
from utils import plot_confusion_matrix

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB, BernoulliNB
from sklearn.neighbors import KNeighborsRegressor, KNeighborsClassifier
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.metrics import confusion_matrix, classification_report

url = "https://goo.gl/vhm1eU"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(url, names=names)
array = dataframe.values
# X = array[:,0:8]
# Y = array[:,8]

X = dataframe.drop(labels='class', axis=1)
# Y = array[:,8]
Y = dataframe['class']
class_names = ['negative', 'positive']
# scaler = MinMaxScaler().fit(dataframe.drop(labels='class', axis=1))
scaler = StandardScaler().fit(dataframe.drop(labels='class', axis=1))
rescaledX = pd.DataFrame(scaler.transform(X), columns=X.columns)

# Split the data into a training set and a test set
X_train, X_test, y_train, y_test = train_test_split(rescaledX, Y, random_state=0)

classifier = LogisticRegression()
y_pred = classifier.fit(X_train, y_train).predict(X_test)

# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, y_pred)
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure(figsize=(5,5))
plot_confusion_matrix(cnf_matrix, classes=class_names,
                      title='Confusion matrix, without normalization')

# Plot normalized confusion matrix
plt.figure(figsize=(5,5))
plot_confusion_matrix(cnf_matrix, classes=class_names, normalize=True,
                      title='Normalized confusion matrix')

plt.show()<jupyter_output>Automatically created module for IPython interactive environment
Confusion matrix, without normalization
[[118  12]
 [ 26  36]]
Normalized confusion matrix
[[0.91 0.09]
 [0.42 0.58]]<jupyter_text>Classification Report PrecisionThe precision is the ratio `tp / (tp + fp)` where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative. RecallThe recall is the ratio `tp / (tp + fn)` where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples. F1-scoreAlso known as balanced F-score or F-measure. The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The formula for the F1 score is:```F1 = 2 * (precision * recall) / (precision + recall)```In the multi-class and multi-label case, this is the weighted average of the F1 score of each class.<jupyter_code>print(classification_report(y_test, y_pred, target_names=class_names))<jupyter_output>precision    recall  f1-score   support

   negative       0.82      0.91      0.86       130
   positive       0.75      0.58      0.65        62

avg / total       0.80      0.80      0.79       192<jupyter_text>Practice using RMSE and RSquared metrics on a regression problem<jupyter_code>scaler.transform
y_test.shape


12./(118+12)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09371241641410497, 'fraction_numerical': 0.07384422956655602, 'mean_word_length': 4.149368342828834, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '483484', 'license': 'MIT', 'n_tokens_mistral': 7465, 'n_tokens_neox': 6512, 'n_words': 2366, 'path': 'mlm.ipynb', 'repo_name': 'rokoplestina/MLM', 'size': '20787'}