<jupyter_start><jupyter_text>NumPy<jupyter_code>import numpy as np
x = np.array([[1,2,3], [4,5,6]])
print("x:\n{}".format(x))<jupyter_output>x:
[[1 2 3]
 [4 5 6]]<jupyter_text>SciPy<jupyter_code>from scipy import sparse
# create a 2D numpy array with a diagonal of ones (1's), and zeros (0's) everywhere else.
eye = np.eye(4)
print("Numpy array:\n{}".format(eye)) #without {} nothing is printed
#Covert a numpy array to a psarse matrix in CSR format
#only the non zero enteries are stored
sparse_matrix = sparse.csr_matrix(eye)
print("\nScipy sparse matrix in CSR format:\n{}".format(sparse_matrix))
data = np.ones(4)
row_indices = np.arange(4)
col_indices = np.arange(4)
eye_coo = sparse.coo_matrix((data, (row_indices, col_indices)))
print("COO representation:\n{}".format(eye_coo))<jupyter_output>COO representation:
  (0, 0)	1.0
  (1, 1)	1.0
  (2, 2)	1.0
  (3, 3)	1.0<jupyter_text>matplotlib<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt

#generate sequence numbers from -10 to 10 with 100 steps in between
x = np.linspace(-10, 10, 100)

#Create a second array using sinus
y = np.sin(x)

#The plot function makes a line chart of one array against another
plt.plot(x, y, marker="x")<jupyter_output><empty_output><jupyter_text>pandas<jupyter_code>import pandas as pd
#create a simple dataset of people
data = { 'Name':['john', 'anna', 'peter', 'linda'],
         'Location': ['New York', 'Paris', 'London', 'Boston'],
         'Age': ['24', '13', '53', '33'],
         'Education':['BS', 'Matric', 'PhD', 'MS' ]        
}
data_pandas = pd.DataFrame(data)
display(data_pandas)
#Ipython.display or other to display nicely
#import mglearn # create your own library
#Query
#Select all rows that have an age column greater than 30
#print("" + data_pandas[data_pandas.Age > 30])
#versions used in this code
import sys
print("Python Version: {}".format(sys.version))

import pandas as pd
print("Pandas version: {}".format(pd.__version__))

import matplotlib as mp
print("matplotlib version: {}".format(mp.__version__))

import numpy as np
print("numpy version: {}".format(np.__version__))

import scipy as sp
print("scipy version: {}".format(sp.__version__))

import IPython
print("Ipython version: {}".format(IPython.__version__))

import sklearn
print("sklearn version: {}".format(sklearn.__version__))

import jupyter_core
print("jupyter_core version: {}".format(jupyter_core.__version__))<jupyter_output>Python Version: 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 10:22:32) [MSC v.1900 64 bit (AMD64)]
Pandas version: 0.22.0
matplotlib version: 2.1.2
numpy version: 1.14.0
scipy version: 1.0.0
Ipython version: 6.2.1
sklearn version: 0.19.1
jupyter version: 4.4.0<jupyter_text>First Application of ML: Classifying iris species<jupyter_code>from sklearn.datasets import load_iris
iris_dataset = load_iris() #it returns a dictionary which contains keys and values
print("Keys of iris dataset: \n{}".format(iris_dataset.keys()))
print(iris_dataset['DESCR'][:193] + "\n...")
print(iris_dataset['DESCR'][:212] + "\n...") #As we increase the last position, we see more characters
print("Target names: \n{}".format(iris_dataset['target_names'])) #'setosa' 'versicolor' 'virginica': the species of the flower we want to predict
print("Feature names: \n{}".format(iris_dataset['feature_names'])) #Features
print("Type of Data: \n{}".format(type(iris_dataset['data'])))<jupyter_output>Type of Data: 
<class 'numpy.ndarray'><jupyter_text>The rows in teh data array correspond to flowerswhile teh column represent teh four measurements that were taken for each flowerThe data itself is contained in target and data fields. data contaisn teh numeric measurements of: a. sepal lengthb. sepal widthc. petal lengthd. petal widthin a Numpy array<jupyter_code>print("Shape of Data: \n{}".format(iris_dataset['data'].shape))<jupyter_output>Shape of Data: 
(150, 4)<jupyter_text>We can see that array contains measurements for 150 flowers (rows), The individual items are called samples in machine Learning and their properties are called features. The shape of teh data array is the number of samples multiplied by the number of features<jupyter_code>print("Data: \n{}".format(iris_dataset['data'][:6]))<jupyter_output>Data: 
[[5.1 3.5 1.4 0.2]
 [4.9 3.  1.4 0.2]
 [4.7 3.2 1.3 0.2]
 [4.6 3.1 1.5 0.2]
 [5.  3.6 1.4 0.2]
 [5.4 3.9 1.7 0.4]]<jupyter_text>The feature values for teh first 6 samplesFrom this data we can notice that first 5 samples ahve petal width of 0.2 cm while 6th has 0.4 cmwhile last sample has the longest sepal (5.4 cm) The target array contains the species of each of the flowers that were also measured as a Numpy array<jupyter_code>print("Type of Target: \n{}".format(type(iris_dataset['target']))) #target is a 1-D array with one entry epr flower
print("Shape of Target: \n{}".format(iris_dataset['target'].shape)) #
print("Target: \n{}".format(iris_dataset['target'])) #The species are encoded as integers from 0 to 2<jupyter_output>Target: 
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
 2 2]<jupyter_text>The meanings of the numbers are given by iris['target_names'] array:        0 means setosa        1 means versicolor        2 means virginica  Training and Testing data<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    iris_dataset['data'], iris_dataset['target'], random_state = 0)
print("X_train shape: {}".format(X_train.shape))
print("y_train shape: {}".format(y_train.shape))
print("X_test shape: {}".format(X_test.shape))
print("y_test shape: {}".format(y_test.shape))<jupyter_output>X_test shape: (38, 4)
y_test shape: (38,)<jupyter_text>Data Exploration<jupyter_code>#Create dataframe from data in X_train
#label the columns using strings in iris_dataset.feature_names


iris_dataframe = pd.DataFrame(X_train, columns=iris_dataset.feature_names)

#create a scatter matrix from teh dataframe color by y_train

grr = pd.scatter_matrix(iris_dataframe, c=y_train, figsize=(15, 15), marker='o',
                        hist_kwds = {'bins':20}, s = 60, alpha = .8) #cmap = mglearn.cm3<jupyter_output>C:\Users\cnaseeb\AppData\Local\Continuum\anaconda2\envs\py36\lib\site-packages\ipykernel_launcher.py:10: FutureWarning: pandas.scatter_matrix is deprecated. Use pandas.plotting.scatter_matrix instead
  # Remove the CWD from sys.path while we load stuff.<jupyter_text>K-Nearest Neighbours<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=1)
#Build the model
knn.fit(X_train, y_train)
#making Predictions
X_new = np.array([[5, 2.9, 1, 0.2]])
print("X_new.shape: {}".format(X_new.shape))
#predict

prediction = knn.predict(X_new)
print("Prediction: {}".format(prediction))
print("Predicted target name: {}".format(iris_dataset['target_names'][prediction]))<jupyter_output>Prediction: [0]
Predicted target name: ['setosa']<jupyter_text>Evaluating the model<jupyter_code>y_pred = knn.predict(X_test)
print("Test set predictions:\n {}".format(y_pred))
print("Test set score: {:.2f}".format(np.mean(y_pred == y_test)))
#test set accuracy
print("Test set score: {:.2f}".format(knn.score(X_test, y_test)))<jupyter_output>Test set score: 0.97<jupyter_text>Summary<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(
    iris_dataset['data'], iris_dataset['target'], random_state = 0)

knn= KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)

print("Test set score: {:.2f}".format(knn.score(X_test, y_test)))
## Thanks to <NAME>
@amueller: Using yoru book and repository, I have created this. I'll expand the exercises (notebooks) further with my 
    comments and additions.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1311743951612903, 'fraction_numerical': 0.04662298387096774, 'mean_word_length': 5.215348472983555, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '307078', 'license': 'Apache-2.0', 'n_tokens_mistral': 3166, 'n_tokens_neox': 2851, 'n_words': 909, 'path': '01-introduction.ipynb', 'repo_name': 'cnaseeb/AIHub', 'size': '7945'}