<jupyter_start><jupyter_text>Representation of Data in Scikit-learnMachine learning is about creating models from data: for that reason, we'll start bydiscussing how data can be represented in order to be understood by the computer.  Most machine learning algorithms implemented in scikit-learn expect data to be stored in a**two-dimensional array or matrix**.  The arrays can beeither ``numpy`` arrays, or in some cases ``scipy.sparse`` matrices.The size of the array is expected to be `[n_samples, n_features]`- **n_samples:**   The number of samples: each sample is an item to process (e.g. classify).  A sample can be a document, a picture, a sound, a video, an astronomical object,  a row in database or CSV file,  or whatever you can describe with a fixed set of quantitative traits.- **n_features:**  The number of features or distinct traits that can be used to describe each  item in a quantitative manner.  Features are generally real-valued, but may be boolean or  discrete-valued in some cases.The number of features must be fixed in advance. However it can be very high dimensional(e.g. millions of features) with most of them being zeros for a given sample. This is a casewhere `scipy.sparse` matrices can be useful, in that they aremuch more memory-efficient than numpy arrays.   A Simple Example: the Iris DatasetAs an example of a simple dataset, we're going to take a look at theiris data stored by scikit-learn.The data consists of measurements of three different species of irises.There are three species of iris in the dataset, which we can picture here:      For the purposes of this exercise, let's pretend we're working for a startup that just got funded to create a smartphone app that automatically identifies species of flowers from pictures taken on the smartphone. We're working with a moderately-sized team of data scientists and will be building part of the data analysis pipeline for this app.We've been tasked by our company's Head of Data Science to create a demo machine learning model that takes four measurements from the flowers (sepal length, sepal width, petal length, and petal width) and identifies the species based on those measurements alone. Data set :We've been given a data set from our field researchers to develop the demo, which only includes measurements for three types of Iris flowers:                                 Iris setosa                             Iris versicolor                             Iris virginica                                                                                                                                        The four measurements we're using currently come from hand-measurements by the field researchers, but they will be automatically measured by an image processing model in the future.  Quick Question:**If we want to design an algorithm to recognize iris species, what might the data be?**Remember: we need a 2D array of size `[n_samples x n_features]`.- What would the `n_samples` refer to?- What might the `n_features` refer to?Remember that there must be a **fixed** number of features for each sample, and featurenumber ``i`` must be a similar kind of quantity for each sample.  Loading the Iris Data with Scikit-LearnScikit-learn has a very straightforward set of data on these iris species.  The data consist ofthe following:- Features in the Iris dataset:  1. sepal length in cm  2. sepal width in cm  3. petal length in cm  4. petal width in cm- Target classes to predict:  1. Iris Setosa  2. Iris Versicolour  3. Iris Virginica  ``scikit-learn`` embeds a copy of the iris CSV file along with a helper function to load it into numpy arrays:<jupyter_code>#import sys
#sys.path
import sklearn
sklearn.datasets.load_iris
from sklearn.datasets import load_iris
iris = load_iris()
iris.keys()
X = iris.data
X.shape
Y = iris.target
Y.shape
iris.target_names
set(iris.target)
for item in iris.target:
    print(iris.target_names[item])
iris.DESCR
n_samples, n_features = iris.data.shape
print((n_samples, n_features))
print(iris.data[0])
print(iris.data.shape)
print(iris.target.shape)
print(iris.target)
print(iris.target_names)<jupyter_output>['setosa' 'versicolor' 'virginica']<jupyter_text>This data is four dimensional, but we can visualize two of the dimensionsat a time using a simple scatter-plot:<jupyter_code>X = iris.data
X
print(X)
X=1
x_index = 0
y_index = 1
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

x_index = 0
y_index = 1

# this formatter will label the colorbar with the correct target names
formatter = plt.FuncFormatter(lambda i, *args: iris.target_names[int(i)])

X = iris.data

#for i in range(0,6):
#    plt.subplot(1,6,i+1)
#    plt.scatter(X[:,i],Y,c=iris.target, cmap=plt.cm.get_cmap('RdYlBu', 3))
plt.scatter(X[:,0],X[:,1],c=iris.target, cmap=plt.cm.get_cmap('RdYlBu', 3))


# plt.colorbar(ticks=[0, 1, 2], format=formatter)
# plt.clim(-0.5, 2.5)
# plt.xlabel(iris.feature_names[x_index])
# plt.ylabel(iris.feature_names[y_index]);
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07114467408585055, 'fraction_numerical': 0.005961844197138315, 'mean_word_length': 3.82088122605364, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '262415', 'license': 'MIT', 'n_tokens_mistral': 1446, 'n_tokens_neox': 1402, 'n_words': 660, 'path': 'Lectures/Lecture-03/SK02.ipynb', 'repo_name': 'khalooei/TMLDL2018', 'size': '5032'}