<jupyter_start><jupyter_text>Week 1: Getting Started with Jupyter NotebooksIn this notebook, we will make sure all the packages required for this course are properly installed and working. To use this notebook, select the input cells (shown as `In [x]`) in order and press Shift-Enter to execute the code. Your installation is properly working if none of the cells below return any errors. Loading and Testing the Course's Required PackagesIn this notebook we will load and test each of packages that we will mainly be using during the course: Numpy, Matplotlib, Pandas, and SciKit-Learn.  NumpyFirst, we will import NumPy. NumPy is a linear algebra library, and provides useful vector and matrix functionality, similar to MATLAB. It is convention to define NumPy as `np` for the sake of brevity:<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>Make a vector with 6 elements:<jupyter_code>a = np.array([1,2,3,4,5,6])

# Print the contents of a
a<jupyter_output><empty_output><jupyter_text>Get some information about the vector:<jupyter_code>print("The vector a has " + str(a.ndim) + " dimension(s) and has the shape " + str(a.shape) + ".")<jupyter_output>The vector a has 1 dimension(s) and has the shape (6,).<jupyter_text>Create a matrix like this:<jupyter_code>m = np.array([[1,2,3], [4,5,6]])

m<jupyter_output><empty_output><jupyter_text>Get some information about the matrix:<jupyter_code>print("The matrix m has " + str(m.ndim) + " dimension(s) and has the shape " + str(m.shape) + ".")<jupyter_output>The matrix m has 2 dimension(s) and has the shape (2, 3).<jupyter_text>A very powerful feature of NumPy and Python are _List Comprehensions_. These can replace many `for` loops and are much more efficient to run. Here we square every element in the vector `a` from above:<jupyter_code>a_squared = [i**2 for i in a]

a_squared<jupyter_output><empty_output><jupyter_text>Using NumPy we can select rows and columns of data very easily (known as _array slicing_). For example, we can print the first row of the matrix, `m`:<jupyter_code>m[0]<jupyter_output><empty_output><jupyter_text>Or we can slice the first column only. Using the `,` symbol we can ask for specfic rows and columns. The first integer __always specifies the rows__, which is followed by `,` and the second integer __specifies the columns__. The colon character `:` is shorthand for _all rows_ or _all columns_. Here we select all rows of matrix `m` using `:` and select the first column, using `0`:<jupyter_code>m[:,0]<jupyter_output><empty_output><jupyter_text>We can select a specific element using the matrix's column and row index, for example we want to select the item in the second column's second row:<jupyter_code>m[1,1]<jupyter_output><empty_output><jupyter_text>Entire books have been written about NumPy. Let's move on to Matplotlib.  MatplotlibFirst we will load Matplotlib, a 2D plotting library, and run `%matplotlib inline`. This magic function (functions beginning with the % symbol are called magic functions) will display any plots inline in the notebook.<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Using the `plt.plot()` function, you can plot many types of data and Matplotlib will try to figure out what you want to do with the data:<jupyter_code>plt.plot([1,2,3,4,5])<jupyter_output><empty_output><jupyter_text>The `plt.plot()` function will take an arbritrary number of `x` `y` argument pairs:<jupyter_code># x**y is shorthand for x to the power of y
plt.plot([1,2,3,4,5],[1**2,2**2,3**2,4**2,5**2])<jupyter_output><empty_output><jupyter_text>Or plot three lines at once by supplying three pairs of `x` `y` values:<jupyter_code>p = np.arange(1,10) # Get a range of numbers form 1 to 10
plt.plot(p, p, p, p**2, p, p**3)<jupyter_output><empty_output><jupyter_text>Combined with functions, more complex curves can be plotted. Let's plot the sigmoid curve:<jupyter_code>import math
def sigmoid(x):
    a = []
    for item in x:
        a.append(1/(1+math.exp(-item)))
    return a

x = np.arange(-10., 10., 0.1)
sig = sigmoid(x)

plt.plot(x,sig)<jupyter_output><empty_output><jupyter_text>PandasLet's now import Pandas. This library provides R-style dataframe table functionality. Like NumPy, it is convention to import the Pandas library as `pd` for brevity.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Let's load the well known Wisconsin Breast Cancer dataset:<jupyter_code>df = pd.read_csv("https://raw.githubusercontent.com/mdbloice/Machine-Learning-for-Health-Informatics/master/data/breast-cancer-wisconsin.csv")<jupyter_output><empty_output><jupyter_text>You can view a summary of the data using the `describe()` function:<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Let's rename the column names:<jupyter_code>df.columns = ["ID","Clump_Thickness","Size_Uniformity","Shape_Uniformity","Marginal_Adhesion","Epithelial_Size","Bare_Nucleoli","Bland_Chromatin","Normal_Nucleoli","Mitoses","Class"]

# Print the new header names:
df.columns<jupyter_output><empty_output><jupyter_text>See the first few rows:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Notice how the `Class` column (the last column in the table) consists of 2s and 4s. In this case 2 stands for malignant and 4 stands for benign. You can check this quickly using:<jupyter_code>df.Class.unique()<jupyter_output><empty_output><jupyter_text>So you see only 2s and 4s are contained in this column. See a breakdown of the counts using the `value_counts()` function:<jupyter_code>df.Class.value_counts()<jupyter_output><empty_output><jupyter_text>However, it is convention in machine learning to use a 0-based index to represent classes. Let's replace the 2s with 0s and the 4s with 1s:<jupyter_code>df = df.replace({"Class": {2: 0, 4: 1}})
df.Class.value_counts()<jupyter_output><empty_output><jupyter_text>Perhaps now we would like to drop the `ID` column:<jupyter_code>df = df.drop(["ID"], axis=1)
df.describe()<jupyter_output><empty_output><jupyter_text>As you can see you can use Pandas to quickly manipulate and access tabular data. Here we access the first 10 rows of the `Size_Uniformity` column:<jupyter_code>df.Size_Uniformity[0:10]<jupyter_output><empty_output><jupyter_text>Columns can also be accessed using the name of the column as an index:<jupyter_code>df['Size_Uniformity'][0:10]<jupyter_output><empty_output><jupyter_text>You can examine the data types (Pandas dataframes can contain multiple types):<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>The column `Bare_Nucleoli` appears as type `object` as it contains some missing data, which appear as `?` in the dataset. Later in the course we will learn how to handle missing data.You can also use Pandas to perform quick statistical analyses. Here we calculate the standard deviation for each column:<jupyter_code>df.std()<jupyter_output><empty_output><jupyter_text>Or calculate the standard deviation for a certain column:<jupyter_code>df.Clump_Thickness.std()<jupyter_output><empty_output><jupyter_text>Pandas also provides useful plotting tools. To look for correlations in data, a scatter matrix is often useful. Here we will plot __only three columns of the data__, and only the __first 100 rows of the data__, as a scatter plot with so many columns can take some time to render and can result in a very large plot.<jupyter_code>from pandas.tools.plotting import scatter_matrix

# Manually select three of the table's columns by passing an array of column names:
df_subset = df[['Clump_Thickness','Size_Uniformity', 'Shape_Uniformity']]

# The semi colon at the end of this line is to suppress informational output (we only want to see the plot)
scatter_matrix(df_subset.head(100), alpha=0.2, figsize=(6,6), diagonal='kde');<jupyter_output><empty_output><jupyter_text>SciKit-LearnLast but not least, we shall import some modules from SciKit-Learn. SciKit-Learn is the main machine learning library for Python. It is a large library and is not normally loaded entirely; in general you load only the modules you need from the main library. Here we will load the `datasets` module and the _k_-nearest neighbours module (`KNeighborsClassifier`):<jupyter_code>from sklearn import datasets
from sklearn.neighbors import KNeighborsClassifier<jupyter_output><empty_output><jupyter_text>Load the Iris dataset (a flower data set often used for demonstration purposes):<jupyter_code>iris = datasets.load_iris()<jupyter_output><empty_output><jupyter_text>Convention states that matrices are represented using uppercase letters, often the letter `X`, and label vectors are represented using lower case letters, often `y`:<jupyter_code>X = iris.data
y = iris.target<jupyter_output><empty_output><jupyter_text>The k-Nearest Neighbour algorithm is possibly the simplest classifier. Given a new observation, take the label of the sample closest to it in the _n_-dimensional feature space. First we must randomise the data, but we must ensure we randomise the labels as well __in sync__. We can use a NumPy feature to create indices that then correlate to both the targets and the data:<jupyter_code>np.random.seed(376483)
random_indices = np.random.permutation(len(y))

random_indices<jupyter_output><empty_output><jupyter_text>Then, the data must be split into a test set and a training set (again we are using naming conventions here for the training and test data `X_train` and `X_test` and their labels `y_train` and `y_test`):<jupyter_code>X_train = X[random_indices[:-10]]
X_test  = X[random_indices[-10:]]

y_train = y[random_indices[:-10]]
y_test  = y[random_indices[-10:]]

print("Number of training samples: %d. Number of test samples: %d." % (len(X_train), len(X_test)) )<jupyter_output>Number of training samples: 140. Number of test samples: 10.<jupyter_text>Now we will try to fit the _k_-nearest neighbours classifier to the training data:<jupyter_code>knn = KNeighborsClassifier() # Initialise the classifier.
knn.fit(X_train, y_train)    # Fit the classifier.<jupyter_output><empty_output><jupyter_text>The classifier has now been trained on the training data (`X_train`). We can now check how well it predicts newly seen data (using our test set, `X_test`):<jupyter_code>y_pred = knn.predict(X_test)

# The classifier's predicted labels are now contained in y_pred:
y_pred<jupyter_output><empty_output><jupyter_text>We can now element-wise compare our predicted results, in `y_pred`, with the true labels stored in `y_test`:<jupyter_code>y_pred == y_test<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09258037145281418, 'fraction_numerical': 0.010841896860563778, 'mean_word_length': 6.486238532110092, 'pattern_counts': {'":': 1, '<': 146, '<?xml version=': 0, '>': 146, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '218466', 'license': 'MIT', 'n_tokens_mistral': 3500, 'n_tokens_neox': 3345, 'n_words': 1188, 'path': 'Assignment1-Complete.ipynb', 'repo_name': 'mdbloice/Machine-Learning-for-Health-Informatics', 'size': '10607'}