<jupyter_start><jupyter_text>Numpy and matplotlibMulti-dimensional arrays and 2D visualization in Python.Author: <NAME> <EMAIL>    With many thanks to [<NAME>](http://alexandre.gramfort.net/) (Telecom ParisTech).__This notebook contains 5 small problems. Make sure you have done them before moving on to the next notebook.__  1. Numpy`numpy` is a python package used in most scientific and numerical computing in Python. It provides performing data structures for manipulating vectors, matrices and tensors of arbitrary dimensions.It is written in C and Fortran, and therefore performs very well on vector/matrix computations.<jupyter_code># Import the numpy module
import numpy as np<jupyter_output><empty_output><jupyter_text>1.1 Numpy arraysNumPy arrays are a fundamental structure for scientific computing. Numpy arrays are homogeneous (i.e. all objects it contains have the same type) multi-dimensional arrays, which we’ll use among other things to represent vectors and matrices.Let us explore some basic Numpy commands. 1.2 Creating arrays<jupyter_code># Create a 1D vector from a list
v = np.array([1, 2, 3, 4])
print v
print type(v)
# Create a random array of size 3 x 5
X = np.random.random((3, 5))
print X
print type(X)
# Create an array of zeros of size 3 x 5
np.zeros((3, 5))
# Create an array of ones of size 3 x 5
np.ones((3, 5))
# Create the identity matrix of size 4 x 4
np.eye(4)
# Create a range
np.arange(0, 10, 2) # arguments: start, stop, step
# Create n points evenly spreading the [stop, start] interval 
# with linspace, both start and stop are included
np.linspace(0, 10, 3) # start, stop, n
# Same in log scale
np.logspace(0, 10, 3, base=np.e)<jupyter_output><empty_output><jupyter_text>1.3 Information about arrays<jupyter_code># The dimensions of an array are accessible via shape
print v.shape
print X.shape
# The total number of elements of an array are accessible via size
print v.size
print X.size
# The number of dimensions of an array
print v.ndim
print X.ndim<jupyter_output><empty_output><jupyter_text>1.4 Array typesProceed with caution!All elements of an array have the same type, accessible via dtype<jupyter_code>print v.dtype
print X.dtype
a = np.array([1, 2, 3])
print a.dtype

a[0] = 3.2
print a.dtype
print a<jupyter_output><empty_output><jupyter_text>The type of the elements of an array can be explicitely defined with the `dtype` keyword, to be chosen among `int`, `float`, `complex`, `bool`, `int64`, etc.<jupyter_code># A first solution
a = np.array([1,2,3], dtype=np.float)
print a.dtype

a[0] = 3.2
print a.dtype
print a<jupyter_output><empty_output><jupyter_text>The type of the elements of an array can be modified with the `astype` function:<jupyter_code># A second solution
a = np.array([1,2,3])
print a.dtype

a = a.astype(float)

a[0] = 3.2
print a.dtype
print a<jupyter_output><empty_output><jupyter_text>1.5 Accessing elements, rows, and columns of arraysRemember, in Python indices start at 0.<jupyter_code># Get a single element of a vector
v[0]
# Get a single element of a matrix
print X[0, 1]
# Get a row
print X[0, :]
print X[0]
print "shape of a row vector:", X[0].shape
# Get a column
print X[:, 3]<jupyter_output><empty_output><jupyter_text>1.6 SlicingThe `[start:stop:step]` syntax applicable to strings and lists also applies to arrays.<jupyter_code># Access a sub-matrix
print X[1:3, :]
# Modify a slice
X[1:3] = np.zeros((2, 5))
print X
X[::2, ::2]<jupyter_output><empty_output><jupyter_text>1.7 Fancy indexingIt is possible to use lists or arrays to define slices<jupyter_code>A = np.array([range(5), range(0, 10, 2), range(1, 6), range(5, 15, 2)])
print A
row_indices = [0, 1, 3]
print A[row_indices]
print(A[[1, 2], [3, 4]])
A[np.ix_([1, 2], [3, 4])] = 0
print A<jupyter_output><empty_output><jupyter_text>1.8 Masks[Masks](https://docs.scipy.org/doc/numpy/reference/maskedarray.generic.html) provide a convenient way to deal with array entries that should not be used in computations, and are used in particular to deal with missing data. Those entries are not removed from the array but simply masked.<jupyter_code>B = np.arange(5)
print B
row_mask = np.array([True, False, True, False, False])
print B[row_mask]
# Equivalently
row_mask = np.array([1,0,1,0,0], dtype=bool)
print B[row_mask]
print B<3
# Convert a binary mask in position indices using *where*
positions = np.where(B<3)
print positions

print B[positions]
a = np.array([1, 2, 3, 4, 5])
print a < 3
print B[a < 3]
print A[:, a<3]<jupyter_output><empty_output><jupyter_text>1.9 Reshaping arrays<jupyter_code>print A
n, m = A.shape
# Reshape A
B = A.reshape((1, n*m))
print B
print A
B[0:5] = -1
print B
print A<jupyter_output><empty_output><jupyter_text>The original variable was also modified! B is only a new _view_ of A.To create a new array that's the vector (1D) version of A, we use `flatten`:<jupyter_code>B = A.flatten()
print B
B.shape
B[0:5] = 10
print B
print A<jupyter_output><empty_output><jupyter_text>1.10 Concatenating arrays  1.10.1 Repeat and tile<jupyter_code>a = np.array([[1, 2], [3, 4]])
print a
# repeat each item 3 times
print np.repeat(a, 3)
# The result was 1D!
print np.repeat(a, 3, axis=1)
# repeat the matrix 3 times
print np.tile(a, 3)<jupyter_output><empty_output><jupyter_text>1.10.2 Concatenate<jupyter_code>b = np.array([[5, 6]])
print b.shape
print np.concatenate((a, b), axis=0)
print np.concatenate((a, b.T), axis=1)<jupyter_output><empty_output><jupyter_text>1.10.3 Stack arrays<jupyter_code>np.vstack((a,b))
np.hstack((a,b.T))<jupyter_output><empty_output><jupyter_text>1.11 Array manipulationWe use 2-dimensional arrays to represent matrices, and can do basic linear algebra operations on them. 1.11.1 Transposing an array<jupyter_code>print X.T<jupyter_output><empty_output><jupyter_text>1.11.2 Applying a transformation to all entries of an array<jupyter_code># Multiply all entries of X by 2:
print 2*X<jupyter_output><empty_output><jupyter_text>Problem 2.1Compute the following matrices:* the one obtained by adding 1 to all entries of X* the one which entries are the logarithm (base 2) of the previous one* the one which entries are the squares of the entries of X<jupyter_code># Add 1 to all entries of x
# TODO

# Compute the array that has as entries the logarithm (base 2) of the entries of X
# TODO

# Square all entries of X
# TODO<jupyter_output><empty_output><jupyter_text>1.11.3 Matrix multiplication<jupyter_code># Element-wise matrix multiplication
print X*X
# Matrix multiplication 
print np.dot(X, X.T)
print X.dot(X.T)<jupyter_output><empty_output><jupyter_text>Problem 2.2* Create a random array B of size 5 x 4* Multiply X by B<jupyter_code># TODO<jupyter_output><empty_output><jupyter_text>1.11.4 Diagonal of a matrix<jupyter_code>X = np.random.random((3, 5))
# Get the diagonal of  X. Note that X is not square.
np.diag(X)
# Compute the trace of  X
np.trace(X)<jupyter_output><empty_output><jupyter_text>1.11.5 Linear algebraMore complex linear algebra operations are available via numpy.linalg:http://docs.scipy.org/doc/numpy/reference/routines.linalg.html<jupyter_code># Compute the determinant of M=XX'
M = np.dot(X, X.T)
np.linalg.det(M)
# Compute the eigenvalues and eigenvectors of M
v, w = np.linalg.eig(M)
print "eigenvalues:", v
print "eigenvectors:", w
# Compute the inverse of M
print np.linalg.inv(M)

# Check the product of M by its inverse is the identity
print np.dot(M, np.linalg.inv(M))<jupyter_output><empty_output><jupyter_text>1.12 Simple statisticsWe'll start from a [Vandermonde matrix](http://mathworld.wolfram.com/VandermondeMatrix.html)<jupyter_code># Generate a Vandermonde matrix
data = np.vander([1, 2, 3, 4])
print data<jupyter_output><empty_output><jupyter_text>1.12.1 Mean<jupyter_code>print np.mean(data)
print np.mean(data, axis=0) # column-wise
print np.mean(data, axis=1) # row-wise
print np.mean(data[:, 2]) # for a given column<jupyter_output><empty_output><jupyter_text>1.12.2 Standard deviation<jupyter_code># variance
print np.var(data[:, 2]) # for a given column

# standard deviation
print np.std(data[:, 2])

# Check the relationship between variance and standard deviation
print np.sqrt(np.var(data[:, 2]))
print np.var(data) # for the entire matrix
print np.std(data, axis=1) # row-wise<jupyter_output><empty_output><jupyter_text>1.12.3 Min and max<jupyter_code>data.min()
data.max(axis=0)
data.sum()<jupyter_output><empty_output><jupyter_text>1.12.4 sum and product<jupyter_code>print np.sum(data[:, 2])
data[:, 2].sum()
data[:, 2].prod()
# cumulative sum
np.cumsum(data[:, 2])
np.cumsum(data[:, 2]+1)<jupyter_output><empty_output><jupyter_text>Problem 2.3Using Numpy and no loop, compute an approximation of $\pi$ with Wallis' product: $\pi = 2 \prod_{n=1}^\infty \frac{4 n^2}{4 n^2 - 1}$<jupyter_code># TODO<jupyter_output><empty_output><jupyter_text>1.13 Reading/writing arrays from/to csv files`data.csv` is a toy data set (4 rows of 5 columns). Let us read it:<jupyter_code>!cat data.csv
# Create a numpy array directly from a csv file
M = np.genfromtxt('data.csv', delimiter=',', dtype='int')
print M
# Save a numpy array to a csv file
M = np.random.rand(3, 3)
print M
np.savetxt("random-matrix.txt", M)
! cat random-matrix.txt
np.savetxt("random-matrix.txt", M, fmt='%.3f', delimiter=',')

! cat random-matrix.txt<jupyter_output><empty_output><jupyter_text>For more on arrays, you can refer to http://docs.scipy.org/doc/numpy/reference/arrays.html.For more about NumPy, you can refer to:* The Tentative NumPy Tutorial at http://wiki.scipy.org/Tentative_NumPy_Tutorial;* The NumPy documentation at http://docs.scipy.org/doc/numpy/index.html.  2. MatplotlibVisualization is an important part of machine learning. Plotting your data will allow you to have a better feel for it (how are the features distributed, are there outliers, etc.). Plotting measures of performance (whether ROC curves or single-valued performance measures, with error bars) allows you to rapidly compare methods.matplotlib is a very flexible data visualization package, partially inspired by MATLAB. matplotlib can be imported with    from matplotlib import *    but in a Jupyter notebook, using the     % pylab inline    will include your plots in your notebook.It is equivalent to:     import numpy as np    from matplotlib import pyplot as plt<jupyter_code>% pylab inline<jupyter_output><empty_output><jupyter_text>2.1 Lines<jupyter_code># Plotting a sinusoide

# Create an array of 100 equally-spaced points between 0 and 10 (to serve as x coordinates)
x = np.linspace(0, 10, 100)

# Create the y coordinates
y = np.sin(x)

# Create the plot
plt.plot(x, y)
plt.show() # only necessary if plt.ion() has been called (default in pylab)
# Change plot style
plt.plot(x, y, color='orange', linestyle='--', linewidth=3)
# Plot the individual points
plt.plot(x, y, color='orange', marker='x', linestyle='')
plt.scatter(x, y, marker='x')
# Plot multiple lines
plt.plot(x, y, linewidth=2, label='sine')
plt.plot(x, np.cos(x), linewidth=2, label='cosine')
plt.legend(fontsize=16)
# Add a title and caption and label the axes
plt.plot(x, y, linewidth=2, label='sine')
plt.plot(x, np.cos(x), linewidth=2, label='cosine')
plt.legend(loc='lower left', fontsize=14)
plt.title("Sinusoides", fontsize=14)
plt.xlabel("$f(x)$", fontsize=16)
plt.ylabel("$\sin(x)$", fontsize=16)
# Save the plot
plt.plot(x, y, linewidth=2, label='sine')
plt.plot(x, np.cos(x), linewidth=2, label='cosine')
plt.legend(loc='lower left', fontsize=14)
plt.title("Sinusoides", fontsize=14)
plt.xlabel("$f(x)$", fontsize=16)
plt.ylabel("$sin(x)$", fontsize=16)
plt.savefig("my_sinusoide.png")<jupyter_output><empty_output><jupyter_text>Problem 2.4Plot a sinusoide of half the amplitude and twice the frequency of the above sine curve.Make it green.<jupyter_code>plt.plot(x, y, linewidth=2, label='sine')
# TODO<jupyter_output><empty_output><jupyter_text>2.2 Clouds of points<jupyter_code># Create a cloud of points, uniformely distributed over [0, 1] x [0, 1]
X = np.random.rand(1000, 1000)
print X.shape
# Create a square figure
fig = plt.figure(figsize=(4, 4))

# Plot them
plt.scatter(X[0], X[1], marker="+")

# Use the same ranges for both axes
plt.xlim([0, 1])
plt.ylim([0, 1])<jupyter_output><empty_output><jupyter_text>2.3 Histograms<jupyter_code># Plot the histogram of the first dimension of X
hh = plt.hist(X[0], bins=50)<jupyter_output><empty_output><jupyter_text>Problem 2.5Create a cloud of 500 normally distributed points. Plot them on a square figure, with orange markers.<jupyter_code># TODO<jupyter_output><empty_output><jupyter_text>2.4 Visualizing vectors<jupyter_code>v = np.array([1, 3, 2, 4])
x = np.array(range(len(v)))

plt.figure()
plt.scatter(x, v, label='v(x)')

plt.xlabel('x', fontsize='16')
plt.ylabel('v', fontsize='16')<jupyter_output><empty_output><jupyter_text>2.5 Heatmaps and matrix visualizationMatplotlib will automatically assign a color to each numerical value, based on a color map. For more about color maps see:http://matplotlib.org/users/colormaps.htmlhttp://matplotlib.org/1.2.1/examples/pylab_examples/show_colormaps.html<jupyter_code>C = np.random.random((50, 100))

# View matrix (row indices from top to botton)
plt.imshow(C, interpolation='nearest')
plt.colorbar()
plt.show()
# View matrix (row indices from bottom to top)
heatmap = plt.pcolor(C) #, cmap=plt.cm.Blues)
plt.colorbar(heatmap)<jupyter_output><empty_output><jupyter_text>2.6 2D plots of functions<jupyter_code># Create a grid of points
xx, yy = np.mgrid[0:5, 0:5]
print xx
print yy
# Create a grid of points
xx, yy = np.mgrid[-50:50, -50:50] 

# Plot |x+iy| on this grid 
plt.imshow(np.abs(xx + 1j*yy))
plt.axis('on')
cc = plt.colorbar()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12467988585644253, 'fraction_numerical': 0.02707251042657496, 'mean_word_length': 5.647859922178989, 'pattern_counts': {'":': 0, '<': 161, '<?xml version=': 0, '>': 156, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '178853', 'license': 'MIT', 'n_tokens_mistral': 5151, 'n_tokens_neox': 4874, 'n_words': 1537, 'path': 'Day1_Introduction/2. Numpy and matplotlib.ipynb', 'repo_name': 'chagaz/lsml2018', 'size': '13727'}