<jupyter_start><jupyter_text>Lecture 3:  Numpy ArraysIn this lecture, we will learn a new type: `array`(which is like matrix/tensor in MATLAB)  Multi-dimensional arrays. Why do we need them?For example, let's say I have a whole bunch of data points like the preference for movies, like    Person    The Matrix      Inception      Force Awakens     Wall-E ...         A         5              4                5              3         B         4              ?                4              ?         C         ?              ?                5              5         ...         We would like to guess how Person C would rate Inception. Not an easy problem, Netflix actually posted a million dollar prize for this problem.BUT in the first place, how would we store this kind of data?If we wanted to just store the matrix, we can do a list of lists (nested lists), let's see an example.<jupyter_code>l1 = [1,2]
l2 = [3,4]
l3 = [l1, l2]
print(l3)
type(l3)
l3[0][0] # l3[0,0] does NOT work for list of lists<jupyter_output><empty_output><jupyter_text>Back to the movie pref example. There are 3 lists (one per person) and each one has 4 entries. So it's a `3x4` matrix. We can do a lot with lists of lists, but it will be slow. We would eventually need a more efficient and flexible ways of using multi-dimensional arrays:    xs = [[5, 4, 5, 3], [4, -1, 4, -1], [-1, -1, 5, 5]]  list of listsThe standard way of working with data sets in Python is to use the **Numpy** library. A Numpy array is a multi-dimensional array.   Run the following line before you proceed<jupyter_code># the almighty linear algebra package in Python
import numpy as np # np means numpy in any Python code<jupyter_output><empty_output><jupyter_text>To see why `numpy` array is superior to lists, let us consider the following example:<jupyter_code># np.array()
x = np.array([1, 2, 3, 4]) # the input can be a list
print(x/3)
# what if y is a list
y = [1,2,3,4]
print(y/3)<jupyter_output><empty_output><jupyter_text>Now let us get back to the `xs` earlier.<jupyter_code>arr = np.array([[5, 4, 5, 3], [4, -1, 4, -1], [-1, -1, 5, 5]])
print(arr)
arr
type(arr)
#arr is now an object
arr.T # transpose<jupyter_output><empty_output><jupyter_text>TupleEvery array has a `shape` (dimension), and the shape of an array is a tuple which is initialized by `()`, opposing to list initialized by `[]`.<jupyter_code>arr.shape
# the shape has type tuple which we will get to that later
type(arr.shape)<jupyter_output><empty_output><jupyter_text>How to access elements<jupyter_code>arr = np.array([[1,2,3], [4,5,6]])
print(arr)
# accessing elements:
arr[0]    # 1st row
arr[1]     # 2nd row
arr[0,0]    # very different from lists of lists, for those, we would have done arr[0][0]
arr[0,1]
arr = np.array([[1,2,3],[4,5,6],[7,8,9]])
print(arr)<jupyter_output><empty_output><jupyter_text>SlicingWe can use slicing `:` which is similar to `:` in MATLAB, which is a tricking to obtain certain indexed elements very fast. The syntax for slicing is to put `start:stop:step` as indices for an array, if there is no `step`, it is 1.<jupyter_code>arr[0,:]
arr[:,0]<jupyter_output><empty_output><jupyter_text>Remark: Index `i`, returns the same values as `i:i+1`. In particular, a selection tuple with the $p$-th element an integer (and all other entries `:`) returns the corresponding sub-array with dimension $N - 1$. If $N = 1$ then the returned object is an array scalar.If the selection tuple has all entries : except the $p$-th entry which is a slice object `i:j:k`, then the returned array has dimension $N$ formed by concatenating the sub-arrays returned by integer indexing of elements $i, i+k, ..., i + (m - 1) k < j$.<jupyter_code>print(arr)
arr[1:3,1:3]
arr[1:2,1:2]
arr[1:-1,:]<jupyter_output><empty_output><jupyter_text>The indexing tricks above are enough for our class. For advanced slicing and indexing tricks, please refer to [https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.indexing.htmladvanced-indexing](https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.indexing.htmladvanced-indexing)  nd-array3 or more dimensional arrays too<jupyter_code>arr3 = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])
print(arr3)
arr1 = np.array([[1, 2, 3], [4, 5, 6]])
arr2 = np.array([[7, 8, 9], [10, 11, 12]])
print(arr1)
print(arr2)
arr3 = np.array([arr1,arr2])
print(arr3)
arr3.shape  # in matlab it would be shape(arr3)
arr3[0,1,2]<jupyter_output><empty_output><jupyter_text>Building Arrays<jupyter_code>arr = np.zeros(3)
arr = np.zeros([3,3])   # you put the shape in as a list
arr = np.zeros([3,3,2])
arr<jupyter_output><empty_output><jupyter_text>The identity matrix, for example, $\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ 0 & 0 & 1\end{pmatrix}$<jupyter_code>arr = np.identity(3)
arr # 1. means 1.0, which is a float<jupyter_output><empty_output><jupyter_text>Equally distant points:<jupyter_code>arr = np.arange(10) # this is not arrange!!!!
print(arr)
arr = np.arange(1,2,0.1)
print(arr) # be careful with the indexing!
arr = np.linspace(1,2,11)   # 11 points between 0 and and 1 (inclusive)
print(arr)<jupyter_output><empty_output><jupyter_text>Vectorization:<jupyter_code># guess what will happen?
arr = np.zeros([3,3])
arr = arr + 1
print(arr)<jupyter_output><empty_output><jupyter_text>Similarly<jupyter_code>def f(x):
    return x*x + x + 1
f(arr)    # again this would never work for lists<jupyter_output><empty_output><jupyter_text>Remark: for library functions you may need to use a function called `np.vectorize`. Let us see the following example:<jupyter_code>np.array([1,2,3]) + np.array([4,5,6])<jupyter_output><empty_output><jupyter_text>Try the following code: if these were lists, it would be concatenation<jupyter_code>[1,2,3]+[4,5,6]<jupyter_output><empty_output><jupyter_text>It added the arrays as if they were vectors. Numpy figures out how to use the function with the array you gave.   ReshapeAlternatively, maybe you want to control it youself by `reshape`:<jupyter_code>arr = np.array(range(16))
arr.reshape(4,4)
arr.reshape(2,2,-1)    # if you put -1, it figures out what the shape should be
arr
np.reshape(arr, (2,2,-1)) # -1 means unspecified<jupyter_output><empty_output><jupyter_text>Vectorization<jupyter_code>arr = np.array(range(9)).reshape(3,3) + 1
arr
np.sum(arr)
np.mean(arr)
np.apply_along_axis()
np.apply_along_axis(np.sum, 0, arr)
np.apply_along_axis(np.sum, 1, arr)
np.sum(arr, axis=1)<jupyter_output><empty_output><jupyter_text>There is also: `np.apply_over_axes`, be aware of this plural `axes`.<jupyter_code>np.apply_over_axes(np.sum, arr, 0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12827502650310466, 'fraction_numerical': 0.03483265182492806, 'mean_word_length': 4.404255319148936, 'pattern_counts': {'":': 0, '<': 78, '<?xml version=': 0, '>': 77, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '110920', 'license': 'Apache-2.0', 'n_tokens_mistral': 2446, 'n_tokens_neox': 2333, 'n_words': 770, 'path': 'matematica-para-datascience/Lectures/Lecture-03-Numpy.ipynb', 'repo_name': 'ddiestra/OpenSource-RoadMap-DataScience', 'size': '6603'}