<jupyter_start><jupyter_text>![](http://www.numpy.org/_static/numpy_logo.png) Introduction to NumPy NumPy supports arrays which are very useful to numerical computations* Arrays are N dimensional: 1d (vector), 2d (plane),...,N dim* Many packages use numpy arrays to store data* Arrays can be used to make calculations in one command, without loops or list comprehension.  * This is known as *vectorisation** Using vectorisation will make your code faster, and easier to read.  * Arrays are (generally) faster than lists.  See more in this guide: https://github.com/rougier/from-python-to-numpy  What is vectorisation?Say you want to calculate an array of values `array2`, based on your original data in `array1`. Vectorised example:```pythonarray2 = array1 * k + c```Non-vectorised example, requires a loop:```pythonfor i in range(len(array1)):    array2[i] = array1[i] * k + c   ```Without vectorisation, multi-dimensional arrays would need multiple nested loops.  Do we still need lists? * Lists can have different objects as elements. Arrays are homogenous.```pythonexample_list = [number, string, cat, dog]example_array = [cat1, cat2, cat3]```* Lists can be nested ```pythonnested_list = [[1, 2], ['a', 'b', 'qwerty'], [1]]```Arrays can also be nested but it negates some of the advantages of n-dimensional arrays  Let's get started ...<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>Looking for help?* Documentation: http://docs.scipy.org/doc/numpy/reference/* Use help function (remember tab will show options available)```python    help(np.mean)```* Interactive help: NumPy has an a built-in search engine<jupyter_code>np.lookfor('weighted average')<jupyter_output><empty_output><jupyter_text>Creating an array from a list<jupyter_code>a1d = np.array([3, 4, 5, 6])
a1d
a2d = np.array([[10., 20,  30], [9, 8, 5]])
a2d
print( type( a1d[0] ) )
print( type( a2d[0,0] ) )
type(a1d)<jupyter_output><empty_output><jupyter_text>The **core class** of NumPy is the `ndarray` (homogeneous n-dimensional array). To find methods or attributes:```a1d.   ->tab```More on this below... --- Array attributes<jupyter_code># a reminder of what we're working with:
a2d = np.array([[10., 20,  30], [9, 8, 5]])
type(a2d)<jupyter_output><empty_output><jupyter_text>ndarray.ndimthe number of dimensions (axes) of the array. In NumPy, the number of dimensions is referred to as rank.<jupyter_code>a2d.ndim<jupyter_output><empty_output><jupyter_text>ndarray.shapethe dimensions of the array<jupyter_code>a2d.shape<jupyter_output><empty_output><jupyter_text>This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m).  The length of the shape tuple is therefore the rank, or number of dimensions, ndim.<jupyter_code># Let's use those values

NLines,NCols = a2d.shape
print('NLines:', NLines,'NCols:',NCols)<jupyter_output><empty_output><jupyter_text>ndarray.sizethe total number of elements in the array<jupyter_code>a2d.size<jupyter_output><empty_output><jupyter_text>Note that for an ND array `size` is not equal to `len()`. The latter returns the length of just the *first* dimension.<jupyter_code>len(a2d)<jupyter_output><empty_output><jupyter_text>ndarray.dtypetype of data within the array<jupyter_code>a2d.dtype<jupyter_output><empty_output><jupyter_text>--- Aside: Common mistakes when creating arrays Say we forget to use the square brackets:<jupyter_code>try:
    a = np.array(1,2,3,4)   # WRONG, only 2 non-keyword arguments accepted
except ValueError as err:
    print('ValueError: ',err)

# help(np.array)
a = np.array([1,2,3,4]) # RIGHT
print(a)<jupyter_output><empty_output><jupyter_text>Arrays can be created using `np.ndarray`, but this works differently to `np.array`. Here you specify the dimensions within the first set of square backets.<jupyter_code>np.ndarray([1,2,3,4]) #  This is not the recommended way to create arrays<jupyter_output><empty_output><jupyter_text>Result: an "empty" 4-D array.This is not the recommended method for creating arrays. `ndarray` is a class, with `array` the recommended function. Below is a number of other possible methods for allocating arrays.  --- Functions for creating arrays  ``np.arange([start,] stop[, step,], dtype=None)``evenly spaced, *defined by step*<jupyter_code>np.arange(1, 9, 2)
# for integers, np.arange is same as range but returns an array instead of a list 
np.array( range(1,9,2) )<jupyter_output><empty_output><jupyter_text>``np.linspace(start, stop, num=50, endpoint=True, retstep=False, dtype=None)``evenly spaced, *defined by length*<jupyter_code>np.linspace(0, 1, 10)   # start, end, num-points<jupyter_output><empty_output><jupyter_text>Note that for ``np.linspace`` the upper value is included within the array. This is different to the result for ``range`` or ``np.arange``.   --- Exercise 1<jupyter_code># a) Create array with units of seconds through the day, from 00:00 to 24:00, inclusive 
# b) Change the length of your array, to have either 1 second, 1 minute, or 1 hour intervals.
# (enter your code below)<jupyter_output><empty_output><jupyter_text>--- More functions for creating arrays  Empty array<jupyter_code>np.empty((2,2))<jupyter_output><empty_output><jupyter_text>As seen in the example with `ndarray` above, the array is not truly empty. Result is an array of *uninitialised* (arbitrary) values. This method should only be used if all values will be allocated at a later stage i.e. use with caution.    Array filled with zeros<jupyter_code>np.zeros((2, 2))<jupyter_output><empty_output><jupyter_text>By default, the dtype of the created array is float64 but other dtypes can be used:<jupyter_code>np.zeros((2, 2), dtype=int)<jupyter_output><empty_output><jupyter_text>Filled with ones<jupyter_code>np.ones((2, 3))<jupyter_output><empty_output><jupyter_text>Filled with random numbers<jupyter_code>np.random.rand(4)       # uniform in [0, 1]
np.random.normal(0, 10, size=4)      # Gaussian (mean, std dev, size/num samples)
np.random.gamma(1, 1, (2,2))      # Gamma (shape of distribution, scale, size/num samples)<jupyter_output><empty_output><jupyter_text>As you can see, you can get multiple outputs in whatever distribution you'd like. The keyword for 'size' is optional. Note that in the third example, size is given as a tuple.  Grid generation * A common task is to generate a pair of 2D (or ND) arrays that represent data coordinates.   * Useful for interpolation of mapping contours.* When orthogonal 1D coordinate arrays already exist, NumPy's `meshgrid` function is very useful:<jupyter_code>x = np.linspace(-5, 5, 3)
y = np.linspace(10, 40, 4)
print(x)
print(y)
x2d, y2d = np.meshgrid(x, y)
print(x2d)
print(y2d)<jupyter_output><empty_output><jupyter_text>Transpose arrays This can be very useful when dealing with grids; there are several ways:<jupyter_code>print(y2d,'\n')
print(np.transpose(y2d),'\n') # using a numpy function
print(y2d.transpose(),'\n')   # using a method of y2d
print(y2d.T)                  # using a property of y2d (i.e. a specific version of general methods above)<jupyter_output><empty_output><jupyter_text>------  Exploring arrays: Array indexing * Indices begin at 0, like other Python sequences and C/C++.   * Note that many languages, such as Matlab, R and Fortran, start with 1  * In 2D, the first dimension corresponds to rows, the second to columns. This is known as [row-major indexing](https://numpy.org/doc/stable/glossary.htmlterm-row-major). * For multi-dimensional arrays, the order of axes in python follows C-style indexing.     * *The fastest varying dimension is the last dimension.* As a simple example, consider an array with 2 rows and 4 columns. In a row-major (or C-style) index system, values will be read in the order (from 0-7): ```| 0 | 1 | 2 | 3 || 4 | 5 | 6 | 7 | ```In terms of nested loops (which we should avoid if possible in our code!), this array is being accessed using: ```for row in rows:   for column in columns:     ...```    This has an impact on the time taken to access each index e.g. it is quicker to access all items in the first axes. A nice explanation of this can be found [here](https://agilescientific.com/blog/2018/12/28/what-is-the-fastest-axis-of-an-array).  1D ExamplesFor 1D arrays, indexing is exactly the same as discussed for lists [here](https://github.com/ueapy/pythoncourse2020-materials/blob/master/notebooks/07-Built-in-Data-Structures.ipynb)<jupyter_code>a = np.arange(10, 100, 10)
a
a[0]
a[2:9:3] # [start:end:step]<jupyter_output><empty_output><jupyter_text>Notice that the 'end' number, 9, actually lies out of bounds (try `a[9]` and see that it gives an error).  The end-index is never inluded.<jupyter_code>a[:3] # last is not included
a[-2] # negative index counts from the end<jupyter_output><empty_output><jupyter_text>Here we started counting 'down' from -1 (and not -0!).  Indexing in practice: How to calculate x[ i ] - x[ i-1 ] without a loop?<jupyter_code>x = np.random.rand(6) # create an array of random numbers (0-1)
x = np.sort(x)        # sort them in order of ascending value
print(x)
x[1:] - x[:-1]
# Note there is actually a function in NumPy that can do this for us
np.diff(x)<jupyter_output><empty_output><jupyter_text>--- Exercise 2 Create a 2D NumPy array from the following list and assign it to the variable "a":<jupyter_code># [[2, 3.2, 5.5, -6.4, -2.2, 2.4],
#  [1, 22, 4, 0.1, 5.3, -9],
#  [3, 1, 2.1, 21, 1.1, -2]]<jupyter_output><empty_output><jupyter_text>a) Can you guess what the following slices are equal to? Print them to check your understanding.<jupyter_code># a[:, 3]
# a[1:4, 0:4]
# a[1:, 2]<jupyter_output><empty_output><jupyter_text>b) How would you extract: i) the last column; ii) the row before last?<jupyter_code># a[]
# a[]<jupyter_output><empty_output><jupyter_text>--- Fancy indexing NumPy arrays can be indexed with slices, but also with boolean orinteger arrays (masks)<jupyter_code>a = np.random.randint(1, 100, 6) # array of 6 random integers between 1 and 100
a<jupyter_output><empty_output><jupyter_text>First, an example with an array of boolean values:<jupyter_code>mask = ( a % 3 == 0 ) # Where divisible by 3 (% is the modulus operator).
mask
a[mask]<jupyter_output><empty_output><jupyter_text>Now an example with an array of integers:<jupyter_code>b = np.array([0,3,5])
a[b]<jupyter_output><empty_output><jupyter_text>--- Copies and Views (a warning)<jupyter_code>original = np.array([99,98,97])

other = original
other[0] = 0<jupyter_output><empty_output><jupyter_text>What do we think has happened to the two arrays?<jupyter_code>print('other is now ',other)
print('original is now ',original)<jupyter_output><empty_output><jupyter_text>NumPy, in its frugality, will create a *view* by default, unless told to make a copy.<jupyter_code>original = np.array([99,98,97])
copy = original.copy()
copy[0] = 0

print('copy is now ',copy)
print('original is now ',original)<jupyter_output><empty_output><jupyter_text>Be aware that views are also used when you create slices of arrays ...<jupyter_code>original = np.ones((4,3))

row = original[:,0]
row[2:] = 10 

print('row is now ',row)
print('original is now \n',original)<jupyter_output><empty_output><jupyter_text>... but not when you use "fancy indexing".<jupyter_code>original = np.ones((4,3))

fancy = original[0,[0,1,2]]
fancy[:] = 10 

print('fancy is now ',fancy)
print('original is now \n',original)<jupyter_output><empty_output><jupyter_text>**To avoid this leading to errors propagating through your data, be sure to check whether a copy is needed.**   Copies in functions vs. methodsFrom help(numpy):```Most of the functions in `numpy` return a copy of the array argument(e.g., `np.sort`).  In-place versions of these functions are oftenavailable as array methods, i.e. ``x = np.array([1,2,3]); x.sort()``.Exceptions to this rule are documented.```<jupyter_code>original = np.array([99,98,97])

# Function np.sort
sortedCopy = np.sort(original)
print('original:',original,'returned:',sortedCopy)

# Method sort()
original.sort()
print('original:', original)<jupyter_output><empty_output><jupyter_text>No new variable (or copy) is created with the method - methods act on the array they're attached to.  So using `.sort()` is equivalent to: ```orginal = np.sort(original)``` --- NumPy Statistics NumPy has a large number of useful methods and functions, enabling you to perform statistical operations on arrays. A full list can be found in the [NumPy documentation](https://numpy.org/doc/stable/reference/index.html), but a selection of useful methods and functions are provided below.   Statistical methods of arrays<jupyter_code>a1d=np.random.normal(0,10,5) 
print('array a1d                       :', a1d)
print('Minimum and maximum             :', a1d.min(), a1d.max())
print('Index of minimum and maximum    :', a1d.argmin(), a1d.argmax())
print('Sum and product of all elements :', a1d.sum(), a1d.prod())
print('Mean and standard deviation     :', a1d.mean(), a1d.std())<jupyter_output><empty_output><jupyter_text>A full list of available methods can be found [here](https://numpy.org/doc/1.18/reference/generated/numpy.ndarray.html).  Statistical functionshttps://numpy.org/doc/stable/reference/routines.statistics.html<jupyter_code>print('Median and percentile           :', np.median(a1d), np.percentile(a1d,75))<jupyter_output><empty_output><jupyter_text>Operations over a given axis<jupyter_code>print(a2d)
print('sum  :',a2d.sum())
print('sum  :',a2d.sum(axis=0))
print('sum  :',a2d.sum(axis=1))<jupyter_output><empty_output><jupyter_text>What about NaN values?A series of functions are available that will excluding NaN values (missing numbers) from your calculations.<jupyter_code>a1d[2] = np.nan
print(a1d)
print('mean  :',np.mean(a1d))
print('nanmean  :',np.nanmean(a1d))<jupyter_output><empty_output><jupyter_text>Vectorisation: operations on whole arrays<jupyter_code>a=np.random.rand(4)
print(a)

result = np.exp(a/100.)/a

print(result)
# Non-vectorised
result=np.zeros(a.shape)   # create an array to hold the results

for i in range(a.size): 
    result[i] = np.exp(a[i]/100.)/a[i]
    
print(result)<jupyter_output><empty_output><jupyter_text>Vectorization is generally faster than using `for` loops. However, for more complicated algorithms it might not always be possible, or the most readable --- Exercise 3 Consider a 4 x 5 2D array of negative integers:<jupyter_code>a = np.arange(-100, 0, 5).reshape(4, 5)
a<jupyter_output><empty_output><jupyter_text>Suppose you want to return an array `result`, which has the squared value when an element in array `a` is greater than `-90` and less than `-40`, and is 1 otherwise. Using a `for` loop, the result would look like this:<jupyter_code>result = np.zeros(a.shape, dtype=a.dtype)    # pre-allocate the result array

for i in range(a.shape[0]):                  # loop over rows
    for j in range(a.shape[1]):              # loop over columns
        if a[i, j] > -90 and a[i, j] < -40:  # only square the number if within the chosen limits
            result[i, j] = a[i, j]**2
        else:                                # set to 1 otherwise
            result[i, j] = 1
            
result<jupyter_output><empty_output><jupyter_text>**Can you write a vectorised solution?**Hint: use np.logical_and() to create a condition for indexing (information on all NumPy's logic functions can be found [here](https://numpy.org/doc/stable/reference/routines.logic.html)).<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>------ Masked arrays - how to handle (propagating) missing values ![](../figures/masked_array.png) All operations related to masked arrays live in `numpy.ma` submodule. The simplest example of manual creation of a masked array:<jupyter_code>a = np.ma.masked_array(data=[1, 2, 3],
                       mask=[True, True, False],
                       fill_value=-999)
a<jupyter_output><empty_output><jupyter_text>Often, a task is to mask array depending on a criterion.<jupyter_code>a = np.linspace(1, 15, 15)
masked_a = np.ma.masked_greater_equal(a, 11)
masked_a<jupyter_output><empty_output><jupyter_text>Note: In a masked array, unlike setting values to NaN, the raw data still exists, should you need it.<jupyter_code>masked_a.data<jupyter_output><empty_output><jupyter_text>--- Exercise 4 1. Create a "data" array of evenly spaced numbers, in the interval (-10, 20) spaced by 0.52. Calculate the (natural) logarithm of the data3. Create a condition i.e. a True/False (boolean) array, that you can use to mask these results    - The resulting array should be masked when either of the following conditions apply        - larger or equal than 10        - larger than -1 and smaller than 1         - data is not a real number4. Mask the array depending on these conditions<jupyter_code># Your code:
# 1. Hint: use `np.linspace` or `np.arange` functions

# 2.
# 3. Hint: use np.isfinite

# 4. Hint: use np.ma.masked_where(condition,arr)<jupyter_output><empty_output><jupyter_text>------ Shape manipulation<jupyter_code>a = np.array([[1, 2, 3], [4, 5, 6]])
print( f'array: \n {a}' )
print( f'shape: {a.shape}' )
a.flatten()
a.repeat(4)
a.reshape((3, 2)) # NB. new shape must be consistent with number of values. 
print( f'Old shape: {a.shape}' )
print( f'New shape: {a.reshape((3, 2)).shape}' )<jupyter_output><empty_output><jupyter_text>--- Exercise 5 Generate a 2d array with 5x5. The first value is 0 and it grows left to right and top to bottom in increments on 0.1.<jupyter_code># Your code here<jupyter_output><empty_output><jupyter_text>--- Further axes manipulation We saw earlier that it is possible to *transpose* an array i.e. flip the order of axes. The `transpose` function can also be used more generally to change the order of axes within an array.<jupyter_code>a3d = np.ones((4,3,2))
print(a3d.shape)

new_order = np.transpose(a3d,[1, 2, 0]) # numbers in list refer to axes
print(new_order.shape)<jupyter_output><empty_output><jupyter_text>Dimensions can also be added<jupyter_code>a3d[..., np.newaxis].shape<jupyter_output><empty_output><jupyter_text>--- Broadcasting The fact that NumPy operates on an element-wise basis means that in principle arrays must always match one another's shape. However, NumPy will also helpfully "broadcast" dimensions when possible. **The Broadcasting Rule**  In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one.<jupyter_code># Example

a1d = np.arange(0,5)
a2d = np.zeros((5,3))
print(a1d)
print(a2d)

# try:
# a1d + a2d

# add a trailing axis: 
# a1d[..., np.newaxis] + a2d<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12190680235672201, 'fraction_numerical': 0.02244242099625067, 'mean_word_length': 5.001607200257152, 'pattern_counts': {'":': 1, '<': 218, '<?xml version=': 0, '>': 219, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '54', 'dir': 'jupyter-structured-clean-dedup', 'id': '148197', 'license': 'MIT', 'n_tokens_mistral': 6517, 'n_tokens_neox': 6174, 'n_words': 2063, 'path': 'notebooks/14-Numpy-Intro.ipynb', 'repo_name': 'ueapy/pythoncourse2021-materials', 'size': '18670'}