<jupyter_start><jupyter_text>The Basics of NumPy Arrays NumPy provides a multidimensional array class as well as a _large_ number of functions that operate on arrays.NumPy arrays allow you to write fast (optimized) code that works on arrays of data.  To do this, there are some restrictions on arrays:* all elements are of the same data type (e.g. float)* the size of the array is fixed in memory, and specified when you create the array (e.g., you cannot grow the array like you do with lists)The nice part is that arithmetic operations work on entire arrays&mdash;this means that you can avoid writing loops in python (which tend to be slow).  Instead the "looping" is done in the underlying compiled codeWe'll cover a few categories of basic array manipulations here:- *Attributes of arrays*: Determining the size, shape, memory consumption, and data types of arrays- *Indexing of arrays*: Getting and setting the value of individual array elements- *Slicing of arrays*: Getting and setting smaller subarrays within a larger array- *Reshaping of arrays*: Changing the shape of a given array- *Joining and splitting of arrays*: Combining multiple arrays into one, and splitting one array into many Note that the traditional way to import numpy is to rename it np.  This saves on typing and makes your code a little more compact.<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>Array Creation and PropertiesThere are a lot of ways to create arrays.  Let's look at a fewHere we create an array using `arange` and then change its shape to be 3 rows and 5 columns.  Note the _row-major ordering_&mdash;you'll see that the rows are together in the inner `[]` (more on this in a bit)<jupyter_code>a = np.arange(15)
a<jupyter_output><empty_output><jupyter_text>we can also create an array from a list<jupyter_code>b = np.array( [1.0, 2.0, 3.0, 4.0] )<jupyter_output><empty_output><jupyter_text>we can create a multi-dimensional array of a specified size initialized all to 0 easily, using the `zeros()` function.<jupyter_code>b = np.zeros((10,8))
b<jupyter_output><empty_output><jupyter_text>There is also an analogous ones() and empty() array routine.  Note that here we can explicitly set the datatype for the array in this function if we wish. Unlike lists in python, all of the elements of a numpy array are of the same datatype<jupyter_code>c = np.eye(10, dtype=np.float64)
c<jupyter_output><empty_output><jupyter_text>`linspace` creates an array with evenly space numbers.  The `endpoint` optional argument specifies whether the upper range is in the array or not<jupyter_code>d = np.linspace(0, 1, 10, endpoint=False)
print(d)<jupyter_output><empty_output><jupyter_text>Quick Exercise:Analogous to `linspace()`, there is a `logspace()` function that creates an array with elements equally spaced in log.  Use `help(np.logspace)` to see the arguments, and create an array with 10 elements from $10^{-6}$ to $10^3$. we can also initialize an array based on a function<jupyter_code>f = np.fromfunction(lambda i, j: i + j, (3, 3), dtype=int)
f<jupyter_output><empty_output><jupyter_text>NumPy Array Attributes We'll start by defining three random arrays, a one-dimensional, two-dimensional, and three-dimensional array.We'll use NumPy's random number generator, which we will *seed* with a set value in order to ensure that the same random arrays are generated each time this code is run:<jupyter_code>np.random.seed(0)  # seed for reproducibility

x1 = np.random.randint(10, size=6)  # One-dimensional array
x2 = np.random.randint(10, size=(3, 4))  # Two-dimensional array
x3 = np.random.randint(10, size=(3, 4, 5))  # Three-dimensional array
x3<jupyter_output><empty_output><jupyter_text>Each array has attributes ``ndim`` (the number of dimensions), ``shape`` (the size of each dimension), and ``size`` (the total size of the array):<jupyter_code>print("x3 ndim: ", x3.ndim)
print("x3 shape:", x3.shape)
print("x3 size: ", x3.size)<jupyter_output><empty_output><jupyter_text>Another useful attribute is the ``dtype``, the data type of the array:<jupyter_code>print("dtype:", x3.dtype)<jupyter_output><empty_output><jupyter_text>Other attributes include ``itemsize``, which lists the size (in bytes) of each array element, and ``nbytes``, which lists the total size (in bytes) of the array:<jupyter_code>print("itemsize:", x3.itemsize, "bytes")
print("nbytes:", x3.nbytes, "bytes")<jupyter_output><empty_output><jupyter_text>In general, we expect that ``nbytes`` is equal to ``itemsize`` times ``size``.  Array Indexing: Accessing Single Elements If you are familiar with Python's standard list indexing, indexing in NumPy will feel quite familiar.In a one-dimensional array, the $i^{th}$ value (counting from zero) can be accessed by specifying the desired index in square brackets, just as with Python lists:<jupyter_code>x1
x1[0]
x1[4]<jupyter_output><empty_output><jupyter_text>To index from the end of the array, you can use negative indices:<jupyter_code>x1[-1]
x1[-2]<jupyter_output><empty_output><jupyter_text>In a multi-dimensional array, items can be accessed using a comma-separated tuple of indices:<jupyter_code>x2
x2[0, 0]
x2[2, 0]
x2[2, -1]<jupyter_output><empty_output><jupyter_text>Values can also be modified using any of the above index notation:<jupyter_code>x2[0, 0] = 12
x2<jupyter_output><empty_output><jupyter_text>Keep in mind that, unlike Python lists, NumPy arrays have a fixed type.This means, for example, that if you attempt to insert a floating-point value to an integer array, the value will be silently truncated. Don't be caught unaware by this behavior!<jupyter_code>x1[0] = 3.14159  # this will be truncated!
x1
x = np.array([1, 2, 2.5])
x
x.astype(int)<jupyter_output><empty_output><jupyter_text>Array Slicing: Accessing Subarrays Just as we can use square brackets to access individual array elements, we can also use them to access subarrays with the *slice* notation, marked by the colon (``:``) character.The NumPy slicing syntax follows that of the standard Python list; to access a slice of an array ``x``, use this:``` pythonx[start:stop:step]```If any of these are unspecified, they default to the values ``start=0``, ``stop=``*``size of dimension``*, ``step=1``.We'll take a look at accessing sub-arrays in one dimension and in multiple dimensions.Remember, python uses 0-based indexing![](slicing.png)  One-dimensional subarrays<jupyter_code>x = np.arange(10)
x
x[:5]  # first five elements
x[5:]  # elements after index 5
x[4:7]  # middle sub-array
x[::2]  # every other element
x[1::2]  # every other element, starting at index 1<jupyter_output><empty_output><jupyter_text>A potentially confusing case is when the ``step`` value is negative.In this case, the defaults for ``start`` and ``stop`` are swapped.This becomes a convenient way to reverse an array:<jupyter_code>x[::-1]  # all elements, reversed
x[5::-2]  # reversed every other from index 5<jupyter_output><empty_output><jupyter_text>Multidimensional ArraysMultidimensional arrays are stored in a contiguous space in memory -- this means that the columns / rows need to be unraveled (flattened) so that it can be thought of as a single one-dimensional array.  Different programming languages do this via different conventions:![](row_column_major.png)Storage order:* Python/C use *row-major* storage: rows are stored one after the other* Fortran/matlab use *column-major* storage: columns are stored one after anotherThe ordering matters when * passing arrays between languages* looping over arrays -- you want to access elements that are next to one-another in memory  * e.g, in Fortran:  ```  double precision :: A(M,N)  do j = 1, N     do i = 1, M        A(i,j) = …     enddo  enddo  ```    * in C  ```  double A[M][N];  for (i = 0; i < M; i++) {     for (j = 0; j < N; j++) {        A[i][j] = …     }  }    ```  In python, using NumPy, we'll try to avoid explicit loops over elements as much as possibleLet's look at multidimensional arrays:Multi-dimensional slices work in the same way, with multiple slices separated by commas.For example:<jupyter_code>x2
x2[:2, :3]  # two rows, three columns
x2[:3, ::2]  # all rows, every other column<jupyter_output><empty_output><jupyter_text>Finally, subarray dimensions can even be reversed together:<jupyter_code>x2[::-1, ::-1]<jupyter_output><empty_output><jupyter_text>Sometimes we want a one-dimensional view into the array -- here we see the memory layout (row-major) more explicitly<jupyter_code>a = np.arange(15).reshape(3,5)
a
a.flatten()<jupyter_output><empty_output><jupyter_text>we can also iterate -- this is done over the first axis (rows)<jupyter_code>for row in a:
    print(row)<jupyter_output><empty_output><jupyter_text>Generally speaking, we want to avoid looping over the elements of an array in python&mdash;this is slow.  Instead we want to write and use functions that operate on the entire array at once.  Quick Exercise:Consider the array defined as:<jupyter_code>q = np.array([[1, 2, 3, 2, 1],
              [2, 4, 4, 4, 2],
              [3, 4, 4, 4, 3],
              [2, 4, 4, 4, 2],
              [1, 2, 3, 2, 1]])<jupyter_output><empty_output><jupyter_text>* using slice notation, create an array that consists of only the `4`'s in `q` (this will be called a _view_, as we'll see shortly)  * zero out all of the elements in your view  * how does `q` change?  Subarrays as no-copy viewsOne important–and extremely useful–thing to know about array slices is that they return *views* rather than *copies* of the array data.This is one area in which NumPy array slicing differs from Python list slicing: in lists, slices will be copies.Consider our two-dimensional array from before:<jupyter_code>print(x2)<jupyter_output><empty_output><jupyter_text>Let's extract a $2 \times 2$ subarray from this:<jupyter_code>x2_sub = x2[:2, :2]
print(x2_sub)<jupyter_output><empty_output><jupyter_text>Now if we modify this subarray, we'll see that the original array is changed! Observe:<jupyter_code>x2_sub[0, 0] = 99
print(x2_sub)
print(x2)<jupyter_output><empty_output><jupyter_text>This default behavior is actually quite useful: it means that when we work with large datasets, we can access and process pieces of these datasets without the need to copy the underlying data buffer.  Creating copies of arraysDespite the nice features of array views, it is sometimes useful to instead explicitly copy the data within an array or a subarray. This can be most easily done with the ``copy()`` method:<jupyter_code>x2_sub_copy = x2[:2, :2].copy()
print(x2_sub_copy)<jupyter_output><empty_output><jupyter_text>If we now modify this subarray, the original array is not touched:<jupyter_code>x2_sub_copy[0, 0] = 42
print(x2_sub_copy)
print(x2)<jupyter_output><empty_output><jupyter_text>Reshaping of ArraysAnother useful type of operation is reshaping of arrays.The most flexible way of doing this is with the ``reshape`` method.For example, if you want to put the numbers 1 through 9 in a $3 \times 3$ grid, you can do the following:<jupyter_code>grid = np.arange(1, 10).reshape((3, 3))
print(grid)<jupyter_output><empty_output><jupyter_text>Note that for this to work, the size of the initial array must match the size of the reshaped array. Where possible, the ``reshape`` method will use a no-copy view of the initial array, but with non-contiguous memory buffers this is not always the case.Another common reshaping pattern is the conversion of a one-dimensional array into a two-dimensional row or column matrix.This can be done with the ``reshape`` method, or more easily done by making use of the ``newaxis`` keyword within a slice operation:<jupyter_code>x = np.array([1, 2, 3])

# row vector via reshape
x.reshape((1, 3))
# row vector via newaxis
x[np.newaxis, :]
# column vector via reshape
x.reshape((3, 1))
# column vector via newaxis
x[:, np.newaxis]<jupyter_output><empty_output><jupyter_text>Array Concatenation and SplittingAll of the preceding routines worked on single arrays. It's also possible to combine multiple arrays into one, and to conversely split a single array into multiple arrays. We'll take a look at those operations here.  Concatenation of arraysConcatenation, or joining of two arrays in NumPy, is primarily accomplished using the routines ``np.concatenate``, ``np.vstack``, and ``np.hstack``.``np.concatenate`` takes a tuple or list of arrays as its first argument, as we can see here:<jupyter_code>x = np.array([1, 2, 3])
y = np.array([3, 2, 1])
np.concatenate([x, y])<jupyter_output><empty_output><jupyter_text>You can also concatenate more than two arrays at once:<jupyter_code>z = [99, 99, 99]
print(np.concatenate([x, y, z]))<jupyter_output><empty_output><jupyter_text>It can also be used for two-dimensional arrays:<jupyter_code>grid = np.array([[1, 2, 3],
                 [4, 5, 6]])
# concatenate along the first axis
np.concatenate([grid, grid])
# concatenate along the second axis (zero-indexed)
np.concatenate([grid, grid], axis=1)<jupyter_output><empty_output><jupyter_text>For working with arrays of mixed dimensions, it can be clearer to use the ``np.vstack`` (vertical stack) and ``np.hstack`` (horizontal stack) functions:<jupyter_code>x = np.array([1, 2, 3])
grid = np.array([[9, 8, 7],
                 [6, 5, 4]])

# vertically stack the arrays
np.vstack([x, grid])
# horizontally stack the arrays
y = np.array([[99],
              [99]])
np.hstack([grid, y])<jupyter_output><empty_output><jupyter_text>Similary, ``np.dstack`` will stack arrays along the third axis.  Splitting of arraysThe opposite of concatenation is splitting, which is implemented by the functions ``np.split``, ``np.hsplit``, and ``np.vsplit``.  For each of these, we can pass a list of indices giving the split points:<jupyter_code>x = [1, 2, 3, 99, 99, 3, 2, 1]
x1, x2, x3 = np.split(x, [3, 5])
print(x1, x2, x3)<jupyter_output><empty_output><jupyter_text>Notice that *N* split-points, leads to *N + 1* subarrays.The related functions ``np.hsplit`` and ``np.vsplit`` are similar:<jupyter_code>grid = np.arange(16).reshape((4, 4))
grid
upper, lower = np.vsplit(grid, [2])
print(upper)
print(lower)
left, right = np.hsplit(grid, [2])
print(left)
print(right)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1028149190710767, 'fraction_numerical': 0.018296973961998593, 'mean_word_length': 5.358389261744967, 'pattern_counts': {'":': 0, '<': 147, '<?xml version=': 0, '>': 145, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '515302', 'license': 'Unlicense', 'n_tokens_mistral': 4657, 'n_tokens_neox': 4415, 'n_words': 1775, 'path': 'day2/01. numpy - basic.ipynb', 'repo_name': 'ubutnux/bosscha-python-workshop-2022', 'size': '14210'}