<jupyter_start><jupyter_text>Vectorization in `numpy`As programmers, we're used to writing for-loops to iterate over collections. This is quite OK but in Python makes the code slow (because it's an interpreted, dynamically-typed language). For example, a "standard" way of summing an array would be```pythonx = [2, 3, 8, -2.3, 0, 15]sum = 0for i in range(len(x)):    sum += x[i]print(sum)```However, there are better ways to do this. `numpy` works in C "behind the scenes". This means that:1. Operations in C are very, very, VERY fast2. Communication between C and Python is slowThis means we should prepare our code to use `numpy` arrays as much as possible. First of all, this gives us a great computational advantage: the code is very fast. Second, it will look simpler and more beautiful. Compare the previous code with this one:```pythonx = [2, 3, 8, -2.3, 0, 15]numpy_sum = np.sum(x)print(numpy_sum)```Of course, the for-loop is still done, it's just hidden.The basic rule is **whenever possible, avoid looping and use vectors and matrices**. Sometimes it's impossible to avoid loops and that's OK.Let's create a performance test. Create a large array of random numbers. You can use `np.random.random()`. Sum the array using `numpy` and using the for-loop. Compare the times. In some cases, the performance difference will be several hundred times (e.g. length = $1.10^7$, difference $\approx 2000$ times: $\approx 0.01ms$ for the `sum()` and $\approx 2s$ for the loop).Don't forget to see that the sums are equal. A fast but incorrect algorihm is not an option :).Next, call the function for different lengths and create two plots showing the time it takes to multiply different-length arrays. **Idea:** You can plot them on two separate y-axes on the same plot. Look at the `twiny()` function. See how much time it takes to perform both operations.Plot another plot: speedup versus length. Plot the length on the x-axis and the speedup (`np_sum_time / for_loop time`) on the y-axis.<jupyter_code>x = np.random.random_sample(10000000)

now = datetime.datetime.now()

sum = 0
for i in range(len(x)):
    sum = x[i]

middle = datetime.datetime.now()

sum = np.sum(x)

end = datetime.datetime.now()

print("For time: ", middle - now)
print("Sum time: ", end - middle)<jupyter_output>For time:  0:00:03.989540
Sum time:  0:00:00.017107<jupyter_text>There's one slight warning to using vectors. If we don't know what we're doing we might get very hard-to-detect bugs.Let's look at vector multiplication. In algebra we may write:$$ \begin{bmatrix}2 \\3 \\4\end{bmatrix}. \begin{bmatrix}5 \\-2 \\3\end{bmatrix}=16$$This is the same as$$ \begin{bmatrix}2 & 3 & 4\end{bmatrix}. \begin{bmatrix}5 & -2 & 3\end{bmatrix}=16$$In vector multiplication, rows and columns don't really matter. However, most of the time we want to use **the matrix convention**: "rows times columns". This means that both products above are undefined. Also, the inner product is$$ \begin{bmatrix}2 & 3 & 4\end{bmatrix}. \begin{bmatrix}5 \\-2 \\3\end{bmatrix}=16$$The inverse operation, following our convention, will return a matrix (this is called **outer product**):$$ \begin{bmatrix}2 \\3 \\4\end{bmatrix}. \begin{bmatrix}5 & -2 & 3\end{bmatrix}=\begin{bmatrix}10 & 15 & 20 \\-4 & -6 & -8 \\6 & 9 & 12\end{bmatrix}$$Let's compare how the default `numpy` behaviour does and how we can impose our convention.First, let's create the arrays. Next, multiply them. Everything should look fine... until we look at the shapes of `x` and `y` which are `(3,)`. This kind of array is called **rank-1 array**. The matrix convention **DOES NOT** apply to it. One big error is evident when we try to transpose them.<jupyter_code>x = np.array([2, 3, 4])
y = np.array([5, -2, 3])
print("x.y =", str(x.dot(y)))

print("x.shape:", x.shape)
print("y.shape:", y.shape)
print("x:", x)
print("x transpose:", x.T)<jupyter_output>x.y = 16
x.shape: (3,)
y.shape: (3,)
x: [2 3 4]
x transpose: [2 3 4]<jupyter_text>Transposition **DID NOT** turn our row-vector into a column vector! This is correct but **does not follow the matrix convention**.How do we follow the matrix convention then? Simple, just represent the vectors as matrices (2D arrays).<jupyter_code>x = np.array([[2, 3, 4]]) # Row vector
y = np.array([[5], [-2], [3]]) # Column vector

print("x.shape:", x.shape)
print("y.shape:", y.shape)
print("x.y:\n", x.dot(y)) # Dot product -> still looks like a matrix
print("y.x:\n", y.dot(x)) # Outer product -> matrix<jupyter_output>x.shape: (1, 3)
y.shape: (3, 1)
x.y:
 [[16]]
y.x:
 [[10 15 20]
 [-4 -6 -8]
 [ 6  9 12]]
{'fraction_non_alphanumeric': 0.1368306010928962, 'fraction_numerical': 0.032568306010928964, 'mean_word_length': 4.296296296296297, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '140059', 'license': 'MIT', 'n_tokens_mistral': 1566, 'n_tokens_neox': 1425, 'n_words': 629, 'path': 'math/linear-algebra/vectorization.ipynb', 'repo_name': 'VladiGGeorgiev/ml-playground', 'size': '4575'}