<jupyter_start><jupyter_text>Introduction to Numpy (Numerical Python) **Note: Numpy is considered as the backbone of data science, machine learning and numerical python**  Why Numpy ?- It's fast- Behind the scenes optimizations written in C- Vectorization via broadcasting (avoiding loops)- Backbone of other Python scientific packages  What are we going to cover?- Most useful functions- NumPy datatypes & attributes (ndarray)- Creating arrays- Viewing arrays & matrices- Manipulating & comparing arrays- Sorting arrays- Use cases  Anatomy of a Numpy Array ![Anatomy of a Numpy Array ](https://cdn.fs.teachablecdn.com/ADNupMnWyR7kCWRvm76Laz/resize=width:1500/https://www.filepicker.io/api/file/wiL4GtSsS5uqS5pAqpbS)  Importing Numpy<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>DataType & Attributes<jupyter_code># NumPy's main datatype is ndarray
a1 = np.array([1,2,3])
# To see the type of a1
type(a1)
a2 = np.array([[1,2,3],[44,5,3]])
# n dimension array: n can be any number
a3 = np.array([[[1,2,3], [2,3,4]],[[2,3,4],[2,3,4]]])
a3<jupyter_output><empty_output><jupyter_text>Shape of Array<jupyter_code>a1.shape
a2.shape
a3.shape<jupyter_output><empty_output><jupyter_text>Dimension of arrays<jupyter_code>a1.ndim , a2.ndim , a3.ndim # Refers to n dimesions in an array<jupyter_output><empty_output><jupyter_text>Data types of the array<jupyter_code>a1.ndim , a2.ndim , a3.ndim<jupyter_output><empty_output><jupyter_text>Size of the array<jupyter_code>a1.size , a2.size , a3.size<jupyter_output><empty_output><jupyter_text>Type of array<jupyter_code>type(a1), type(a2), type(a3)<jupyter_output><empty_output><jupyter_text>Creating the Pandas data from numpy 2D-array<jupyter_code>import pandas as pd
data = pd.DataFrame(a2)
data<jupyter_output><empty_output><jupyter_text>Creating Arrays<jupyter_code>sample_array = np.array([1,2,3])
sample_array
ones = np.ones((2,3))
ones
ones_like = np.ones_like(sample_array) # This will take an array and will return the array of one's of same size and type
ones_like
zeros = np.zeros((2,3))
zeros<jupyter_output><empty_output><jupyter_text>Creating a Array of Range<jupyter_code>#arange is same as range function in python
rangeArr = np.arange(0,20)
rangeArr
rangeArr_step2 = np.arange(0,20,2)
rangeArr_step2<jupyter_output><empty_output><jupyter_text>Creating a Random Array<jupyter_code>random_array = np.random.randint(0,10,size=(3,5)) #Always return int numbers
random_array
random_array_2 = np.random.rand(3,5) ##
random_array_2<jupyter_output><empty_output><jupyter_text>The Numbers from random actually come from np.random.seed<jupyter_code>randInt = np.random.randint(0,100,size=(10)) # This will always give random number because seed is random
randInt
np.random.seed(10)
randInt_seed = np.random.randint(0,100,size=(10)) # This will always give same random number 
randInt_seed<jupyter_output><empty_output><jupyter_text>Run Multiple time the cell above you will see the same result while the cell above random seed will give random number  Viewing Arrays and matrices  Viewing the unique numbers from array<jupyter_code>np.unique(random_array_2)<jupyter_output><empty_output><jupyter_text>Indexing is same as python list in Numpy<jupyter_code>print(a1)
a1[1]<jupyter_output>[1 2 3]<jupyter_text>Manipulating & comparing arrays  Arithmetic<jupyter_code>a1
one = np.ones_like(a1)
a1 + one
a1 - one
one = one * 2
one
a1 / one
a1 // one
a1 ** 2<jupyter_output><empty_output><jupyter_text>Aggregation<jupyter_code>a1
np.sum(a1) # 1+2+3 => 6<jupyter_output><empty_output><jupyter_text>Let's explore a linspace function which is often used to generate a data (mostly for ml tasks)<jupyter_code>ten = np.linspace(1,10,num=10)
ten
sum(ten)<jupyter_output><empty_output><jupyter_text>Let's create a 10000000 numbers of array<jupyter_code>millions = np.linspace(1,10,num=10000000)
millions
millions.size<jupyter_output><empty_output><jupyter_text>Let's explore the sum method and numpy.sum method<jupyter_code>%timeit sum(millions)
%timeit np.sum(millions)<jupyter_output>5.25 ms ± 383 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)<jupyter_text>As we can see how powerful numpy is. The same operation we performed with python sum function took way more time.<jupyter_code># mean 
np.mean(ten)<jupyter_output><empty_output><jupyter_text>Standard Deviation :A measure of how spread out data values are around the mean, defined as the square root of the variance.<jupyter_code># Standard deviation
np.std(ten)<jupyter_output><empty_output><jupyter_text>VarianceIn probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its mean. Informally, it measures how far a set of numbers are spread out from their average value.<jupyter_code># Variance
np.var(ten)<jupyter_output><empty_output><jupyter_text>One of the best article i found explaining standard deviation and variance is [here](https://www.mathsisfun.com/data/standard-deviation.html)  Reshaping and Transposing<jupyter_code>a2.shape
a3.shape<jupyter_output><empty_output><jupyter_text>Broadcasting rules  according to numpy are BroadCastingThe term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.  General Broadcasting Rules¶When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing (i.e. rightmost) dimensions and works its way left. Two dimensions are compatible when - they are equal, or - one of them is 1          You can read more about broadcating [here](https://numpy.org/doc/stable/user/basics.broadcasting.html)  It hards to get let's see it in action<jupyter_code>a1.shape
a3.shape
# It works, because the dimension is one. Now let's see 2d by 3d
a1 * a3
a2.shape
a3.shape
a2 * a3<jupyter_output><empty_output><jupyter_text>What if we have an array of (2,3) and (2,3,1)<jupyter_code>rand3D = np.random.rand(2,3,1)
rand3D
rand3D.shape
a2.shape
a2 * rand3D
#### Now's Let see rand3D with (4)
randD = np.random.rand(4)
randD
randD.shape
randD * rand3D<jupyter_output><empty_output><jupyter_text>Let's see some more example  Example 1|  Array       | Size   || :---        | ---: || A(4d array): | 8 x 1 x 6 x 1 |  | B(3d array): | 7 x 1 x 5 |   | Result (4d array):|8 x 7 x 6 x 5|    If you want to see more example go to numpy offical [documentation](https://numpy.org/doc/stable/user/basics.broadcasting.html)  Reshape the array<jupyter_code>a2.shape
rand3D.shape
a2 = a2.reshape(2,3,1)<jupyter_output><empty_output><jupyter_text>Array a2 has been reshaped all the elements are same but the shape is changed now<jupyter_code>a2.shape<jupyter_output><empty_output><jupyter_text>As the Array2 has been reshaped now we can multiple it with rand3D<jupyter_code>a2 * rand3D<jupyter_output><empty_output><jupyter_text>Transpose the matrix<jupyter_code>a2.shape
a2.T<jupyter_output><empty_output><jupyter_text>It's same as matrix transpose<jupyter_code>a2.T.shape<jupyter_output><empty_output><jupyter_text>dot Product of two array ![Dot Product vs. element-vise](https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fd6lIaqQI0UE%2Fmaxresdefault.jpg&f=1&nofb=1)<jupyter_code>np.random.seed(0)
arr1 = np.random.randint(10,size=(5,3))
arr2 = np.random.randint(10,size=(3,5))
arr1
arr2
arr1.shape, arr2.shape
np.dot(arr1,arr2)<jupyter_output><empty_output><jupyter_text>To explore more about dot product, check out this amazing [website](http://matrixmultiplication.xyz/)  Comparison Operators<jupyter_code>a1
a2[0][:].T[0]
bool2 = a1 > a2[0][:].T[0]
bool2
bool1 = a1 == a2[0][:].T[0]
bool1<jupyter_output><empty_output><jupyter_text>Sorting Arrays<jupyter_code>rand_arr = np.random.randint(10,size=(3,5))
rand_arr
rand_arr.shape
np.sort(rand_arr)<jupyter_output><empty_output><jupyter_text>Returns the indices that would sort an array.<jupyter_code>np.argsort(rand_arr)
a1.sort()
a1<jupyter_output><empty_output><jupyter_text>Return indices of the maximum values along the given axis.<jupyter_code>a1.argmax()
np.argmax(a1)<jupyter_output><empty_output><jupyter_text>Return indices of the minimum values along the given axis.<jupyter_code>a1.argmin()
np.argmin(a1)
rand3D.shape
np.argmax(rand3D,axis=1)<jupyter_output><empty_output><jupyter_text>If you want to dig dive, you can use these resources- [The Basics of NumPy Arrays by <NAME>](https://jakevdp.github.io/PythonDataScienceHandbook/02.02-the-basics-of-numpy-arrays.html)- [A Visual Introduction to NumPy by <NAME>](https://jalammar.github.io/visual-numpy/)- [NumPy Quickstart tutorial (part of the NumPy documentation)](https://numpy.org/doc/1.17/user/quickstart.html)As always, remember the best way to learn a new coding concept is to write it yourself. If in doubt, run the code.  Practical Example - Turning image into numpy array<jupyter_code>from matplotlib.image import imread
plane = imread('./airplane.jpg')
plane
plane.size, plane.shape, plane.ndim<jupyter_output><empty_output><jupyter_text>The dimension of image is 3 because 1 = height, 2 = width, 3 = RGB  Let's change some values and try to view the image back<jupyter_code>img_1 = plane ** 2

img_1
from matplotlib import pyplot as plt
plt.imshow(img_1,interpolation='nearest')
plt.show()
plt.imshow(plane,interpolation='nearest')
plt.show()
plt.imshow(plane[100:][400:800],interpolation='nearest')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10877340626972438, 'fraction_numerical': 0.030717441615821586, 'mean_word_length': 6.679321486268175, 'pattern_counts': {'":': 0, '<': 153, '<?xml version=': 0, '>': 155, 'https://': 9, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '405918', 'license': 'MIT', 'n_tokens_mistral': 3571, 'n_tokens_neox': 3414, 'n_words': 867, 'path': 'Intro-to-Numpy.ipynb', 'repo_name': 'sameerkatija/MachineLearningNotes', 'size': '9520'}