<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;NumPy1.0.1&nbsp;&nbsp;Numpy Arrays: overview1.1&nbsp;&nbsp;Array Creation1.1.1&nbsp;&nbsp;Generating numpy arrays from lists1.1.2&nbsp;&nbsp;Array vs list cration: which is faster?1.1.3&nbsp;&nbsp;Generating arrays using special methods1.1.4&nbsp;&nbsp;Summary of common array creation methods1.2&nbsp;&nbsp;Indexing, slicing and shaping arrays1.2.1&nbsp;&nbsp;Vectorized operations with numpy1.2.2&nbsp;&nbsp;Aggregation1.2.3&nbsp;&nbsp;Reshaping arrays1.2.4&nbsp;&nbsp;Broadcasting rules of numpy arrays1.2.5&nbsp;&nbsp;Pandas1.2.6&nbsp;&nbsp;Exercises  NumPy NumPy is the core library for numerical and scientific computing in Python. It provides a high-performance multidimensional array object, and tools for working with these arrays.  To use NumPy, we first need to import the `numpy` package:<jupyter_code>import numpy as np
print(np.__version__)<jupyter_output>1.19.2<jupyter_text>Numpy Arrays: overview - A numpy array is a grid of values, all of the same type, and is indexed by nonnegative integers. - The array can have any number of dimensions 1D, 2D, 3D, ...- The shape of an array is a tuple of integers giving the size of the array along each dimension. For example a 1D vector of size 4 is (4,). a matrix of size 2 is (2,2), a matrix with size 2x5 is (2,5) - Numpy arrays can be generates either by feeding lists to numpy or on the fly using numpy special methods  Array Creation  Generating numpy arrays from lists<jupyter_code>data=np.array([1,2,3])
data
data.shape<jupyter_output><empty_output><jupyter_text>![](./figs/numpy1.png)<jupyter_code>print(data[0], data[1], data[2]) 
data[0] = 10                 # Change an element of the array
print(data)                   
b = np.array([[1,2,3],[4,5,6]])   # Create a 2D array
print(b)
print(b.shape)                    
print(b[0, 0], b[0, 1], b[1, 0])
# Question: what does this do?
np.array( [ xx**2 for x in range(100) if x%3==0 ])<jupyter_output><empty_output><jupyter_text>Array vs list cration: which is faster?Jupyter notebooks have a nice built-in method to time how long a line of code takes to execute. In a Jupyter notebook, when a line starts with ```%timeit``` followed by code, the kernel runs the line of code multiple times and outputs an average of the time spent to execute the line of code.We can use ```%timit``` to compare a mathematical operation on a Python list using a for loop to the same mathematical operation on a NumPy array.<jupyter_code>lst = list(range(10000))
%timeit for i, item in enumerate(lst): lst[i] = lst[i]*2
nparray = np.arange(0,10000,1)
%timeit 2*nparray<jupyter_output>2.85 µs ± 71.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)<jupyter_text>With 10,000 integers, the Python list and for loop takes an average of single milliseconds, while the NumPy array completes the same operation in tens of microseconds. This is a speed increase of over 100x by using the NumPy array (1 millisecond = 1000 microseconds).For larger lists of numbers, the speed increase using NumPy is considerable.  Generating arrays using special methods ![](./figs/numpy2.png)<jupyter_code>a = np.zeros((5,8))  # Create an array of all zeros
print(a)
b = np.ones((1,5))   # Create an array of all ones
print(b) 
e = np.random.random((4,4)) # Create an array filled with random values
print(e)
x = np.linspace(1,100,10) # create an array between 1 and 100 divided by 10 segments
print(x)
y = np.arange(1,100,5) # create an array strting from 1 to 100 in 10 incremenets
print(y)
c = np.full((2,2), 7.5) # Create a constant array
print(c)  
d = np.eye(5)        # Create a 3x3 identity matrix
print(d) 
k = np.tile(d,3)  # repeat the array d 3 times
k<jupyter_output><empty_output><jupyter_text>Summary of common array creation methodsBelow is a list of NumPy functions and associated descriptions used in this section.| Function | Description || --- | --- || ```np.array([list, of, numbers])``` | Array from a list || ```np.arange(start, stop, step)``` | Array with know step || ```np.linspace(start, stop, num)``` | Array with known num || ```np.logspace(start, stop, num)``` | Logorithmically spaced array ||```np.zeros((rows, cols))``` | Array of zeros || ```np.ones((rows, cols))``` | Array of ones || ```np.random.randint(start, stop, num)``` | Random integers || ```np.random.rand(num)``` | Random float 0 to 1 || ```np.random.choice(list, num)``` | Randome choice from a list | | ```np.random.randn(num)``` | Random normal distribution || ```np.meshgrid(array1, array2)``` | Two 2D arrays from two 1D arrays  || ```np.mgrid[start:stop:step, start:stop:step]``` | MATLAB meshgrid |  Indexing, slicing and shaping arrays **Slicing:** Similar to Python lists, numpy arrays can be sliced. Since arrays may be multidimensional, you must specify a slice for each dimension of the array:<jupyter_code>data=np.array([1,2,3])
data[0:3]<jupyter_output><empty_output><jupyter_text>![](./figs/numpy_index.png)<jupyter_code>data = np.array([[1,3,5], [2,4,6]])
data.T<jupyter_output><empty_output><jupyter_text>![](./figs/numpy-matrix-indexing.png)<jupyter_code>a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
a.shape
a.shape
a[1,:4]  #
a[1,3]
a[:,-1] # last column
a[-1,:] # last row<jupyter_output><empty_output><jupyter_text>**Same principles of slicing and shapes applies to the N-dimensional arrays.** ![](./figs/numpy_3d.png)  Vectorized operations with numpy Basic mathematical functions operate **elementwise on arrays**, and are available both as operator overloads and as functions in the numpy module:<jupyter_code>x = np.array([1,2,3,4])
y = np.array([5,6,7,8])
# Elementwise sum; both produce the array
print(x + y) 
# Elementwise difference; both produce the array
print(x - y)  
# Elementwise product; both produce the array
print(x * y) 
print(x / y) 
print(np.sqrt(x)) 
1.5*x  # elementwise multiplication!
y+3    # elementwise addition.<jupyter_output><empty_output><jupyter_text>As last two examples show can also do operations on arrays with unequal shapes! These are powerful operations which follow set of rules called **broadcasting.** See the end for these rules and examples **To use vector,matrix dot product between A and B use A@B**<jupyter_code>x = np.array([[1,2],[3,4]])
y = np.array([[5,6],[7,8]])

v = np.array([9,10])
w = np.array([11, 12])

# Inner product of vectors; both produce 219
print(v@w) 
# Matrix / vector product; both produce the rank 1 array [29 67]
print(x@v) 
# Matrix / matrix product; both produce the rank 2 array
print(x@y)<jupyter_output>20<jupyter_text>Aggregation Numpy provides many useful functions for performing computations on arrays; one of the most useful is `sum`:<jupyter_code>x = np.array([[1,2],[3,4]])
np.sum(x,axis=1)
print(np.sum(x))   # Compute sum of all elements; prints "10"
print(np.sum(x, axis=0))  # Compute sum of each column; prints "[4 6]"
print(np.sum(x, axis=1))   # Compute sum of each row; prints "[3 7]"
print(x.max())
print(x.min())<jupyter_output>4
1<jupyter_text>Reshaping arrays<jupyter_code>x=np.array([1,2,3,4,5,6,7,8,9,10])
x=x.reshape(2,5)
x
x=x.reshape(5,2)
x
x.T # transpose matrix<jupyter_output><empty_output><jupyter_text>Broadcasting rules of numpy arrays Broadcasting is a powerful mechanism that allows numpy to work with arrays of different shapes when performing arithmetic operations. Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.  The rules of broadcasting are:- **Rule 1:** If the two arrays differ in their number of dimensions, the shape of the one with fewer dimensions is padded with ones on its leading (left) side.- **Rule 2:** If the shape of the two arrays does not match in any dimension, the array with shape equal to 1 in that dimension is stretched to match the other shape.- **Rule 3:** If in any dimension the sizes disagree and neither is equal to 1, an error is raised. ![](./figs/numpy-matrix-broadcast.png) **Examples of broadcasting**<jupyter_code>data     = np.array([[1,2],[3,4],[5,6]])
ones_row = np.array([1,1])
data.shape, ones_row.shape
data
ones_row
data.shape, ones_row.shape
data+ones_row<jupyter_output><empty_output><jupyter_text>**Let us see both rules in action on another example**<jupyter_code>a = np.arange(3).reshape((3, 1))
print(a)
print(a.shape)
b = np.arange(3)
print(b)
print(b.shape)<jupyter_output>[0 1 2]
(3,)<jupyter_text>Lets predict a+b sum. By first rule the sum of arrays with shapes **(3,1)+(3,)** are broadcast to **(3,1)+(1,3)** then by second rule dimensions one are padded to match the shape **(3,3)+(3,3)**<jupyter_code>a+b<jupyter_output><empty_output><jupyter_text>Pandas You may thinkg of numpy as enhancing functionality of lists for numerical computations. In the same vein you can think of pandas as enahcnign dicitonaires to deal with heteogenuous categorical data. Pandas is widely used by data analysts from all disciplines to carry out rapid data cleaning, statistical analysis and plotting. The DataFrame is the ore object of pandas whihc stores observables as columns whith rows indicating measurments or samples. Lets create an example<jupyter_code>import pandas as pd
import numpy as np
A = pd.DataFrame({'Time': [1,2,3,4,5],
                 'Energy': [10,20,30,40,50]
                 })
A
A['velocity'] = np.zeros(5)
A.columns
A.index
# acess underlying values as numpy arrays
A['Energy'].values<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12535642623297075, 'fraction_numerical': 0.03347766395606717, 'mean_word_length': 5.133419689119171, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '244435', 'license': 'MIT', 'n_tokens_mistral': 3283, 'n_tokens_neox': 3087, 'n_words': 1123, 'path': '00_python/.ipynb_checkpoints/intro2numpy-checkpoint.ipynb', 'repo_name': 'DPotoyan/Statmech4ChemBio', 'size': '9469'}