<jupyter_start><jupyter_text>Python NumPy library  NumPy is the most fundamental package for numeric computing with Python!* a powerful N-dimensional array object* sophisticated (broadcasting) functions* tools for integrating C/C++ and Fortran code* useful linear algebra, Fourier transform, and random number capabilitiesNumPy is licensed under the BSD license, enabling reuse with few restrictions.https://numpy.org/* [NumPy cheat sheet](https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Numpy_Python_Cheat_Sheet.pdf)---This notebook is based on https://github.com/ValRCS/LU-pysem/blob/master/Python%20NumPy.ipynb  Comes with Anaconda, otherwise pip install NumPy<jupyter_code>import numpy as np  # extremely common naming for Numpy to be used as np
np.arange(10)
a = np.arange(20).reshape(4,5)
a
b = np.arange(5,500,10)
b<jupyter_output><empty_output><jupyter_text>NumPy arraysNumPy’s main object is the homogeneous multidimensional array. It is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers. In NumPy dimensions are called axes. In other words think of matrix (2-D, 3-D, etc)  with elements given their coordinates NumPy’s array class is called ndarray. It is also known by the alias array. Note that numpy.array is not the same as the Standard Python Library class array.array, which only handles one-dimensional arrays and offers less functionality.<jupyter_code>type(a),type(b),a.dtype,b.dtype<jupyter_output><empty_output><jupyter_text>When should you use ndarray instad of normal Python data structures such as list? * NumPy's arrays are more compact than Python lists (about 3-5x difference)* Faster access for reading/writing items / less Python "bookkeeping"* Convencience operations matrix operations/Fourier Transforms/histograms/statistics/filtering/etc* Spatial locality in memory access patterns results in performance gains notably due to the CPU cache* Items stored contiguously in memory allow NumPy to take advantage of vectorized instructions of modern CPUs, such as Intel's SSE and AVX, AMD's XOP* Restriction(uniform data types)<jupyter_code>## Litmus test: for group operations use NumPy, if you have many individual operations
## then regular Python lists might suffice

## NOTE: For multiple data types better to use Pandas library(another lecture)
d=a*2
d
c=a**2
c
# individual elements
c[1,1]
c[1]
c[:2]
c[:,1]
c[::-1]
c[::-1,::-1]<jupyter_output><empty_output><jupyter_text>Filling arrays with values<jupyter_code>np.zeros(5)
np.zeros((5,3))
np.ones(5)
np.ones(5, dtype=int)
np.eye(5)<jupyter_output><empty_output><jupyter_text>Ranges<jupyter_code># integer numbers
np.arange(10)
np.arange(10000).reshape(100,100)
# floating point numbers
np.linspace(0.0, 1.0, num=5)<jupyter_output><empty_output><jupyter_text>Random numbers<jupyter_code>my_csv = np.random.random(36)
my_csv = my_csv.reshape(9,4)

my_csv
# Let's roll dice

np.random.randint(1, 7, 50)
# How long time will *many* dice rolls take?

%timeit rolls = np.random.randint(1, 7, 1_000_000)
# Let's do the same without NumPy

import random
%timeit rolls_list = [random.randrange(1, 7) for i in range(1_000_000)]<jupyter_output>1.44 s ± 31.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Array attributes and operations<jupyter_code>dir(a)
my_csv.ndim
my_csv
my_csv.T
my_csv.shape
my_csv.dtype
my_csv.itemsize
my_csv.data<jupyter_output><empty_output><jupyter_text>Most important array attributes* ndarray.ndimthe number of axes (dimensions) of the array.* ndarray.shapethe dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m). The length of the shape tuple is therefore the number of axes, ndim.* ndarray.sizethe total number of elements of the array. This is equal to the product of the elements of shape.* ndarray.dtypean object describing the type of the elements in the array. One can create or specify dtype’s using standard Python types. Additionally NumPy provides types of its own. numpy.int32, numpy.int16, and numpy.float64 are some examples.* ndarray.itemsizethe size in bytes of each element of the array. For example, an array of elements of type float64 has itemsize 8 (=64/8), while one of type complex32 has itemsize 4 (=32/8). It is equivalent to ndarray.dtype.itemsize.* ndarray.datathe buffer containing the actual elements of the array. Normally, we won’t need to use this attribute because we will access the elements in an array using indexing facilities.<jupyter_code>a.ndim
a.shape
a.size
a.dtype
a.itemsize
## NumPy ndarray advantage over normal Python data structures:
#   * efficient and fast 
my_csv.mean()
a
a[2,0]=20
a
np.median(a)
np.median(a, axis=1)
np.median(a, axis=0)
a.mean(axis=1)
a.mean(axis=0)
a.std() ## standard deviation
a.std(axis=1)
# reshape: 3 dimensions

d=np.arange(90).reshape(3,3,10)
d
d[2,1,5]
# reshape: 4 dimensions

d.reshape(3,3,2,5)<jupyter_output><empty_output><jupyter_text>NumPy data types https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html<jupyter_code># A frequent error consists in calling array with multiple numeric arguments,
# rather than providing a single list of numbers as an argument.

# c = np.array(2,3,4,5,6,7)  # WRONG 

# ---------------------------------------------------------------------------
# ValueError                                Traceback (most recent call last)
# <ipython-input-183-6548e3003d99> in <module>
# ----> 1 c = np.array(2,3,4,5,6,7)  # WRONG

# ValueError: only 2 non-keyword arguments accepted


# CORRECT would be
c = np.array([2,3,4,6,7,9,44])
c
c.dtype
t = np.array([4000000000000,"R"])
t
t = np.array((5,6,7,9,1111111111112,0.4))
t
s = np.array(["Hello",'World','''Aha something strange'''])
s
s.itemsize
s.dtype
## Can specify array type at creation
c = np.array([[3,4,5], [3,1,6]], dtype=complex)
c
c[0,0].real
c[0,0].imag
# c[1,1].imag = 5

# This will not work as .imag only returns the value:

# AttributeError: attribute 'imag' of 'numpy.generic' objects is not writable
c[1,1] = 2 + 3j # use Python complex numbers instead (j for the imaginary part)
c
c.dtype, c.size, c.shape
d = np.array([[3+4j,4,5+2.1j], [3,1-3j,6]], dtype=complex)
d
d[0,0].real,d[0,0].imag
z = np.zeros((5,6))
z[3,3] = 42
print(type(z))
print(z.dtype)
z
## Data Type conversion IN PLACE!
z = z.astype(np.int32, copy=True)
print(z.dtype)
z
## Matrix operations
ones = np.full((5,6), 1, dtype=np.int32)
ones
ones=ones+z
ones
ones*z
twos = np.full((5,6), 2, dtype=np.int32)
twos
ones*twos
# Anyone remembers the algorithm for Matrix Multiplication? 
# I don't but NumPy lets us not worry about it :)
np.matmul(ones, twos)
## We have mismatching dimensions for matrix multiplication 

# the number of columns in the first matrix must be equal 
# to the number of rows in the second matrix.
(ones.shape, twos.shape)
twos.transpose()
twos.transpose().shape
mult = np.matmul(ones, twos.transpose())
mult
mult1 = np.matmul(ones.transpose(), twos)
mult1
mult.sum(), mult1.sum()
np.sqrt(mult)
mult
mult**2
# np.sin, np.cos, np.exp(power), and so on<jupyter_output><empty_output><jupyter_text>Array stacking and splitting<jupyter_code>[1,3,65] + [4,6,7]
vstack = np.vstack((ones,twos,ones)) # notice the double parenthesis, ie we supply a tuple as an argument!
vstack
hstack = np.hstack((twos,ones,twos))
hstack
hsplit = np.hsplit(hstack, 3)
print(type(hsplit))
hsplit
h2 = np.hsplit(hstack, 6)
print(type(h2))
h2
## vsplit is similar along vertical axis and array_split lets you specify axis<jupyter_output><empty_output><jupyter_text>View or Shallow Copy<jupyter_code>a = np.arange(20).reshape(4,5)
a
v = a.view()
v
v is a
v == a
v.base is a.base
v[2,2]=999
v
a
s = a[:,1:3]
s
s.base is a.base
s[:] = 55
s
a<jupyter_output><empty_output><jupyter_text>Remember: Slice is a shallow copy to original (ie reference)  Deep Copy<jupyter_code>d = a.copy()
d
d.base is a.base
d[1,1]=777
d
a
a == d<jupyter_output><empty_output><jupyter_text>Now d is completely separate object from a !!<jupyter_code>## More on copy vs sliceing: https://stackoverflow.com/questions/47181092/numpy-views-vs-copy-by-slicing
# key depends on how you slice
# slice by rows -> view (shallow copy)
# slice by column -> deep copy because not contigous<jupyter_output><empty_output><jupyter_text>Useful NumPy Functions and Methods (there are more!) * Array Creationarange, array, copy, empty, empty_like, eye, fromfile, fromfunction, identity, linspace, logspace, mgrid, ogrid, ones, ones_like, r, zeros, zeros_like* Conversionsndarray.astype, atleast_1d, atleast_2d, atleast_3d, mat* Manipulationsarray_split, column_stack, concatenate, diagonal, dsplit, dstack, hsplit, hstack, ndarray.item, newaxis, ravel, repeat, reshape, resize, squeeze, swapaxes, take, transpose, vsplit, vstack* Questionsall, any, nonzero, where* Orderingargmax, argmin, argsort, max, min, ptp, searchsorted, sort* Operationschoose, compress, cumprod, cumsum, inner, ndarray.fill, imag, prod, put, putmask, real, sum* Basic Statisticscov, mean, std, var* Basic Linear Algebracross, dot, outer, linalg.svd, vdot<jupyter_code>f=np.ones((5,6), dtype='int32')
f[2,3]=0
f
f.nonzero()
g = np.random.randint(0,20,16).reshape(4,4)
g
g.sort(axis=0)
g<jupyter_output><empty_output><jupyter_text>Boolean Arrays<jupyter_code>a
b = a > 17
b
a[b] # return values from a where b is True
## Useful in assignments
a[b] = 500
a<jupyter_output><empty_output><jupyter_text>Fancy indexing Python Data Science Handbook:https://jakevdp.github.io/PythonDataScienceHandbook/02.07-fancy-indexing.html  Automatic Reshaping<jupyter_code>arr = np.arange(24)
arr<jupyter_output><empty_output><jupyter_text>To change the dimensions of an array, you can omit one of the sizes which will then be deduced automatically:<jupyter_code>arr.shape = 6, -1
arr
arr.shape = -1, 2
arr<jupyter_output><empty_output><jupyter_text>Broadcasting * The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes* usually leads to efficient algorithm implementations.<jupyter_code>e = np.arange(6)
f = np.arange(3)*10  #same as np.arange(0,30,10)

e
f
e + f<jupyter_output><empty_output><jupyter_text>General Broadcasting Rules When operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions, and works its way forward. Two dimensions are compatible when* they are equal, or* one of them is 1<jupyter_code>e.shape, f.shape
e.reshape(6,1)
f.reshape(1,3)
e.reshape(6,1) + f.reshape(1,3)
e.reshape(6,1) * f.reshape(1,3)
mtab = np.arange(10).reshape(10,1) * np.arange(10).reshape(1,10)
mtab
g=np.arange(12)
h=np.arange(4)*100
g,h
g.reshape(4,3)
g.reshape(4,3)+h.reshape(4,1)
g.reshape(4,3)+50<jupyter_output><empty_output><jupyter_text>Histograms np version only builds the histogram but does not plot it (unlike mathplotlib version)<jupyter_code>import matplotlib.pyplot as plt
# Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2
mu, sigma = 2, 0.5
v = np.random.normal(mu,sigma,20000)

v[:20], v.mean(),v.max(),v.min()
# Plot a normalized histogram with 50 bins
plt.hist(v, bins=500)       # matplotlib version (plot)
plt.show()
# Compute the histogram with numpy and then plot it
(n, bins) = np.histogram(v, bins=500)  # NumPy version (no plot)
n, bins
plt.plot(.5*(bins[1:]+bins[:-1]), n)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.121737622474638, 'fraction_numerical': 0.034509667909477154, 'mean_word_length': 5.323464912280702, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 86, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '227576', 'license': 'MIT', 'n_tokens_mistral': 4259, 'n_tokens_neox': 4006, 'n_words': 1273, 'path': 'NumPy/Python NumPy.ipynb', 'repo_name': 'CaptSolo/LU_BigData_2021', 'size': '11533'}