<jupyter_start><jupyter_text>NumPy Tutorial  What is NumPy?- A Python librrary to efficiently store and operate data in-memory- A core library of the ecosystem of data science tools in Python- Its main object is the homogeneous multidimensional array    * array -- a block of elements with the same data type and indexed by a tuple of integers    * multidimensional    * homogeneous  NumPy Basics<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>np.ndarray versus np.arrayRead the documentation of `np.ndarray` and `np.array`. Then answer the following questions.1. Is `np.array` a function or a class? How about `np.ndarray`?1. Can you explain the meaning of the following attributes of `np.ndarray`?   + T   + data   + dtype   + size   + nbytes   + itemsize   + nbytes   + ndim   + shape   + base   + stride   + real   + image1. What objects can be the first argument of `np.array()`?1. How to create a $2 \times 2$ np.ndarray with values 1, 2, 3, 4?1. Does Python have a buil-in array?   + list   + array module<jupyter_code>?np.array
?np.ndarray
import inspect

a = np.arange(5)

print("Attributes:")
for m in inspect.getmembers(a):
    if not m[0].startswith('_'):
        if not inspect.isbuiltin(m[1]):
            print("\t", m[0])

print("Methods:")
for m in inspect.getmembers(a):
    if not m[0].startswith('_'):
        if inspect.isbuiltin(m[1]):
            print("\t", m[0])<jupyter_output>Attributes:
	 T
	 base
	 ctypes
	 data
	 dtype
	 flags
	 flat
	 imag
	 itemsize
	 nbytes
	 ndim
	 real
	 shape
	 size
	 strides
Methods:
	 all
	 any
	 argmax
	 argmin
	 argpartition
	 argsort
	 astype
	 byteswap
	 choose
	 clip
	 compress
	 conj
	 conjugate
	 copy
	 cumprod
	 cumsum
	 diagonal
	 dot
	 dump
	 dumps
	 fill
	 flatten
	 getfield
	 item
	 itemset
	 max
	 mean
	 min
	 newbyteorder
	 nonzero
	 partition
	 prod
	 ptp
	 put
	 ravel
	 repeat
	 reshape
	 resize
	 round
	 searchsorted
	 setfield
	 setflags
	 sort
	 squeeze
	 std
	 sum
	 swapaxes
	 take
	 tobytes
	 tofile
	 tolist
	 tostring
	 trace
	 transpose
	 var
	 view<jupyter_text>Concepts<jupyter_code>x = np.array([[1, 2, 3], [4, 5, 6]], dtype=np.int32)
x
type(x)
x.shape
x.dtype
x.T<jupyter_output><empty_output><jupyter_text>Array reshaping<jupyter_code>a = np.arange(24)
a
b = a.reshape(3, 8)
b
c = b.reshape(3, 2, 4)
c
print("a[12] = ", a[12])
print("b[1][4] = ", b[1][4])
print("c[1][1][0] = ", c[1][1][0] )

c.base
c[-1, -1, -1] = 100
a[-1]<jupyter_output><empty_output><jupyter_text>ndarray memory layout<jupyter_code>import numpy as np
import pandas as pd
import collections
a = np.arange(10, dtype=np.float32)
b = a.reshape(2, 5)
d = collections.defaultdict(dict)
for n, c in {"a": a, "b": b}.items():
    for i, f in enumerate(str(c.flags).split('\n')):
        if f != '':
            k, v = f.split(":")
            d[n]["flags." + k] = v
    d[n]['shape'] = c.shape
    d[n]['strides'] = c.strides
    d[n]['ndim'] = c.ndim
    d[n]['size'] = c.size
    d[n]['itemsize'] = c.itemsize
    d[n]['nbytes'] = c.nbytes
    d[n]['base'] = c.base    
df = pd.DataFrame({"a": d["a"], "b": d["b"]})
df<jupyter_output><empty_output><jupyter_text>ndarray creation  Converting Python array_like Objects to NumPy Arrays<jupyter_code>x = np.array([1, 2, 3, 4], dtype=np.float32)
x
y = np.array([range(i, i+3) for i in [2, 4, 6]])
y<jupyter_output><empty_output><jupyter_text>Creating Arrays from scratch<jupyter_code>np.zeros((2, 3))
np.full((3, 3), 100)
np.arange(2, 3, 0.1)
np.arange(6)
np.eye(3)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

a = np.random.normal(0, 1, (100, 2))
_ = sns.scatterplot(x=a[:, 0], y=a[:, 1])<jupyter_output><empty_output><jupyter_text>Array Indexing and Slicing<jupyter_code>x = np.arange(24).reshape(2, 3, 4)
print('x =\n', x)
print('x[1] =\n', x[1])
print('x[1, 2] =\n', x[1, 2])
print('x[1, 2, 3] =', x[1, 2, 3])
print('x[1, 1:3, 0:2] =\n', x[1, 1:3, 0:2])
help(np.newaxis)
x = np.arange(8)
print('x =', x)
print('x[1:7:2] = ', x[1:7:2])
print('x[1::2] = ', x[1::2])
print('x[::2] = ', x[::2])
print('x[-1::] = ', x[-1::])
print('x[-1::-1] = ', x[-1::-1])<jupyter_output>x = [0 1 2 3 4 5 6 7]
x[1:7:2] =  [1 3 5]
x[1::2] =  [1 3 5 7]
x[::2] =  [0 2 4 6]
x[-1::] =  [7]
x[-1::-1] =  [7 6 5 4 3 2 1 0]<jupyter_text>Array Manipulation Routines  Concatenation of Arrays<jupyter_code>x = np.array([1, 2, 3])
y = np.array([3, 2, 1])
z = np.array([0, 0])
w = np.concatenate([x, y, z])
print('x=', x)
print('y=', y)
print('z=', z)
print('w=', w)
x = np.arange(12).reshape(3, 4)
y = np.arange(4).reshape(1, 4)
z = np.concatenate([x, y])
print('x=\n', x)
print('y=\n', y)
print('z=\n', z)
x = np.arange(12).reshape(3, 4)
y = np.arange(4).reshape(1, 4)
z = np.vstack([x, y])
print('x=\n', x)
print('y=\n', y)
print('z=\n', z)
# This will trigger an error
# w = np.hstack([x, y])
x = np.arange(12).reshape(4, 3)
y = np.arange(4).reshape(4, 1)
# now we should use np.hstack
z = np.hstack([x, y])
print('x=\n', x)
print('y=\n', y)
print('z=\n', z)
# This will trigger an error
# w = np.vstack([x, y])<jupyter_output>x=
 [[ 0  1  2]
 [ 3  4  5]
 [ 6  7  8]
 [ 9 10 11]]
y=
 [[0]
 [1]
 [2]
 [3]]
z=
 [[ 0  1  2  0]
 [ 3  4  5  1]
 [ 6  7  8  2]
 [ 9 10 11  3]]<jupyter_text>Splitting<jupyter_code>a = np.arange(10)
np.split(a, [1, 3, 6, ])
grid = a = np.arange(16).reshape(4, 4)
np.hsplit(grid, [1, 3])
np.vsplit(grid, [1, 3])<jupyter_output><empty_output><jupyter_text>Universal FunctionsFor many types of operations, NumPy provides a convenient interface into just this kind of statically typed, compiled routine. This is known as a vectorized operation. This can be accomplished by simply performing an operation on the array, which will then be applied to each element. This vectorized approach is designed to push the loop into the compiled layer that underlies NumPy, leading to much faster execution.  The Performance of UFunc versus For-loop<jupyter_code>import math
theta = np.linspace(0, np.pi, 10)
y = np.zeros(theta.size)
%%timeit
for i in range(theta.size):
    y[i] = math.sin(theta[i])
%%timeit
y = np.sin(theta)
# Performance improvement
4.77 * 1000 / 703<jupyter_output><empty_output><jupyter_text>Array arithmeticNumPy's ufuncs make use of Python's native arithmetic operators. NumPy implements the standard addition, subtraction, multiplication, and division operators for ndarray objects. You can use either the standard operators or the equivalent ufuncs. | Operator | Equivalent ufunc	| Description || :------------- | :----------: | -----------: || +	| np.add	| Addition (e.g., 1 + 1 = 2) || -	| np.subtract |	Subtraction (e.g., 3 - 2 = 1) || -	| np.negative |	Unary negation (e.g., -2) || *	| np.multiply |	Multiplication (e.g., 2 * 3 = 6) || /	| np.divide	| Division (e.g., 3 / 2 = 1.5) || // |	np.floor_divide |	Floor division (e.g., 3 // 2 = 1) || ** |	np.power	| Exponentiation (e.g., 2 ** 3 = 8) || %	| np.mod	| Modulus/remainder (e.g., 9 % 4 = 1) |<jupyter_code>x = np.arange(5)
print("x =", x)
print("x + 10 =", x + 10)
print("np.add(x, 10) = ", np.add(x, 10))
print("x ** 2 =", x ** 2)
print("np.power(x, 2) = ", np.power(x, 2))<jupyter_output>x = [0 1 2 3 4]
x + 10 = [10 11 12 13 14]
np.add(x, 10) =  [10 11 12 13 14]
x ** 2 = [ 0  1  4  9 16]
np.power(x, 2) =  [ 0  1  4  9 16]<jupyter_text>Reduce<jupyter_code>x = np.arange(1, 6)
print("x =", x)
print("np.add.reduce(x) = ", np.add.reduce(x))
print("np.multiply.reduce(x) = ", np.multiply.reduce(x))
print("np.add.accumulate(x) = ", np.add.accumulate(x))
print("np.multiply.accumulate(x) = ", np.multiply.accumulate(x))
print("np.cumprod(x) = ", np.cumprod(x))<jupyter_output>x = [1 2 3 4 5]
np.add.reduce(x) =  15
np.multiply.reduce(x) =  120
np.add.accumulate(x) =  [ 1  3  6 10 15]
np.multiply.accumulate(x) =  [  1   2   6  24 120]
np.cumprod(x) =  [  1   2   6  24 120]<jupyter_text>Custome UFunc<jupyter_code>?np.frompyfunc
def discount(x):
    if x < 5:
        return 0
    if x < 20:
        return math.floor(0.2 * x)
    return math.floor(0.4 * x)
uf_discount = np.frompyfunc(discount, 1, 1)
price = np.arange(1, 40, 8)
discounts = uf_discount(price)
print("price = ", price)
print("discounts = ", discounts)
# help(np.random.randint)<jupyter_output>Help on built-in function randint:

randint(...) method of numpy.random.mtrand.RandomState instance
    randint(low, high=None, size=None, dtype=int)
    
    Return random integers from `low` (inclusive) to `high` (exclusive).
    
    Return random integers from the "discrete uniform" distribution of
    the specified dtype in the "half-open" interval [`low`, `high`). If
    `high` is None (the default), then results are from [0, `low`).
    
    .. note::
        New code should use the ``integers`` method of a ``default_rng()``
        instance instead; see `random-quick-start`.
    
    Parameters
    ----------
    low : int or array-like of ints
        Lowest (signed) integers to be drawn from the distribution (unless
        ``high=None``, in which case this parameter is one above the
        *highest* such integer).
    high : int or array-like of ints, optional
        If provided, one above the largest (signed) integer to be drawn
        from the distribution (see above fo[...]<jupyter_text>Broadcasting<jupyter_code>a = np.array([1.0, 2.0, 3.0])
b = np.array([2.0, 2.0, 2.0])
print(a * b)
b = 2.0
print(a * b)
a = np.array([[ 0.0,  0.0,  0.0],
           [10.0, 10.0, 10.0],
           [20.0, 20.0, 20.0],
           [30.0, 30.0, 30.0]])
b = np.array([1.0, 2.0, 3.0])
print(a + b)
a = np.array([1.0, 2.0, 3.0])
b = np.array([1.0, 2.0, 3.0])
print(a[:, np.newaxis] + b)<jupyter_output>[[2. 3. 4.]
 [3. 4. 5.]
 [4. 5. 6.]]<jupyter_text>k-Nearst NeighborThe k-nearest neighbors algorithm (k-NN) is a learning algorithm that can be used for both classification and regression.+ In k-NN classification, the output is a class membership of an object decided by a plurality vote of its neighbors.+ In k-NN regression, the output is the average of the values of k nearest neighbors.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline
# Get data
X = np.random.rand(4, 2)
X
# Plot the data
plt.scatter(X[:, 0], X[:, 1], s=100)
X[:, np.newaxis, :]
X[np.newaxis, :, :]
# compute squared distances of any pair of points
pair_difference = X[:, np.newaxis, :] - X[np.newaxis, :, :]
pair_difference.shape
pair_difference
sq_pair_difference = pair_difference ** 2
sq_pair_difference.shape
sq_pair_difference
sq_dist = np.sum(sq_pair_difference, axis=2)
sq_dist.shape
sq_dist
# Verify: the diagnal of the sq_dist should all be zero

d = sq_dist.diagonal()
d
# You can use the assert to ensure all items in the diagonal of sq_dist are zeros
assert(np.all(d == 0))
?assert
# Sort each row to find the nearest neighbor
nearest = np.argsort(sq_dist, axis=1)
print(nearest)
# The leftmost item of each row is the index of the row. 
# This is what we expected since the nearest point of any point is itself.
# connect each point with its nearest neighbor
plt.scatter(X[:, 0], X[:, 1], s=100)
for i in range(X.shape[0]):
    j = nearest[i][1]
    pair_x = [X[i][0], X[j][0]]
    pair_y = [X[i][1], X[j][1]]
    plt.plot(pair_x, pair_y, color='red')
    
# We want to connect the two nearest neighbor
K = 2
plt.scatter(X[:, 0], X[:, 1], s=100)
for i in range(X.shape[0]):
    for j in nearest[i, 1:K+1]:
        pair_x = [X[i, 0], X[j, 0]]
        pair_y = [X[i,1], X[j, 1]]
        plt.plot(pair_x, pair_y, color='red')
# put it togther

def plot_knn(X, K):
    pair_difference = X[:, np.newaxis, :] - X[np.newaxis, :, :]
    sq_pair_difference = pair_difference ** 2
    sq_dist = np.sum(sq_pair_difference, axis=2)
    nearest = np.argsort(sq_dist, axis=1)

    plt.scatter(X[:, 0], X[:, 1], s=100)
    for i in range(X.shape[0]):
        for j in nearest[i, 1:K+1]:
            pair_x = [X[i, 0], X[j, 0]]
            pair_y = [X[i,1], X[j, 1]]
            plt.plot(pair_x, pair_y, color='red')
    
# Test teh function with existing data
plot_knn(X, 1)
#plot_knn(X, 2)
#plot_knn(X, 3)
plot_knn(X, 2)
plot_knn(X, 3)
plot_knn(X, 4)
# Testing the function with some new data 
Y = np.random.rand(20, 2)
plot_knn(Y, 2)
# What are about higher dimensional data?
Y = np.random.rand(20, 4)
# We only draw the the coordinates of the first two dimensions
plot_knn(Y, 2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1689390883420734, 'fraction_numerical': 0.04292053247277128, 'mean_word_length': 3.4461979913916787, 'pattern_counts': {'":': 5, '<': 60, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '270847', 'license': 'MIT', 'n_tokens_mistral': 5332, 'n_tokens_neox': 4895, 'n_words': 1438, 'path': 'numpy_pandas_matplotlib/numpy.ipynb', 'repo_name': 'cdhumme/notebooks', 'size': '12395'}