<jupyter_start><jupyter_text>Introduction to NumpyThis workbook aims to provide a quick and in-depth introduction to `numpy`.<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>``from numpy import *``This is not a good idea as `numpy` namespace is large and contains a number of functions which can conflict with standard built-in Python functions (like min and max). As a best practice, it is good to alias the namespace as np, so all the methods of `numpy` would be accessible via the np. syntax.<jupyter_code>my_arr = np.arange(1000000)

my_list = list(range(1000000))
%time for _ in range(10): my_arr2 = my_arr * 2
%time for _ in range(10): my_list2 = [x * 2 for x in my_list]<jupyter_output>CPU times: user 650 ms, sys: 156 ms, total: 806 ms
Wall time: 816 ms<jupyter_text>*** Generating Random Data ***<jupyter_code>data = np.random.randn(2,3) #Sample from Standard Normal Distribution
data
data * 10  # batch mathematical operations can be performed directly. Element-wise multiplication.
data + data # Element-wise addition.<jupyter_output><empty_output><jupyter_text>`ndarray` Multi-dimensional ArraysAn `ndarray` is a generic-multidimensional container for homogenous data. In simple terms, all the elements should be of same type. For example, all ints, all floats etc.Every array has a `shape` property, which is a tuple describing the dimensions.<jupyter_code>data.shape #2 rows, 3 columns<jupyter_output><empty_output><jupyter_text>The `dtype` property describes the data type of the array.<jupyter_code>data.dtype # in this case it is float64<jupyter_output><empty_output><jupyter_text>Creating `ndarray`To create an array the `array` function can be used. A simple list can be converted into `ndarray`.<jupyter_code>data1 = [1,2,3,4,5,6]

arr1 = np.array(data1)

print('Array: {}'.format(arr1))
print('Type : {}'.format(type(arr1)))<jupyter_output>Array: [1 2 3 4 5 6]
Type : <class 'numpy.ndarray'><jupyter_text>Nested sequences will be converted into multi-dimensional lists.<jupyter_code>data2 = [[1,2,3,4,5],[6,7,8,9,10]]

arr2 = np.array(data2)

print('Array: {}'.format(arr2))
print('Type: {}'.format(type(arr2)))
print('Type (alternative): {}'.format(arr2.dtype))
print('Dimension: {}'.format(arr2.ndim))
print('Shape: {}'.format(arr2.shape))<jupyter_output>Array: [[ 1  2  3  4  5]
 [ 6  7  8  9 10]]
Type: <class 'numpy.ndarray'>
Type (alternative): int64
Dimension: 2
Shape: (2, 5)<jupyter_text>In previous example, the nested lists were of same length. What will happen if it is not the case?<jupyter_code>data3 = [[1,2,3,4,5],[6,7,8,9]]

arr3 = np.array(data3)

print('Array: {}'.format(arr3))
print('Type: {}'.format(type(arr3)))
print('Type (alternative): {}'.format(arr3.dtype))
print('Dimension: {}'.format(arr3.ndim))
print('Shape: {}'.format(arr3.shape))<jupyter_output>Array: [list([1, 2, 3, 4, 5]) list([6, 7, 8, 9])]
Type: <class 'numpy.ndarray'>
Type (alternative): object
Dimension: 1
Shape: (2,)<jupyter_text>*** Notice, since the length of the lists is not the same. The numpy module created a single dimensional array of lists, instead of multi-dimensional array. ***> *** Important: *** While creating a numpy array from individual list remember this check. Additionally, you can create initialized arrays using helper methods like `zeros`, `ones` and `empty`.1. `zeros` : Initializes an array of requested shape with all zeros.2. `ones`  : Initializes an array of requested shape with all ones.3. `empty` : Initializes an array of requested shape with *** random values. ***  To create higher order arrays pass in tuple with shape specification.  Let's look at an illustration.<jupyter_code>z1 = np.zeros(10)
print('Array : {}'.format(z1))
print('Dimension: {}'.format(z1.ndim))
print('Shape: {}'.format(z1.shape))
z2 = np.zeros((2,3))
print('Array : {}'.format(z2))
print('Dimension: {}'.format(z2.ndim))
print('Shape: {}'.format(z2.shape))
o1 = np.ones((2,3))
print('Array : {}'.format(o1))
print('Dimension: {}'.format(o1.ndim))
print('Shape: {}'.format(o1.shape))
e1 = np.empty((2,3))
print('Array : {}'.format(e1))
print('Dimension: {}'.format(e1.ndim))
print('Shape: {}'.format(e1.shape))<jupyter_output>Array : [[1. 1. 1.]
 [1. 1. 1.]]
Dimension: 2
Shape: (2, 3)<jupyter_text>*** Note: *** It is not necessary that the values will be either 0, or 1 they can be random in case of `np.empty` (garbage values). `arange` is array-valued version of Python's `range` function.<jupyter_code>np.arange(10)<jupyter_output><empty_output><jupyter_text>*** Array Creation Functions ***| Function | Description ||----------|-------------|| array    | Convert input data (list, tuple, array, or other sequence to array) || asarray  | Comvert input to ndarray, but do not copy if it is already a ndarray || arange   | Built-in range function, returns ndarray instead of list.    || ones     | Array of 1's with given shape and size. || ones_like | Create ndarray of ones matching shape and size of input array || zeros    | Array of O's with given shape and size. || zeros_like | Create ndarray of zeros matching shape and size of input array || empty    | Array of random (garbage) data with given shape and size. || empty_like | Create ndarray of random (garbage) data matching shape and size of input array || full    | Array of given element data indicated by "fill value" of given shape and size. || full_like | Create ndarray of rgiven element data indicated by "fill value" matching shape and size of input array || eye, identity | Create a square NxN matrix ()  Data Types for ndarraysThe *data type* or *dtype* is a special object containing the information (or metadata about the data). Different types of Numpy data types supported are:int8, uint8, int16, uint16, int32, uint32, int64, uint64, float16, float32, float64, float128, complex64, complex128, complex256, bool, object, string_, unicode_Explicit, type-casting of an array can be done using the `astype` method.<jupyter_code>arr = np.array([1,2,3,4,5])
print('Initial DataType : {}'.format(arr.dtype))
float_arr = arr.astype(np.float64)
print('Converted DataType : {}'.format(float_arr.dtype))<jupyter_output>Initial DataType : int64
Converted DataType : float64<jupyter_text>The above example is converting int to floating point. *** Note: *** When casting from floating-point to int, the decimal part is truncated. This is a typical *down-casting* behaviour which is lossy in nature (irreversible). That is you loose the information represented by the decimal part, and it cannot be recovered by converting int to float again. The context of the problem to be solved is important.<jupyter_code>arr = np.array([3.7,2.8, -0.5, 0.5, 12.1, 11.4])
print('Initial DataType : {}'.format(arr.dtype))
int_arr = arr.astype(np.int)
print('Converted DataType: {}'.format(int_arr)) # Note there is no rounding.
numeric_string = np.array(['1.25', '-9.6', '42'], dtype=np.string_)
numeric_string.astype(np.float)<jupyter_output><empty_output><jupyter_text>*** Calling `astype` *always* creates a new array (a copy of the data), even if the new dtype is teh same as the old dtype.  Arithmetic with NumPy ArraysBatch operations can be done on data without writing any for loops. This is called as *vectorization*.  The arrays need to be of equal-size for these operations.<jupyter_code>arr = np.array([[1., 2., 3.], [4., 5., 6.]])
arr
arr * arr #Element-wise operations or vectorization
arr - arr<jupyter_output><empty_output><jupyter_text>*** Scalar Operations ***<jupyter_code>1/arr
arr ** 0.5<jupyter_output><empty_output><jupyter_text>Comparison between ndarrays result in ndarray ofsame shape and dimension.<jupyter_code>arr2 = np.array([[0., 4., 1.],[7., 2., 12.]])
arr2
arr2 > arr<jupyter_output><empty_output><jupyter_text>Basic Indexing and Slicing<jupyter_code>arr = np.arange(10)
 arr
arr[5] #6th element in the array
arr[5:8]
arr[5:8] = 12 # This will broadcast the value to the selection, and update the list.
arr # Notice the index positions 5:8 are updated in original array.<jupyter_output><empty_output><jupyter_text>*Note:* Array slices are *views*, the data is not copied and any modifications to the view will be reflected in the source array.<jupyter_code>arr_slice = arr[5:8]
arr_slice
arr_slice[1] = 12345
arr
arr_slice[:] = 64 # all indexes in slice will be assigned 64
arr<jupyter_output><empty_output><jupyter_text>If you want a copy of a slice of an ndarray instead of a view, you will need to explicitly copy the array - for example, `arr[5:8].copy()`.<jupyter_code>arr2d = np.array([[1,2,3],[4,5,6],[7,8,9]])
arr2d
arr2d[2]
arr2d[0][2]
arr2d[0, 2] # (rows, column) same as above and concise. Axis 0 = "Rows", Axis 1 = "Columns".<jupyter_output><empty_output><jupyter_text>Indexing with SlicesIn Numpy the indexing with slices works much like standard Python like slices.<jupyter_code>arr
arr[1:6]
arr2d
arr2d[:2]<jupyter_output><empty_output><jupyter_text>Multiple slices can be passed just like multiple indexes, and the slices would work on respective axis.<jupyter_code>arr2d[:2, ]
arr2d[:2, 1:]<jupyter_output><empty_output><jupyter_text>Assignements also work with slices.<jupyter_code>arr2d[:2, 1:] = 0
arr2d<jupyter_output><empty_output><jupyter_text>Boolean Indexing<jupyter_code>names = np.array(['Bob', 'Joe', 'Will', 'Bob', 'Will', 'Joe', 'Joe'])
data = np.random.randn(7,4)
names
data<jupyter_output><empty_output><jupyter_text>Suppose each name corresponds to a row in the data array and we wanted to select all the rows with corresponding name 'Bob'. Like arithmetic operations '==' comparison can be vectorized.<jupyter_code>names == 'Bob' #yields array of booleans<jupyter_output><empty_output><jupyter_text>This boolean array can be passed when indexing the array. *Note*: The boolean array *must* be of same length as the array axis it's indexing. Otherwise inconsistent results will be obtained which are hard to track.<jupyter_code>data[ names== 'Bob']<jupyter_output><empty_output><jupyter_text>Filtering rows and indexing is possible in one shot.<jupyter_code>data[names == 'Bob', 2:]
names != 'Bob'
data[~(names == 'Bob')] # Notice the ~ operator which is used for negation.<jupyter_output><empty_output><jupyter_text>Selecting multiple names using & and | operators. Note the standard python `and` and `or` operators do not work with boolean arrays.<jupyter_code>mask = (names == 'Bob') | (names == 'Will')
mask
data[mask]<jupyter_output><empty_output><jupyter_text>Setting boolean values.<jupyter_code>data[data < 0] = 0
data
data[ names != 'Joe'] = 7
data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1309332070108953, 'fraction_numerical': 0.02965419232591189, 'mean_word_length': 5.697969543147208, 'pattern_counts': {'":': 0, '<': 107, '<?xml version=': 0, '>': 108, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '502555', 'license': 'MIT', 'n_tokens_mistral': 3603, 'n_tokens_neox': 3423, 'n_words': 1192, 'path': 'numpy/01_numpy.ipynb', 'repo_name': 'satyasrikantmantha/python-essentials-for-data-science', 'size': '10555'}