<jupyter_start><jupyter_text>Lab 00 - A - Data Analysis In Python - First StepsMachine learning and data analysis (that is usually also a pre-step for learning) deal with data. Thererefore we need tools to manipulate it and extract the information we want. In the Python environment there are two very useful packages for this: [`numpy`](https://numpy.org/doc/1.19/) and [`pandas`](https://pandas.pydata.org/docs/reference/index.htmlapi).In this lab, we will take the first steps into using these packages and see some of their functionalities. These will be needed throughout the course. Both packages contain many useful functionalities, of which we will introduce only a few. To find out more about the other functionalities, the documentations, **Google** and **StackOverflow** are your best friends.<jupyter_code># Load commonly used imports (such as numpy and pandas) and several utils functions that 
# are used throughout different labs and code examples

import sys
sys.path.append("../")
from utils import *<jupyter_output><empty_output><jupyter_text>Numpy - The BasicsLet us start with numpy. This package supports vector, matrix and tensor operations over numerical (but not just) data. It is very comfortable and much faster than `for` loops and classic `list` manipulations.  Array CreationThere are multiple ways to create an array. We can create it from an existing list, load it from a file or generate a new array.<jupyter_code>array_1D = np.array([6, 2, 8, 4, 5, 10, 7, 143, 9, 10])
print(array_1D)
print(array_1D.shape)
array_2D = np.array(
    [[10, 20, 30, 40],
     [100, 200, 300, 400], 
     [1000, 2000, 3000, 4000]])
print(array_2D)
print(array_2D.shape)
array_3D = np.array(
    [[[10, 20, 30, 40], [100, 200, 300, 400], [1000, 2000, 3000, 4000]],
    [[11, 21, 31, 41], [101, 201, 301, 401], [1001, 2001, 3001, 4001]]])
print(array_3D)
print(array_3D.shape)<jupyter_output>[[[  10   20   30   40]
  [ 100  200  300  400]
  [1000 2000 3000 4000]]

 [[  11   21   31   41]
  [ 101  201  301  401]
  [1001 2001 3001 4001]]]
(2, 3, 4)<jupyter_text>Using `numpy`'s functions for creating new arrays requires specifying the shape of the desired output array. This is an n-array tuple specifying the sizes of the different dimensions. *   Specifying the shape `(3)` will create a 1D array with 3 entries.*   Specifying the shape `(10, 3)` will create a 2D matrix with 10 rows and 3 columns.*   Specifying the shape `(10, 28, 28)` will create a 3D matrix (a tensor) which we can think of in the following manner: it is an object holding 10 2D matrices of size 28x28.<jupyter_code># Initalize arrays with built-in numpy functions
zeros_3D = np.zeros((4, 5, 2)) # Create a 3D arrays of 0's
ones_2D = np.ones((4, 5)) # Create a 2D arrays of 1's
print(zeros_3D)
print(ones_2D)
np.arange(50) # Create the vector [0, 1, 2, 3, 4, .. , 48, 49]
# Create a vector of random integers from 5 to 50, with shape (2, 3)
np.random.randint(5, 50, (2, 3))<jupyter_output><empty_output><jupyter_text>There are many other functions such as `np.full`, `np.eye`, `np.random.uniform`, etc. Next, let us load an existsing dataset into a numpy array. This specific dataset represents images.<jupyter_code>img_array =\
    np.loadtxt(open("../datasets/MNIST_Images.csv", "rb"), delimiter=",").reshape(-1, 28, 28)
img_array.shape
px.imshow(img_array[0]).show()<jupyter_output><empty_output><jupyter_text>Array indexing and slicingA great strength of `numpy` is the ease in subsetting an array to retrieve only specific parts of it. We do so by indexing and slicing the arrays. For 1D arrays these operations are very similar to those over lists.  For arrays of higher dimensions, we use a comma to separate the slicing of each dimension. For example, accessing an element in the array `arr` in the first row and second column is done by: `arr[0, 1]` (recall indexing in python begins from zero). To select only the second column, over all rows write: `arr[:, 2]`<jupyter_code>array_1D = np.array([10, 11, 12, 13, 14, 15, 16])
# Select 1st element
print("Select 1st element") 
print(array_1D[1])

# Select all the elements from 1st to 4th element
print("\nSelect all the elements from 1st to 4th element")
print(array_1D[1:5])

# Select elements [1, 4]
print("\nSelect elements [1, 4]")
print(array_1D[[1, 4]])


array_2D = np.array([[1, 2, 3, 4],
                     [5, 6, 7, 8],
                     [9, 10, 11, 12],
                     [13, 14, 15, 16],
                     [17, 18, 19, 20]])
# Select 1st row, 2nd column  
print(array_2D[1, 2])


randint_2D = np.random.randint(5, 50, (10, 20))
print("\nPrint random array")
print(randint_2D)

# Select from 3rd row to 5th, all the columns
print("\nSelect from 3rd row to 5th, all the columns")
print(randint_2D[2:5, :])

# Select from 3rd row to 5th, columns 1 and 2
print("\nSelect from 3rd row to 5th, columns 1 and 2")
print(randint_2D[2:5, 1:3])

# Select from 3d row to 5th, columns 3, 5, 6, and  11
print("\nSelect from 3rd row to 5th, columns 3, 5, 6, and  11")
print(randint_2D[2:5, [3, 5, 6, 11]])<jupyter_output>Select 1st element
11

Select all the elements from 1st to 4th element
[11 12 13 14]

Select elements [1, 4]
[11 14]
7

Print random array
[[48 21 15 11 27 42 30 41 19  6 48 24 42 30 17 28  7 19 36 49]
 [23 34 18  9 30 40 49 12 12 35 39 41 31 45 11 24 24  8 49 17]
 [44 32 17 13 29 38  9 37 18 34 27 26 23 16 27 15 15 13 20  6]
 [16  9 33 31  5 40 26 14 31 11 38 39 14 24 28 38 47 12 25 18]
 [28 34 19 34 27 43 26 23 27 22 43 18 43  6 27 23 39 17 43  5]
 [23 23 44 31  9 37 20 43  5 36 41 25 38 38  7 35 32 46 15 13]
 [ 7 39 16 49  7  8 44 35 23 41  8 35 30 11 10 27 35 36 23 29]
 [38 45 41 42 31 48 22 49 13 33 10 41 15 25 37 42 21 48  6 40]
 [12 40 45 15 48 37 29 44 46 32 47 36 28 44 16 32  6 47 36 30]
 [17 49 34 44 27 36 26 37 21  7 44 26 33 29 16 44 23 41 42 31]]

Select from 3rd row to 5th, all the columns
[[44 32 17 13 29 38  9 37 18 34 27 26 23 16 27 15 15 13 20  6]
 [16  9 33 31  5 40 26 14 31 11 38 39 14 24 28 38 47 12 25 18]
 [28 34 19 34 27 43 26 23 27 22 43 18 43  6 27 23 39 17 43 [...]<jupyter_text>Matrix OperationsAnother strength of the `numpy` package is that all matrix operations you can think of (and even more) are already implemented. For example, element-wise addition of scalar, multiplication, powering up a matrix, log-transformations and much more.As `numpy` overloads the different mathematical operators, it is easy to write mathematical expressions over 2 (or more) vectors/matrices, such as summing or multiplying and also comparing their elements.<jupyter_code># Operations on 1 matrix
A = np.arange(1, 33).reshape(4, 8) # Create an array of numbers from 1 ot 32, and then make a 2d array of 4 rows to 8 columns
print(A)

print("\n\nA + 1:")
print(A + 1)
print("\n2 * A:")
print(2 * A)
print("\nA*A*A:")
print(np.power(A, 3))
print("\nlog(A):")
print(np.log(A))
print("\nA Transpose:")
print(A.transpose()) # also np.transpose(A) and A.T are valid syntaxes
print("\n A > 10:")
print(A > 10)

# Operations on 2 matrices
A = np.arange(6).reshape([2, 3])
B = np.random.randint(1, 10, (2, 3))
print("A")
print(A)
print("\n\nB")
print(B)

print("\nA+B:")
print(A + B) # Equivalent to np.add(a_array, b_array)
print("\n A * B (element-wise multiplication):")
print(np.multiply(A, B))
print("\n AB (matrix multiplication):")
print(A @ B.T) # Equivalent to np.dot(a_array, b_array)

print("\nA > B")
print(A > B)<jupyter_output>[[ 1  2  3  4  5  6  7  8]
 [ 9 10 11 12 13 14 15 16]
 [17 18 19 20 21 22 23 24]
 [25 26 27 28 29 30 31 32]]


A + 1:
[[ 2  3  4  5  6  7  8  9]
 [10 11 12 13 14 15 16 17]
 [18 19 20 21 22 23 24 25]
 [26 27 28 29 30 31 32 33]]

2 * A:
[[ 2  4  6  8 10 12 14 16]
 [18 20 22 24 26 28 30 32]
 [34 36 38 40 42 44 46 48]
 [50 52 54 56 58 60 62 64]]

A*A*A:
[[    1     8    27    64   125   216   343   512]
 [  729  1000  1331  1728  2197  2744  3375  4096]
 [ 4913  5832  6859  8000  9261 10648 12167 13824]
 [15625 17576 19683 21952 24389 27000 29791 32768]]

log(A):
[[0.         0.69314718 1.09861229 1.38629436 1.60943791 1.79175947
  1.94591015 2.07944154]
 [2.19722458 2.30258509 2.39789527 2.48490665 2.56494936 2.63905733
  2.7080502  2.77258872]
 [2.83321334 2.89037176 2.94443898 2.99573227 3.04452244 3.09104245
  3.13549422 3.17805383]
 [3.21887582 3.25809654 3.29583687 3.33220451 3.36729583 3.40119738
  3.4339872  3.4657359 ]]

A Transpose:
[[ 1  9 17 25]
 [ 2 10 18 26]
 [ 3 11 19 27]
 [[...]<jupyter_text>For many `numpy` operations we can specify the `axis` over which to perform the operation:<jupyter_code># Concatenate matrices
print("\nConcatenate 2 arrays by the rows:")
print(np.concatenate((A, B), axis = 0)) # Concatenate rows - look at the shape
print("\nConcatenate 2 arrays by the columns:")
print(np.concatenate((A, B), axis = 1)) # Concatenate columns - look at the shape<jupyter_output>Concatenate 2 arrays by the rows:
[[0 1 2]
 [3 4 5]
 [1 6 7]
 [3 3 9]]

Concatenate 2 arrays by the columns:
[[0 1 2 1 6 7]
 [3 4 5 3 3 9]]<jupyter_text>Basic StatisticsYou can easily calculate a lot of basic statistics from an array, such as the sum, mean, variance, maximum, argmax, etc. All of these can be retrieved either over the entire array or over rows/columns.For each of these functions, you can get the statistic for: - the whole array: `np.stat(arr)`- by row: `np.stat(arr, axis = 1)`- by column: `np.stat(arr, axis = 0)`<jupyter_code># Basic statistics of a matrix
A = np.arange(6).reshape([2, 3])
print("A")
print(A)

print("\n\nSum of the array")
print(np.sum(A)) # Sum all the matrix (return a scalar)
print("\nSum of the array by column")
print(np.sum(A, axis = 0)) # Sum by column (return a vector)
print("\nSum of the array by row")
print(np.sum(A, axis = 1)) # Sum by row (return a vector)

print("\nMax of each row")
print(np.max(A, axis = 1)) # Max by column (return a vector)

print("\nMax of each column")
print(np.max(A, axis = 0)) # Max by row (return a vector)

print("\nAverage of all the array")
print(np.mean(A))<jupyter_output>A
[[0 1 2]
 [3 4 5]]


Sum of the array
15

Sum of the array by column
[3 5 7]

Sum of the array by row
[ 3 12]

Max of each row
[2 5]

Max of each column
[3 4 5]

Average of all the array
2.5<jupyter_text>Sampling From Distributions`numpy` provides a broad set of distributions to sample from. We will cover this in more depth in lab 2.<jupyter_code>A = np.random.randint(1, 15, size = 20)
print("A")
print(A)

print("\n\nThe values that appear in the array")
print(np.unique(A))
print("\nHow many values are between [0, 5), [5, 10), [10, 15]")
np.histogram(A, bins = [0, 5, 10, 15])<jupyter_output>A
[ 3  7 10  8  4 10 12  5 14 14  8 13 13 11  8  3  8  5 12  8]


The values that appear in the array
[ 3  4  5  7  8 10 11 12 13 14]

How many values are between [0, 5), [5, 10), [10, 15]<jupyter_text>Linear AlgebraOne of the most important mathematical fields in machine learning is linear algebra. You can perform many of these operations using `numpy`. You can calculate the eigenvectors of a matrix, or its inverse, the rank of the matrix and so on.<jupyter_code>A = np.array([[1., 2.], [3., 4.]])
print("A")
print(A)
print("\n\nInverse of A")
print(np.linalg.inv(A))

B = np.diag((1, 2, 3))
print("\n\nB")
print(B)

eigvalues, eigvectors = np.linalg.eig(B)
print("\neigenvalues, eigenvectors")
print(eigvalues, eigvectors)

print("\nRank of the matrix")
print(np.linalg.matrix_rank(B))<jupyter_output>A
[[1. 2.]
 [3. 4.]]


Inverse of A
[[-2.   1. ]
 [ 1.5 -0.5]]


B
[[1 0 0]
 [0 2 0]
 [0 0 3]]

eigenvalues, eigenvectors
[1. 2. 3.] [[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]

Rank of the matrix
3<jupyter_text>ReshapingYou can change the shape of your array, with transpose, flattening, reshape, adding a new axis (see `np.newaxis`)<jupyter_code>B = np.array([[1, 2, 3], [4, 5, 6]])

print("\n\nB")
print(B)

print("\nFlatten matrix")
print(np.ravel(B))


print("\nTranspose matrix")
print(B.transpose())<jupyter_output>B
[[1 2 3]
 [4 5 6]]

Flatten matrix
[1 2 3 4 5 6]

Transpose matrix
[[1 4]
 [2 5]
 [3 6]]<jupyter_text>SortingYou can sort the matrix by row or by column<jupyter_code>B = np.array([[3, 6, 1, 4, 10], [5, 1, 8, 3, 65]])

print("B")
print(B)

print("\n\nSort by row")
print(np.sort(B, axis = 1))<jupyter_output>B
[[ 3  6  1  4 10]
 [ 5  1  8  3 65]]


Sort by row
[[ 1  3  4  6 10]
 [ 1  3  5  8 65]]<jupyter_text>Indexing By Condition<jupyter_code>A = np.random.randint(1, 30, (5, 4))
print("A")
print(A)

print("\nGet the numbers that are greater than 5:")
print(A[A > 5])

print("\nGet the numbers that are divisible by 4:")
print(np.extract(np.mod(A, 4)==0, A))<jupyter_output>A
[[ 6 21 25 27]
 [ 5 17 19 14]
 [19 29 17 16]
 [ 1  2 17  7]
 [22 27  7 22]]

Get the numbers that are greater than 5:
[ 6 21 25 27 17 19 14 19 29 17 16 17  7 22 27  7 22]

Get the numbers that are divisible by 4:
[16]<jupyter_text>Let's practice!To get you a bit more accustomed to `numpy` you are encouraged to solve the following challenges. If you choose to not solve the following challenges, be sure to understand the solutions. Do not use loops or list comprehensions. Write a program to create a `7x10` matrix that has `0` and `1` staggered:``` 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1```Hint: use slice operations on different axes.<jupyter_code>staggered = np.zeros((7, 10))
staggered[::2, 1::2] = 1
staggered[1::2, ::2] = 1

print(staggered)<jupyter_output>[[0. 1. 0. 1. 0. 1. 0. 1. 0. 1.]
 [1. 0. 1. 0. 1. 0. 1. 0. 1. 0.]
 [0. 1. 0. 1. 0. 1. 0. 1. 0. 1.]
 [1. 0. 1. 0. 1. 0. 1. 0. 1. 0.]
 [0. 1. 0. 1. 0. 1. 0. 1. 0. 1.]
 [1. 0. 1. 0. 1. 0. 1. 0. 1. 0.]
 [0. 1. 0. 1. 0. 1. 0. 1. 0. 1.]]<jupyter_text>Calculate the volume of a cylinder with the following diameters and lengths:<jupyter_code>diameters = np.array([1, 3, 5, 2, 4])
lengths = np.array([10, 20, 3, 10, 5])
print(np.array(np.power(diameters/2, 2)*lengths*np.pi))<jupyter_output>[  7.85398163 141.37166941  58.90486225  31.41592654  62.83185307]<jupyter_text>Write a function that receives 2 vectors and returns their cartesian product:```def create_cartesian_product(vec1, vec2):  pass```<jupyter_code>def cartesian_product(vec1, vec2):
    # np.repeat([1, 2, 3], 4) -> [1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3]
    # np.tile([1, 2, 3], 4)   -> [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
    return np.transpose(np.array([np.repeat(vec1, len(vec2)), np.tile(vec2, len(vec1))]))

print(cartesian_product([1, 2, 3], [4, 5, 6, 7]))<jupyter_output>[[1 4]
 [1 5]
 [1 6]
 [1 7]
 [2 4]
 [2 5]
 [2 6]
 [2 7]
 [3 4]
 [3 5]
 [3 6]
 [3 7]]<jupyter_text>Given an array `a` and a number `n`, find the closest number to `n` in `a`:```def find_closest(a, n):  pass```<jupyter_code>def find_closest(a, n):
  a = np.array(a)
  return a[np.argmin(np.abs(a - n))]
                     
print(find_closest([1, 24, 12, 13, 14], 10))<jupyter_output>12<jupyter_text>Check if the sudoku grid is valid:*   Check that each row contains all the numbers from 1 to 9*   Check that each column contains all the numbers from 1 to 9*   Check that each of the 9 non-overlapping `3x3` blocks composing grid contain 1 to 9You can assume it contains only integers and that the shape of the array is `9x9````def check_sudoku(grid):  pass```<jupyter_code>def is_1_to_9(array_):
  # Return True if the array contains all the numbers from 1 to 9, False otherwise
    return np.all(np.sort(array_, axis = None) == np.arange(1, 10))

def check_sudoku(grid):
    def check_block(coords_block):
        return is_1_to_9(grid[coords_block[0]*3 : coords_block[0]*3+3, 
                              coords_block[1]*3 : coords_block[1]*3+3])

    grid = np.array(grid)

    # Check that the grid contains only 1 to 9
    if (not is_1_to_9(np.unique(grid))):
        return False

    # Check that each line/column contains 1 to 9:
    # Sort each column. We expect it to be
    # [[1, 1, 1, 1, 1, 1, 1, 1, 1], [2, 2, 2, 2, 2, 2, 2, 2] ... [9, 9, 9, 9, 9, 9, 9, 9, 9]]
    # Thus we expect the sums of the rows: 1*9, 2*9....
    if  np.any(np.sum(np.sort(grid, axis = 0), axis = 1) != np.arange(1, 10)*9):
        return False

    # Make the same for the columns
    if np.any(np.sum(np.sort(grid.transpose(), axis = 0), axis = 1) != np.arange(1, 10)*9):
        return False


    # 0,0 0,0 0,0    0,1 0,1 0,1    0,2 0,2 0,2
    # 0,0 0,0 0,0    0,1 0,1 0,1    0,2 0,2 0,2
    # 0,0 0,0 0,0    0,1 0,1 0,1    0,2 0,2 0,2
    #
    #
    # 1,0 1,0 1,0    1,1 1,1 1,1    1,2 1,2 1,2
    # 1,0 1,0 1,0    1,1 1,1 1,1    1,2 1,2 1,2
    # 1,0 1,0 1,0    1,1 1,1 1,1    1,2 1,2 1,2
    #
    #
    # 2,0 2,0 2,0    2,1 2,1 2,1    2,2 2,2 2,2
    # 2,0 2,0 2,0    2,1 2,1 2,1    2,2 2,2 2,2
    # 2,0 2,0 2,0    2,1 2,1 2,1    2,2 2,2 2,2


    # For each block of 9, check if it contains all the numbers 1 to 9
    blocks_are_valid = np.apply_along_axis(check_block, 1, cartesian_product([0, 1, 2], [0, 1, 2]))
    return np.all(blocks_are_valid)<jupyter_output><empty_output><jupyter_text>Given a matrix, check if some row is a scalar multplication of another```def check_dependencies(matrix_):  pass```<jupyter_code>def check_dependencies(matrix_):
    def rows_are_dependent(indices):
        if indices[0] == indices[1]:
            return False
        return np.unique(matrix_[indices[0],] / matrix_[indices[1],]).shape[0] == 1

    return np.any(np.apply_along_axis(rows_are_dependent, 1,
                                      cartesian_product(np.arange(matrix_.shape[0]), np.arange(matrix_.shape[0]))))<jupyter_output><empty_output><jupyter_text>Write a function that gets a 1D array and check if there is no local extrema point in addition to the global one.```def have_an_extrema(array):  pass```<jupyter_code>def have_a_maxima(array):
    argmax_arr = np.argmax(array[1:-1]) + 1
    before_max_neg = np.all(array[:argmax_arr - 1] - array[1:argmax_arr] < 0)
    after_max_neg = np.all(array[argmax_arr:-1] - array[argmax_arr + 1:] > 0)
    return before_max_neg and after_max_neg

def have_an_extrema(array):
    # Check if it is a monotonic series
    if np.unique(array[:-1] - array[1:]).shape[0] == 1: return True

    # If there is a minimum, we can look for a maximum in the negated array
    return have_a_maxima(array) or have_a_maxima(-array)<jupyter_output><empty_output><jupyter_text>Note that instead of `array[:argmax_arr - 1] - array[1:argmax_arr]`, you could use the `np.diff` function  PandasUntil now, we have only looked at numerical data. But in real-world problems, we also have textual and categorical data. To manipulate this type of data, we will use the `pandas` library. One of the basic data structures of `pandas` is called a `DataFrame`. Generally, in a `DataFrame`, each row is a different sample and each column is a feature.For example, each row can represent a student, with columns of the ID, birthday, and gender of the student. `pandas` has a lot of possibilities of which we are going to introduce a very small subset.  Array CreationIn addition to creating an array from lists or randomly generated data frames, we are going to use an existing dataset of house prices (you will get back to this dataset in excercise 2).<jupyter_code># Load https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data?select=train.csv
df = pd.read_csv('../datasets/house_train.csv', index_col=0)
df.head()
print("\nRows Names")
print(df.index)
print("\nColumns Names")
print(df.columns)
print("\nDf train shape")
print(df.shape)<jupyter_output>Rows Names
Int64Index([   1,    2,    3,    4,    5,    6,    7,    8,    9,   10,
            ...
            1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459, 1460],
           dtype='int64', name='Id', length=1460)

Columns Names
Index(['MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'Street', 'Alley',
       'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope',
       'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle',
       'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', 'RoofStyle',
       'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'MasVnrArea',
       'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', 'BsmtCond',
       'BsmtExposure', 'BsmtFinType1', 'BsmtFinSF1', 'BsmtFinType2',
       'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'Heating', 'HeatingQC',
       'CentralAir', 'Electrical', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF',
       'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath',
       'Bedroo[...]<jupyter_text>Indexing And SlicingJust like when using `numpy` you can select a subset of rows and columns. You can do it using indices or using names of the rows and columns. You can easily add a new column based on existing ones.<jupyter_code>print("\ndf[['GrLivArea', 'SalePrice', 'BedroomAbvGr']]")
print(df[['GrLivArea', 'SalePrice', 'BedroomAbvGr']]) # Select  GrLivArea columns SalePrice BedroomAbvGr

print("\ndf.loc[3:10,['GrLivArea', 'SalePrice', 'BedroomAbvGr'] ]")
print(df.loc[3:10,['GrLivArea', 'SalePrice', 'BedroomAbvGr'] ])

print("\ndf.iloc[[3, 4, 5]]")
print(df.iloc[[3, 4, 5]])
print("\ndf.iloc[3:10,[6, 7, 8]]")
print(df.iloc[3:10,[6, 7, 8]])
individual_df = pd.DataFrame(np.array([np.random.randint(2000000, 3000000,50), np.random.uniform(1.50, 1.70, size = 50), np.random.uniform(45, 90, size = 50)]).transpose(), columns=['ID','Height','Weight'])

individual_df["BMI"] = individual_df["Weight"] / individual_df["Height"].pow(2)
print("\nIndividual DF")
print(individual_df.head())<jupyter_output>Individual DF
          ID    Height     Weight        BMI
0  2432787.0  1.665154  76.526591  27.599659
1  2731829.0  1.586409  62.971467  25.021518
2  2080881.0  1.635798  59.939918  22.400441
3  2213397.0  1.601308  88.755578  34.613543
4  2162701.0  1.665256  48.703103  17.562829<jupyter_text>Basic Statistics`pandas` provides different statistical functions over `DataFrame`s.<jupyter_code>print("\nMedian of the SalePrice column")
print(df.SalePrice.median())

print("\nSelecting rows by condition")
median_price = df.SalePrice.median()
print(df[df.SalePrice > median_price].head())<jupyter_output>Median of the SalePrice column
163000.0

Selecting rows by condition
    MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \
Id                                                                    
1           60       RL         65.0     8450   Pave   NaN      Reg   
2           20       RL         80.0     9600   Pave   NaN      Reg   
3           60       RL         68.0    11250   Pave   NaN      IR1   
5           60       RL         84.0    14260   Pave   NaN      IR1   
7           20       RL         75.0    10084   Pave   NaN      Reg   

   LandContour Utilities LotConfig  ... PoolArea PoolQC Fence MiscFeature  \
Id                                  ...                                     
1          Lvl    AllPub    Inside  ...        0    NaN   NaN         NaN   
2          Lvl    AllPub       FR2  ...        0    NaN   NaN         NaN   
3          Lvl    AllPub    Inside  ...        0    NaN   NaN         NaN   
5          Lvl    AllPub       FR2  ...        [...]<jupyter_text>Group-byWhen working with data that contains also a categorical feature, we are often interested in performing some kind of calculation over all rows containing the same categorical value. For example, given a data frame of student grades for different courses, we can calculate the students' average grade.<jupyter_code>students_df = pd.DataFrame(np.array([np.random.choice(["Zohar", "Shelly", "Omer", "Avi"],50), np.random.choice(["Linearit", "Intro", "Infi", "Probabilistic"], 50), np.random.randint(80, 101, 50)]).transpose(), columns=['Name','Course','Grade'])
students_df["Grade"] = students_df["Grade"].astype(int)

print("\n\nStudents df")
print(students_df.head())

print("\n\nCalculate average by student and by course")
print(students_df.groupby(['Name', 'Course']).mean().reset_index())<jupyter_output>Students df
    Name         Course  Grade
0   Omer           Infi     85
1  Zohar  Probabilistic     91
2    Avi  Probabilistic     94
3   Omer          Intro     95
4    Avi          Intro     83


Calculate average by student and by course
      Name         Course      Grade
0      Avi           Infi  91.400000
1      Avi          Intro  89.666667
2      Avi       Linearit  84.000000
3      Avi  Probabilistic  91.000000
4     Omer           Infi  85.000000
5     Omer          Intro  89.875000
6     Omer       Linearit  92.000000
7     Omer  Probabilistic  88.666667
8   Shelly          Intro  87.750000
9   Shelly       Linearit  95.250000
10  Shelly  Probabilistic  84.000000
11   Zohar           Infi  92.000000
12   Zohar          Intro  92.000000
13   Zohar       Linearit  86.500000
14   Zohar  Probabilistic  90.000000<jupyter_text>SortingAs in `numpy` you can sort data frame based on a column<jupyter_code>students_df.sort_values(by='Grade').head()<jupyter_output><empty_output><jupyter_text>Executing Functions By ColumnsIn `pandas`, you can select columns and apply functions to them. You also can apply functions by elements.<jupyter_code>df = pd.DataFrame(np.random.randint(1, 10, (5,3)),columns=['col1','col2','col3'])

print("\n\ndf")
print(df)
print("\nCalculate the difference between the min and the max of each column")
print(df.apply(lambda x: x.max() - x.min()))

# Apply by element
print("\nMultiply elements by 100")
print(df.applymap(lambda x:x*100))

print("\n\nIterate over the columns")
print("\n-----------------------")
for key,value in df.iteritems():
   print(key,value)

print("\n\nIterate over the rows")
print("\n-----------------------")
for row_idx,row in df.iterrows():
   print(row_idx,row)<jupyter_output>df
   col1  col2  col3
0     1     7     6
1     1     3     4
2     9     2     5
3     7     6     2
4     3     7     6

Calculate the difference between the min and the max of each column
col1    8
col2    5
col3    4
dtype: int32

Multiply elements by 100
   col1  col2  col3
0   100   700   600
1   100   300   400
2   900   200   500
3   700   600   200
4   300   700   600


Iterate over the columns

-----------------------
col1 0    1
1    1
2    9
3    7
4    3
Name: col1, dtype: int32
col2 0    7
1    3
2    2
3    6
4    7
Name: col2, dtype: int32
col3 0    6
1    4
2    5
3    2
4    6
Name: col3, dtype: int32


Iterate over the rows

-----------------------
0 col1    1
col2    7
col3    6
Name: 0, dtype: int32
1 col1    1
col2    3
col3    4
Name: 1, dtype: int32
2 col1    9
col2    2
col3    5
Name: 2, dtype: int32
3 col1    7
col2    6
col3    2
Name: 3, dtype: int32
4 col1    3
col2    7
col3    6
Name: 4, dtype: int32<jupyter_text>Merging Data Frames<jupyter_code># Concatenate data frames
import pandas as pd

# Create 2 data frames
ids_df1 = pd.DataFrame({
        'ID': ['336097897', '32109678', '25976389', '32438509', '36790307'],
         'name': ['Amos', 'Eran', 'Sapir', 'Amichai', 'Hadar'], 
        'gender': ["M", "M", "F", "M", "F"]})

ids_df2 = pd.DataFrame({
        'ID': ['21370565', '34256798', '3908412', '326780578'],
        'name': ['Matan', 'Gabriel', 'Anael', 'Liora'], 
        'gender': ["M", "M", "F", "F"]})

print("\n\ndf1")
print(ids_df1)
print("\ndf2")
print(ids_df2)
print("\n\nJoin the two dataframes along rows:")
concatenate_data = pd.concat([ids_df1, ids_df2])
print(concatenate_data)
birthdates_id = pd.DataFrame({
        'ID': ['336097897', '32109678', '25976389', '32438509', '36790307', '21370565', '34256798', '3908412', '326780578'],
        'birth_year': [1995, 1996, 1993, 1994, 1997, 1991, 1994, 1992, 1996]})

print("\nNow join the result_data and df_exam_data along ID:")
pd.merge(concatenate_data, birthdates_id, on='ID')<jupyter_output>Now join the result_data and df_exam_data along ID:<jupyter_text>Let's Practice! Let us create a table of flight companies' flights. Each row will represent a single flight and will have 3 features: city of departure, city of destination and price. Implement a function `create_flight_df` that recieves a collection of cities and creates a dataset of randomly selected flights and a price in the range of 100-400.```    def create_flight_df(cities_poss, nrows = 100):        pass```The output data frame must not have more than a single record for any pair of cities. There are no flights from a city to itself.<jupyter_code>import numpy as np
import pandas as pd

def create_flight_df(cities, nrows = 20):
    df = pd.DataFrame([], columns=["Departure", "Destination", "Price"])
    
    while df.shape[0] < nrows:
        dep, dest = np.random.choice(cities, size=2, replace=False)
        price = np.random.randint(100, 400)
        
        if not ((df["Departure"] == dep) & (df["Destination"] == dest)).any():
            df = df.append({"Departure": dep, "Destination": dest, "Price": price}, ignore_index=True)
    return df

cities = ["Beijing", "Moscow", "New-York", "Tokyo", "Paris", "Cairo", "Santiago", "Lima", "Kinshasa", "Singapore", 
          "New-Delhi", "London", "Ankara", "Nairobi", "Ottawa", "Seoul", "Tehran", "Guatemala", "Caracas", "Vienna"]

flights = create_flight_df(cities)
flights.head()<jupyter_output>C:\Users\Matanel\AppData\Local\Temp/ipykernel_6420/3998128772.py:12: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

C:\Users\Matanel\AppData\Local\Temp/ipykernel_6420/3998128772.py:12: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

C:\Users\Matanel\AppData\Local\Temp/ipykernel_6420/3998128772.py:12: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

C:\Users\Matanel\AppData\Local\Temp/ipykernel_6420/3998128772.py:12: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

C:\Users\Matanel\AppData\Local\Temp/ipykernel_6420/3998128772.py:12: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pand[...]<jupyter_text>As there are pairs of cities with no direct flight between them, let us find the pairs of cities that have a single connection flight between them and calculate the total price of the flgihts. To do so merge the two data frames. This operation is often referred to as "joining" with the options of inner, outer, left, right and cross joining. For more about merging `pandas` data frames read the [documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.htmldatabase-style-dataframe-or-named-series-joining-merging).<jupyter_code>df = pd.merge(flights, flights, left_on=["Destination"], right_on=["Departure"], how="inner")
df = df[df.Departure_x != df.Destination_y]
df["Total_Price"] = df["Price_x"] + df["Price_y"]

df.head()<jupyter_output><empty_output><jupyter_text>Create a data frame with all flights of no connection and single connection.<jupyter_code>flights = flights.append(df[["Departure_x", "Destination_y", "Total_Price"]]\
                         .rename(columns={"Departure_x":"Departure", "Destination_y":"Destination", "Total_Price":"Price"}))
flights<jupyter_output>C:\Users\Matanel\AppData\Local\Temp/ipykernel_6420/1123988012.py:1: FutureWarning:

The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.<jupyter_text>Since now we might have more than one way to flight between each pair of cities, let us find the cheapest flight option, with one connection, between two cities.<jupyter_code>min_by_group = df.groupby(["Departure_x", "Destination_y"], as_index=False)["Total_Price"].min()
min_by_group<jupyter_output><empty_output><jupyter_text>And if we want to know on average what is the most expensive city to fly to then:<jupyter_code>mean_by_dest = min_by_group.groupby("Destination_y")["Total_Price"].mean()
expensive_city = mean_by_dest.idxmax()
expensive_city<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12750335864029744, 'fraction_numerical': 0.10666416721342206, 'mean_word_length': 3.3483222388262464, 'pattern_counts': {'":': 6, '<': 109, '<?xml version=': 0, '>': 116, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '37410', 'license': 'MIT', 'n_tokens_mistral': 14010, 'n_tokens_neox': 11492, 'n_words': 4146, 'path': 'lab/Lab 00 - A - Data Analysis In Python - First Steps.ipynb', 'repo_name': 'MatanelP/IML.HUJI', 'size': '32007'}