<jupyter_start><jupyter_text>Python: Handling missing values **Goal**: Clean and organise your data! Table of Contents1&nbsp;&nbsp;Introduction to dataset2&nbsp;&nbsp;Find missing values3&nbsp;&nbsp;Problems with missing values3.1&nbsp;&nbsp;Training4&nbsp;&nbsp;Introduction to pivot table4.1&nbsp;&nbsp;Training5&nbsp;&nbsp;Remove missing values6&nbsp;&nbsp;Iloc to access rows7&nbsp;&nbsp;Column indexes8&nbsp;&nbsp;Re-index rows in a dataframe9&nbsp;&nbsp;Apply functions to a dataframe9.1&nbsp;&nbsp;Training10&nbsp;&nbsp;Practice: compute the percentage of survival by classe group  Introduction to dataset In this chapter, we will clean and analyze the data of a dataset containing the ``survivors of titanic``.<jupyter_code>import pandas as pd
titanic_survival = pd.read_csv("titanic_survival.csv")
titanic_survival.head()<jupyter_output><empty_output><jupyter_text>We will proceed to a brief presentation of some columns. The column ``pclass`` is the class of the passenger cabin which goes from 1 to 3 knowing that 1 is the highest class. The column ``survived`` represents the survival of a passenger. It takes the value 1 if the passenger survived and 0 otherwise. The ``fare`` column represents the amount paid by the passenger for the boarding ticket. The column ``embarked`` represents the place of boarding of the passenger, it takes 3 values: C, Q and S. In this dataset, it should be noted that many columns like ``age`` and ``sex`` have missing values. These ``missing values`` can cause numerical errors in our calculations. So we have to handle with them before starting the analyses. So it is important to learn how to handle with missing values and this is what we will do throughout this chapter.  Find missing values In this section, we will proceed to the ``discovery of missing values``. As a reminder, there are several types of missing values. There are values of type ``None`` which indicates no value, there is also the value ``Nan`` which means ``not a number`` which indicates a missing value. In general, we can consider None and Nan as ``null values``. In pandas, there is a function that allows to see which values are null or none, it is the ``isnull()`` function.<jupyter_code># example with sex column
sex = titanic_survival["sex"]
sex_is_null = pd.isnull(sex)
sex_is_null
sex_null = sex[sex_is_null]
sex_null
# example with age column
age_null = titanic_survival["age"][pd.isnull(titanic_survival.age)]
age_null<jupyter_output><empty_output><jupyter_text>We can clearly see that our age column has 264 missing values that we can recheck using the ``len()`` function.<jupyter_code>count_age_null = len(age_null)
count_age_null<jupyter_output><empty_output><jupyter_text>Problems with missing values We have seen previously that there are columns with missing values. In this section, we will show what ``problems`` these missing values ``cause``.<jupyter_code># example of problem
mean_age = sum(titanic_survival["age"]) / len(titanic_survival["age"])
mean_age<jupyter_output><empty_output><jupyter_text>This example above simply illustrates that a computation containing a missing value will return a missing value. So it is necessary to ``filter`` out the missing values before proceeding with the computations.<jupyter_code># filter the nan values
age_not_null = titanic_survival["age"][pd.isnull(
    titanic_survival["age"]) == False]
age_not_null
# compute the mean of age column without nan values
mean_age = sum(age_not_null) / len(age_not_null)
mean_age<jupyter_output><empty_output><jupyter_text>For ``information``, there is a method of pandas to compute more simply an average directly on a column. This function which the ``mean()`` method ignores the missing values and computes directly the average of a numerical series.<jupyter_code># example with age column
mean_age = titanic_survival["age"].mean()
mean_age<jupyter_output><empty_output><jupyter_text>Training In this practice, we will try to answer the following questions:* create an empty dictionary that we will name fares_by_class* create the list passenger_classes which contains the elements [1,2,3]* use a for loop to browse the passenger_classes list:    * select just the rows of titanic_survival for which the column pclass is equal to the temporary variable (the iterator) of the for loop, i.e. corresponding to the class number (1, 2 or 3)    * select only the fare column for this subset of rows (corresponding to the class)    * use the series.mean() method to calculate the average of this subset    * add this calculated average of the class to the fares_by_class dictionary with the class number as key (and thus the average fare as value)* once the loop is completed, the dictionary fares_by_class should have 1,2 and 3 as keys with the corresponding average values* display the result<jupyter_code>fares_by_class = {}
passenger_classes = [1, 2, 3]

for this_class in passenger_classes:
    pclass_rows = titanic_survival[titanic_survival["pclass"] == this_class]
    pclass_fares = pclass_rows["fare"]
    fare_for_class = pclass_fares.mean()
    fares_by_class[this_class] = fare_for_class
fares_by_class<jupyter_output><empty_output><jupyter_text>Introduction to pivot table In this section we will look at pivot tables which are a way of creating a subset from a column and performing computations, for example calculating an average, etc. The idea of pivot tables is to group and then apply a function. The ``pivot_table()`` method of pandas allows to perform these operations. This following example shows how to perform our previous task just by using the ``pivot_table()`` method.<jupyter_code>import numpy as np
fares_by_pclass = titanic_survival.pivot_table(index="pclass",
                                               values="fare",
                                               aggfunc=np.mean)
fares_by_pclass<jupyter_output><empty_output><jupyter_text>The first parameter ``index`` indicates the column you want to group. The second parameter ``values`` indicates the column on which we want to apply a function (sum, average, etc). And the last parameter ``aggfunc`` indicates the function we want to apply on the ``values`` parameter. With this same method ``pivot_table()``, we can also perform calculations on several columns.  Training In this practice, we will try to answer the following questions:* make a table pivot that calculates the total money collected ("fare") and the total number of survivors ("survived") for each embarked port ("embarked") using the numpy.sum function* assign the result to the variable port_stats* display the result<jupyter_code>port_stats = titanic_survival.pivot_table(index="embarked",
                                          values=["fare", "survived"],
                                          aggfunc=np.sum)
port_stats<jupyter_output><empty_output><jupyter_text>Remove missing values To ``remove`` missing values directly from a dataframe, we use the ``dropna()`` method. This method allows to ``delete`` all rows or columns with at least one missing value.<jupyter_code>drop_na_rows = titanic_survival.dropna(axis=0)
drop_na_rows
drop_na_columns = titanic_survival.dropna(axis=1)
drop_na_columns<jupyter_output><empty_output><jupyter_text>These examples above show that all rows or columns in our dataset have at least one missing value. It is also possible to remove missing values by specifying a ``subset of variables`` using the ``subset`` parameter.<jupyter_code>titanic_survival.shape
name_drop_na_rows = titanic_survival.dropna(axis=0, subset=["name"])
name_drop_na_rows.shape<jupyter_output><empty_output><jupyter_text>This last example says that there is a row that has at least one missing value for the ``name`` column.  Iloc to access rows In this section we will see the ``iloc`` method for accessing rows. More advanced than the ``loc`` method, the ``iloc`` method allows to display the elements according to the position where it is located. This method also allows to do other tasks much more advanced than the ``loc`` method.<jupyter_code>unordered_titanic_survival = titanic_survival.sort_values(
    by="age", ascending=False)
unordered_titanic_survival.head()
unordered_titanic_survival.loc[0]
unordered_titanic_survival.iloc[0]
unordered_titanic_survival.loc[0:5]
unordered_titanic_survival.iloc[0:5]<jupyter_output><empty_output><jupyter_text>The above example clearly shows that the ``iloc`` method does not take into account the order of the elements on the dataframe but only focuses on the positions of the elements in the dataframe. So when you are slicing a dataframe, you should use the ``iloc`` method because slicing using the ``loc`` method assumes that the index of our dataframe is sorted in ascending order. And on the other hand, if you want to access elements, you should use the ``loc`` method if you are accessing by index number, otherwise you should use the ``iloc`` method if you are accessing by the position of the element in the dataframe.  Column indexes As we have seen in the previous chapters, the indexing of columns is done with the ``loc`` and ``iloc`` methods. With the ``loc`` method, we use the name of the column, and for the ``iloc`` method, we use an integer that corresponds to the positions of the column in the dataframe.<jupyter_code># example
unordered_titanic_survival.head()
unordered_titanic_survival.iloc[0, 2]
unordered_titanic_survival.loc[14, "name"]<jupyter_output><empty_output><jupyter_text>Re-index rows in a dataframe In this section, we will see how to re-index the rows of a dataframe. We have seen that sorting changes the order of a dataframe but keeps the indexes of each row. However, it can sometimes be useful to re-index a dataframe by starting the indexes at 0. To do this, we use the ``reset_index()`` method of pandas.<jupyter_code># example
unordered_titanic_survival.reset_index().head()<jupyter_output><empty_output><jupyter_text>In the example above, we know that our dataframe re-indexed has kept the old index, to remove it, we can use the ``drop`` parameter which will take the value ``True``.<jupyter_code>unordered_titanic_survival.reset_index(drop=True).head()<jupyter_output><empty_output><jupyter_text>Apply functions to a dataframe In this section, we will see how to apply functions on a dataframe. To do so, we will learn the ``apply()`` method. By default, this method applies a function on each column or row of the dataframe.<jupyter_code># example 1
fare = titanic_survival["fare"]
fare
squared_fare = fare.apply(lambda x: x**2)
squared_fare<jupyter_output><empty_output><jupyter_text>By changing the ``axis (0 by default)`` parameter of the ``apply()`` method, we can also apply a function on the rows of the datadrame by setting the parameter to ``1``.<jupyter_code># example 2
def is_minor(row):
    if row["age"] < 18:
        return True
    else:
        return False
minors = titanic_survival.apply(is_minor, axis=1)
minors
# example 3
def which_class(row):

    pclass = row["pclass"]

    if pd.isnull(pclass):
        return "Unknown"
    elif pclass == 1:
        return "First Class"
    elif pclass == 2:
        return "Second Class"
    else:
        return "Third Class"
classes = titanic_survival.apply(which_class, axis=1)
classes<jupyter_output><empty_output><jupyter_text>Training In this practice, we will try to answer the following questions:* write a function that counts the number of missing values of a series object* use the DataFrame.apply() method to apply your function to titanic_survival* assign the result to the column_null_value_count variable* display the result<jupyter_code>def null_value_count(column):
    null = column[pd.isnull(column)]
    null_value = len(null)
    return null_value
column_null_value_count = titanic_survival.apply(null_value_count)
column_null_value_count<jupyter_output><empty_output><jupyter_text>All of the above code could be simplified to a single line of code by using the methods already predefined on pandas namely ``isnull()`` and ``sum()`` which determine respectively the rows with null values and the sum of these null values as shown in the following code.<jupyter_code>titanic_survival.isnull().sum()<jupyter_output><empty_output><jupyter_text>Practice: compute the percentage of survival by classe group In this practical case, we will apply a pivot table to compute the percentage of survival by classe group. So, we will try to answer the following questions:* add the column "classe_labels" to the dataframe titanic_survival containing the variable classes that we created in the previous example 3* create a pivot table that calculates the average chance of survival (column "survived") for each class (column "classes") of the dataframe titanic_survival* assign the resulting series object to the classe_group_survival variable* display the result<jupyter_code>titanic_survival["classe_labels"] = classes
classe_group_survival = titanic_survival.pivot_table(
    index="classe_labels", values="survived")
classe_group_survival<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07492331288343558, 'fraction_numerical': 0.004984662576687116, 'mean_word_length': 5.427304090685067, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '63777', 'license': 'MIT', 'n_tokens_mistral': 3744, 'n_tokens_neox': 3580, 'n_words': 1558, 'path': 'courses/5. Handling missing values in Python.ipynb', 'repo_name': 'Niangmohamed/Introduction-to-Data-Science-With-Python', 'size': '13040'}