<jupyter_start><jupyter_text>Cleaning Data Table of Contents1&nbsp;&nbsp;Imports2&nbsp;&nbsp;les doublons2.1&nbsp;&nbsp;Les titres en doublons ?3&nbsp;&nbsp;les variables (à supprimer, à modifier etc...)3.1&nbsp;&nbsp;Les années3.2&nbsp;&nbsp;color3.3&nbsp;&nbsp;Suppression de variables3.4&nbsp;&nbsp;director_name4&nbsp;&nbsp;Les valeurs manquantes (NaN)4.1&nbsp;&nbsp;supprimer la colonne qui a trop de manquants ?5&nbsp;&nbsp;les zéros5.1&nbsp;&nbsp;Suppression des variables5.2&nbsp;&nbsp;Traitement de lignes6&nbsp;&nbsp;Sauvegarde  Imports ```%load imports.py``` permet d'importer le contenu du fichier imports.py créé lors de l'étape : [Import des libraries](env.ipynb)<jupyter_code># %load imports.py
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import missingno as msno
plt.rcParams["figure.figsize"] = (16,10)
data = pd.read_csv('data/5000_movies_bis.csv')<jupyter_output><empty_output><jupyter_text>les doublons<jupyter_code>start_data_shape = data.shape
start_data_shape
sum(data.duplicated())
#duplicated_rows_movies_df = data[data.duplicated()]
#data.drop_duplicates(inplace = True)
data.drop_duplicates(ignore_index=True, inplace=True)
data.shape<jupyter_output><empty_output><jupyter_text>Les titres en doublons ?<jupyter_code>sum(data.duplicated(subset='movie_title', keep=False))<jupyter_output><empty_output><jupyter_text>On ne les supprime pas car il peut il y avoir des films différents qui portent le même nom (version animation et film ...)  les variables (à supprimer, à modifier etc...)  Les années<jupyter_code>data['title_year'].head(3)
data['title_year'] = data['title_year'].astype(str)
data['title_year'] = data['title_year'].str[0:4]
data['title_year'].head(3)<jupyter_output><empty_output><jupyter_text>* Convertir les années en ```datetime```<jupyter_code>#data['title_year'] = pd.to_datetime(data['title_year'])<jupyter_output><empty_output><jupyter_text>color<jupyter_code>data.groupby('color')['color'].count()<jupyter_output><empty_output><jupyter_text>Suppression de variables On supprime les variables qui nous semblent inutiles<jupyter_code>data.drop(columns=['plot_keywords',
                'movie_imdb_link',
                'color',
                'language'
                ], inplace=True)
data.shape<jupyter_output><empty_output><jupyter_text>director_name<jupyter_code>data.groupby('director_name').imdb_score.agg(['max','min']).reset_index().head(10)<jupyter_output><empty_output><jupyter_text>Les valeurs manquantes (NaN) > ```isnull``` is an alias for ```isna```* To detect NaN values numpy uses ```np.isnan()```* To detect NaN values pandas uses either ```.isna()``` or ```.isnull()``````df.notnull()```   Opposite of ```.isnull()```<jupyter_code>data.isna().sum().sort_values(ascending=False)
va1 = data.isna().sum().sort_values(ascending=False)
va2 = ((data.isna().sum()*100/data.isna().count()).sort_values(ascending=False))
va2 = va2.round(1)
vadf = pd.concat([va1, va2], axis=1)
vadf = vadf.rename(columns={0: 'nb', 1: '%'},)
vadf
data.shape
data.isna().sum().sum()
sns.heatmap(data.isna(), cbar=False)
msno.matrix(data)
msno.heatmap(data)<jupyter_output><empty_output><jupyter_text>* A value near -1 means if one variable appears then the other variable is very likely to be missing.* A value near 0 means there is no dependence between the occurrence of missing values of two variables.* A value near 1 means if one variable appears then the other variable is very likely to be present. * https://www.geeksforgeeks.org/python-visualize-missing-values-nan-values-using-missingno-library/<jupyter_code>data = data.dropna(how='any')<jupyter_output><empty_output><jupyter_text>> ```df.dropna(how='all')``` Drop the rows where all elements are missing* ```any``` : If any NA values are present, drop that row or column.* ```all``` : If all values are NA, drop that row or column.<jupyter_code>#data = data.dropna()
print("total null values: ", data.isna().sum().sum())
data.shape<jupyter_output><empty_output><jupyter_text>supprimer la colonne qui a trop de manquants ? test 1 : non  les zéros<jupyter_code>data.shape
(data == 0).sum().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Suppression des variables* qui présentent trop de 0<jupyter_code>data = data.drop(['movie_fb_likes', 'director_fb_likes'], axis='columns')
data.shape<jupyter_output><empty_output><jupyter_text>Traitement de lignes * On supprime les lignes avec des 0* ! Perte de beaucoup d'information TODO: pour ne pas perdre autant de data, refaire le traitement des 0 en supprimant la variable "facenumber_in_poster"<jupyter_code>#data = data.loc[(data!=0).any(axis=1)]
#data = data.loc[(data !=0).any(1)]
# https://stackoverflow.com/questions/22649693/drop-rows-with-all-zeros-in-pandas-data-frame
data = data.replace(0, np.nan)
data = data.dropna(axis=0)
data = data.replace(np.nan, 0)
(data == 0).sum().sum()
data.shape<jupyter_output><empty_output><jupyter_text>Sauvegarde<jupyter_code>final_data_shape = data.shape
print("data.shape au départ : ", start_data_shape, "et en fin de traitement : ", final_data_shape)
data.to_csv('data/data_cleaning.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14814814814814814, 'fraction_numerical': 0.01408179012345679, 'mean_word_length': 7.204113924050633, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '413435', 'license': 'MIT', 'n_tokens_mistral': 1895, 'n_tokens_neox': 1846, 'n_words': 381, 'path': 'cleaning.ipynb', 'repo_name': 'DROMZEE/imdb_5000_Movie_Dataset', 'size': '5184'}