<jupyter_start><jupyter_text>Projet n°3 : Concevez une application au service de la santé publique  Import des librairies<jupyter_code>import pandas as pd
import missingno as msno
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Import du jeux de données<jupyter_code>df = pd.read_csv('en.openfoodfacts.org.products.csv', nrows=10, sep='\t')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>En m'appuyant sur mon idée d'application, je sélectionne différentes variables :- macro-nutriments- micro-nutriments- energy et qualité nutritionnelle- informations diverses sur le produit<jupyter_code>columns = ['code',           
'product_name',
            
'brands',
'brands_tags',
            
'categories',
'categories_tags',
'main_category',
'ingredients_text',
'traces_tags',

            
'labels',
'labels_tags',
           
'origins',
'origins_tags',
            
'countries',
'countries_tags',
            
'purchase_places',
'cities',
            
'emb_codes',
'emb_codes_tags',
          
'nutrition-score-fr_100g',
'nutriscore_grade',
'nutriscore_score',
'energy_100g',
'energy-kcal_100g',
'fat_100g',
           'saturated-fat_100g',
           'polyunsaturated-fat_100g',
           'monounsaturated-fat_100g',
           'trans-fat_100g',
'omega-3-fat_100g',
'potassium_100g',
'sodium_100g',
'carbohydrates_100g',
           'sugars_100g',
           'fiber_100g',
'proteins_100g',
'vitamin-a_100g',
'vitamin-c_100g',
'calcium_100g',
'iron_100g'
]
df = pd.read_csv('en.openfoodfacts.org.products.csv', sep='\t', usecols=columns, low_memory=False)
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1555491 entries, 0 to 1555490
Data columns (total 40 columns):
 #   Column                    Non-Null Count    Dtype  
---  ------                    --------------    -----  
 0   code                      1555491 non-null  object 
 1   product_name              1475940 non-null  object 
 2   brands                    855624 non-null   object 
 3   brands_tags               855568 non-null   object 
 4   categories                789900 non-null   object 
 5   categories_tags           789898 non-null   object 
 6   origins                   65893 non-null    object 
 7   origins_tags              65782 non-null    object 
 8   labels                    351083 non-null   object 
 9   labels_tags               351107 non-null   object 
 10  emb_codes                 103382 non-null   object 
 11  emb_codes_tags            103353 non-null   object 
 12  cities                    0 non-null        float64
 13  purchase_places           1[...]<jupyter_text>Choix parmi les doublons des colonnes<jupyter_code>df.head(30)
df[['countries', 'countries_tags']].tail(50)<jupyter_output><empty_output><jupyter_text>Les variables avec le suffixe tags sont standardisées. Elle semble donc plus facile à exploiter:- les catégories peuvent être renseignée en plusieurs langues- les tags sont soient traduits en anglais, soient ils indiquent la langue, exemple : 'de:tiefkühl-kuchen' De plus, le remplissage semble identique à celui de la variable originale.Pour toutes ces raisons, je décide de garder les variables avec le suffixe tag lorsque j'ai le choix entre deux variables<jupyter_code>msno.matrix(df)<jupyter_output><empty_output><jupyter_text>Autre nettoyage : - Je me débarasse également des variables non-nutritionnelles qui ont un taux de remplissage très faible (grapiquement).- Je garde energy_100g (KJ) à la place de energy_kcal  Deuxième sélection<jupyter_code>columns_v2 = ['code',           
'product_name',        
'brands_tags',
'labels_tags',
'countries_tags',
'main_category',
'ingredients_text',
          
'nutrition-score-fr_100g',
'nutriscore_grade',
'energy_100g',
'fat_100g',
           'saturated-fat_100g',
           'polyunsaturated-fat_100g',
           'monounsaturated-fat_100g',
           'trans-fat_100g',
'omega-3-fat_100g',
'potassium_100g',
'sodium_100g',
'carbohydrates_100g',
           'sugars_100g',
           'fiber_100g',
'proteins_100g',
'vitamin-a_100g',
'vitamin-c_100g',
'calcium_100g',
'iron_100g'
]
df2 = df[columns_v2]<jupyter_output><empty_output><jupyter_text>Valeurs manquantes<jupyter_code>df2 = df2.dropna(subset=['brands_tags',  'energy_100g',
       'fat_100g', 'saturated-fat_100g', 'carbohydrates_100g', 'sugars_100g',
       'proteins_100g', 'nutrition-score-fr_100g'], how='all')

print(df2.shape)
df2 = df2.dropna(subset=['fat_100g', 'carbohydrates_100g', 'proteins_100g'], how='any')
print(df2.shape)
msno.matrix(df2)
print(df2.code.duplicated().sum()) 

df2 = df2.drop_duplicates(subset='code')
pd.Series(df2.code).is_unique<jupyter_output><empty_output><jupyter_text>Valeurs aberrantes<jupyter_code>print(df2.shape)

df2.describe()<jupyter_output>(1224745, 26)<jupyter_text>Maximum Les variables sont exprimés par 100g. Par conséquent aucune de ces variables de peut être supérieure à 100.<jupyter_code>col_num_max = list(df2.describe().columns)
col_num_max.remove('energy_100g')
for x in col_num_max :
    df2.drop(axis=0, index=df2[df2[x] > 100].index, inplace=True)
# la densité d'énergie maximale est de 37 KJ/g (lipide). Par conséquent, aucune variable ne peut avoir un maximum supérieur à 3700

df2.drop(axis=0, index=df2[df2['energy_100g'] > 3700].index, inplace=True )<jupyter_output><empty_output><jupyter_text>Minimum<jupyter_code>col_num_min = list(df2.describe().columns)
col_num_min.remove('nutrition-score-fr_100g') # le score min est -15
for x in col_num_min :
    df2.drop(axis=0, index=df2[df2[x] < 0 ].index, inplace=True)
print(df2.shape)

df2.describe()<jupyter_output>(1213948, 26)<jupyter_text>Total La somme entre les protéines, les glucides et les lipides doit être inférieur à 100g<jupyter_code>df2['total_mac'] = df2.fat_100g + df2.carbohydrates_100g + df2.proteins_100g
(df2['total_mac'] > 100).sum()
df2[df2['total_mac'] > 100]
index_to_sup = df2[df2['total_mac'] > 100].index

df2.drop(index=index_to_sup, inplace=True)<jupyter_output><empty_output><jupyter_text>Version simplifiée (sans les micro-nutriments)<jupyter_code>df2.columns
df2_simp = df2[['product_name', 'brands_tags', 'main_category', 'ingredients_text', 'labels_tags', 'countries_tags', 'energy_100g', 'fat_100g', 'proteins_100g', 'carbohydrates_100g', 'nutrition-score-fr_100g', 'sodium_100g', 'sugars_100g', 'saturated-fat_100g']]
print(df2_simp.shape)
df2_simp.head()
msno.matrix(df2_simp)
%%time
df2_simp.info()
608950 / df2_simp.shape[0]<jupyter_output><empty_output><jupyter_text>NaN  Product_name manquants<jupyter_code>df2_simp.isna().sum()
df2_simp.dropna(subset=['product_name'], inplace=True)
df2_simp.isna().sum()
df2_simp.dropna(subset=['sodium_100g', 'sugars_100g', 'saturated-fat_100g'], inplace=True)
df2_simp.isna().sum()
df2_simp
df2_simp['energy_100g'] = df2_simp['fat_100g'] * 37 + df2_simp['proteins_100g'] * 17 + df2_simp['carbohydrates_100g'] * 17
df2_simp.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1110702 entries, 0 to 1555489
Data columns (total 14 columns):
 #   Column                   Non-Null Count    Dtype  
---  ------                   --------------    -----  
 0   product_name             1110702 non-null  object 
 1   brands_tags              656666 non-null   object 
 2   main_category            630178 non-null   object 
 3   ingredients_text         543877 non-null   object 
 4   labels_tags              282735 non-null   object 
 5   countries_tags           1109363 non-null  object 
 6   energy_100g              1110702 non-null  float64
 7   fat_100g                 1110702 non-null  float64
 8   proteins_100g            1110702 non-null  float64
 9   carbohydrates_100g       1110702 non-null  float64
 10  nutrition-score-fr_100g  606602 non-null   float64
 11  sodium_100g              1110702 non-null  float64
 12  sugars_100g              1110702 non-null  float64
 13  saturated-fat_100g       1110702 non-null [...]<jupyter_text>variable catégorielle<jupyter_code>for x in ['product_name', 'brands_tags', 'main_category', 'labels_tags', 'countries_tags']:
    print(len(df2_simp[x].unique()))
df2[df2.countries_tags.str.contains(',')== True]['countries_tags']
df2_simp = df2_simp.copy()
index = df2_simp[df2_simp.countries_tags.str.contains(',')== True].index
df2_simp.drop(index=index, inplace=True)
df2_simp['countries_tags'] = df2_simp.countries_tags.str.replace('en:', '')
df2_simp.describe()
index = df2_simp[df2_simp.countries_tags.str.contains(':')== True].index
df2_simp.drop(index=index, inplace=True)
df2_simp.shape
df2_simp.head()<jupyter_output><empty_output><jupyter_text>Valeurs dupliquées<jupyter_code>print(df2_simp.duplicated().sum())
df2_simp.columns
print(df2_simp.duplicated(subset=['product_name','fat_100g',
       'proteins_100g', 'carbohydrates_100g', 'nutrition-score-fr_100g',
       'sodium_100g', 'sugars_100g']).sum())
df2_simp.duplicated(subset=['product_name','fat_100g',
       'proteins_100g', 'carbohydrates_100g', 'nutrition-score-fr_100g',
       'sodium_100g', 'sugars_100g']).sum() / df2.shape[0]
df2_simp = df2_simp.drop_duplicates(subset=['product_name','fat_100g',
       'proteins_100g', 'carbohydrates_100g', 'nutrition-score-fr_100g',
       'sodium_100g', 'sugars_100g'])
df2_simp.shape<jupyter_output><empty_output><jupyter_text>Export<jupyter_code>df2_simp.to_csv('products.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.123421415685026, 'fraction_numerical': 0.07439244401995118, 'mean_word_length': 3.952180767209669, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '565970', 'license': 'Unlicense', 'n_tokens_mistral': 4025, 'n_tokens_neox': 3481, 'n_words': 630, 'path': 'notebooks/ds_p3_p1.ipynb', 'repo_name': 'j30v/j30v.github.io', 'size': '9423'}