<jupyter_start><jupyter_text>Prunning research question: categories  Set-up<jupyter_code>try:
    # If we are not on google colab we need to upgrade pandas...
    from google.colab import widgets
    !pip install pandas --upgrade
except ModuleNotFoundError:
    # If we are not on google colab we pass
    pass


from distutils.version import LooseVersion
import pandas as pd
# As we are using features from pandas 0.23, we need to 
# check that the correct version is used.
assert LooseVersion(pd.__version__) > LooseVersion('0.23'), """
If you are on Google Colab and this fails, make sure you "restart runtime"
after running the cell that install the newest version of Pandas
If you are not on Google Colab and this fails, please update your
Pandas version: pip install --upgrade pandas
"""
# The block of code is here to make the notebook Google Colab compatible
try:
    # If we are not on google colab (we assume that you don't have 
    # the google.colab module on your computer...), 
    # the line below will raise an error that will be catch :)
    from google.colab import widgets
    # We clean the content of the directory, so that we can use
    #  git clone directly in it
    !rm -rf *
    !rm -rf .*
    !git clone https://github.com/striantafyllouEPFL/healthy-candies.git .
    # We get the data and initialize everything
    !python ./init_project.py
except ModuleNotFoundError:
    pass
import pandas as pd
import numpy as np

# Project specific module/functions
from healthy_candies.load import load_data
from healthy_candies.path import OFF_CAT_CSV, WIKIDATA_CAT_TREE_CSV
from healthy_candies.utils import plot_settings, log_bins


import seaborn as sns
%pylab inline
%config InlineBackend.figure_format = 'retina'
# further plot settings
sns.set()  # Use default seaborn styling for all plots
sns.set_context('notebook')
plt.rcParams['figure.dpi'] = 100<jupyter_output><empty_output><jupyter_text>*One of the first objective we had internally in the team was to be able to generalize our approach to as many products categories as possible. The aim of this notebook is to explain why we dropped this idea.*<jupyter_code>COL_NUTRI_CAT = 'nutrition_grade_fr'
COL_NUTRI_FLOAT = 'nutrition-score-fr_100g'
COL_MAIN_CAT = 'main_category'
COL_PRODUCT_NAME = 'product_name'<jupyter_output><empty_output><jupyter_text>Exploring the categories Of all the products that are contained in the dataset, we have only $\approx 140\,000$ products for which we have information on the associated nutriscore.<jupyter_code>df = load_data(usecols=[COL_NUTRI_FLOAT,
                        COL_NUTRI_CAT,
                        COL_MAIN_CAT,
                        COL_PRODUCT_NAME],
               limit_have_nutri_score=True)
df.shape<jupyter_output><empty_output><jupyter_text>In the Open Food Facts database, all products are associated with one or more categories. Our first objective was therefore to see how many categories are present among the data. We can do that by looking at the `main_category` column of the dataset. What we first found is that the number of categories is huge with $\approx 4\,742$ different categories present in the data.<jupyter_code>categories = df[COL_MAIN_CAT]
cat_counts = categories.value_counts()
cat_counts.shape<jupyter_output><empty_output><jupyter_text>Also, the number of products in each category follows a distribution close to a power law: many categories have very few products whereas few categories are associated with many categories.<jupyter_code>bins = log_bins(cat_counts, 30)
cat_counts.hist(bins=bins, density=True)

plot_settings({
    'title': 'Distribution of the categories count',
    'xlabel': 'Categories count (log scale)',
    'ylabel': 'Number of categories concerned / bin width (log scale)',
    'xscale': 'log',
    'yscale': 'log'})<jupyter_output><empty_output><jupyter_text>When we look at the categories that are the most frequent, we find one called `en:plant-based-foods-and-beverages`.<jupyter_code>cat_counts.head(10)<jupyter_output><empty_output><jupyter_text>If we look at a sample from of the products from that category, we can find very diverse things that are not comparable (olive oil, jam, butter, nuggets, etc.). This category must be the default one in the app that helps to collect information on the products.<jupyter_code>df[df[COL_MAIN_CAT] == 'en:plant-based-foods-and-beverages'][COL_PRODUCT_NAME] \
    .sample(10, random_state=10)<jupyter_output><empty_output><jupyter_text>**As a result we won't be able to assume that the content of all categories is homogenous.**  Grouping the categories Since there are many categories inside the data, we were still interested into grouping those categories automatically, to have relevant bigger categories.To do this we used two other datasets:* The first one is the list of categories [provided by  Open Food Facts](https://world.openfoodfacts.org/categories.json) that contains also for some categories a matching *Wikidata* entry,* The second one is dataset build from a [hierarchy of entries](https://tools.wmflabs.org/wikidata-todo/tree.html?q=2095&rp=279) related to food on *Wikidata*.Using the hierarchy, we tried to group categories of open food facts. We built the necessary script to perform this task, but the results are not successful. Let's explore. The head of the first (custom) dataset is shown below. For each category on Open Food Facts, the matching wiki data id is given if it exists.<jupyter_code># Load the 2 new datasets (custom formatted)
df_off_cat = pd.read_csv(OFF_CAT_CSV)
df_off_cat = df_off_cat.fillna('')
df_off_cat.head()<jupyter_output><empty_output><jupyter_text>We can see that we have $998$ products with a matching wikidataId, which is pretty good !<jupyter_code>df_off_cat.wikidataId.value_counts().shape<jupyter_output><empty_output><jupyter_text>The head of the second (custom) dataset is shown below. For each `wikidataId` we have the matching category parent according to the wikidata. In column `0` we have the closest parent to the food category, in column `1` we have the closest parent separated by distance 1 from the food category, etc.<jupyter_code>df_h = pd.read_csv(WIKIDATA_CAT_TREE_CSV)
df_h.head(5)<jupyter_output><empty_output><jupyter_text>Doing the merge between the main dataset and the one with all the categories is performed with a negligible loss of data.<jupyter_code>df_with_wikiId = df.merge(
    df_off_cat, left_on=COL_MAIN_CAT, right_on='catIdOff')
df_with_wikiId.shape<jupyter_output><empty_output><jupyter_text>However, the mapping from a product to wikidataId concerns only $\approx 42 \,000$ products.<jupyter_code>sum(df_with_wikiId['wikidataId'] != '')<jupyter_output><empty_output><jupyter_text>And we are left with only $23$ different categories.<jupyter_code>tmp = df_with_wikiId['wikidataId'].value_counts()
tmp.shape
tmp<jupyter_output><empty_output><jupyter_text>Now, we continue by merging the data with the second dataset also.<jupyter_code>df_final = df_with_wikiId.merge(df_h, on='wikidataId')<jupyter_output><empty_output><jupyter_text>Again we lose some products and we are left with $\approx 34\,000$ products.<jupyter_code>df_final.shape<jupyter_output><empty_output><jupyter_text>At the end, we are left with $18$ categories from Open Food Facts (shown below).<jupyter_code>df_final[COL_MAIN_CAT].value_counts()<jupyter_output><empty_output><jupyter_text>Using the data from the second dataset, we can match those $18$ categories to only $11$ categories.<jupyter_code>df_final['0'].value_counts().shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07772637144745538, 'fraction_numerical': 0.008460013218770655, 'mean_word_length': 5.278838174273859, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '333129', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 2365, 'n_tokens_neox': 2263, 'n_words': 838, 'path': 'notebooks/exploration/pruning_categories.ipynb', 'repo_name': 'striantafyllouEPFL/healthy-candies', 'size': '7565'}