<jupyter_start><jupyter_text>Salary recommandation using a scoreAuthor: <NAME>The purpose of this Notebook is to find the best score in order to select the optimal salary change in the salary-based recommendation. The best score will be the one that allows the best compromise between a salary decrease and a raise in job offers, for every job groups & experience buckets.We first use the 10% sample (~1.2Go) dataset called `sample_10perc.csv` because the full dataset is too big.<jupyter_code>import numpy as np
import pandas as pd
from pandas.tseries import offsets
from paul_emploi.lib import cleaned_data
from paul_emploi.modeling import salary_recommendation_model, job_offers_optimal_buckets
import matplotlib.pyplot as plt
from matplotlib import gridspec

%matplotlib inline
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Import & clean job postings datasetLet's first `import_clean_job_offers` to import a clean database of job_offers.<jupyter_code>etalab_path = "/var/data/bayes_impact/"
postings = cleaned_data.job_offers(data_folder=etalab_path, filename_offers="job_offers/sample_10perc.csv")
postings.head(2)<jupyter_output>/usr/local/lib/python3.4/dist-packages/IPython/core/interactiveshell.py:3006: DtypeWarning: Columns (1,2,6,7,10,11,20,21,23,24,27,28,30,31,32,37,38,39,40,41,42,43,44,45,46,47,48,49,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,92,94,95,96,97,98,100,102,104,105,106,107,108,109,110,111,112,115,119,120,124,126,127,128,129) have mixed types. Specify dtype option on import or set low_memory=False.
  if self.run_code(code, result):<jupyter_text>Filter postingsRemove outliers & filter date.These modules will not be used in production since the filtering will be done before without pandas.<jupyter_code># Since our sample has data until early 2016, we use for now SMIC
# 2015 to run some tests. As a maximum salary, we use 50.000€ (which is
# arbitrary.)
# SMIC amounts available here: http://www.salairebrutnet.fr/
SMIC_2015 = 17496
SMIC_2016 = 17604
_MIN_ANNUAL_SALARY = SMIC_2015
# Arbitrary.
_MAX_ANNUAL_SALARY = 50000


def filter_postings(
        table_offers,
        min_salary=_MIN_ANNUAL_SALARY,
        max_salary=_MAX_ANNUAL_SALARY,
        nb_months=6):
    """Returns a table containing only job offers that fulfill specific
    criteria. It will be used for salary recommendations.
    Args :
        table_offers: pandas DataFrame containing job postings.
        min_salary (int.): exclude all salary under min_salary (ex : SMIC).
        max_salary (int.): exclude all salary above max_salary (ex: 50000).
        nb_months: number of months to look back to get job offers.
    Returns:
        a pandas DataFrame containing only job offers that fulfill specific
        criteria. It will be used for salary recommendations.
    """
    clean_postings = table_offers.copy()
    # Exclude outliers.
    clean_postings = _filter_outliers(
        clean_postings, min_salary, max_salary)

    # Grab every offers until nb_months before.
    if not clean_postings.empty:  # Or it will crash.
        clean_postings = _filter_date(clean_postings, nb_months)

    # Check if table is not empty.
    if clean_postings.empty:
        raise ValueError('No job offers match the given criteria.')

    interesting_cols = [
        'rome_id',
        'annual_minimum_salary',
        'experience_min_duration']
    return clean_postings[interesting_cols]


def _filter_date(table_offers, nb_months):
    """ Keep only the latest nb_months of the postings dataset.
    Args :
        table_offers: pandas DataFrame containing job postings.
        nb_months (int.): number of months to look back to get job offers.
    Returns:
        pandas DataFrame contaning offers of the latest nb_months
        of table_offers.
    """
    table_offers.set_index(pd.DatetimeIndex(
        table_offers.date_debut_imputed), inplace=True)
    end_date = table_offers.date_debut_imputed.max()
    start_date = (end_date - offsets.DateOffset(months=nb_months)).strftime(
        '%Y-%m-%d')
    date_mask = (table_offers['date_debut_imputed'] >= start_date)
    return table_offers.loc[date_mask].reset_index(drop=True)

def _filter_outliers(
        table_offers,
        min_salary,
        max_salary):
    """Returns a table containing only job offers without outliers.
    Args :
        table_offers: pandas DataFrame containing job postings.
        min_salary (int.): exclude all salary under min_salary (ex : SMIC).
        max_salary (int.): exclude all salary above max_salary (ex: 50000).
    Returns:
        pandas DataFrame containing only job offers with a salary between
        [min_salary, max_salary[.
    """
    valid_salary_mask = (table_offers.annual_minimum_salary >= min_salary) & (
        table_offers.annual_minimum_salary < max_salary)
    return table_offers.loc[valid_salary_mask]
clean_postings = filter_postings(postings)
clean_postings.head(2)<jupyter_output><empty_output><jupyter_text>Make experience buckets<jupyter_code># Select some job groups (some contaning offers, other without so many)
job_groups_examples = [
    'G1602',
    'K2204',
    'K1303',
    'K1304',
    'G1803',
    'A1202',
    'B1401']

postings_example = clean_postings.loc[clean_postings.rome_id.isin(job_groups_examples)]

# Bucketize
postings_example_bucketized = postings_example.groupby('rome_id').apply(
    job_offers_optimal_buckets.apply_bucketize)

postings_example_bucketized.head(5)<jupyter_output><empty_output><jupyter_text>Let's create 2 small subsetsEach of them contains a single `exp_bucket` of a `rome_id`.<jupyter_code># Make a dataset contaning only 1 job group & 1 experience bucket

postings_mask_1 = (postings_example_bucketized.rome_id == 'G1602') &\
    (postings_example_bucketized.exp_bucket == '[0, 1[')
    
postings_mask_2 = (postings_example_bucketized.rome_id == 'G1803') &\
    (postings_example_bucketized.exp_bucket == '[0, 1[')
    
postings_1 = postings_example_bucketized.loc[postings_mask_1]
postings_2 = postings_example_bucketized.loc[postings_mask_2]

postings_2.head(2)<jupyter_output><empty_output><jupyter_text>Cumulative countsFor each salary, we count the number of offers available.<jupyter_code>num_offers_with_higher_salary = salary_recommendation_model._compute_job_offers_salary(postings_1)
num_offers_with_higher_salary.head(2)<jupyter_output><empty_output><jupyter_text>Chosing the best scoreWe use the following modules to compute graphs on score.<jupyter_code>def compute_result_as_df(table_offers, score_label='sqrtO_salary'):
    """ Compute every metrics of the score and store them in a pandas Dataframe
    """
    num_offers_with_higher_salary = salary_recommendation_model._compute_job_offers_salary(
        table_offers)
    cumul_offers = num_offers_with_higher_salary.reset_index()
    def _scoring(idx):
        return _apply_score(num_offers_with_higher_salary, idx, score_label=score_label)

    result_as_df = pd.DataFrame(cumul_offers.reset_index()['index'].apply(_scoring).tolist())
    result_as_df = pd.concat([cumul_offers, result_as_df], axis=1)
    
    return result_as_df


def _apply_score(
        num_offers_with_higher_salary, idx, score_label):
    """ Calculate a score to each salaries of table_offers, maximize it and return the amount of
    gained offers for the optimal decrease of salary + additional metrics to compute 
    score comparison.

    Args:
        num_offers_with_higher_salary: Pandas Series containing the amount of job offers (value)
            by salary (index).
        idx: the index of the salary on which to compute the score.
        score_label: label of the score we decided to compute.

    Returns:
        a dictionnary containing all the metrics

    """
    # Cumulative count.
    cumul_offers = num_offers_with_higher_salary.reset_index()
    if idx == 0:
        return _fill_dict_of_res(0, cumul_offers.annual_minimum_salary.iloc[idx], 0, 0)

    delta_salaries = salary_recommendation_model._compute_delta_from_index(
        cumul_offers.annual_minimum_salary, idx)
    delta_offers = salary_recommendation_model._compute_delta_from_index(
        cumul_offers.num_offers_with_higher_salary, idx)
    # Compute score.
    scores = _compute_scores(delta_offers, delta_salaries, score_label)
    # Best score = max(score).
    idx_max_score = scores.idxmax()
    # Compute results.
    final_num_offers = cumul_offers.num_offers_with_higher_salary.iloc[idx_max_score]
    final_salary = cumul_offers.annual_minimum_salary.iloc[idx_max_score]
    gained_offers = delta_offers.iloc[idx_max_score]
    decrease_of_salary = delta_salaries.iloc[idx_max_score]

    return _fill_dict_of_res(idx_max_score, final_salary, decrease_of_salary, gained_offers)


def _fill_dict_of_res(idx_max_score, final_salary, decrease_of_salary, gained_offers):
    return {
        'idx_max_score': idx_max_score,
        'final_salary': final_salary,
        'decrease_of_salary': decrease_of_salary,
        'gained_offers': gained_offers}


def _compute_scores(offers, delta_salaries, score_label):
    """Compute different scores
    NOTE: 'O' stands for 'delta(O)' and 'S' stands for 'delta(S)', with 'delta(O)' being the variation
    of number of offers and 'delta(S)' the respective variation of salary.
    """
    if score_label == 'sqrt(O)_S':
        score = np.sqrt(offers) / (delta_salaries)
    if score_label == 'sqrt(O)_S²':
        score = np.sqrt(offers) / (delta_salaries ** 2)
    if score_label == 'log(O)_S':
        score = np.log(offers) / (delta_salaries)
    if score_label == 'log(O)_S²':
        score = np.log(offers) / (delta_salaries ** 2)
    if score_label == 'O_S²':
        score = offers / (delta_salaries ** 2)

    return score
table_offers = postings_1.copy()
#table_offers = postings_2.copy()
# Output example
result_as_df = compute_result_as_df(table_offers=table_offers, score_label='sqrt(O)_S')
result_as_df.head(2)<jupyter_output><empty_output><jupyter_text>Score comparaisonWe're going to compare scores relatively to 3 metrics :* Recommended salaries* Recommended decrease of salary* Gained offers  Recommended salaries* x-axis: salary indexes in ascending values (0: the lowest salary)* y-axis: recommended salary indexes (also in ascending values).<jupyter_code>score_label_list = ['sqrt(O)_S', 'sqrt(O)_S²', 'log(O)_S', 'log(O)_S²', 'O_S²']
col_list = ['blue', 'green', 'red', 'darkturquoise', 'purple']
fig = plt.figure(figsize=(15, 15))
gs = gridspec.GridSpec(len(score_label_list), 1)
num_fig = 0
num_col = 0
for score_label in score_label_list:
    ax = plt.subplot(gs[num_fig])
    result_as_df = compute_result_as_df(table_offers=table_offers, score_label=score_label)
    result_as_df.idx_max_score.plot(label=score_label, ax=ax, color=col_list[num_col])
    plt.xlabel('Salary indexes (ascending values)')
    plt.ylabel('Recommended salary indexes')
    plt.legend(loc='upper left')
    num_fig += 1
    num_col += 1<jupyter_output><empty_output><jupyter_text>Recommended decrease of salary* x-axis: annual_minimul salary in ascending values* y-axis: recommended decrease of salary<jupyter_code>fig = plt.figure(figsize=(15, 15))
gs = gridspec.GridSpec(len(score_label_list), 1)
num_fig = 0
num_col = 0
for score_label in score_label_list:
    ax = plt.subplot(gs[num_fig])
    result_as_df = compute_result_as_df(table_offers=table_offers, score_label=score_label)
    result_as_df.set_index('annual_minimum_salary').decrease_of_salary.plot(
        label=score_label, ax=ax, color=col_list[num_col])
    plt.ylabel('Recommended decrease of salary')
    plt.legend(loc='upper left')
    num_fig += 1
    num_col += 1<jupyter_output><empty_output><jupyter_text>Gained offers* x-axis: annual_minimul salary in ascending values* y-axis: Gained offers in percent.<jupyter_code>fig = plt.figure(figsize=(15, 15))
gs = gridspec.GridSpec(len(score_label_list), 1)
num_fig = 0
num_col = 0
for score_label in score_label_list:
    ax = plt.subplot(gs[num_fig])
    result_as_df = compute_result_as_df(table_offers=table_offers, score_label=score_label)
    result_as_df.set_index('annual_minimum_salary').gained_offers.plot(label=score_label, ax=ax, color=col_list[num_col])
    plt.ylabel('Gained offers (in %)')
    plt.legend(loc='upper left')
    num_col += 1
    num_fig += 1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09386834259930707, 'fraction_numerical': 0.0292482475223592, 'mean_word_length': 4.773023255813953, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '28588', 'license': 'MIT', 'n_tokens_mistral': 4551, 'n_tokens_neox': 4211, 'n_words': 959, 'path': 'recommandations_salariales/algo_reco_salaries_score.ipynb', 'repo_name': 'SGMAP-AGD/recommandations_emploi', 'size': '12421'}