<jupyter_start><jupyter_text>Prescriptive part of the problem: divide & conquer deterministic approach Optimization problem to solve: maximize total profit on the portfolio by finding personalized rebate/surcharge coefficients while maintaining a certain percentage of customers in the portfolio.**Note: for the rest of this study, we will search personalized surcharge coefficients** ```delta``` **between -5% and 10% for each customer**.Difficulties:- large scale problem (one variable to optimize per client i: the rebate/surcharge coefficient delta_i)- nonlinear problem, both in the objective (profit_i = retention_proba_i(delta_i) . [premium_i . (1+delta_i) - cost_i]) and the global constraint (avg(retention_proba_i for i in range(N)) > min_global_retention_rate)Divide & conquer deterministic resolution:- **(a)**: cluster clients, then optimize the problem by giving all the members of each cluster the same rebate/surcharge coefficient - **(b)**: start from the results of **(a)**, and solve independent optimization subproblems in each cluster by initializing them with the solution found with **(a)**; the constraint of each subproblem has to be adapted such that the aggregate solution remains a solution of the original problem.  Packages importations<jupyter_code>import pandas as pd
import sys
import numpy as np
import nlopt
import os
import matplotlib.pyplot as plt
from collections import Counter
from copy import deepcopy, copy

from sklearn.cluster import MiniBatchKMeans

import scipy.sparse as sp
from scipy.optimize import minimize<jupyter_output><empty_output><jupyter_text>Dataset importation<jupyter_code>os.chdir("### path to the dataset ###")
contract_df = pd.read_csv("ptf_mrh_2020_elast_plr.csv")
contract_df = contract_df.set_index("id_police").dropna()<jupyter_output><empty_output><jupyter_text>Crafting features to cluster clients To cluster clients into small groups where we will prescribe the same rebate/surcharge, we compute 3 features for each client:- ```profitability```- ```elasticity_10``` (the client's price elasticity for a 10% surcharge)- ```proba_resil_10``` (the client's resiliation probability for a 10% surcharge)<jupyter_code>def getResiliationProba(row, delta):
    return 1/(1+np.exp(-row['coeff_non_prix']-100*delta*row['coeff_prix']))
# we devise a surcharge of 10% for all customers to estimate how their price elasticity and churn probability would evolve (and cluster them afterwards)
delta = 0.1   
proba_resil_delta_series = contract_df.apply(getResiliationProba, args=(delta,), axis=1)
df_for_clustering = pd.DataFrame()
delta_for_col_names = delta*100

df_for_clustering['profitabilite'] = contract_df['prime_profit']/contract_df['pcc']
df_for_clustering['elasticite_%d' % delta_for_col_names] = (
    proba_resil_delta_series-contract_df['proba_resil_0%']
)/(contract_df['proba_resil_0%']*delta)
df_for_clustering['proba_resil_%d' % delta_for_col_names] = proba_resil_delta_series

df_for_clustering = (df_for_clustering-df_for_clustering.mean())/df_for_clustering.std()
# the 3 features according to which we will cluster our clients
df_for_clustering<jupyter_output><empty_output><jupyter_text>Visualization of the clients along the 3 clustering axes<jupyter_code>viz = plt.figure().gca(projection='3d')
viz.scatter(df_for_clustering['profitabilite'], df_for_clustering['elasticite_%d' % delta_for_col_names], df_for_clustering['proba_resil_%d' % delta_for_col_names])
plt.show()<jupyter_output><empty_output><jupyter_text>(2 axes seem quite correlated... Could be fixed)  1. Clustering via K-Means mini-batch Given the size of the portfolio, using K-Means mini-batch for clustering is very efficient.<jupyter_code># The number of clusters should be a hyperparameter of the optimization pipeline (and not necessarily using the elbow method); 
# however, for lack of computational power, we fix K to 150 (based on business knowledge) and do not tune this hyperparameter further.
clusteringMiniBatchKMeans = MiniBatchKMeans(n_clusters=150)  
clusteringMiniBatchKMeans.fit(sp.csr_matrix(df_for_clustering))
labelList = list(clusteringMiniBatchKMeans.labels_)

counter = Counter(labelList)
nb_of_elem_per_cluster = list(counter.values())
print("Min nb of clients in a cluster : "+str(min(nb_of_elem_per_cluster)))
print("Max nb of clients in a cluster : "+str(max(nb_of_elem_per_cluster)))
print("Avg nb of clients in the clusters : "+str(int(np.mean(nb_of_elem_per_cluster))))
print("Total nb of clients in all the clusters (sanity check) : "+str(np.sum(nb_of_elem_per_cluster)))<jupyter_output>125
9738
4379
656858<jupyter_text>Let us then creat a Python dictionary to store the clustering results (client affiliations for each cluster):<jupyter_code>def createClusters(X, labelList):
    
    X_copy = deepcopy(X)
    X_copy['label'] = pd.Series(labelList)
    resDict = {}
    
    for label in set(labelList):
        resDict[str(label)] = X_copy[X_copy['label'] == label].drop('label', axis=1).reset_index()
    
    return resDict
resDict = createClusters(contract_df, labelList)<jupyter_output><empty_output><jupyter_text>This way, we can access the 6th cluster this way for instance:<jupyter_code>resDict['5']<jupyter_output><empty_output><jupyter_text>2. Resolution of **(a)**: prescribing the same rebate/surcharge to all customers within a cluster Creation of an "average individual" per cluster:<jupyter_code>cols = list(contract_df)
cluster_averages_df = pd.DataFrame(
    columns=cols
)

for k in range(len(resDict)):
    
    df = resDict[str(k)]
    n = len(df)     
    # the feature values of each "average individual" (one per cluster) take into account the size of the cluster
    cluster_averages_df = cluster_averages_df.append(
        dict(
            ((col, df[col].mean()*n if (col == 'prime_profit' or col == 'pcc') 
              else df[col].mean()) 
             for col in cols)
        ),
        ignore_index=True
    )              

   
cluster_averages_df<jupyter_output><empty_output><jupyter_text>Let us now start setting up the optimization problems with nlopt. We first create the objective function that will be used to solve all our maximization problems (only the ```cluster_df``` argument will change from one problem to another):<jupyter_code>def buildObjectiveFunctionForSpecificCluster(cluster_df):
    
    df = cluster_df
    prime_profit = df['prime_profit']
    pcc = df['pcc']
    A = df['coeff_non_prix']
    B = df['coeff_prix']
    
    
    def objectiveFunction(delta, grad):
        # delta is the majorations vector, of size equal to cluster_df's row nb
        
        retention_proba = 1-(1/(1+np.exp(-A-100*pd.Series(delta)*B)))
        
        if grad.size > 0:
            
            grad[:] = (
                -(
                    1-retention_proba
                )*(
                    prime_profit+B/(1+np.exp(pd.Series(delta)*100*B+A))*(prime_profit*(1+pd.Series(delta))-pcc)
                )+prime_profit
            ).values            
        
        return ((prime_profit*(1+pd.Series(delta))-pcc)*retention_proba).sum()
        
    return objectiveFunction<jupyter_output><empty_output><jupyter_text>We then create the function implementing the global inequality constraint forthe optimization problem that will prescribe the same rebate/surcharge to all members within a cluster (ie global constraint of pb **(a)**):<jupyter_code>def buildConstraintsForAverageClients(
    cluster_dict, min_retention_rate, delta_min, delta_max
):

    K = len(cluster_dict)
    N = len(contract_df)
    
    def globalConstraint(delta, grad):
        # delta is the majorations vector, *here* of size equal to the number K of clusters!
        
        total_mean_retention_proba = 0

        if grad.size > 0:
        
            for k in range(K):
                
                cluster_k = cluster_dict[str(k)]
                a = cluster_k['coeff_non_prix']
                b = cluster_k['coeff_prix']
                cluster_k_resiliation_probas = (1/(1+np.exp(-a-100*delta[k]*b)))
                
                grad[k] = (1/N)*(cluster_k_resiliation_probas*b*100/(1+np.exp(a+b*delta[k]*100))).sum()
                
                total_mean_retention_proba += (1-cluster_k_resiliation_probas).sum()
                
        return min_retention_rate-(1/N)*total_mean_retention_proba

    
    lower_bound = [delta_min for i in range(K)]
    upper_bound = [delta_max for i in range(K)]
    delta_bounds = (lower_bound, upper_bound)         
    

    return delta_bounds, globalConstraint<jupyter_output><empty_output><jupyter_text>We then create the function implementing the global inequality constraint for all the optimization subproblems (cf pb **(b)**: one subproblem per cluster, initialized with the solution of the optimization problem that prescribed the same coefficient to all individuals within a cluster):<jupyter_code>def buildConstraintsForSpecificCluster(
    cluster_df, min_retention_rate, delta_min, delta_max
):
    
    df = cluster_df
    A = df['coeff_non_prix']
    B = df['coeff_prix']   
    N = len(df)
    
    
    def globalConstraint(delta, grad):
    # delta is the majorations vector, of size equal to total_df's length!

        retention_proba = 1-(1/(1+np.exp(-A-100*pd.Series(delta)*B)))

        if grad.size > 0:

            grad[:] = (
                (1/N)*(1-retention_proba)*B*100/(1+np.exp(A+B*pd.Series(delta)*100))
            ).values

        return min_retention_rate-(1/N)*retention_proba.sum()   
             
        
    lower_bound = [delta_min for i in range(N)]
    upper_bound = [delta_max for i in range(N)]
    delta_bounds = (lower_bound, upper_bound)
    
    return delta_bounds, globalConstraint<jupyter_output><empty_output><jupyter_text>We then define the following function which we will use to optimize the optimization problem **(a)** on the K "average individuals":(Note: we solve the problem for different initial coefficients, as the solution returned by the algorithm will depend on these initial values)<jupyter_code>def optimizeMarginForAverageClients(
    cluster_averages_df, cluster_dict, global_min_retention_rate, delta_min, delta_max, delta_init_list
):
    
    objectiveFunction = buildObjectiveFunctionForSpecificCluster(cluster_averages_df)
    #preconditioning = buildPreconditioningForSpecificCluster(cluster_averages_df)   #à tester
    delta_bounds, globalConstraint = buildConstraintsForAverageClients(
        cluster_dict, global_min_retention_rate, delta_min, delta_max
    )
    N = len(cluster_averages_df)

    opt = nlopt.opt(nlopt.LD_CCSAQ, N)
    opt.set_lower_bounds(delta_bounds[0])
    opt.set_upper_bounds(delta_bounds[1])
    opt.set_max_objective(objectiveFunction)
    #opt.set_precond_min_objective(objectiveFunction, preconditioning)   #à tester
    opt.add_inequality_constraint(globalConstraint)
    opt.set_xtol_abs(1e-20)
    
    print("Starting cluster average clients optimization.")
    
    delta_opt_for_various_delta0s = []
    margin_for_various_delta0s = []
    ineq_cons_for_various_delta0s = []
    
    # we run the solver of (a) for different initial coefficients and will keep the best result
    for delta_init_scalar in delta_init_list:
        
        rounded_delta_init_scalar = round(delta_init_scalar,4)
        delta0 = rounded_delta_init_scalar*np.ones(N)
        delta_opt = opt.optimize(delta0)
        margin = opt.last_optimum_value()
        ineq_cons = globalConstraint(delta_opt,np.zeros(N))
        print("Cluster average clients optimization done for "+str(rounded_delta_init_scalar)+" initial point!")
        print("Optimal margin for this init point: "+str(margin))
        print("Inequality constraint gap for this init point: "+str(ineq_cons)+" (has to be negative or quasi-null)")
        print("\n")
        
        delta_opt_for_various_delta0s.append(delta_opt)
        margin_for_various_delta0s.append(margin)
        ineq_cons_for_various_delta0s.append(ineq_cons)
        
    deltas = copy(delta_opt_for_various_delta0s)
    margins = copy(margin_for_various_delta0s)
    ineqs = copy(ineq_cons_for_various_delta0s)
    
    while True:
        
        if len(margins) == 0:
            
            i = ineq_cons_for_various_delta0s.index(min(ineq_cons_for_various_delta0s))
            print("No result found satisfies inequality constraint")
            print("Closest to acceptable margin: "+str(margin_for_various_delta0s[i]))
            print("\n")
            return margin_for_various_delta0s[i], delta_opt_for_various_delta0s[i]
        
        
        max_margin = max(margins)
        i = margins.index(max_margin)
        
        # our criterion to consider the constraints tight
        if ineqs[i] < 10**(-3):  
            
            print("Optimal margin: "+str(margins[i]))
            print("\n")
            return max_margin, deltas[i]
        
        else:
            
            deltas.pop(i)
            margins.pop(i)
            ineqs.pop(i)<jupyter_output><empty_output><jupyter_text>Test of the previously defined function solving problem **(a)** for a set of parameter values:<jupyter_code>delta_min, delta_max = -0.05, 0.1
step = 0.05
delta_init_list = np.arange(delta_min, delta_max+step, step)
global_min_retention_rate = 0.919
cluster_dict = resDict
margin, delta = optimizeMarginForAverageClients(
    cluster_averages_df, cluster_dict, global_min_retention_rate, delta_min, delta_max, delta_init_list
)<jupyter_output>Starting cluster average clients optimization.
Cluster average clients optimization done for -0.05 initial point!
Optimal margin for this init point: 94515239.42375694
Inequality constraint gap for this init point: 0.00031462672944160275 (has to be negative or quasi-null)


Cluster average clients optimization done for 0.0 initial point!
Optimal margin for this init point: 94253193.47011122
Inequality constraint gap for this init point: 4.5823592890181786e-06 (has to be negative or quasi-null)


Cluster average clients optimization done for 0.05 initial point!
Optimal margin for this init point: 94811555.42447811
Inequality constraint gap for this init point: 0.0006739724739438513 (has to be negative or quasi-null)


Cluster average clients optimization done for 0.1 initial point!
Optimal margin for this init point: 94280926.54697937
Inequality constraint gap for this init point: 4.486126373892674e-05 (has to be negative or quasi-null)


Optimal margin: 94811555.42447811<jupyter_text>We obtain here "a priori" a total profit of 94811555 euros. However, as seen in the report, this is not exactly the total margin obtained by prescribing the solution ```delta``` to our customers, solution which we found by optimizing problem **(a)**:<jupyter_code>delta<jupyter_output><empty_output><jupyter_text>Indeed, since problem **(a)** relies on the "average individuals" that have weighted feature values to compute objective function, the optimal objective returned by the algorithm is not exactly the optimal total profit here. We go back to the optimal total profit by simply applying a ```delta```[k] surcharge to all individuals of cluster k. We therefore define a function to calculate the real profit based on the rebate/surcharge coefficient ```delta```[k] prescribed to each cluster in the previously printed ```delta``` vector:<jupyter_code>def computeMarginEntireDataset(cluster_dict, delta_opt_for_each_cluster):
    
    res = 0
    K = len(cluster_dict)
    
    for k in range(K):
        
        df = cluster_dict[str(k)]
        n = len(df)
        objectiveFunction = buildObjectiveFunctionForSpecificCluster(df)
        delta_opt = delta_opt_for_each_cluster[k]*np.ones(n)
        
        res += objectiveFunction(delta_opt, np.empty((len(df),0)))
    
    return res

res = computeMarginEntireDataset(resDict, delta)
res<jupyter_output><empty_output><jupyter_text>The optimal total profit is thus 93150079 euros and not 94811555. Before turning to problem **(b)** which improves over problem **(a)**, let us finally analyze the histogram of the ```delta``` vector whose components indicate the rebate/surcharge to apply to each of the K clusters:<jupyter_code>plt.hist(list(delta), bins=10)
plt.title("Distribution of surcharge coefficients for the clusters")
plt.xlabel("Surcharge %")
plt.ylabel("Nb of occurences")
plt.savefig("### exportation path ###")
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that this histogram (although it does not multiply each ```delta```[k] component by n[k], the cardinal of cluster k) is not really polarized... There is still room for optimization! Hence the problem **(b)** right after **(a)** which we just tested.  3. Resolution of **(b)**: extending the results of **(a)** by solving per-cluster optimization subproblems Let us exploit the previous results by extracting the new averages of the retention probabilities in each cluster, obtained by prescribing the ```delta```[k] surcharge (found in the solution to **(a)**) to each cluster k.This will be used to modify the constraint of each subproblem and guarantee that the aggregated solution of all the subproblems remains a solution of the initial optimization problem. First, we define a function that allows, for each cluster k, to compute its new average retention probability for a ```delta```[k] surcharge:<jupyter_code>def getNewMeanRetentionRates(cluster_dict, delta_opt_for_each_cluster):
    
    K = len(cluster_dict)
    mean_retention_rate_for_delta_opt = np.zeros(K)
    
    for k in range(K):
        
        df = cluster_dict[str(k)]
        mean_retention_rate_for_delta_opt[k] = (
            1-df.apply(getResiliationProba, args=(delta_opt_for_each_cluster[k],), axis=1)
        ).mean()
        
    return mean_retention_rate_for_delta_opt
    
mean_retention_rate_for_delta_opt = getNewMeanRetentionRates(resDict, delta)<jupyter_output><empty_output><jupyter_text>We then define the function that will perform the successive optimization of the K subproblems (each subproblem being solved on one of the K clusters), starting from the initial point (```delta```[k],...,```delta```[k]) for each cluster k, and respecting the global constraint of inequality for a min retention threshold worth the kth average calculated just above.Note: as all subproblems are independent, it could have been possible to parallelize their resolution (ideally).<jupyter_code>def optimizeMarginForEntireDataset(
    cluster_dict, mean_retention_rate_for_delta_opt, delta_min, delta_max, delta_opt_for_each_cluster
):
    
    total_margin = 0
    delta_list = []
    K = len(cluster_dict)
    
    # we use the mean_retention_rate_delta_opt computed previously to formulate the constraint of each subproblem
    print('Starting cluster margin optimization.')
    for k in range(K):
        
        print("Starting cluster "+str(k)+" optimization.")
        cluster_df = cluster_dict[str(k)]
        objectiveFunction = buildObjectiveFunctionForSpecificCluster(cluster_df)
        delta_bounds, globalConstraint = buildConstraintsForSpecificCluster(
            cluster_df, mean_retention_rate_for_delta_opt[k], delta_min, delta_max
        )
        N = len(cluster_df)
        
        opt = nlopt.opt(nlopt.LD_CCSAQ, N)
        opt.set_lower_bounds(delta_bounds[0])
        opt.set_upper_bounds(delta_bounds[1])
        opt.set_max_objective(objectiveFunction)
        opt.add_inequality_constraint(globalConstraint)
        opt.set_xtol_rel(1e-7)
        
        delta0 = delta_opt_for_each_cluster[k]*np.ones(N)
        delta_opt = opt.optimize(delta0)
        maxf = opt.last_optimum_value()
        print("Cluster "+str(k)+" optimization done!")
        print("Optimal margin: "+str(maxf))
        print("Inequality constraint gap: "+str(globalConstraint(delta_opt,np.zeros(N)))+" (has to be negative or quasi-null)")
        
        total_margin += maxf
        delta_list.append(delta_opt)
        
    print('Optimization of total margin done!')
    print("\n")
    print("\n")
    
    return total_margin, np.concatenate(delta_list, axis=0)<jupyter_output><empty_output><jupyter_text>Let us try to run the solver of **(b)** defined just above on the example we used to test the solver of **(a)** (with a ```global_min_retention_rate``` of 0.919):<jupyter_code>total_margin, total_delta = optimizeMarginForEntireDataset(
    resDict, mean_retention_rate_for_delta_opt, delta_min, delta_max, delta
)<jupyter_output>Starting cluster margin optimization.
Starting cluster 0 optimization.
Cluster 0 optimization done!
Optimal margin: 1543493.4970666803
Inequality constraint gap: 1.4432899320127035e-15 (has to be negative or quasi-null)
Starting cluster 1 optimization.
Cluster 1 optimization done!
Optimal margin: 729457.9136564162
Inequality constraint gap: -1.1102230246251565e-15 (has to be negative or quasi-null)
Starting cluster 2 optimization.
Cluster 2 optimization done!
Optimal margin: 530911.706202318
Inequality constraint gap: -4.522067527767959e-06 (has to be negative or quasi-null)
Starting cluster 3 optimization.
Cluster 3 optimization done!
Optimal margin: 500804.1788422904
Inequality constraint gap: 4.440892098500626e-16 (has to be negative or quasi-null)
Starting cluster 4 optimization.
Cluster 4 optimization done!
Optimal margin: 365240.1050948454
Inequality constraint gap: 9.81783366316158e-06 (has to be negative or quasi-null)
Starting cluster 5 optimization.
Cluster 5 optimization don[...]<jupyter_text>Let us now display the histogram of the values of the aggregated surcharge vector ```delta```, obtained by concatenating the optimal surcharge vectors ```delta```_k of each of the K clusters:<jupyter_code>plt.hist(list(total_delta), bins=10)
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the vector of truly personalized surcharge coefficients (for the entire portfolio) is much more polarized here than in the solution of **(a)**.<jupyter_code>total_margin<jupyter_output><empty_output><jupyter_text>We were indeed able to squeeze out a few millions more in terms of total profit compared to the result obtained by solving **(a)**.Thus, extending the results of **(a)** by solving per-cluster optimization subproblems was worth it.  4. Drawing the "efficient frontier" using **(a)** and **(b)** To draw the efficient frontier, we want to solve problem **(a)** + all the per-cluster subproblems **for each** ```global_min_retention_rate``` threshold (which intervenes in the expression of the global constraint of the inital problem) **between 88% and 97%** **(b)**. Let us first set our study domain for ```global_min_retention_rate``` thresholds between 88% and 97%:<jupyter_code>minimal_min_retention_rate = 0.88
maximal_min_retention_rate = 0.97
step = 0.005
global_min_retention_rates = np.arange(minimal_min_retention_rate,maximal_min_retention_rate+step,step)
total_margins = np.zeros(len(global_min_retention_rates))
diffs = np.zeros(len(global_min_retention_rates))
cluster_dict, delta_min, delta_max = resDict, -0.05, 0.10<jupyter_output><empty_output><jupyter_text>Then, for each ```global_min_retention_rate``` threshold between 88% and 97%, we run:- the resolution of problem **(a)** (which leverages "average individuals" and prescribes the same surcharge coefficient to all the members of a cluster k)- the resolution of the K independent subproblems (each adapted and initialized using the solutions of **(a)**), i.e. **(b)**The function takes less than an hour to run.<jupyter_code>for i in range(len(total_margins)):
    
    global_min_retention_rate = round(global_min_retention_rates[i],4)
    
    print("*************")
    print("Starting optimization of margin for a "+str(global_min_retention_rate*100)+"% average retention rate")
    print("*************")
    print("\n")
    
    margin, delta = optimizeMarginForAverageClients(
        cluster_averages_df, cluster_dict, global_min_retention_rate, delta_min, delta_max, delta_init_list
    )
    total_margin_same_delta = computeMarginEntireDataset(resDict, delta)
    
    mean_retention_rate_for_delta_opt = getNewMeanRetentionRates(resDict, delta)
    total_margin, total_delta = optimizeMarginForEntireDataset(
        cluster_dict, mean_retention_rate_for_delta_opt, delta_min, delta_max, delta
    )

    total_margins[i] = total_margin
    diffs[i] = total_margin-total_margin_same_delta<jupyter_output>*************
Starting optimization of margin for a 88.0% average retention rate
*************


Starting cluster average clients optimization.
Cluster average clients optimization done for -0.05 initial point!
Optimal margin for this init point: 97309836.97090256
Inequality constraint gap for this init point: -0.03495592507125622 (has to be negative or quasi-null)


Cluster average clients optimization done for 0.0 initial point!
Optimal margin for this init point: 97309836.97090256
Inequality constraint gap for this init point: -0.03495592507125622 (has to be negative or quasi-null)


Cluster average clients optimization done for 0.05 initial point!
Optimal margin for this init point: 97309836.97090256
Inequality constraint gap for this init point: -0.03495592507125622 (has to be negative or quasi-null)


Cluster average clients optimization done for 0.1 initial point!
Optimal margin for this init point: 97309836.97090256
Inequality constraint gap for this init point: -0.034955925071[...]<jupyter_text>Graph of the "efficient frontier" (ie optimal total profit obtained for different ```global_min_retention_rate```s):<jupyter_code>plt.plot(global_min_retention_rates, total_margins)
plt.title("Efficient frontier of the optimal total profits")
plt.xlabel("Min global retention rate admittible")
plt.ylabel("Total profit over the portfolio")
plt.savefig("### exportation path ###")<jupyter_output><empty_output><jupyter_text>It was expected to see that the optimal total profit would reduce when we the min global retention rate imposed as a threshold in the global constraint increases. Indeed, the more we aim to keep a large proportion of clients in our portfolio, the more we have to prescribe fairly low surcharges for them (and even rebates), as most clients see their resiliation probability increase with the surcharge coefficient to prescribe.The leftward plateau can be explained by the fact that, at a min global retention rate of 0.91, all surcharge coefficients are saturated at 10%, so the total profit cannot be increased anymore. The rightward plateau can be explained by the fact that, at a min global retention rate of 0.94, all surcharge coefficients are saturated at -5%, so the total profit cannot be decreased anymore.  Optimal total profits obtained for the different global min retention rates **using only (a)** (and not the subsequent subproblems):<jupyter_code>[-diffs[i]+total_margins[i] for i in range(len(total_margins))]<jupyter_output><empty_output><jupyter_text>Graph of the gain extracted using the per-cluster subproblems (i.e. **(b)**) right after **(a)**:<jupyter_code>plt.plot(global_min_retention_rates, diffs)
plt.title("Profit difference between (b) and (a)")
plt.xlabel("Min global retention rate admittible")
plt.ylabel("Profit difference on the whole portfolio between (b) and (a) ((b) builds upon (a))")
plt.savefig("### exportation path ###")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08943476350856641, 'fraction_numerical': 0.026797481329623663, 'mean_word_length': 4.494167337087691, 'pattern_counts': {'":': 1, '<': 102, '<?xml version=': 0, '>': 105, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '58230', 'license': 'MIT', 'n_tokens_mistral': 8725, 'n_tokens_neox': 8065, 'n_words': 2504, 'path': 'prescriptive_deterministic_divide&conquer.ipynb', 'repo_name': 'mathieu-sibue/optimizing-insurance-renewal-prices', 'size': '27316'}