<jupyter_start><jupyter_text>Example of use of the Concurrent_neural_network package<jupyter_code>import pandas as pd
import numpy as np
import datetime as dt
import os
from Concurrent_Neural_Network.preprocessing import filter_index_from_dataframe, compute_proportion, add_temporal_features, dataframe_to_data_loader
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F<jupyter_output><empty_output><jupyter_text>I DataThe data we use are 92400 weekly sales of 550 smartphones on CDiscount website. It has 5 columns: - product_id : the name of the product - monday_date : the monday of the week (week are taken from monday to sunday) - sales : number of sales of the week - price : average price of the week - margin : average relative margin of the week  I.A Reading Data<jupyter_code>CUR_DIR = os.getcwd()
DATA_FILE = os.path.join(CUR_DIR,'data','smartphone_data.csv' )
data =  pd.read_csv(DATA_FILE, sep =';')
data.set_index(['product_id', 'monday_date'], inplace= True)<jupyter_output><empty_output><jupyter_text>I.B Preprocess_dataWe first use the function filter_index_from_dataframe to filter only the interesting products (with enough sales and enough data)<jupyter_code># filter parameters

minimal_sum=100 # A product must have at least minimal_sum to be considered
minimal_positive_length=15 # A product must have been sold during at least minimal_positive_length to be considered
filtered_data, products =  filter_index_from_dataframe(data, "sales", minimal_sum_target=minimal_sum, minimal_positive_length=minimal_positive_length )

print('Number of products before filter : %s' % len(set(data.index.get_level_values(0))) )
print('Number of products after filter : %s' % len(products) )
print('Number of sales before filter : %s' % sum(data['sales']))
print('Number of sales after filter : %s' % sum(filtered_data['sales']))
print('Keeped_sales_pourcentages: %.3f  ' %(100 * sum(filtered_data['sales'])/sum(data['sales'])) )<jupyter_output>Number of products before filter : 1304
Number of products after filter : 1055
Number of sales before filter : 1434188.0
Number of sales after filter : 1403184.0
Keeped_sales_pourcentages: 97.838<jupyter_text>We then compute the market share of the products (we scale it to 100) :<jupyter_code>market_shares, weekly_global_sales, products = compute_proportion(filtered_data, 'sales', resize=1)<jupyter_output><empty_output><jupyter_text>I.C Exploration of the sales dataHere we present some visualisations of the data<jupyter_code># temporal feature manipulation

def format_time_index(df, index_col_name):
    adf = df.reset_index()
    adf[index_col_name] = adf[index_col_name].apply(lambda x: dt.datetime.strptime(x,'%Y-%m-%d'))
    return adf.set_index(index_col_name)

f_weekly_global_sales = format_time_index(weekly_global_sales, 'monday_date')
f_weekly_total_sales  = format_time_index(data['sales'].groupby(level=1).sum(),'monday_date')
plt.figure(figsize=(12,7))
plt.plot(f_weekly_global_sales, label = 'Selected_sales')
plt.plot(f_weekly_total_sales, label = 'Total_sales')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>There is a decreasing trend. It is caused by the way the products are selected. We will therefore ignore the last week.  II Learning II.A Data Learning Setting<jupyter_code>#parameters
horizon = 6 #Prediction horizon
features = ['proportion_shift_6', 'proportion_shift_7','price', 'margin'] # Here are the features used for prediction
target ='proportion'
n_input = len(features)
data_restrained = add_temporal_features(market_shares, horizon=horizon)
data_restrained['price'] = data_restrained['price'] /100
data_restrained.head()
date_learning =  '2020-06-01' # This date separate learning & testing set
date_stop = '2021_01_01' # This is the end of the testing set
X_train, X_test = data_restrained[data_restrained.index.get_level_values(1) < date_learning ], data_restrained[data_restrained.index.get_level_values(1) >= date_learning ]
X_test = X_test[X_test.index.get_level_values(1) < date_stop]
data_train_loader = dataframe_to_data_loader(X_train,features,target) 
data_test_loader = dataframe_to_data_loader(X_test,features,target)<jupyter_output><empty_output><jupyter_text>Now we present the prediction MAPE of some simple predictor :    - null : 0    - simple shift : predicting X_{t-h} for X_t    - shifted 2-mean :  predicting (X_{t-h}+X_{t-h-1})/2  for X_tNote that due to the scaling factor present on the Concurrent Neural network model they are not strictly equivalent, due to the arrival of new products. It gives however an reference<jupyter_code>#Some reference_MAPE
print('Null MAPE %.2f' %   (100*sum(abs(  X_test['proportion']))/ sum(X_test['proportion'])))
print('Simple_shift MAPE %.2f' %   (100  *sum(abs(X_test['proportion_shift_6'] - X_test['proportion']))/ sum(X_test['proportion'])))
print('Simple 2-mean MAPE %.2f' %  (100  *sum(abs(X_test['proportion_shift_6']/2 + X_test['proportion_shift_7']/2 - X_test['proportion']))/ sum(X_test['proportion'])))<jupyter_output>Null MAPE 100.00
Simple_shift MAPE 88.65
Simple 2-mean MAPE 86.96<jupyter_text>II.B Inner Neural Network modelHere we define the inner submodel we will use to compute the "competitiveness function". We stay simple and use a 2 layered neural network<jupyter_code>class Feed_forward_model(nn.Module):
    "2 layers neural network used for for testing"

    def __init__(self, n_input, n_hidden):
        super().__init__()
        self.linear_1 = nn.Linear(n_input, n_hidden, bias=False)
        self.linear_2 = nn.Linear(n_hidden, 1, bias=False)
        self.n_input = n_input
        self.n_hidden = n_hidden

    def forward(self, x):
        return F.softplus(self.linear_2(F.relu(self.linear_1(x))), )

n_hidden = 4
submodel = Feed_forward_model(n_input,n_hidden)<jupyter_output><empty_output><jupyter_text>II.C Concurrent neural networkHere is an example using a Concurrent neural network defined in the previous session Note: sometimes learning doesn't work, as the prediction stay constant. In this case, you should recompile both this and previous section (this is due to a bad inialisation of the submodel).<jupyter_code>from Concurrent_Neural_Network.models import Concurrent_Module

learning_rate = 0.005
epochs = 200
model= Concurrent_Module(submodel, sum_factor=1, loss='L1', learning_rate =learning_rate)
model.train(data_train_loader,eval_dataset=data_test_loader, max_epochs=epochs, batch_print = epochs//10)<jupyter_output>Epoch 0
Train MAPE: 133.7591
Test MAPE: 120.7434 

Epoch 20
Train MAPE: 95.8357
Test MAPE: 87.9909 

Epoch 40
Train MAPE: 91.4303
Test MAPE: 83.2091 

Epoch 60
Train MAPE: 89.9918
Test MAPE: 82.3293 

Epoch 80
Train MAPE: 89.7786
Test MAPE: 82.6094 

Epoch 100
Train MAPE: 89.4753
Test MAPE: 82.2045 

Epoch 120
Train MAPE: 89.4623
Test MAPE: 82.8056 

Epoch 140
Train MAPE: 89.4397
Test MAPE: 84.1233 

Epoch 160
Train MAPE: 89.1585
Test MAPE: 83.5027 

Epoch 180
Train MAPE: 89.1718
Test MAPE: 82.6605<jupyter_text>Concurrent Model MAPE<jupyter_code>_ = model.eval(data_test_loader)<jupyter_output>Test MAPE: 82.0284<jupyter_text>III Analysis of the modelWe show in this section how to use the model we present in the last parts II.A Retrieving results from the model :In this section, an example of output is provided and the prediction plotted. We first need some function to retrieve the comportement for a unique product<jupyter_code>def data_to_position_dict(data, prod):
    date_index = list(set(data.index.get_level_values(1)))#List of the date existing in the dataset
    date_index.sort()
    dict_index_prod = dict()
    for i,dt in enumerate(date_index):
        sub_data = data[data.index.get_level_values(1) == dt]
        l_prod_dt = list(sub_data.index.get_level_values(0))
        if prod in l_prod_dt:
            dict_index_prod[i] = l_prod_dt.index(prod)
    return dict_index_prod

def get_prediction_series(model, dict_index_prod):
    l_prevs = []
    for key in dict_index_prod.keys():
        l_prevs.append(float(model(data_test_loader[key][0])[dict_index_prod[key]]))
    return l_prevs
    
prod = 'HUAWPSMART64BLEU' # The product sales and prediction we want to plot
dict_index_prod = data_to_position_dict(X_test, prod)
sales_product_series = X_test.loc[prod]['proportion']
pred_product_series = pd.Series( data = get_prediction_series(model, dict_index_prod), index = sales_product_series.index)

plt.figure(figsize=(12,7))
plt.plot(format_time_index(sales_product_series, 'monday_date'), label ='market_share')
plt.plot(format_time_index(pred_product_series, 'monday_date'), label ='prediction')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Remember : it is a 6 week in advance prediction  II.B Covariates influenceIn this section, we compile partial dependance of the competitivity of the submodel<jupyter_code># Let's recall the feature we use :
features
selected_features = "margin"
bins = 70
_, selected_fet_bin = pd.qcut(X_train[selected_features], bins, duplicates='drop', retbins=True)
avg_imp = []
lis_x = []
for j,b in  enumerate(selected_fet_bin[:-1]):
    X_test_rep = X_test.copy()
    X_test_rep[selected_features] = (b+selected_fet_bin[j+1])/2
    lis_x.append((b+selected_fet_bin[j+1])/2)
    tt_enter =torch.tensor(X_test_rep[features].values.astype(np.float32)) 
    comp = submodel(tt_enter)
    avg_imp.append(sum(comp)/len(comp))
plt.figure(figsize=(12, 7))
plt.plot( lis_x, avg_imp, '+')
#plt.xscale('log')
#plt.yscale('log')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10063025210084034, 'fraction_numerical': 0.032563025210084036, 'mean_word_length': 6.011045655375552, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '533416', 'license': 'MIT', 'n_tokens_mistral': 3521, 'n_tokens_neox': 3171, 'n_words': 847, 'path': 'example_notebook.ipynb', 'repo_name': 'garnier94/Concurrent_Neural_NEtwork', 'size': '9520'}