<jupyter_start><jupyter_text>Preparando os dadosDados em [UCI Repository](https://archive.ics.uci.edu/ml/datasets/Sales_Transactions_Dataset_Weekly).   este estudo é baseado no Workshop de Time-Series do curso do Mario Filho<jupyter_code>data = pd.read_csv('Sales_Transactions_Dataset_Weekly.csv')
# para esquecer das colunas normalizadas
data = data.filter(regex=r'Product|W')
data.head()<jupyter_output><empty_output><jupyter_text>Usando a função melt do pandas para deixar o dataframe em um formato melhor.<jupyter_code>melt = data.melt(id_vars='Product_Code', var_name='Week', value_name='Sales')

# regex para pegar somente os números do código do produto
melt['Product_Code'] = melt['Product_Code'].str.extract('(\d+)', expand=False).astype(int)
# regex para pegar somente os números da semana
melt['Week'] = melt['Week'].str.extract('(\d+)', expand=False).astype(int)

# ordenando os valores por semana e código do produto
melt = melt.sort_values(['Week', 'Product_Code'])
# dimensões do dataframe
print(melt.shape)
melt.head()<jupyter_output>(42172, 3)<jupyter_text>simples de variáveis simples<jupyter_code>melt2 = melt.copy()
# o argumento default de shift é 1
# vendas da última semana por produto: pegando as vendas da última semana
melt2['Sales_Ultima_Semana'] = melt2.groupby(['Product_Code'])['Sales'].shift(1)
# vendas da última semana por produto: pegando a diferença das vendas da última semana
melt2['Diff_Sales_Ultima_Semana'] = melt2.groupby(['Product_Code'])['Sales_Ultima_Semana'].diff()
# tirando os dados que contém nan
# tirando os valores faltantes. temos poucos features
melt2 = melt2.dropna()
melt2.head()
melt2['Week'].value_counts().sort_values()<jupyter_output><empty_output><jupyter_text>Os valores de venda para cada semana são igualmente distribuídos!   Baseline e Validação: obtendo um resultado simples<jupyter_code># root mean squared log error metrics
def rmsle(ytrue, ypred):
    return np.sqrt(mean_squared_log_error(ytrue, ypred))<jupyter_output><empty_output><jupyter_text>baseline simples usando o valor da semana a ser previsto como sendo o valor da semana anterior!<jupyter_code>mean_error = []
for week in range(40,52):
    # treina nas semanas anteriores 
    train = melt2[melt2['Week'] < week]
    # valida na semana atual
    val = melt2[melt2['Week'] == week]
    
    p = val['Sales_Ultima_Semana'].values
    
    error = rmsle(val['Sales'].values, p)
    print('Semana %d - Erro %.5f' % (week, error))
    mean_error.append(error)
print('Média Erro = %.5f' % np.mean(mean_error))
# fazendo um histograma do número de vendas
melt2['Sales'].hist(bins=40, figsize=(8,5))<jupyter_output><empty_output><jupyter_text>as vendas estão concentradas no intervalo 0-10  Criando modelos interessantes!<jupyter_code>melt4 = melt.copy()
melt4['Sales_Ultima_Semana'] = melt4.groupby(['Product_Code'])['Sales'].shift()
melt4['Media_movel_1'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana'].rolling(1).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_Ultima_Semana'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana'].diff()
melt4['Sales_Ultima_Semana-1'] = melt4.groupby(['Product_Code'])['Sales'].shift(2)
melt4['Media_movel_2'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-1'].rolling(2).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_Ultima_Semana-1'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-1'].diff()
melt4['Sales_Ultima_Semana-2'] = melt4.groupby(['Product_Code'])['Sales'].shift(3)
melt4['Media_movel_3'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-2'].rolling(3).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_Ultima_Semana-2'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-2'].diff()
melt4 = melt4.dropna()
melt4.tail()

mean_error = []
for week in range(40,52):
    train = melt4[melt4['Week'] < week]
    val = melt4[melt4['Week'] == week]
    
    xtr, xts = train.drop(['Sales'], axis=1), val.drop(['Sales'], axis=1)
    ytr, yts = train['Sales'].values, val['Sales'].values
    
    mdl = RandomForestRegressor(n_estimators=1000, n_jobs=-1, random_state=0)
    mdl.fit(xtr, ytr)
    
    p = mdl.predict(xts)
    
    error = rmsle(yts, p)
    print('Semana %d - Erro %.5f' % (week, error))
    mean_error.append(error)
print('Média Erro = %.5f' % np.mean(mean_error))<jupyter_output>Semana 40 - Erro 0.42007
Semana 41 - Erro 0.42680
Semana 42 - Erro 0.42025
Semana 43 - Erro 0.41694
Semana 44 - Erro 0.42772
Semana 45 - Erro 0.39780
Semana 46 - Erro 0.42112
Semana 47 - Erro 0.42974
Semana 48 - Erro 0.41538
Semana 49 - Erro 0.40337
Semana 50 - Erro 0.44168
Semana 51 - Erro 0.44709
Média Erro = 0.42233<jupyter_text>Modificando a função otimizada (e distribuição do alvo)  temos valores mais uniformemente distribuídos<jupyter_code>np.log1p(melt4['Sales']).hist(bins=40, figsize=(10,5))<jupyter_output><empty_output><jupyter_text>recalculando!<jupyter_code>mean_error = []
for week in range(40,52):
    train = melt4[melt4['Week'] < week]
    val = melt4[melt4['Week'] == week]
    
    xtr, xts = train.drop(['Sales'], axis=1), val.drop(['Sales'], axis=1)
    ytr, yts = train['Sales'].values, val['Sales'].values
    
    mdl = RandomForestRegressor(n_estimators=1000, n_jobs=-1, random_state=0)
    mdl.fit(xtr, np.log1p(ytr))
    
    p = np.expm1(mdl.predict(xts))
    
    error = rmsle(yts, p)
    print('Week %d - Error %.5f' % (week, error))
    mean_error.append(error)
print('Média Erro = %.5f' % np.mean(mean_error))<jupyter_output>Week 40 - Error 0.40767
Week 41 - Error 0.40515
Week 42 - Error 0.40197
Week 43 - Error 0.39968
Week 44 - Error 0.41825
Week 45 - Error 0.38842
Week 46 - Error 0.41281
Week 47 - Error 0.39931
Week 48 - Error 0.40716
Week 49 - Error 0.40968
Week 50 - Error 0.43622
Week 51 - Error 0.46366
Média Erro = 0.41250<jupyter_text>engraçado, piorou muito nos últimas semanas!!!  Um modelo mais sofisticado<jupyter_code>mean_error = []
for week in range(40,52):
    train = melt4[melt4['Week'] < week]
    val = melt4[melt4['Week'] == week]
    
    xtr, xts = train.drop(['Sales'], axis=1), val.drop(['Sales'], axis=1)
    ytr, yts = train['Sales'].values, val['Sales'].values
    
    mdl = LGBMRegressor(n_estimators=1000, learning_rate=0.01)
    mdl.fit(xtr, np.log1p(ytr))
    
    p = np.expm1(mdl.predict(xts))
    
    error = rmsle(yts, p)
    print('Semana %d - Erro %.5f' % (week, error))
    mean_error.append(error)
print('Média Erro = %.5f' % np.mean(mean_error))<jupyter_output>Semana 40 - Erro 0.40073
Semana 41 - Erro 0.38563
Semana 42 - Erro 0.38627
Semana 43 - Erro 0.38782
Semana 44 - Erro 0.40600
Semana 45 - Erro 0.37642
Semana 46 - Erro 0.39674
Semana 47 - Erro 0.38189
Semana 48 - Erro 0.39265
Semana 49 - Erro 0.39501
Semana 50 - Erro 0.41345
Semana 51 - Erro 0.44514
Média Erro = 0.39731<jupyter_text>colocando o target como sqrt(x)<jupyter_code>mean_error = []
for week in range(40,52):
    train = melt4[melt4['Week'] < week]
    val = melt4[melt4['Week'] == week]
    
    xtr, xts = train.drop(['Sales'], axis=1), val.drop(['Sales'], axis=1)
    ytr, yts = train['Sales'].values, val['Sales'].values
    
    mdl = LGBMRegressor(n_estimators=1000, learning_rate=0.01)
    mdl.fit(xtr, np.sqrt(ytr))
    
    p = (mdl.predict(xts))**2
    
    error = rmsle(yts, p)
    print('Semana %d - Erro %.5f' % (week, error))
    mean_error.append(error)
print('Média Erro = %.5f' % np.mean(mean_error))<jupyter_output>Semana 40 - Erro 0.41173
Semana 41 - Erro 0.38888
Semana 42 - Erro 0.39358
Semana 43 - Erro 0.39307
Semana 44 - Erro 0.41669
Semana 45 - Erro 0.38529
Semana 46 - Erro 0.40727
Semana 47 - Erro 0.37940
Semana 48 - Erro 0.39922
Semana 49 - Erro 0.40028
Semana 50 - Erro 0.41889
Semana 51 - Erro 0.44841
Média Erro = 0.40356<jupyter_text>log para o target é melhor!<jupyter_code>import matplotlib.pyplot as plt
val.loc[:, 'Prediction'] = np.round(p)
val.plot.scatter(x='Prediction', y='Sales', figsize=(15,10), title='Prediction vs Sales', 
                 ylim=(0,40), xlim=(0,40))

xx=[0,10,20,30,40]
yy=[0,10,20,30,40]
plt.plot(xx,yy,linewidth=5,color='black')<jupyter_output><empty_output><jupyter_text>o modelo está errando mais para vendas de 20 unidades.<jupyter_code>sel.shape
df_aux=melt.groupby(['Week'])['Sales'].sum()

#df_aux
melt.shape[0]
melt['Week'][0]
df_aux=melt.groupby(['Week'])['Sales'].sum()
list_week=[]
for i in range(melt.shape[0]):
    week=melt['Week'][i]
    list_week.append(df_aux[week])
    

#list_week
melt4 = melt.copy()
melt4['Sales_Ultima_Semana'] = melt4.groupby(['Product_Code'])['Sales'].shift()
melt4['Media_movel_1'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana'].rolling(1).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_Ultima_Semana'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana'].diff()

#melt4['SalesperWeek']=melt4.groupby(['Week'])['Sales'].shift()
#melt4['PerweekMedia_movel_1'] = melt4.groupby(['Product_Code'])['SalesperWeek'].rolling(2).mean().reset_index(level=0, drop=True)
#melt4['Perweek_Diff_Sales_Ultima_Semana'] = melt4.groupby(['Product_Code'])['SalesperWeek'].diff()
df_aux=melt.groupby(['Week'])['Sales'].sum()
list_week=[]
for i in range(melt.shape[0]):
    week=melt['Week'][i]
    list_week.append(df_aux[week])
melt4['weekallsales']=list_week

melt4['weekallsales'] = melt4.groupby(['Week'])['weekallsales'].shift()
melt4['Media_movel_weeksales'] = melt4.groupby(['Week'])['weekallsales'].rolling(2).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_weekallsales'] = melt4.groupby(['Week'])['weekallsales'].diff()


melt4['Sales_Ultima_Semana-1'] = melt4.groupby(['Product_Code'])['Sales'].shift(2)
melt4['Media_movel_2'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-1'].rolling(2).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_Ultima_Semana-1'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-1'].diff()

#melt4['SalesperWeek_2']=melt4.groupby(['Week'])['Sales'].shift(2)
#melt4['PerweekMedia_movel_2'] = melt4.groupby(['Product_Code'])['SalesperWeek'].rolling(4).mean().reset_index(level=0, drop=True)
#melt4['Perweek_Diff_Sales_Ultima_Semana_2'] = melt4.groupby(['Product_Code'])['SalesperWeek'].diff()


melt4['weekallsales_2'] = melt4.groupby(['Week'])['weekallsales'].shift(2)
melt4['Media_movel_weeksales_2'] = melt4.groupby(['Week'])['weekallsales'].rolling(4).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_weekallsales_2'] = melt4.groupby(['Week'])['weekallsales'].diff(2)


melt4['Sales_Ultima_Semana-2'] = melt4.groupby(['Product_Code'])['Sales'].shift(3)
melt4['Media_movel_3'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-2'].rolling(3).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_Ultima_Semana-2'] = melt4.groupby(['Product_Code'])['Sales_Ultima_Semana-2'].diff()

melt4['weekallsales_3'] = melt4.groupby(['Week'])['weekallsales'].shift(3)
melt4['Media_movel_weeksales_3'] = melt4.groupby(['Week'])['weekallsales'].rolling(8).mean().reset_index(level=0, drop=True)
melt4['Diff_Sales_weekallsales_3'] = melt4.groupby(['Week'])['weekallsales'].diff(3)



melt4 = melt4.dropna()
melt4.tail()


#melt4['SalesperWeek_3']=melt4.groupby(['Week'])['Sales'].shift(3)
#melt4['PerweekMedia_movel_3'] = melt4.groupby(['Product_Code'])['SalesperWeek'].rolling(6).mean().reset_index(level=0, drop=True)
#melt4['Perweek_Diff_Sales_Ultima_Semana_3'] = melt4.groupby(['Product_Code'])['SalesperWeek'].diff()


mean_error = []
for week in range(40,52):
    train = melt4[melt4['Week'] < week]
    val = melt4[melt4['Week'] == week]
    
    xtr, xts = train.drop(['Sales'], axis=1), val.drop(['Sales'], axis=1)
    ytr, yts = train['Sales'].values, val['Sales'].values
    
    mdl = LGBMRegressor(n_estimators=1000, learning_rate=0.01)
    mdl.fit(xtr, np.log1p(ytr))
    
    p = np.expm1(mdl.predict(xts))
    
    error = rmsle(yts, p)
    print('Semana %d - Erro %.5f' % (week, error))
    mean_error.append(error)
print('Média Erro = %.5f' % np.mean(mean_error))
melt<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17181499415595258, 'fraction_numerical': 0.06136249791284021, 'mean_word_length': 5.699664429530201, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '619165', 'license': 'Unlicense', 'n_tokens_mistral': 5248, 'n_tokens_neox': 4745, 'n_words': 819, 'path': 'previsaodedemanda_vagner_v1.ipynb', 'repo_name': 'vzeizer/Workshop_TimeSeries', 'size': '11978'}