<jupyter_start><jupyter_text>**Setup**  Clone the dataset<jupyter_code># Clone commodity-predictor repo.
!git clone -l -s git://github.com/ferdinand-lanvino/commodity-predictor.git
%cd commodity-predictor
!ls<jupyter_output>Cloning into 'commodity-predictor'...
warning: --local is ignored
remote: Enumerating objects: 141, done.[K
remote: Counting objects:   0% (1/141)[K
remote: Counting objects:   1% (2/141)[K
remote: Counting objects:   2% (3/141)[K
remote: Counting objects:   3% (5/141)[K
remote: Counting objects:   4% (6/141)[K
remote: Counting objects:   5% (8/141)[K
remote: Counting objects:   6% (9/141)[K
remote: Counting objects:   7% (10/141)[K
remote: Counting objects:   8% (12/141)[K
remote: Counting objects:   9% (13/141)[K
remote: Counting objects:  10% (15/141)[K
remote: Counting objects:  11% (16/141)[K
remote: Counting objects:  12% (17/141)[K
remote: Counting objects:  13% (19/141)[K
remote: Counting objects:  14% (20/141)[K
remote: Counting objects:  15% (22/141)[K
remote: Counting objects:  16% (23/141)[K
remote: Counting objects:  17% (24/141)[K
remote: Counting objects:  18% (26/141)[K
remote: Counting objects:  19% (27/141)[K
remote: Counting objects:  20% (29/141[...]<jupyter_text>Pull the updates if needed<jupyter_code>!git pull<jupyter_output>Already up to date.<jupyter_text>Import Dependency<jupyter_code>#basic packages
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O
import datetime # manipulating date formats

#visualization
import matplotlib.pylab as plt # basic plotting
import seaborn as sns # for prettier plots

#statistic
from statsmodels.tsa.stattools import adfuller
from statsmodels.tsa.stattools import acf, pacf
from statsmodels.tsa.seasonal import seasonal_decompose
from statsmodels.tsa.arima_model import ARIMA
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 14, 9<jupyter_output><empty_output><jupyter_text>**Preprocessing**  Import combined dataset located in Dataset/Converted/commodity.csv<jupyter_code># pd.set_option('display.max_rows', 2000)
#Importing dataset 
dataset = pd.read_csv('Dataset/Converted/commodity.csv')
dataset['tanggal'] = pd.to_datetime(dataset['tanggal'],format='%d/%m/%Y')
#Drop all NaN values
dataset = dataset.dropna(how='any',axis=0)
print(dataset)<jupyter_output>komoditas    tanggal    harga
0                 Beras 2016-07-25  10900.0
1                 Beras 2016-07-26  10950.0
2                 Beras 2016-07-27  10950.0
3                 Beras 2016-07-28  10950.0
4                 Beras 2016-07-29  10950.0
...                 ...        ...      ...
29327  <NAME> 2020-05-20  17500.0
29328  <NAME> 2020-05-26  17650.0
29329  <NAME> 2020-05-27  17650.0
29330  <NAME> 2020-05-28  17650.0
29331  <NAME> 2020-05-29  17650.0

[29242 rows x 3 columns]<jupyter_text>Grouping dataset by commodity type and then aggregating it by min-max date and min-max price<jupyter_code>monthly_data = dataset.groupby("komoditas")["tanggal","harga"].agg({"tanggal":["min","max"]})
monthly_data<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.
  """Entry point for launching an IPython kernel.<jupyter_text>Creating method for imputating dataset per commodity type<jupyter_code>def imputating(commodity_name,dataset):
  print("-----------------------------------")
  print("Imputing "+commodity_name)

  #Get sub dataset per commodity type
  subdataset = dataset[dataset.komoditas==commodity_name]
  #Set index by tanggal (date)
  subdataset = subdataset[['tanggal','harga']].set_index('tanggal')
  #reindex
  subdataset.index = pd.DatetimeIndex(subdataset.index)

  subdataset_imputed = subdataset.asfreq('D')
  #filling NaN values with LOCF (Last Observation Carried Forward) algorithm using method Forward Fill
  subdataset_imputed['harga-imputed'] = subdataset.asfreq('D', method='ffill')
  plt.plot(subdataset_imputed)
  plt.title(commodity_name)
  plt.xlabel('Tanggal (Harian)')
  plt.ylabel('Harga (Kg)')
  subdataset_imputed = subdataset_imputed.drop(['harga'], axis = 1) 
  print(subdataset_imputed.info())
  return subdataset_imputed<jupyter_output><empty_output><jupyter_text>Running the imputation<jupyter_code>subdataset = imputating("Cabai Rawit Merah",dataset)<jupyter_output>-----------------------------------
Imputing Cabai Rawit Merah
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1405 entries, 2016-07-25 to 2020-05-29
Freq: D
Data columns (total 1 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   harga-imputed  1405 non-null   float64
dtypes: float64(1)
memory usage: 22.0 KB
None<jupyter_text>Rolling Windows<jupyter_code>subdataset_rollmean = subdataset['harga-imputed'].rolling(7, center=True).mean()
subdataset_rollstd = subdataset['harga-imputed'].rolling(7, center=True).std()
plt.plot(subdataset_rollmean)
plt.title("Rolling Mean")
plt.xlabel('Tanggal (Harian)')
plt.ylabel('Harga (Kg)')<jupyter_output><empty_output><jupyter_text>Rolling Mean & Std<jupyter_code>orig = plt.plot(subdataset, color='blue', label='Original')
mean = plt.plot(subdataset_rollmean, color='red', label='Rolling Mean')
std = plt.plot(subdataset_rollstd, color='black', label='Rolling Std')
plt.legend(loc='best')
plt.title('Rolling Mean & Standard Deviation dari Cabai Merah')
plt.show(block=False)<jupyter_output><empty_output><jupyter_text>**ACDF Test**<jupyter_code>print('Results of Dickey Fuller Test:')
dftest = adfuller(subdataset['harga-imputed'], autolag='AIC')

dfoutput = pd.Series(dftest[0:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])
for key,value in dftest[4].items():
    dfoutput['Critical Value (%s)'%key] = value
    
print(dfoutput)<jupyter_output>Results of Dickey Fuller Test:
Test Statistic                   -3.238660
p-value                           0.017853
#Lags Used                       14.000000
Number of Observations Used    1390.000000
Critical Value (1%)              -3.435063
Critical Value (5%)              -2.863622
Critical Value (10%)             -2.567878
dtype: float64<jupyter_text>**Data Transformation to achieve Stationarity**<jupyter_code>indexedDataset_logScale = np.log(subdataset)
plt.plot(indexedDataset_logScale)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1421889616463985, 'fraction_numerical': 0.065325849703773, 'mean_word_length': 4.39983164983165, 'pattern_counts': {'":': 1, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '317776', 'license': 'MIT', 'n_tokens_mistral': 2541, 'n_tokens_neox': 2231, 'n_words': 510, 'path': 'commodity_predictor_research.ipynb', 'repo_name': 'ferdinand-lanvino/AES-Encryption-Classes', 'size': '6464'}