<jupyter_start><jupyter_text>Purchase Time Series PreparationThe main purpose here is to prepare for sample data to try different forecasting methods later.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
from pandas.plotting import autocorrelation_plot
from statsmodels.graphics.tsaplots import plot_pacf<jupyter_output><empty_output><jupyter_text>Prepare Sample Data* Find specific merchants, create training, testing<jupyter_code>ts_purchase_df = pd.read_pickle('ts_purchase.pkl')
print(ts_purchase_df.shape, ts_purchase_df['merchant'].nunique())

ts_purchase_df.head()
agg_df = ts_purchase_df[['merchant', 'purchase_date']].astype('str').drop_duplicates()\
          .groupby(['merchant'], as_index=False)['purchase_date']\
          .agg(['count']).reset_index()\
          .sort_values(['count'], ascending=False)

print(min(agg_df['count']), max(agg_df['count']))
agg_df.head()<jupyter_output>110 114<jupyter_text>Yeah, I assigned the purchase dates in uniformal distribution. This mean choose whatever merchant here is similar.  Sample 1<jupyter_code>sample_ts_df = ts_purchase_df.loc[ts_purchase_df['merchant']=='49th Parallel Grocery']
print(sample_ts_df.shape)

sample_ts_df.head()
import warnings
warnings.filterwarnings("ignore")

sample_ts_df['purchase_date'] = sample_ts_df['purchase_date'].astype('datetime64[ns]')

print(sample_ts_df.dtypes)
sample_ts_df.head()
# Plotting data
sample_ts_df.index = sample_ts_df['purchase_date']
sample_ts_df['purchase_amount'].plot(figsize=(10,4), title= 'Daily Purchase $', fontsize=14, color='green')
plt.show()
sample_ts_df['client_count'].plot(figsize=(10, 4), title= 'Daily Clients', fontsize=14, color='purple')
plt.show()
# Autocorrelation plot
autocorrelation_plot(sample_ts_df['purchase_amount'])
plt.show()
plot_pacf(sample_ts_df['purchase_amount'], lags=20);
autocorrelation_plot(sample_ts_df['client_count'])
plt.show()
plot_pacf(sample_ts_df['client_count'], lags=20);<jupyter_output><empty_output><jupyter_text>Notes* In the data, the pattern of client_count and purchase_amount are almost the same.* The autocorrelation plot (ACF)  * It summarizes the relationship between an observation in a time series with observations at prior time steps. The relationship here include direct and indirect correlations.  * From above plots, it's exponential, decaying to 0, so it can be AR (Autoregressive) model* The partial autocorrelation (PACF)  * It is similar to autocorrelation plot, but removes those indirect correlations.* We can start with `AR(2)` or `AR(3)`, since it's showing the first 2,3 lags have strong correlations.<jupyter_code>sample_ts_df.drop(['merchant', 'purchase_date'], inplace=True, axis=1)
sample_train_df = sample_ts_df.iloc[0:100, :]
sample_test_df = sample_ts_df.iloc[100:, :]

print(sample_train_df.shape, sample_test_df.shape)
sample_train_df.head()
sample_train_df.to_pickle('sample_train_df1.pkl')
sample_test_df.to_pickle('sample_test_df1.pkl')<jupyter_output><empty_output><jupyter_text>Sample 2<jupyter_code>sample_ts_df = ts_purchase_df.loc[ts_purchase_df['merchant']=="Vince's Market"]
print(sample_ts_df.shape)

sample_ts_df.head()
sample_ts_df['purchase_date'] = sample_ts_df['purchase_date'].astype('datetime64[ns]')

print(sample_ts_df.dtypes)
sample_ts_df.head()
# Plotting data
sample_ts_df.index = sample_ts_df['purchase_date']
sample_ts_df['purchase_amount'].plot(figsize=(10,4), title= 'Daily Purchase $', fontsize=14, color='green')
plt.show()
sample_ts_df['client_count'].plot(figsize=(10, 4), title= 'Daily Clients', fontsize=14, color='purple')
plt.show()
# Autocorrelation plot
autocorrelation_plot(sample_ts_df['purchase_amount'])
plt.title('Autocorrelation Plot (ACF)')
plt.show()
plot_pacf(sample_ts_df['client_count'], lags=20);<jupyter_output><empty_output><jupyter_text>Notes* In the data, the pattern of client_count and purchase_amount are almost the same.* The autocorrelation plot (ACF)  * It summarizes the relationship between an observation in a time series with observations at prior time steps. The relationship here include direct and indirect correlations.  * From above plots, it's exponential, decaying to 0, so it can be AR (Autoregressive) model* The partial autocorrelation (PACF)  * It is similar to autocorrelation plot, but removes those indirect correlations.* We can start with `AR(2)` or `AR(3)`, since it's showing the first 2,3 lags have strong correlations.<jupyter_code>sample_ts_df.drop(['merchant', 'purchase_date'], inplace=True, axis=1)
sample_train_df = sample_ts_df.iloc[0:100, :]
sample_test_df = sample_ts_df.iloc[100:, :]

print(sample_train_df.shape, sample_test_df.shape)
sample_train_df.head()
sample_train_df.to_pickle('sample_train_df2.pkl')
sample_test_df.to_pickle('sample_test_df2.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12064287205176373, 'fraction_numerical': 0.014610728449175537, 'mean_word_length': 8.414538310412574, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '609694', 'license': 'MIT', 'n_tokens_mistral': 1718, 'n_tokens_neox': 1625, 'n_words': 356, 'path': 'Bank_Fantasy/Golden_Bridge/purchase_times_series_preparation.ipynb', 'repo_name': 'hanhanwu/Hanhan_Break_the_Limits', 'size': '4791'}