<jupyter_start><jupyter_text>Stationary Transformation  why stationary transformation matters A great explaination of time series analysis and the importance of stationary data can be found [here](https://chih-ling-hsu.github.io/2017/03/20/time-series).In summary, in order for our time series forecasting to be effective, our data has to be stationary. In order to be stationary: 1. The mean of the series should be a constant of 02. The variance should be the same over time3. The covariance should be the same over time [see this picture as an example](https://i.imgur.com/3lKCxEn.png)  ways to make stationary 1. log()Taking the log of data ia a good way to convert data with an exponential distribution into a linear distribution and it also helps to stabilize the variance over time. Vector autoregression is a linear model and taking the log enables us to describe an exponential distribution in linear terms.         2. differencingTaking the difference of consecutive data points is another way to make data stationary. DIfferencinghelps to stabilize the mean of a time series by removing changes to the level of a time series, thus reducing seasonality and trends.        3. log & differencingTaking the log of data and then differencing is the same thing as log returns, or difference in log overa time period. Log returns are an industry standard in finance for good reason. Log returns are similar to scaled percent returns but there are some important differences. Log returns are additive while scaled percent returns are not. In other words, the five-day log return is equal to the sum of five one-day log returns.In addition, log returns are symmetrical with respect to gains and losses, where scaled percent returns are biased in favor of gains. For example, if a stock goes down 50%(-0.5 scaled percent) and thengoes up 200% (2.0 scaled percent), the stock is back where it started. If the same situation happened in log, we would first see a log return of log(0.5) = -0.301 followed by log(2) = 0.301.<jupyter_code># import packages
import pandas as pd
import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
from statsmodels.tsa.stattools import adfuller
from statsmodels.tsa.stattools import kpss
# import data
df = pd.read_csv('formatted_data_all.csv', index_col = 0)
df.tail()
# do both of these to make stationary transformation step easier
# cant have any values nan, inf or 0 for tests to work
# fill na
df = df.fillna(1)
# fill value '0' with '1'
df = df.replace(0, 1)
df.head()<jupyter_output><empty_output><jupyter_text>exploratory functions<jupyter_code># descriptive statistics
def descriptive_statistics(df, series):
    stats = df[series].describe()    
    print('\nDescriptive Statistics for', '\'' + series + '\'', '\n\n', stats)
# create graphics function
def get_graphics(df, series, xlabel, ylabel, title, grid = True):
    plt.plot(pd.to_datetime(df.index), df[series])
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.title(title)
    plt.grid(grid)
    return plt.show()<jupyter_output><empty_output><jupyter_text>stationary tests unit root = statistical properties of series are not constant with time.In order to be stationary, series has to be constant with time. So if a series has a unit root, it is not stationary- strict stationary = mean, variance, covariance are not function of time- trend stationary = no root unit, but has a trend. if you remove the trend, it would be strict stationary- difference stationary = series can be made strict stationary by differencing  ADF Augmented Dickey Fuller Test (unit root test)null hypothesis = series has a unit root (a = 1)alt hypothesis = series has no unit rootaccept null = t-score is greter than critical value (there is a unit root)reject null = t-score is less than critical value (there is no unit root)accpet null = bad (not stationary)reject null = good (stationary)adf can be interpreted as a difference stationary test<jupyter_code># ADF Augmented Dickey Fuller Test
def adf_test(df, series):
    results = adfuller(df[series])
    output = pd.Series(results[0:4], index = ['t-score', 'p-value', '# of lags used', '# of observations'])
    for key, value in results[4].items():
        output['critical value (%s)' %key] = value
    # if t-score < critical value at 5%, the data is stationary
    # if t-score > critical value at 5%, the data is NOT stationary
    if output[0] < output[5]:
        print('\nADF: The data', '\'' + series + '\'', 'is STATIONARY \n\n', output)
    elif output[0] > output[5]:
        print('\nADF: The data', '\'' + series + '\'', 'is NOT STATIONARY \n\n', output)
    else:
        print('\nADF: There is something wrong with', '\'' + series + '\'','\n\n', output)<jupyter_output><empty_output><jupyter_text>KPSS Kwiatkowski-Phillips-Schmidt-Shin Test (stationary test)null hypothesis = the series has a stationary trendalt hypothesis = the series has a unit root (series is not stationary)accept null = t-score is less than critical value (series is stationary) reject null = t-score is greater than the critical value (series is not stationary)accpet null = good (stationary)reject null = bad (not stationary)kpss classifies a series as stationary on the absence of a unit root(both strict stationary and trend stationary will be classified as stationary)<jupyter_code># KPSS Kwiatkowski-Phillips-Schmidt-Shin Test (stationary test)
def kpss_test(df, series):
    results = kpss(df[series], regression = 'ct')
    output = pd.Series(results[0:3], index = ['t-score', 'p-value', '# lags used'])              
    for key, value in results[3].items():
        output['critical value (%s)' %key] = value
    # if t-score < critical value at 5%, the data is stationary
    # if t-score > critical value at 5%, the data is NOT stationary
    if output[0] < output[4]:
        print('\nKPSS: The data', '\'' + series + '\'', 'is STATIONARY \n\n', output)
    elif output[0] > output[4]:
        print('\nKPSS: The data', '\'' + series + '\'', 'is NOT STATIONARY \n\n', output)
    else:
        print('\nKPSS: There is something wrong with', '\'' + series + '\'', '\n\n', output)<jupyter_output><empty_output><jupyter_text>Many times, adf and kpss can give conflicting results. if so:1. [adf = stationary], [kpss = stationary] = series is stationary2. [adf = stationary], [kpss = NOT stationary] = series is difference stationary. use differencing to make it stationary3. [adf = NOT stationary], [kpss = stationary] = series is trend stationary. remove trend to make strict stationary4. [adf = NOT STATIONARY], [kpss = NOT STATIONARY] = series is not stationary  everything combined<jupyter_code># all functions above in one
def series_analysis(df, series, xlabel, ylabel, title, grid = True):
    # descriptive stats
    descriptive_statistics(df, series)
    # graphics
    get_graphics(df, series, xlabel, ylabel, title, grid = True)
    # stationary tests
    adf_test(df, series)
    kpss_test(df, series)<jupyter_output><empty_output><jupyter_text>transforming variables<jupyter_code># create new df for stationary data
stationary = pd.DataFrame()<jupyter_output><empty_output><jupyter_text>-  price<jupyter_code># ['price'] 
series_analysis(df, 'price', xlabel = 'year', ylabel = 'Bitcoin Price(USD)', title = 'df[\'price\']')
# ['price'] = log & diff
stationary['price'] = df['price'].apply(np.log).diff().dropna()
# run tests to see if stationary
series_analysis(stationary, 'price', xlabel = 'year', ylabel = 'Bitcoin Price(USD)', title = 'Log_Diff_Price')

stationary.head()<jupyter_output>Descriptive Statistics for 'price' 

 count    2968.000000
mean        0.004005
std         0.069389
min        -0.848765
25%        -0.013854
50%         0.001964
75%         0.022700
max         1.474395
Name: price, dtype: float64<jupyter_text>-  volatility<jupyter_code># ['volatility']
series_analysis(df, 'volatility', xlabel = 'year', ylabel = 'volatility (daily (high-low)/price)', title = 'df[\'volatility\']')
# ['volatility'] = diff
stationary['volatility'] = df['volatility'].diff().dropna()
# run tests to see if stationary
series_analysis(stationary, 'volatility',  xlabel = 'year', ylabel = 'Volatility', title = 'Diff_Volatility')

stationary.head()<jupyter_output>Descriptive Statistics for 'volatility' 

 count    2968.000000
mean       -0.000325
std         0.100633
min        -1.804238
25%        -0.021514
50%        -0.002326
75%         0.016968
max         1.785943
Name: volatility, dtype: float64<jupyter_text>-  volume_price<jupyter_code># ['volume_price']
series_analysis(df, 'volume_price', xlabel = 'year', ylabel = 'Volume_Price(USD)', title = 'df[\'volume_price\']')
# ['volume_price'] = log & diff
stationary['volume_price'] =  df['volume_price'].apply(np.log).diff().dropna()
# run tests to see if stationary
series_analysis(stationary, 'volume_price',  xlabel = 'year', ylabel = 'volume_price', title = 'Log_Diff_Volume_Price')

stationary.head()<jupyter_output>Descriptive Statistics for 'volume_price' 

 count    2968.000000
mean        0.006577
std         0.749558
min       -13.192387
25%        -0.363592
50%        -0.032482
75%         0.358553
max        12.365727
Name: volume_price, dtype: float64<jupyter_text>-  volume_number<jupyter_code># ['volume_number']
series_analysis(df, 'volume_number', xlabel = 'year', ylabel = 'Number of Bitocins exchanged', title = 'df[\'volume_number\']')
# ['volume_number'] = diff
stationary['volume_number'] = df['volume_number'].diff().dropna()
# run tests to see if stationary
series_analysis(stationary, 'volume_number',  xlabel = 'year', ylabel = 'volume_number', title = 'Diff_Volume_Number')

stationary.head()<jupyter_output>Descriptive Statistics for 'volume_number' 

 count      2968.000000
mean         14.045672
std       42092.258606
min     -356062.771140
25%      -13951.065498
50%        -684.245552
75%       12164.821981
max      421763.909910
Name: volume_number, dtype: float64<jupyter_text>-  positive_comment<jupyter_code># ['positive_comment']
series_analysis(df, 'positive_comment', xlabel = 'year', ylabel = 'Number of Positive Comments', title = 'df[\'positive_comment\']')
# ['positive_comment'] = log & diff 
stationary['positive_comment'] = df['positive_comment'].apply(np.log).diff().dropna()
# run tests to see if stationary
series_analysis(stationary, 'positive_comment', xlabel = 'year', ylabel = 'Number of Positive Comments', title = 'Log_Diff_Positive_Comment')

stationary.head()<jupyter_output>Descriptive Statistics for 'positive_comment' 

 count    2968.000000
mean        0.002283
std         0.372720
min        -2.484907
25%        -0.143344
50%         0.000000
75%         0.146424
max         4.382027
Name: positive_comment, dtype: float64<jupyter_text>-  neutral_comment<jupyter_code># ['neutral_comment']
series_analysis(df, 'neutral_comment', xlabel = 'year', ylabel = 'Number of Neutral Comments', title = 'df[\'neutral_comment\']')
# ['neutral_comment'] = log & diff
stationary['neutral_comment'] = df['neutral_comment'].apply(np.log).diff().dropna()
# run tests to see if stationary
series_analysis(stationary, 'neutral_comment', xlabel = 'year', ylabel = 'Number of Neutral Comments', title = 'Log_Diff_Neutral_Comment')

stationary.head()<jupyter_output>Descriptive Statistics for 'neutral_comment' 

 count    2968.000000
mean        0.001792
std         0.449143
min        -3.135494
25%        -0.163068
50%         0.000000
75%         0.158856
max         3.784190
Name: neutral_comment, dtype: float64<jupyter_text>-  negative_comment<jupyter_code># ['negative_comment']
series_analysis(df, 'negative_comment', xlabel = 'year', ylabel = 'Number of Negative Comments', title = 'df[\'negative_comment\']')
# ['negative_comment'] = log & diff 
stationary['negative_comment'] = df['negative_comment'].apply(np.log).diff().dropna()
# run tests to see if stationary
series_analysis(stationary, 'negative_comment', xlabel = 'year', ylabel = 'Number of Negative Comments', title = 'Log_Diff_Negative_Comment')

stationary.head()
stationary.to_csv('stationary_data_all.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11297520661157025, 'fraction_numerical': 0.04090909090909091, 'mean_word_length': 4.305129329241561, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '429566', 'license': 'MIT', 'n_tokens_mistral': 4071, 'n_tokens_neox': 3655, 'n_words': 1263, 'path': '7_stationary_transformation.ipynb', 'repo_name': 'pcann9/Predict_Bitcoin_Using_Reddit_Sentiment', 'size': '12100'}