<jupyter_start><jupyter_text>Your time series may actually be a random walk, and some ways to check this are as follows:> The time series shows a strong temporal dependence (autocorrelation) that decays linearly or in a similar pattern.> The time series is non-stationary and making it stationary shows no obviously learnable structure in the data.> The persistence model (using the observation at the previous time step as what will happen in the next time step) provides the best source of reliable predictions.[Written post](https://towardsdatascience.com/how-not-to-use-machine-learning-for-time-series-forecasting-avoiding-the-pitfalls-19f9d7adf424)<jupyter_code>### IMPORT LIBRARIES ###

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import math
### IMPORT AND PREPROCESS DATA ###

file = open('/content/drive/MyDrive/Projects/2020_Trading finance/Dataset/ES 2005 to 2021/ES_97eqjv0/ES_continuous_adjusted_5min_simplified.csv', "r")
df = pd.read_csv(file,names=['Date', 'Open', 'High','Low', 'Close', 'Volume'])
df.head
# ANALYZING DATA FOR OPEN PRICE
series = df['Open']
series.plot()
plt.show()
series_diff = pd.DataFrame([series[i] - series[i-1] for i in range(1,series.shape[0])])
series_diff.plot()
plt.show()
open_diff = pd.DataFrame([df['Open'][i] - df['Open'][i-1] for i in range(1,df['Open'].shape[0])])
high_diff = pd.DataFrame([df['High'][i] - df['High'][i-1] for i in range(1,df['High'].shape[0])])
low_diff = pd.DataFrame([df['Low'][i] - df['Low'][i-1] for i in range(1,df['Low'].shape[0])])
close_diff = pd.DataFrame([df['Close'][i] - df['Close'][i-1] for i in range(1,df['Close'].shape[0])])

close_diff<jupyter_output><empty_output><jupyter_text>Autocorrelation with non-stationery data and stationery data paint a very different picture<jupyter_code>from statsmodels.graphics.tsaplots import plot_acf
plot_acf(open_diff, zero = False)
plt.show()
plot_acf(high_diff, zero = False)
plt.show()
plot_acf(low_diff, zero = False)
plt.show()
plot_acf(close_diff, zero = False)
plt.show()<jupyter_output><empty_output><jupyter_text>Doing a forecase performance baseline test using a persistent model<jupyter_code>from sklearn.metrics import mean_squared_error

def PersistentModelError(df,split):
  #Transform the univariate dataset into a supervised learning problem.

  values = pd.DataFrame(df.values)
  dataframe = pd.concat([values.shift(1), values], axis=1)
  dataframe.columns = ['t-1', 't+1']
  #print(dataframe.head(5))

  #Establish the train and test datasets for the test harness.

  X = dataframe.values
  train_size = int(len(X) * split)
  train, test = X[1:train_size], X[train_size:]
  train_X, train_y = train[:,0], train[:,1]
  test_X, test_y = test[:,0], test[:,1]

  #Define the persistence model.

  def model_persistence(x):
    return x

  #Make a forecast and establish a baseline performance.

  predictions = list()
  for x in test_X:
    yhat = model_persistence(x)
    predictions.append(yhat)
  test_score = mean_squared_error(test_y, predictions)
  print('Test MSE: %.3f' % test_score)
  return test_score
from sklearn.metrics import mean_squared_error

#Transform the univariate dataset into a supervised learning problem.

values = pd.DataFrame(series_diff.values)
dataframe = pd.concat([values.shift(1), values], axis=1)
dataframe.columns = ['t-1', 't+1']
#print(dataframe.head(5))

#Establish the train and test datasets for the test harness.

X = dataframe.values
train_size = int(len(X) * 0.66)
train, test = X[1:train_size], X[train_size:]
train_X, train_y = train[:,0], train[:,1]
test_X, test_y = test[:,0], test[:,1]

#Define the persistence model.

def model_persistence(x):
	return x

#Make a forecast and establish a baseline performance.

predictions = list()
for x in test_X:
	yhat = model_persistence(x)
	predictions.append(yhat)
test_score = mean_squared_error(test_y, predictions)
print('Test MSE: %.3f' % test_score)

#Review the complete example and plot the output.

plt.figure(figsize=(12,6))
plt.plot(train_y[-50:])
plt.plot([None for i in train_y[-50:]] + [x for x in test_y[:50]])
plt.plot([None for i in train_y[-50:]] + [x for x in predictions[:50]])
plt.show()<jupyter_output>Test MSE: 6.774
{'fraction_non_alphanumeric': 0.13678682263069944, 'fraction_numerical': 0.019097636667462402, 'mean_word_length': 5.661367249602543, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '355526', 'license': 'CC0-1.0', 'n_tokens_mistral': 1502, 'n_tokens_neox': 1445, 'n_words': 400, 'path': 'OHLC_Prediction_Observations.ipynb', 'repo_name': 'jingzhi22/OHLC-Predictions', 'size': '4189'}