<jupyter_start><jupyter_text>**Bitcoin Price and its Movement Direction Prediction***Using Linear Regression, Logistics Regression, ARIMA, Simple RNN and LSTM Models.*<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Mounted at /content/drive<jupyter_text>**Packages**<jupyter_code>import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
%matplotlib inline
import matplotlib.image as mpimg

import seaborn as sns
sns.set(rc={'figure.figsize':(10,6)})
custom_colors = ["#4e89ae", "#c56183","#ed6663","#ffa372"]

import datetime as datetime
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, LSTM, Dropout<jupyter_output><empty_output><jupyter_text>**Data Cleaning** Import Dataset<jupyter_code>df = pd.read_csv("/content/drive/MyDrive/bitcoin.csv")<jupyter_output><empty_output><jupyter_text>Modify Timestamp<jupyter_code># Change the Timestamp column attribute to datetime
df['Timestamp'] = pd.to_datetime(df['Timestamp'],unit='s')
# Enable to refer to the row by index 
df = df.set_index('Timestamp')<jupyter_output><empty_output><jupyter_text>Spot the missing values<jupyter_code>missed = pd.DataFrame()
missed['column'] = df.columns

missed['percent'] = [round(100* df[col].isnull().sum() / len(df), 2) for col in df.columns]
missed = missed.sort_values('percent',ascending=False)
missed = missed[missed['percent']>0]<jupyter_output><empty_output><jupyter_text>Fill in the missing data<jupyter_code>def fill_missing(df):
    ### function to impute missing values using interpolation ###
    df['Open'] = df['Open'].interpolate()
    df['Close'] = df['Close'].interpolate()
    df['Weighted_Price'] = df['Weighted_Price'].interpolate()
    df['Volume_(BTC)'] = df['Volume_(BTC)'].interpolate()
    df['Volume_(Currency)'] = df['Volume_(Currency)'].interpolate()
    df['High'] = df['High'].interpolate()
    df['Low'] = df['Low'].interpolate()
    
fill_missing(df)<jupyter_output><empty_output><jupyter_text>Remove NA rows<jupyter_code>df = df.dropna()<jupyter_output><empty_output><jupyter_text>**Feature Engineering** Add `Price` column by averaging Low and Close<jupyter_code>df["Price"] = df[["Open","High","Low","Close"]].mean(axis=1)<jupyter_output><empty_output><jupyter_text>Add `Return` Column by calculating the difference from now to previous second's `Price`<jupyter_code>df["Return"] = df["Price"].diff()
df["Return"].plot(x="Timestamp", y="Return")<jupyter_output><empty_output><jupyter_text>Add `Direction` column showing if bitcoin price is going up or down.<jupyter_code>df["Direction"] = pd.Series(df["Return"] > 0)
df["Direction"] = "Down"
df.loc[(df["Return"] > 0), "Direction"] = "Up"<jupyter_output><empty_output><jupyter_text>Add `Percentage_Return` column to show the perentage return per seconds<jupyter_code>df["Percentage_Return"] = ((df["Return"].shift(periods=-1))/ (df["Price"])).shift(periods=1)
df["Percentage_Return"].plot()<jupyter_output><empty_output><jupyter_text>Add Lag1, Lag2, Lag3, Lag4, Lag5 columns as t-1, t-2, t-3, t-4, and t-5 features to predict bitcoin price at t (in seconds)<jupyter_code>df["Lag1"]= df["Percentage_Return"].shift(periods=-1)
df["Lag2"]= df["Percentage_Return"].shift(periods=-2)
df["Lag3"]= df["Percentage_Return"].shift(periods=-3)
df["Lag4"]= df["Percentage_Return"].shift(periods=-4)
df["Lag5"]= df["Percentage_Return"].shift(periods=-5)
df.dropna()<jupyter_output><empty_output><jupyter_text>Cleaned Dataset Display<jupyter_code>df.shape
df.head()<jupyter_output><empty_output><jupyter_text>**Exploratory Data Analysis** Zheming's Part So in this cahart, we use candlestick graph to show the open,close,high and low price of each day throughout the 10 years.we can see that  the price of bitcoin has dropped since 2018, but in the middle of 2020. It has risen rapidly.  Furthermore, we also use the blue line to represent the moving average of 30 perio, the the red line for 50 perio. The shorter the period of the moving average, the faster it can reflect the changes in price trends. For example, the 30 perio moving average is closer to the current price level than the 50-period moving average. The 50-period moving average is more able to reflect the long-term overall price trend. bitcoin moving average<jupyter_code>df = df.resample("6H").mean()
import plotly.graph_objects as go
trace1 = {
    'x': df.index,
    'open': df.Open,
    'close': df.Close,
    'high': df.High,
    'low': df.Low,
    'type': 'candlestick',
    'name': 'MSFT',
    'showlegend': False
}
# Calculate and define moving average of 30 periods
avg_30 = df.Close.rolling(window=30, min_periods=1).mean()

# Calculate and define moving average of 50 periods
avg_50 = df.Close.rolling(window=50, min_periods=1).mean()

trace2 = {
    'x': df.index,
    'y': avg_30,
    'type': 'scatter',
    'mode': 'lines',
    'line': {
        'width': 1,
        'color': 'blue'
            },
    'name': 'Moving Average of 30 periods'
}

trace3 = {
    'x': df.index,
    'y': avg_50,
    'type': 'scatter',
    'mode': 'lines',
    'line': {
        'width': 1,
        'color': 'red'
    },
    'name': 'Moving Average of 50 periods'
}
data_trace = [trace1, trace2, trace3]
# Config graph layout
layout = go.Layout({
    'title': {
        'text': 'Bitcoin Moving Averages',
        'font': {
            'size': 15
        } 
    }
})



df = df.resample("24H").mean()
fig = go.Figure(data=data_trace, layout=layout)

fig.show()



from PIL import Image
with Image.open("/content/a.png") as im:
    im.rotate(45).show()
%pylab inline
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
img = mpimg.imread('/content/a.png')
imgplot = plt.imshow(img)
plt.show()
top = plt.subplot2grid((4,4), (0, 0), rowspan=3, colspan=4)
top.plot(df.index, df["Price"])
plt.title('Price from 2011 - 2021')

bottom = plt.subplot2grid((4,4), (3,0), rowspan=1, colspan=4)
bottom.bar(df.index, df['Volume_(BTC)'])
plt.title('Bitcoin Trading Volume')

plt.gcf().set_size_inches(15,8)<jupyter_output><empty_output><jupyter_text>Produce a kernal density plot for Price (refer to Lecture 7 02-Exploratory-Data-Analysis.ipynb) in this graph, we grouped the data by direction,the orange line represents up and the blue line is down. Most of price are below 10000,at around 100 at most.<jupyter_code>def kdeplot(feature):
    plt.figure(figsize=(9, 4))
    plt.title("KDE for {}".format(feature))
    ax0 = sns.kdeplot(df[df['Direction'] == 'Up'][feature].dropna(), color= 'navy', label= 'Churn: No')
    ax1 = sns.kdeplot(df[df['Direction'] == 'Down'][feature].dropna(), color= 'orange', label= 'Churn: Yes')
kdeplot('Price')<jupyter_output><empty_output><jupyter_text>Analyze correlations with seaborn heatmap Each square shows the correlation between the variables on each axis.The close to 1 the correlation is the more positively correlated they are. As we can see in the graph those which are green, they have a perfect correlation, such as price and open price. For open price and volume(BTC),it has a negative correlation which means if one increase, one will decrease.<jupyter_code>corr = df.corr()
plt.figure(figsize=[10,10])
plt.rcParams['font.size']=7
sns.heatmap(corr, cmap='PiYG', annot= True, linewidths=0.3 );<jupyter_output><empty_output><jupyter_text>Next we are gonna talk about the distribution of Opening Price. We used hitagram+KDE, boxplot and violin plot respectively to show the distribution of opening price. I think the box plot tells us more clearly. From the graph we can see the opening price has reached a price close to 17,000. The 17000 should be the maximum, because there are too much data under 10000. So any price beyond 17000 counts as outliers because it happened not much.. In most cases, the opening price will not exceed 7000. But normally, the opening price of bitcoin is around 1000. So we can tell that Bitcoin's opening price fluctuates very much,and most of them are sold at low prices.<jupyter_code>def triple_plot(x, title,c):
    fig, ax = plt.subplots(3,1,figsize=(25,10),sharex=True)
    sns.distplot(x, ax=ax[0],color=c)
    ax[0].set(xlabel=None)
    ax[0].set_title('Histogram + KDE')
    sns.boxplot(x, ax=ax[1],color=c)
    ax[1].set(xlabel=None)
    ax[1].set_title('Boxplot')
    sns.violinplot(x, ax=ax[2],color=c)
    ax[2].set(xlabel=None)
    ax[2].set_title('Violin plot')
    fig.suptitle(title, fontsize=30)
    plt.tight_layout(pad=3.0)
    plt.show()
triple_plot(df['Open'],'Distribution of Opening price',custom_colors[0])

top = plt.subplot2grid((4,4), (0, 0), rowspan=3, colspan=4)
top.plot(df.index, df["Open"])
plt.title('Price from 2011 - 2021')

bottom = plt.subplot2grid((4,4), (3,0), rowspan=1, colspan=4)
bottom.bar(df.index, df['Volume_(BTC)'])
plt.title('Bitcoin Trading Volume')

plt.gcf().set_size_inches(15,8)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14587570621468926, 'fraction_numerical': 0.02553672316384181, 'mean_word_length': 5.479502196193265, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '423622', 'license': 'MIT', 'n_tokens_mistral': 3199, 'n_tokens_neox': 3016, 'n_words': 812, 'path': "Zheming_Xu's part .ipynb", 'repo_name': 'zhouchun0105/Bitcoin-Price-and-Movement-Prediction', 'size': '8850'}