<jupyter_start><jupyter_text>Importing dataset Importing the libraries<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Importing the data from the files<jupyter_code>pd.read_csv('BCH-USD.csv',names=['time','low','high','open','close','volume'])<jupyter_output><empty_output><jupyter_text>Making the final dataframe.The final dataframe contains all the close and volume values from the all the csv files merged together with the timestamp as the index.<jupyter_code>data=['BCH-USD','BTC-USD','ETH-USD','LTC-USD']
main_df=pd.DataFrame()
for dataset in data:
  d=f'{dataset}.csv'
  df=pd.read_csv(d,names=['time','low','high','open','close','volume'])
  #print(df.head(5))
  df.rename(columns={"close":f'{dataset}_close','volume':f"{dataset}_volume"},inplace=True)
  df.set_index("time",inplace=True)
  df=df[[f'{dataset}_close',f"{dataset}_volume"]]
  
  if len(main_df)==0:
    main_df=df
  else:
    main_df=main_df.join(df)

#printing the head of the main_df
main_df.head(10)<jupyter_output><empty_output><jupyter_text>Data Preprocessing  Setting up the initial parameters and conditions  **What do we want to do ?**We want to predict the prices for the future period of time using the current prices of the currency .   **Initial Setup***   We decide a sequence length which is required to train a model,this sequence represents the number of past observations required to make the predictions.(SEQ_LEN)*   Then , we decide a time in the future for which we want to predict the prices.(FUTURE_PERIOD_PREDICT)* We choose a column for which we want to predict the prices.(RATIO_TO_PREDICT)<jupyter_code>SEQ_LEN=60
FUTURE_PERIOD_PREDICT=3
RATIO_TO_PREDICT='LTC-USD'<jupyter_output><empty_output><jupyter_text>The function which helps us detemine whether buying a currency is profitable or not in the future .<jupyter_code>def classify(current,future):
  if( float(future)> float(current)):
    return 1
  else:
    return 0<jupyter_output><empty_output><jupyter_text>Setting up the future and the target column in the main dataframe<jupyter_code>main_df['future']=main_df[f'{RATIO_TO_PREDICT}_close'].shift(-FUTURE_PERIOD_PREDICT)
main_df.head(10)
#we are going to make a list by mapping the columns to a function and then assign the list as a column 
main_df['target']=list(map(classify,main_df[f'{RATIO_TO_PREDICT}_close'],main_df['future']))
main_df.head(10)<jupyter_output><empty_output><jupyter_text>Making the validation set and the main dataframe**We want to carefully seperate the validation data and the training data .In this case getting a random sequence of data and shuffling the data does not help .The validation data has to be an accurate representation of the future** Sorting rows using the timestamp<jupyter_code>times=sorted(main_df.index.values)<jupyter_output><empty_output><jupyter_text>Getting the index of the last 5 percent of the data<jupyter_code>last_5pct=times[-int(0.05*len(times))]
last_5pct # timestamp of the row which marks the beginning of the last 5 percent of the data<jupyter_output><empty_output><jupyter_text>Making the validation set<jupyter_code>validation_main_df=main_df[(main_df.index>=last_5pct)]<jupyter_output><empty_output><jupyter_text>Making the training set<jupyter_code>main_df=main_df[(main_df.index<last_5pct)]<jupyter_output><empty_output><jupyter_text>Preprocessing function We design a pre-processing function that performs the following operations on the data:1.   Normalising the data2.   Scaling the data3. Building the sequential data4. Shuffling the resultant sequences5. Balancing the sequential data 6. Returning the matrix of features and the dependent variable<jupyter_code>from sklearn import preprocessing 
from collections import deque
import random
import numpy as np

def preprocess_df(df):
  df=df.drop('future',1)

#here we are pre-processing the values , normalising and scaling them
  for col in df.columns:
    if col!='target':
      df[col]=df[col].pct_change() #Normalising the data 
      df.dropna(inplace=True)
      df[col]=preprocessing.scale(df[col].values) #scaling
  df.dropna(inplace=True)

#creating sequences
  sequential_data=[]
  prev_days=deque(maxlen=SEQ_LEN) # deque function stops appending new elements as soon as the length equals the maximum length specified 

#building the sequential data
  for i in df.values:
    prev_days.append([n for n in i[:-1]]) # list of values of 60 rows (a sequence), this is the input data for the RNN  
    if len(prev_days)==SEQ_LEN:
      sequential_data.append([np.array(prev_days),i[-1]]) # sequences of 60 each along with their label 
#shuffling the sequences
  random.shuffle(sequential_data)
  
#balancing the data
  buys=[]
  sells=[]
#balancing the data and building again sequential data
  for seq,target in sequential_data:
    if target==0:
      sells.append([seq,target])
    elif target==1:
      buys.append([seq,target])
  
  random.shuffle(buys)
  random.shuffle(sells)
  lower=min(len(buys),len(sells))
  buys=buys[:lower]
  sells=sells[:lower]
  
  sequential_data=buys+sells
  random.shuffle(sequential_data)

#splitting into X,y
  X=[]
  y=[]

  for seq,target in sequential_data:
    X.append(seq)
    y.append(target)
  
  return np.array(X),y<jupyter_output><empty_output><jupyter_text>Applying it on the dataframe<jupyter_code>X_train,y_train=preprocess_df(main_df)
X_validation,y_validation=preprocess_df(validation_main_df)<jupyter_output><empty_output><jupyter_text>Printing out some statistics<jupyter_code>print(f"train data: {len(X_train)} validation: {len(X_validation)}")
print(f"Dont buys: {y_train.count(0)}, buys: {y_train.count(1)}")
print(f"VALIDATION Dont buys: {y_validation.count(0)}, buys: {y_validation.count(1)}")<jupyter_output>train data: 69400 validation: 3688
Dont buys: 34700, buys: 34700
VALIDATION Dont buys: 1844, buys: 1844<jupyter_text>Applying RNN model Importing the packages<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense,Dropout,LSTM,BatchNormalization
from tensorflow.keras.callbacks import TensorBoard,ModelCheckpoint
import tensorflow as tf
import time<jupyter_output><empty_output><jupyter_text>Deciding parameters for the training of the RNN<jupyter_code>EPOCHS=10
BATCH_SIZE=64
NAME=f'{SEQ_LEN}--SEQ--{FUTURE_PERIOD_PREDICT}--PRED--{int(time.time())}'<jupyter_output><empty_output><jupyter_text>Initializing the  model<jupyter_code>model=Sequential()<jupyter_output><empty_output><jupyter_text>Adding Recurrent layers<jupyter_code>X_train.shape<jupyter_output><empty_output><jupyter_text>1st Recurrent Layer or the input layer<jupyter_code>model.add(LSTM(128,input_shape=(X_train.shape[1:]),activation='tanh',return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())<jupyter_output><empty_output><jupyter_text>2nd Recurrent layer<jupyter_code>model.add(LSTM(128,activation='tanh',return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())<jupyter_output><empty_output><jupyter_text>3rd Recurrent layer<jupyter_code>model.add(LSTM(128,activation='tanh'))
model.add(Dropout(0.1))
model.add(BatchNormalization())<jupyter_output><empty_output><jupyter_text>Adding Dense layers<jupyter_code>model.add(Dense(32,activation='relu'))
model.add(Dropout(0.2))<jupyter_output><empty_output><jupyter_text>Adding output layer<jupyter_code>model.add(Dense(2,activation='softmax'))<jupyter_output><empty_output><jupyter_text>Making the optimizer , to apply SGD and optimize the weights of the network<jupyter_code>optimizer=tf.keras.optimizers.Adam(lr=0.001,decay=1e-6)<jupyter_output><empty_output><jupyter_text>Compiling the NN (Putting all the information together)<jupyter_code>model.compile(loss='sparse_categorical_crossentropy',optimizer=optimizer,metrics=['accuracy'])
y_train=np.asarray(y_train)
y_validation=np.asarray(y_validation)<jupyter_output><empty_output><jupyter_text>Fitting the model<jupyter_code>history=model.fit(X_train,y_train,batch_size=BATCH_SIZE,epochs=EPOCHS,validation_data=(X_validation,y_validation))<jupyter_output>Epoch 1/10
1085/1085 [==============================] - 17s 15ms/step - loss: 0.7223 - accuracy: 0.5139 - val_loss: 0.6969 - val_accuracy: 0.5095
Epoch 2/10
1085/1085 [==============================] - 16s 15ms/step - loss: 0.6888 - accuracy: 0.5407 - val_loss: 0.6847 - val_accuracy: 0.5393
Epoch 3/10
1085/1085 [==============================] - 16s 15ms/step - loss: 0.6847 - accuracy: 0.5555 - val_loss: 0.6861 - val_accuracy: 0.5434
Epoch 4/10
1085/1085 [==============================] - 16s 15ms/step - loss: 0.6830 - accuracy: 0.5596 - val_loss: 0.6752 - val_accuracy: 0.5770
Epoch 5/10
1085/1085 [==============================] - 16s 15ms/step - loss: 0.6817 - accuracy: 0.5635 - val_loss: 0.6805 - val_accuracy: 0.5700
Epoch 6/10
1085/1085 [==============================] - 16s 15ms/step - loss: 0.6802 - accuracy: 0.5673 - val_loss: 0.6760 - val_accuracy: 0.5705
Epoch 7/10
1085/1085 [==============================] - 16s 15ms/step - loss: 0.6788 - accuracy: 0.5690 - val_loss: 0.6728 -[...]
{'fraction_non_alphanumeric': 0.13634356370059617, 'fraction_numerical': 0.03775667917862663, 'mean_word_length': 7.07397504456328, 'pattern_counts': {'":': 1, '<': 100, '<?xml version=': 0, '>': 101, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '660000', 'license': 'MIT', 'n_tokens_mistral': 3415, 'n_tokens_neox': 3137, 'n_words': 691, 'path': 'Predicting Cryptocurrency Prices with RNN/Cryptocurrency_RNN.ipynb', 'repo_name': 'mehulfollytobevice/Deep-learning', 'size': '9058'}