<jupyter_start><jupyter_text>Introduction This notebook presents the pre-processing of the poloniex (crypto-currencies) data used to produce the input tensors of the neural network. For each stock, the input is a raw time series of the prices (High, Low, Open, Close). Please note for crypto-currencies, the market never closes, so Close(t) = Open(t+1). The output is a matrix of 3 rows and n (number of available data points) columns. The columns correspond to:- High(t-1)/Open(t-1)- Low(t-1)/Open(t-1)- Open(t)/Open(t-1)    Remark: We don't need to normalize the data since it's already of ratio of 2 prices closed to one.   Imports<jupyter_code>#for navigation in the folders
import os
import pathlib

from time import strptime
from datetime import datetime

from tqdm import tqdm

#for plot
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline

import numpy as np
import pandas as pd
import seaborn as sns 
import PIL
import pickle
from time import strftime



import seaborn as sns
sns.despine()<jupyter_output><empty_output><jupyter_text>Processing<jupyter_code>data_dir = '/poloniex_data/'
directory = os.getcwd() + data_dir # path to the files
files_tags = os.listdir(directory) #these are the differents pdf files

#this is here because hidden files are also shown in the list. 
for file in files_tags:
    if file[0] == '.':
        files_tags.remove(file)
stock_name = [file.split('.')[0] for file in files_tags]
stocks = [file for file in files_tags]
print(len(stock_name) == len(stocks))
print('There are {} different currencies.'.format(len(stock_name)))
for s in stocks:
    df = pd.read_csv('.'+data_dir+s)
    print(s, len(df))<jupyter_output>BITCOIN.csv 1264
BTCUSDT.csv 42010
DASHBTC.csv 60103
DOGEBTC.csv 60915
ETCBTC.csv 17032
ETCETH.csv 17031
ETHBTC.csv 33878
ETHUSDT.csv 33876
GNTBTC.csv 7001
GNTETH.csv 7001
GOLD.csv 1264
LTCBTC.csv 61096
REPBTC.csv 13547
REPETH.csv 13547
XMRBTC.csv 55285
XRPBTC.csv 51113<jupyter_text>We want roughly 1 year of data. So, we drop the data with less than 17000 rows.<jupyter_code>kept_stocks = ['GOLD.csv','BITCOIN.csv']
# kept_stocks = ['BITCOIN.csv']
# kept_stocks = ['GOLD.CSV']
len_stocks = list()

for s in kept_stocks:
    df = pd.read_csv('.'+data_dir+s)
    len_stocks.append(len(df))

    
min_len = np.min(len_stocks)
len_stocks
list_open = list()
list_close = list()
# list_high = list()
# list_low = list()

for s in tqdm(kept_stocks):
    data = pd.read_csv(os.getcwd() + data_dir + s).fillna('bfill').copy()
    # data = data[['open', 'close', 'high', 'low']]
    data = data[['open', 'close']]
    data = data.tail(min_len)
    list_open.append(data.open.values)
    list_close.append(data.close.values)
    # list_high.append(data.high.values)
    # list_low.append(data.low.values)

array_open = np.transpose(np.array(list_open))[:-1]
array_open_of_the_day = np.transpose(np.array(list_open))[1:]
array_close = np.transpose(np.array(list_close))[:-1]
# array_high = np.transpose(np.array(list_high))[:-1]
# array_low = np.transpose(np.array(list_low))[:-1]
print(array_close)
# np.transpose(np.array(list_low)).shape
    # array_high/array_open,
    # array_low/array_open,
X = np.transpose(np.array([
    array_open_of_the_day/array_open]), axes=(0, 2, 1))
X.shape<jupyter_output><empty_output><jupyter_text>The shape corresponds to:- 3: Number of features- 10: Number of stocks- 17030: Number of data points  Save<jupyter_code>np.save('./np_data/test.npy', X)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12299771167048056, 'fraction_numerical': 0.030892448512585814, 'mean_word_length': 4.867449664429531, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '292929', 'license': 'MIT', 'n_tokens_mistral': 1379, 'n_tokens_neox': 1286, 'n_words': 343, 'path': 'Codes/data_pipe_poloniex.ipynb', 'repo_name': 'Reself-C/COMAP-MCM-ICM-2022', 'size': '3496'}