<jupyter_start><jupyter_text>Loads Data from Stock-net DatasetDataset can be found [here](https://github.com/yumoxu/stocknet-dataset).<jupyter_code># Dependencies
import os
import json
import numpy as np
import pandas as pd
from tqdm import tqdm
# Change this filepath to your relative path
stocknet_dataset_filepath = './stocknet-dataset-master'<jupyter_output><empty_output><jupyter_text>Import data into maps`company_to_price_df` maps each company name to a DataFrame containing the stock movement information per date. Schema:```{    company_name: DataFrame ['date', 'open', 'high', 'low', 'close', 'adjust_close', 'volume']    ...}````company_to_tweets` maps each company name to a dictionary of dates and list of tweets. Schema:```{    company_name:     {        date: [list of tweets + metadata]        ...    }    ...}```NOTE: GMRE is not included because it does not include dates before 2017.<jupyter_code># Data import
preprocessed_prices_filepath = stocknet_dataset_filepath + '/price/preprocessed'
preprocessed_tweets_filepath = stocknet_dataset_filepath + '/tweet/preprocessed'

company_to_price_df = {}
company_to_tweets = {}

for filename in os.listdir(preprocessed_prices_filepath):
    with open(preprocessed_prices_filepath + '/' + filename) as file:
        company_name = filename.split('.')[0]
        
        # Not enough data for GMRE
        if company_name == 'GMRE':
            continue
        df = pd.read_csv(file, sep='\t')
        df.columns = ['date', 'open', 'high', 'low', 'close', 'adjust_close', 'volume']
        company_to_price_df[company_name] = df

for filename in tqdm(os.listdir(preprocessed_tweets_filepath)):
    company_name = filename.split('.')[0]
    dates_to_tweets = {}
    for tweet_filename in os.listdir(preprocessed_tweets_filepath + '/' + filename):
        with open(preprocessed_tweets_filepath + '/' + filename + '/' + tweet_filename) as file:
            list_of_tweets = []
            for line in file:
                tweet_json = json.loads(line)
                list_of_tweets.append(tweet_json)
            dates_to_tweets[tweet_filename] = list_of_tweets
    company_to_tweets[company_name] = dates_to_tweets<jupyter_output>100%|██████████| 87/87 [00:05<00:00, 15.54it/s]<jupyter_text>Train / Dev / Test Split for PriceThe dates are as specified for the train / dev / test split:* 01/01/2014 to 31/07/2015 for training* 01/08/2015 to 30/09/2015 for validation* 01/10/2015 to 01/01/2016 for testingThere are 6 companies that don't have 1256 rows in their stock data, 3 of which have 1255 rows (PTR, REX, SNP). The remaining 3 each have varying numbers of rows (BABA, AFGS, ABBV).<jupyter_code># Train / dev / test split
COMPANIES_1255 = ['PTR', 'REX', 'SNP']
DIFF_COMPANIES = ['BABA', 'AGFS', 'ABBV']

TRAIN_IDXS = range(526, 924)
TRAIN_IDXS_1255 = range(525, 923)
TRAIN_IDXS_BABA = range(526, 743)
TRAIN_IDXS_AGFS = range(526, 699)
TRAIN_IDXS_ABBV = range(526, 924)

DEV_IDXS = range(484, 526)
TEST_IDXS = range(419, 484)<jupyter_output><empty_output><jupyter_text>Build price tensors per company`company_to_price_tensors_{dataset}` maps each company name to a numpy tensor of dimension `len({dataset}) x 3`. Each feature vector is structured as `[closing_price, high_price, low_price]`. Schema:```{    company_name:         numpy.array([            [closing_price, high_price, low_price],            ...        ])    ...}```<jupyter_code># Build price tensor per company [closing, highest, lowest] & split data up
company_to_price_tensors_train = {}
company_to_price_tensors_dev = {}
company_to_price_tensors_test = {}

# Helper method for building tensor
def build_price_tensor(company, idxs):
    tensor = []
    for index, row in company_to_price_df[company].iloc[idxs].iterrows():
        tensor.append([row['close'], row['high'], row['low']])
    return np.array(tensor)

# Build training tensors for prices
for company in company_to_price_df.keys():
    # Skip over companies that dont match for now
    if company in COMPANIES_1255 or company in DIFF_COMPANIES:
        continue
    company_to_price_tensors_train[company] = build_price_tensor(company, TRAIN_IDXS)

# Build training tensors for prices
for company in company_to_price_df.keys():
    # Skip over companies that dont match for now
    if company in COMPANIES_1255 or company in DIFF_COMPANIES:
        continue
    company_to_price_tensors_dev[company] = build_price_tensor(company, DEV_IDXS)

# Build training tensors for prices
for company in company_to_price_df.keys():
    # Skip over companies that dont match for now
    if company in COMPANIES_1255 or company in DIFF_COMPANIES:
        continue
    company_to_price_tensors_test[company] = build_price_tensor(company, TEST_IDXS)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1057166876839008, 'fraction_numerical': 0.0308953341740227, 'mean_word_length': 4.22967032967033, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '414153', 'license': 'MIT', 'n_tokens_mistral': 1689, 'n_tokens_neox': 1530, 'n_words': 421, 'path': 'model_notebooks/Data_Extraction.ipynb', 'repo_name': 'Waterkin/MAN-SF_', 'size': '4758'}