<jupyter_start><jupyter_text>We start off by imprting dependencies require for data preparation. This includes:- os, inspect, sys: get the current and the project root directories.- pandas: read and explore data<jupyter_code># import 
import os, inspect, sys
import pandas as pd<jupyter_output><empty_output><jupyter_text>All the notebook files here are stored within the notebooks directory at project root. We need to get access the prject root directory.<jupyter_code># get project root directory
curr_dir = os.path.dirname(inspect.getabsfile(inspect.currentframe()))
root_dir = os.path.dirname(curr_dir)
sys.path.insert(0, 0)<jupyter_output><empty_output><jupyter_text>We load the news datasets to a pandas dataframe:<jupyter_code># load data
fname = os.path.join(root_dir, "stockomen", "data", "ext", "combined_stock_data.csv")
news_df = pd.read_csv(fname)<jupyter_output><empty_output><jupyter_text>Once the data is loaded, we look at the data structure using `head()`. By this, we can get a general idea of the data and the data columns.<jupyter_code>news_df.head(5)<jupyter_output><empty_output><jupyter_text>Lets look at data columns:<jupyter_code>news_df.columns<jupyter_output><empty_output><jupyter_text>This dataset consists of top 25 news headlines followed by their sentiment analysis scores. The sentiments scores include Subjectivity, Objectivity, Positivity, Negativity and Neutrality provided by [USENT](https://github.com/nik0spapp/usent). Next, check on the shape and the data types:<jupyter_code>print("shape: ", news_df.shape, os.linesep)
print("dtypes: ", os.linesep, news_df.dtypes)<jupyter_output>shape:  (1989, 34) 

dtypes:  
 Unnamed: 0        int64
Date             object
Label             int64
Top1             object
Top2             object
Top3             object
Top4             object
Top5             object
Top6             object
Top7             object
Top8             object
Top9             object
Top10            object
Top11            object
Top12            object
Top13            object
Top14            object
Top15            object
Top16            object
Top17            object
Top18            object
Top19            object
Top20            object
Top21            object
Top22            object
Top23            object
Top24            object
Top25            object
Para             object
Subjectivity    float64
Objectivity     float64
Positive        float64
Neutral         float64
Negative        float64
dtype: object<jupyter_text>In the next step, we load stock prices from `DJIA_table.csv` and we check on the data structure, shape and data types"<jupyter_code>fname = os.path.join(root_dir, "stockomen", "data", "ext", "DJIA_table.csv")
stock_prices_df = pd.read_csv(fname)
print("shape: ", stock_prices_df.shape, os.linesep)
print("dtypes: ", os.linesep, stock_prices_df.dtypes)
stock_prices_df.head()<jupyter_output>shape:  (1989, 7) 

dtypes:  
 Date          object
Open         float64
High         float64
Low          float64
Close        float64
Volume         int64
Adj Close    float64
dtype: object<jupyter_text>We see that the `Volume` column is in integer type. Lets cast this column to float:<jupyter_code>stock_prices_df["Volume"] = stock_prices_df["Volume"].astype(float)
stock_prices_df.dtypes<jupyter_output><empty_output><jupyter_text>we see that it has rows equal to the number of rows the news dataframe.Having both dataframes loaded, we build the model data by mergings the five sentiment data column of news data frame and the stock price data frame.<jupyter_code>df = news_df[["Date", "Label", "Subjectivity", "Objectivity", "Positive", "Negative", "Neutral"]] \
    .merge(stock_prices_df, how="inner", on="Date", left_index=True)
fname = os.path.join(root_dir, "stockomen", "data", "ext", "DJIA_table.csv")
print("shape: ", df.shape, os.linesep)
print("dtypes: ", os.linesep, df.dtypes)
stock_prices_df.head()
df.head()<jupyter_output>shape:  (1989, 13) 

dtypes:  
 Date             object
Label             int64
Subjectivity    float64
Objectivity     float64
Positive        float64
Negative        float64
Neutral         float64
Open            float64
High            float64
Low             float64
Close           float64
Volume          float64
Adj Close       float64
dtype: object<jupyter_text>To include any latent seasonal and monthly patterns, we include include to mothly and seasonal columns:<jupyter_code>df.Date = pd.to_datetime(df.Date, format="%Y-%m-%d")
df["Month"] = df.Date.dt.month.astype("float32")
df["Quarter"] = df.Date.dt.quarter.astype("float32")
df.head()<jupyter_output><empty_output><jupyter_text>Lastly, we push the `Lable` column to the end of data frame and sort by index to make it more readable:<jupyter_code>columns = list(df.columns.values)
columns.append(columns.pop(1))
df = df[columns]
df.index = df.index.sort_values()
df.head(5)<jupyter_output><empty_output><jupyter_text>We save the data frame prepared in a separate `csv` file:<jupyter_code>df.to_csv(os.path.join(root_dir, "stockomen", "data", "int", "stock.csv"), index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0915327322618581, 'fraction_numerical': 0.023716189729517837, 'mean_word_length': 3.5849056603773586, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '129532', 'license': 'MIT', 'n_tokens_mistral': 1678, 'n_tokens_neox': 1581, 'n_words': 463, 'path': 'notebooks/0.0.prepare.data.ipynb', 'repo_name': 'aidinhass/stockomen', 'size': '5102'}