<jupyter_start><jupyter_text><NAME> - Historic Stock Data (2000-2020) AnalysisIn this Notebook we will have a quick look at the historic Dow Jones data-set, perform feature engineering and use plots to understand the results.  Load DataFirst, let's load the data and have a look at it.<jupyter_code># Import libs...
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
# ... and set variables
FILE_LOCATION = './data/preprocessed/dow_historic_2000_2020.csv'
# Read dataframe and show last couple of rows...
df = pd.read_csv(FILE_LOCATION)
df.tail()<jupyter_output><empty_output><jupyter_text>...as you can see, we have multiple interesting data columns (you can check the data set description for a detailled explanation of the columns):   Pre-ProcessingNow, let's enginere some date-related features and also add a column which shows the daily change percentage of the stock.<jupyter_code># Engineer some basic features...
# Convert date to datetime format
df.date = pd.to_datetime(df.date)
# Add year
df['year'] = df.date.dt.year
# Add month
df['month'] = df.date.dt.month_name().str[0:3]
# Add weekday
df['day_of_week'] = df.date.dt.day_name()
# Calculate the stock movement on a given day
df['pct_change_day'] = round((df.close / df.open - 1) * 100, 3)<jupyter_output><empty_output><jupyter_text>Now, let's check our data with pandas describe operation...<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>PlottingWe can also do some plotting to learn more about the data. First let us check the average stock movements per day.<jupyter_code># Use a group-by operation to calculate the pct_change mean 
stock_performance = df.groupby('stock').pct_change_day.mean() 

# Chose seaborn style
sns.set_style("darkgrid")
# Use matplotlib and seaborn to plot a lineplot....
plt.figure(figsize=(12,5))
chart = sns.lineplot(x=stock_performance.index, y=stock_performance)
_ = plt.xticks(rotation=45)<jupyter_output><empty_output><jupyter_text>We can observe that the *CRM* stock (*Salesforce*) has the highest *mean* intra-day stock performance in this data-set (tnote that his company is rather young and historic performances should not be extrapolated into the future).Next, let us combine the *pct_change_day* feature with two of the other features we derived from the data-column (month and day-of-week).<jupyter_code># Make lineplot for months...
_ = sns.lineplot(data=df, x="month", y="pct_change_day")
# Make lineplot for day of week...
_ = sns.lineplot(data=df, x="day_of_week", y="pct_change_day")<jupyter_output><empty_output><jupyter_text>From the plots above we can see that - *historically* - in June and September stocks dropped most, while in March and November stocks performed best. It is also interesting to see, that in *average* stocks increased every day of the week, except friday. What did we learn ?- Understand which kind of information is included in this data set- How to easilly engineer some additional features- Use a plot to get an understanding of the different stock performances- Plot some of the newly created features to understand their value (month, day of week)This was only a small example what can be done with the available data set - other features like *dividends*, *trading volume* and *stock splits* could be investigated as well...if you like, **have fun** and success with your own stock data analysis!<jupyter_code>len(df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09045956951716114, 'fraction_numerical': 0.008144269924374637, 'mean_word_length': 5.333333333333333, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '191533', 'license': 'MIT', 'n_tokens_mistral': 1052, 'n_tokens_neox': 1005, 'n_words': 409, 'path': 'dow/dow_features_basic.ipynb', 'repo_name': 'thg-muc/ml-labs', 'size': '3441'}