<jupyter_start><jupyter_text>importing libraries<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Importing and understanding the dataset<jupyter_code>#reading the dataset : "Dataframe.csv"
df = pd.read_csv('./Datasets/DataFrame.csv') 
df.head(10)
#let's get some infromation about the data features
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 22805 entries, 0 to 22804
Data columns (total 8 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   Type        22805 non-null  object 
 1   Date        22805 non-null  int64  
 2   Time        22805 non-null  object 
 3   open        22805 non-null  float64
 4   high        22805 non-null  float64
 5   low         22805 non-null  float64
 6   close       22805 non-null  float64
 7   Unnamed: 7  0 non-null      float64
dtypes: float64(5), int64(1), object(2)
memory usage: 1.4+ MB<jupyter_text>Inshight : We can see that Unnamed column is useless so we better drop it<jupyter_code>df.drop(['Unnamed: 7'],axis=1,inplace=True) #inplace = true as we don't want to store it in another dataset
df.head() #to check wheather changes are reflected to the original dataset or not<jupyter_output><empty_output><jupyter_text>Data Description : Dataset features- Type : NIFTY- Date : Specific trading Date- Time : Time of the trade- Open : Opening price of the trade- High : maximum price at particular time- Low : minimum price at particular time- Close : Closing price- Total change - { Close - Open }  Type of the data<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 22805 entries, 0 to 22804
Data columns (total 7 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   Type    22805 non-null  object 
 1   Date    22805 non-null  int64  
 2   Time    22805 non-null  object 
 3   open    22805 non-null  float64
 4   high    22805 non-null  float64
 5   low     22805 non-null  float64
 6   close   22805 non-null  float64
dtypes: float64(4), int64(1), object(2)
memory usage: 1.2+ MB<jupyter_text>Inshights : Date and Time is listed as int64 and object respectively, so we need to convert it into more readable type, but before we do it let's check out about the missing values and duplicates in the dataset  Finding about the missing values<jupyter_code>df.isnull().values.any()<jupyter_output><empty_output><jupyter_text>Inshight : False means there are no null values in the dataset   Finding about the duplicates<jupyter_code>df[df.duplicated()]<jupyter_output><empty_output><jupyter_text>Inshight : So by looking at the output we can say there are no duplicates in our dataset.  Working with date and time column, it is very important in the data about the datetime that we add month, year, day, hour and other additional information to get better inshights<jupyter_code>df['Date'] = dt = pd.to_datetime(df["Date"],format="%Y%m%d")
df.head()
df["Year"] = df.Date.dt.year
df["month"] = df.Date.dt.month
df["day"] = df.Date.dt.day
df["day_of_week"] = df.Date.dt.dayofweek
df["week"] = df.Date.dt.isocalendar().week
df.head(10)
# df.head(10000)
# Important Note : The ISO definition for week 1 is the week with the first Thursday in it. Using the ISO week number may give unexpected results near the start and end of a year. 
# For instance, in above output 01/01/2021 falls in week 53 of year 2020.
# df.drop(["Date"], axis = 1, inplace = True)  #now we don't need date coulumn 
df.head()
df["hour"] = pd.to_datetime(df["Time"]).dt.hour
df["min"] = pd.to_datetime(df["Time"]).dt.minute
df.head()
df.drop(["Time"], axis = 1, inplace = True)
df.head()
#df.head(1000)<jupyter_output><empty_output><jupyter_text>Calculating the change in percentage that occured since the opening<jupyter_code>df['change'] = df['close'] - df['open']
df['change_in_percentage'] =   round( (( (df['close'] - df['open'] ) / df['open'] ) * 100 ), 3)
df.head()
# df.head(1000)<jupyter_output><empty_output><jupyter_text>Now that we have all the features in our data we need, we can describe our data<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Question : In particular week, month, found out changes in opening, closing, high and low<jupyter_code>df.week.unique()
df.Year.unique()
df.month.unique()
df.day.unique()
df.day_of_week.unique()
# code for : int(datetime.datetime.today().strftime('%w')) to change the start of the week<jupyter_output><empty_output><jupyter_text>Technical Note: here python starts 0 with monday so if you want to change you can refere [Documentation](https://docs.python.org/3/library/datetime.htmlstrftime-and-strptime-format-codes)  Inshights from above : In our data,- For year, We have only year 2021 data- For month, We have only 3 months data which consists of January, February & March- For week, We have only limited weeks from year 2021- For day of week, We have only data of Monday to Thursday- In given data, Data consists of all the days in the particular month.<jupyter_code>short_period = 12
long_period = 26
signal_period = 9
df['ema_12'] = df['close'].ewm(span=short_period, adjust=False).mean()
df['ema_26'] = df['close'].ewm(span=long_period, adjust=False).mean()
df['ema_50'] = df['close'].ewm(span=50, adjust=False).mean()
df['ema_100'] = df['close'].ewm(span=100, adjust=False).mean()
df['ema_200'] = df['close'].ewm(span=200, adjust=False).mean()
ewm_short = df['ema_12']
ewm_long = df['ema_26']
df['MACD'] =  ewm_long - ewm_short
df['Signal'] =  df['close'].ewm(span=signal_period, adjust=False).mean()
df.head()
df['temp'] = df['close'].shift(1)
# df.head()
df['Returns'] = (df['close']/df['temp']) -1
df.drop(['temp'],axis=1,inplace=True)
df.head()
# Plot to view the same
df.plot(x = "Date", y = ['close', 'ema_12', 'ema_26', 'ema_50', 'ema_100', 'ema_200'] , subplots = True, layout = (3, 3), figsize = (15, 15), title = "Close vs EMA", rot = 90)<jupyter_output><empty_output><jupyter_text>Corelation Matrix<jupyter_code># corelation matrix
plt.figure(figsize=(12,8))
sns.heatmap(df.corr(),cmap='Blues',annot=True) 
k = 13 #number of variables for heatmap
cols = df.corr().nlargest(k, 'change')['change'].index
cm = df[cols].corr()
plt.figure(figsize=(14,10))
sns.heatmap(cm, annot=True, cmap = 'viridis')
plt.plot(df.index, df['open'])
plt.grid(True)
plt.xticks(rotation=90)
plt.figure(figsize=(10,10))
plt.show()
plt.plot(df.index, df['close'])
plt.grid(True)
plt.xticks(rotation=90)
plt.figure(figsize=(10,10))
plt.show()<jupyter_output><empty_output><jupyter_text>Inshights : - Open and close both variables are highly corelated to each other as well to high and low - By looking at the visualization we can say that there is not much difference in the trends of the plots  Visualizing the skewness<jupyter_code>Column_List = ['open', 'high', 'low', 'close'] #We don't want to check other columns

fig, ax = plt.subplots(len(Column_List), figsize = (20, 20))

for i, col_list in enumerate(Column_List):
    sns.distplot(df[col_list], hist = True, ax = ax[i])
    ax[i].set_title ("Frequency Distribution of " + col_list, fontsize = 14)
    ax[i].set_xlabel (col_list, fontsize = 14)
    ax[i].set_ylabel ('Distribution Value', fontsize = 14)
    ax[i].grid('on') 
    fig.tight_layout (pad = 5)<jupyter_output>D:\Yas_PC\Currently Installed Softwares\lib\site-packages\seaborn\distributions.py:2557: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)
D:\Yas_PC\Currently Installed Softwares\lib\site-packages\seaborn\distributions.py:2557: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)
D:\Yas_PC\Currently Installed Softwares\lib\site-packages\seaborn\distributions.py:2557: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with si[...]<jupyter_text>Inshights : Frequency distribution of the open, close, high & Low is pretty similar   Outlier<jupyter_code>for i, col_list in enumerate(Column_List):
    fig = plt.figure()
    fig.suptitle('BoxPlot for '+col_list, fontsize=14, fontweight='bold')

    ax = fig.add_subplot(111)
    ax.boxplot(x=df[col_list])
    ax.set_ylabel('Values')
    ax.grid('on')
    plt.show()<jupyter_output><empty_output><jupyter_text>Inshights : By looking at the boxplots, there's no/little difference in the similarity of the boxplots.<jupyter_code>import pickle
pickle_out = open("niftydf.pickle","wb")
pickle.dump(df, pickle_out)
pickle_out.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12903934930754013, 'fraction_numerical': 0.033523851395911186, 'mean_word_length': 4.766159695817491, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '21949', 'license': 'Apache-2.0', 'n_tokens_mistral': 3293, 'n_tokens_neox': 3015, 'n_words': 928, 'path': 'eda_for_dataframe.ipynb', 'repo_name': 'ChaturyaKatragadda/stock-price-prediction', 'size': '9098'}