<jupyter_start><jupyter_text>5. Model the Solution  Preprocessing to get the tidy dataframe<jupyter_code># Import the library we need, which is Pandas and Matplotlib
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
# Set some parameters to get good visuals - style to ggplot and size to 15,10
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (15, 10)
# Read the csv file of Monthwise Quantity and Price csv file we have.
df = pd.read_csv('MonthWiseMarketArrivals_clean.csv')
# Changing the date column to a Time Interval columnn
df.date = pd.DatetimeIndex(df.date)
# Change the index to the date column
df.index = pd.PeriodIndex(df.date, freq='M')
# Sort the data frame by date
df = df.sort_values(by = "date")
df.head()<jupyter_output><empty_output><jupyter_text>Question 3: How is Price and Quantity related for Onion in Bangalore?<jupyter_code>dfBang = df[df.city == 'BANGALORE']
dfBang.head()
dfBang.plot(kind = "scatter", x = "quantity", y = "priceMod", s = 100)
dfBang.plot(kind = "scatter", x = "quantity", y = "priceMod", s = 100, alpha = 0.7, xlim = [0,2000000])<jupyter_output><empty_output><jupyter_text>PRINCIPLE: CorrelationCorrelation refers to any of a broad class of statistical relationships involving dependence, though in common usage it most often refers to the extent to which two variables have a linear relationship with each other.![](../img/corr.svg)<jupyter_code>dfBang.corr()
pd.set_option('precision', 2)
dfBang.corr()
from pandas.tools.plotting import scatter_matrix
scatter_matrix(dfBang, figsize=(15, 15), diagonal='kde', s = 50)<jupyter_output><empty_output><jupyter_text>PRINCIPLE: Linear Regression<jupyter_code>import statsmodels.api as sm
x = dfBang.quantity
y = dfBang.priceMod
lm = sm.OLS(y, x).fit()
lm.summary()<jupyter_output><empty_output><jupyter_text>PRINCIPLE: Visualizing linear relationships<jupyter_code># Import seaborn library for more funcitionality
import seaborn as sns
# We can try and fit a linear line to the data to see if there is a relaltionship
sns.regplot(x="quantity", y="priceMod", data=dfBang);
sns.jointplot(x="quantity", y="priceMod", data=dfBang, kind="reg");<jupyter_output><empty_output><jupyter_text>Question 4: Can we forecast the price of Onion in Bangalore?  TIme Series ModellingHowever, we have our data at constant time intervals of every month. Therefore we can analyze this data  to determine the long term trend so as to forecast the future or perform some other form of analysis. - Instead of using linear regression model where observations are independent, our observations are really time-dependent and we should use that.- Second, we need to account for both a trend component and seasonality component in the time series data to better improve our forecast<jupyter_code># Set some parameters to get good visuals - style to ggplot and size to 15,10
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (15, 10)
dfBang.index = pd.DatetimeIndex(dfBang.date)
dfBang.head()
# Let us create a time series variable for priceMin
ts = dfBang.priceMin
ts.plot()
# We take the log transform to reduce the impact of high values
ts_log = np.log(ts)
ts_log.plot()
# One approach to remove the trend and seasonality impact is to take the difference between each observation
ts_log_diff = ts_log - ts_log.shift()
ts_log_diff.plot()
ts_log.plot()
# For smoothing the values we can use
# 12 month Moving Averages 
ts_log_diff_ma = pd.rolling_mean(ts_log_diff, window = 12)
# Simple Exponential Smoothing
ts_log_diff_exp = pd.ewma(ts_log_diff, halflife=24)
ts_log_diff_ma.plot()
ts_log_diff_exp.plot()
ts_log_diff.plot()<jupyter_output><empty_output><jupyter_text>Now we can fit an ARIMA model on this (Explaining ARIMA is out of scope of this workshop)<jupyter_code>from statsmodels.tsa.arima_model import ARIMA
model = ARIMA(ts_log, order=(0, 1, 2))
results_MA = model.fit(disp=-1)  
plt.plot(ts_log_diff)
plt.plot(results_MA.fittedvalues, color='blue')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09540459540459541, 'fraction_numerical': 0.012987012987012988, 'mean_word_length': 5.929065743944637, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '144975', 'license': 'MIT', 'n_tokens_mistral': 1382, 'n_tokens_neox': 1301, 'n_words': 447, 'path': 'onion/5-Model.ipynb', 'repo_name': 'u6yuvi/intro-python-data-science', 'size': '4004'}