<jupyter_start><jupyter_text>**Avacado Prices : Descriptive Analysis**The dataset represents weekly 2018 retail scan data for National retail volume (units) and price. Retail scan data comes directly from retailers’ cash registers based on actual retail sales of Hass avocados. The Average Price (of avocados) in the data reflects a per unit (per avocado) cost, even when multiple units (avocados) are sold in bags. The Product Lookup codes (PLU’s) in the table are only for Hass avocados. Other varieties of avocados (e.g. greenskins) are not included in this table.Some relevant columns in the dataset:Date - The date of the observationAveragePrice - the average price of a single avocadotype - conventional or organicyear - the yearRegion - the city or region of the observationTotal Volume - Total number of avocados sold4046 - Total number of avocados with PLU 4046 sold4225- Total number of avocados with PLU 4225 sold4770 - Total number of avocados with PLU 4770 sold<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt # standard graphics
import seaborn as sns # fancier graphics
import pandas as pd
import matplotlib
matplotlib.use("Agg", warn=False)
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import pandas_profiling
%matplotlib inline

import plotly.offline as py
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode
init_notebook_mode(connected=True)
from plotly import tools

import warnings
warnings.filterwarnings("ignore")
warnings.filterwarnings("ignore",category=DeprecationWarning)

data=pd.read_csv("/content/drive/MyDrive/avocado (1).csv")
type(data)
#Show the first rows
data.head()<jupyter_output><empty_output><jupyter_text>The first column is an index, so drop it out<jupyter_code>data = data.drop(columns = 'Unnamed: 0')
data.head()
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 18249 entries, 0 to 18248
Data columns (total 13 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   Date          18249 non-null  object 
 1   AveragePrice  18249 non-null  float64
 2   Total Volume  18249 non-null  float64
 3   4046          18249 non-null  float64
 4   4225          18249 non-null  float64
 5   4770          18249 non-null  float64
 6   Total Bags    18249 non-null  float64
 7   Small Bags    18249 non-null  float64
 8   Large Bags    18249 non-null  float64
 9   XLarge Bags   18249 non-null  float64
 10  type          18249 non-null  object 
 11  year          18249 non-null  int64  
 12  region        18249 non-null  object 
dtypes: float64(9), int64(1), object(3)
memory usage: 1.8+ MB<jupyter_text>What are 4046, 4225 and 4770?Reading of avocado varieties: https://producebrands.com/the-avocado/ gives explanations:4046 = Hass – small4225 = Hass – large4770 = Hass Extra LargeWe rename the columns accordingly.<jupyter_code>data = data.rename(columns = {'4046': 'small', '4225': 'large', '4770': 'xl'})
data.head()
# Show the last rows
data.tail()<jupyter_output><empty_output><jupyter_text>Descriptive Statistics<jupyter_code># rows, columns
data.shape
data.columns (It will show the name of ll columns)
data.isnull().sum()
# Show the descriptive statistics using describe()     

data.describe()
# What is the mean AveragePrice? (notice that there is a column of named 'AveragePrice')     

data['AveragePrice'].mean()
# Print the mean with two decimals only     
#Hint: use round()

m = data['AveragePrice'].mean()
print('Mean value of AveragePrice = ', round(m,2))
# What is the standard deviation of Average Price?        

s = data['AveragePrice'].std()
print('Standard Deviation of AveragePrice = ', round(m,2))
# Printing the standard deviation of Average Price with two decimals only   
# print('Standard deviation of Average Price = ',round(s,2)).<jupyter_output><empty_output><jupyter_text>If we compare the data.head() and data.describe(), we notice that 'type' and 'region' columns were dropped out from the descriptive statistics table as they were text (or categorical data). However, we can study how many unique values these variables contain and the distribution of the values.<jupyter_code># What are the unique type values?          

data['type'].unique()
# How many rows i.e., observations there are for each type?         

data['type'].value_counts()
# Display the descriptive statistics grouped by 'type'      

data.groupby('type').describe()
# Display the descriptive statistics for 'AveragePrice' grouped by 'type'               

data_by_type = data.groupby('type')
data_by_type['AveragePrice'].describe()
# Compare the mean of AveragePrice between different types?              

data['AveragePrice'].groupby(data['type']).mean()
# Show the distribution of the average prices using Histogram? (Hint: use bins = 30)        

data.hist(column = 'AveragePrice', bins = 30, figsize = (8, 6))
plt.xlabel('Price')
plt.ylabel('Count')
plt.title('Distribution of avocado average prices')
plt.show()<jupyter_output><empty_output><jupyter_text>Average Price distribution shows that for most cases price of avocado is between 1.1, 1.4.<jupyter_code># Use a seaborn to create distribution plot of 'AveragePrice'            

plt.figure(figsize=(12,5))
plt.title("Distribution of avocado average price")
ax = sns.distplot(data["AveragePrice"], color = 'b')
plt.show()<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/distributions.py:2557: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>If we observe the distribution closely, it seems that we have two peaks in our distribution. Could this be due to different avocado types?<jupyter_code># Can we overlay the distribution of average price grouped by type?
#Hint: plot two distribution plot using seaborn for each type in a single figure           

#set figure size
plt.figure(figsize=(12,5))

# Plot the distribution of conventional type data
sns.distplot(data["AveragePrice"][data['type'] == 'conventional'], color = 'r', label = 'conventional')
sns.distplot(data["AveragePrice"][data['type'] == 'organic'], color = 'g', label = 'organic')

# add legend, show the graphics
plt.legend()
plt.grid()
plt.title("Distribution of average price grouped by type")
plt.show()
# Make a boxplot graph using pandas or seaborn to compare 'Avrage Price' by 'type'    

#boxplot using pandas
data.boxplot(column = 'AveragePrice', by = 'type', figsize = (8,6))
plt.show()
# boxplot with seaborn

plt.figure(figsize=(12,5))
sns.boxplot(y = "type", x = "AveragePrice", data = data)
plt.xlim([0, 4])
plt.show()<jupyter_output><empty_output><jupyter_text>There is no use of the column"Unnamed:0" is of no use hence we will remove it<jupyter_code>data.drop('Unnamed: 0',axis=1,inplace=True)
data.head()<jupyter_output><empty_output><jupyter_text>Above we have observed that Date is Object type not the date type. We have to change its type to date type.<jupyter_code>data['Date']=pd.to_datetime(data['Date'])
data['Month']=data['Date'].apply(lambda x:x.month)
data['Day']=data['Date'].apply(lambda x:x.day)
data.head()<jupyter_output><empty_output><jupyter_text>**Average price distriution as per the region**<jupyter_code>region_list=list(data.region.unique())
average_price=[]

for i in region_list:
    x=data[data.region==i]
    region_average=sum(x.AveragePrice)/len(x)
    average_price.append(region_average)

data1=pd.DataFrame({'region_list':region_list,'average_price':average_price})
new_index=data1.average_price.sort_values(ascending=False).index.values
sorted_data=data1.reindex(new_index)

plt.figure(figsize=(24,10))
ax=sns.barplot(x=sorted_data.region_list,y=sorted_data.average_price)

plt.xticks(rotation=90)
plt.xlabel('Region')
plt.ylabel('Average Price')
plt.title('Average Price of Avocado According to Region')<jupyter_output><empty_output><jupyter_text>As per above graph prices are high in Hartford SpringfieldSanfranciscoNewYorkPhiladelphiaSacromento **Average consumption as per the region**<jupyter_code>filter1=data.region!='TotalUS'
data1=data[filter1]

region_list=list(data1.region.unique())
average_total_volume=[]

for i in region_list:
    x=data1[data1.region==i]
    average_total_volume.append(sum(x['Total Volume'])/len(x))
data3=pd.DataFrame({'region_list':region_list,'average_total_volume':average_total_volume})

new_index=data3.average_total_volume.sort_values(ascending=False).index.values
sorted_data1=data3.reindex(new_index)

plt.figure(figsize=(22,10))
ax=sns.barplot(x=sorted_data1.region_list,y=sorted_data1.average_total_volume)

plt.xticks(rotation=90)
plt.xlabel('Region')
plt.ylabel('Average of Total Volume')
plt.title('Average of Total Volume According to Region')<jupyter_output><empty_output><jupyter_text>As per above graph West,California,South Central,Northeast,Southeast region have high consumption **Distribution of average price as per the date**<jupyter_code>byDate=data.groupby('Date').mean()
plt.figure(figsize=(12,8))
byDate['AveragePrice'].plot()
plt.title('Average Price')<jupyter_output><empty_output><jupyter_text>We can see that there is hike in prices after July 2017 and before Jan 2018.  Correlation of data with each other<jupyter_code>plt.figure(figsize=(12,6))
sns.heatmap(data.corr(),cmap='coolwarm',annot=True)<jupyter_output><empty_output><jupyter_text>As per above, all the Features are not correleted with the Average Price column, instead most of them are correlated with each other. Feature Engineering for Model building<jupyter_code>data['region'].nunique()
data
data['type'].nunique()<jupyter_output><empty_output><jupyter_text>As per above we have 54 regions and 2 unique types, so it's not difficult  to transform the type feature to dummies, but as far as region is concerned its going to be a bit hard, hence we have to drop the entire column.We have to drop the Date Feature because there are 3 other columns for the Year, Month and Day<jupyter_code>data_final=pd.get_dummies(data.drop(['region','Date'],axis=1),drop_first=True)
data_final.head()
data_final.tail()<jupyter_output><empty_output><jupyter_text>Model selection/predictions We will apply Linear Regression because our Target variable 'AveragePrice' is continuous. We will need to first split up our data into an X array that contains the features to train on, and a y array with the target variable<jupyter_code>X=data_final.iloc[:,1:14]
y=data_final['AveragePrice']
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=42)<jupyter_output><empty_output><jupyter_text>Creating and Training the Model<jupyter_code>from sklearn.linear_model import LinearRegression
lr=LinearRegression()
lr.fit(X_train,y_train)
pred=lr.predict(X_test)
from sklearn import metrics
print('MAE:', metrics.mean_absolute_error(y_test, pred))
print('MSE:', metrics.mean_squared_error(y_test, pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, pred)))<jupyter_output>MAE: 0.23297133291683372
MSE: 0.09108802805353655
RMSE: 0.30180793239001613<jupyter_text>The RMSE is low so we can say that we do have a good model, but lets check to be more sure.Lets plot the y_test vs the predictions<jupyter_code>plt.scatter(x=y_test,y=pred)<jupyter_output><empty_output><jupyter_text>Above we don't have a straight line so  this is not the best model we can apply on our dataLets try working with the Decision Tree Regression model<jupyter_code>from sklearn.tree import DecisionTreeRegressor
dtr=DecisionTreeRegressor()
dtr.fit(X_train,y_train)
pred=dtr.predict(X_test)
plt.scatter(x=y_test,y=pred)
plt.xlabel('Y Test')
plt.ylabel('Predicted Y')<jupyter_output><empty_output><jupyter_text>Here we can see that we nearly have a straight line, in other words its better than the Linear regression model, and to be more sure lets check the RMSE<jupyter_code>print('MAE:', metrics.mean_absolute_error(y_test, pred))
print('MSE:', metrics.mean_squared_error(y_test, pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, pred)))<jupyter_output>MAE: 0.1352739726027397
MSE: 0.044339424657534245
RMSE: 0.2105692870708695<jupyter_text>Our RMSE is lower than the previous one we got with Linear Regression. Now I am going to try one last model to see if I can improve my predictions for this data which is the RandomForestRegressor<jupyter_code>from sklearn.ensemble import RandomForestRegressor
rdr = RandomForestRegressor()
rdr.fit(X_train,y_train)
pred=rdr.predict(X_test)
print('MAE:', metrics.mean_absolute_error(y_test, pred))
print('MSE:', metrics.mean_squared_error(y_test, pred))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, pred)))<jupyter_output>MAE: 0.10042646575342466
MSE: 0.021281194005479447
RMSE: 0.14588075269026907<jupyter_text>Well as we can see the RMSE is lower than the two previous models, so the RandomForest Regressor is the best model in this case<jupyter_code>sns.distplot((y_test-pred),bins=50)<jupyter_output><empty_output><jupyter_text>Notice here that our residuals looked to be normally distributed and that's really a good sign which means that our model was a correct choice for the data.<jupyter_code>Lets see final Actual Vs Predicted sample.
data = pd.DataFrame({'Y Test':y_test , 'Pred':pred},columns=['Y Test','Pred'])
sns.lmplot(x='Y Test',y='Pred',data=data,palette='rainbow')
data.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10427337918922874, 'fraction_numerical': 0.03314795843699693, 'mean_word_length': 5.881671701913394, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 89, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '15992', 'license': 'MIT', 'n_tokens_mistral': 4776, 'n_tokens_neox': 4324, 'n_words': 1323, 'path': 'Avocado _EDA (1).ipynb', 'repo_name': 'jagman19/Avacado-EDA-with-ML-predictions', 'size': '13666'}