<jupyter_start><jupyter_text>Pre-processing of the Dataset "Avocado prices" Goal: - Create a new data set that can fit different kind of models, by removing the "NaN" and "String" variables.- Get RMSE of different models to compare later with generated models Avocado data setCharacteristics:    - Number of instances: 18246    - Number of attributes: 13    - Attribute information:        "Date": The date of the observation        "AveragePrice":  the average price of a single avocado        "Type": conventional or organic        "Year": the year        "Region": the city or region of the observation        "Total Volume": Total number of avocados sold        "4046": Total number of avocados with PLU 4046 sold        "4225": Total number of avocados with PLU 4225 sold        "4770": Total number of avocados with PLU 4770 sold        "Total bags": Total number of bags sold        "Small bags": Total number of small bags sold        "Large bags": Total number of large bags sold        "XLarge bags": Total number of XLarge bags sold        This is a copy of the Avocado prices dataset: https://www.kaggle.com/neuromusic/avocado-prices"This data was downloaded from the Hass Avocado Board website in May of 2018 & compiled into a single CSV"<jupyter_code>import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Exploring the Avocado "prices" data set  1. Importing the data set<jupyter_code>df = pd.read_csv('../Data/avocado_price/avocado.csv')
original_df = df
df = df.drop("Unnamed: 0",axis=1)
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>The dataset has 18249 and 13 columns, which means we have information over 18248 dates and 13 differents features. Now, we know what we will work with.  2. Data pre-processing  Missing values Most ML models can't handle specific type of data such as "NaN" (which means Not a Number and usualy stand for missig values) or "String". Therfore, we'll have to handle them by modifying our dataset.<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 18249 entries, 0 to 18248
Data columns (total 13 columns):
Date            18249 non-null object
AveragePrice    18249 non-null float64
Total Volume    18249 non-null float64
4046            18249 non-null float64
4225            18249 non-null float64
4770            18249 non-null float64
Total Bags      18249 non-null float64
Small Bags      18249 non-null float64
Large Bags      18249 non-null float64
XLarge Bags     18249 non-null float64
type            18249 non-null object
year            18249 non-null int64
region          18249 non-null object
dtypes: float64(9), int64(1), object(3)
memory usage: 1.8+ MB<jupyter_text>There is no missing values. Let's work on the string values then.  String values There is 3 different string column: "Date","type" and "region". Let's start with date. We split the date in 2 smaller columns for the day and the month ( the year is aleady provided )<jupyter_code>df['Date']=pd.to_datetime(df['Date'])
df['Month']=df['Date'].apply(lambda x:x.month)
df['Day']=df['Date'].apply(lambda x:x.day)
df = df.drop(['Date'],axis=1)<jupyter_output><empty_output><jupyter_text>Now that Date is done, we can work on Type. We create a fonction allowing us to create "dummie features" to split string features in categoricals features and use it on the "Type" feature<jupyter_code>def create_dummies(df,column_name):
    dummies = pd.get_dummies(df[column_name],prefix=column_name)
    df = pd.concat([df,dummies],axis=1)
    return df

df = create_dummies(df,'type').drop('type',axis=1)
df = create_dummies(df,'type').drop('type_organic',axis=1)<jupyter_output><empty_output><jupyter_text>For the last one "Region" we map each region with a number<jupyter_code>df_region = df
mapping = {'Albany' : '0' , 'Atlanta' : '1' , 'BaltimoreWashington' : '2' , 'Boise' : '3' , 'Boston' : '4' ,'BuffaloRochester' : '5' , 'California' : '6' , 'Charlotte' : '7' , 'Chicago' : '8' ,'CincinnatiDayton' : '9' , 'Columbus' : '10' , 'DallasFtWorth' : '11' , 'Denver' : '12' ,'Detroit' : '13' , 'GrandRapids' : '14' , 'GreatLakes' : '15' , 'HarrisburgScranton' : '16' ,'HartfordSpringfield' : '17' , 'Houston' : '18' , 'Indianapolis' : '19' , 'Jacksonville' : '20' ,'LasVegas' : '21' , 'LosAngeles' : '22' , 'Louisville' : '23' , 'MiamiFtLauderdale' : '24' ,'Midsouth' : '25' , 'Nashville' : '26' , 'NewOrleansMobile' : '27' , 'NewYork' : '28' ,'Northeast' : '29' , 'NorthernNewEngland' : '30' , 'Orlando' : '31' , 'Philadelphia' : '32' ,'PhoenixTucson' : '33' , 'Pittsburgh' : '34' , 'Plains' : '35' , 'Portland' : '36' ,'RaleighGreensboro' : '37' , 'RichmondNorfolk' : '38' , 'Roanoke' : '39' , 'Sacramento' : '40' ,'SanDiego' : '41' , 'SanFrancisco' : '42' , 'Seattle' : '43' , 'SouthCarolina' : '44' ,'SouthCentral' : '45' , 'Southeast' : '46' , 'Spokane' : '47' , 'StLouis' : '48' , 'Syracuse' : '49' ,'Tampa' : '50' , 'TotalUS' : '51' , 'West' : '52' , 'WestTexNewMexico' : '53'}
df.region = df.region.map(mapping)
df.head()<jupyter_output><empty_output><jupyter_text>To conclude, we can change our float to integers, those values are prices, wich means that we just have to change the scale to cents.<jupyter_code>for x in df.columns:
    if (df[x].dtype == 'float64'):
        df[x] = (df[x]*100).astype(int)
df.head()
Our tranformation of this dataset is done, we have no more missing values and no string aswell.<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code>byDate=df.groupby('year').mean()
plt.figure()
byDate['AveragePrice'].plot()
plt.title('Average Price')<jupyter_output><empty_output><jupyter_text>The average price peaked in 2017 and is much lower otherwise<jupyter_code>byMonth = df.groupby("Month").mean()
plt.figure()
plt.plot(["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sept","Oct","Nov","Dec"],byMonth['AveragePrice'])
plt.title('Average Price Per Month')<jupyter_output><empty_output><jupyter_text>The price keep increasing from February to October and then decrease fast<jupyter_code>byDay = df.groupby("Day").mean()
plt.figure()
byDay['AveragePrice'].plot()
plt.title('Average Price Per Day')<jupyter_output><empty_output><jupyter_text>The price keep varying from one day to another, which seems to imply that if the price is high one day, you can wait the day after tommorow for it to decrease significantly.<jupyter_code>import seaborn as sns

byRegion=original_df.groupby('region').mean()
byRegion.sort_values(by=['AveragePrice'], ascending=False, inplace=True)
plt.figure(figsize=(17,8),dpi=250)
sns.barplot(x = byRegion.index,y=byRegion["AveragePrice"],data = byRegion,palette='rocket')
plt.xticks(rotation=80)
plt.xlabel('Region')
plt.ylabel('Average Price')
plt.title('Average Price According to Region')
plt.savefig('Avocado_Average-Price-According-to-Region.svg')<jupyter_output><empty_output><jupyter_text>The average price by region shows us that the region seems to be correlated to the price<jupyter_code>plt.style.use(style='ggplot')
corr = df.corr()
corr_AveragePrice = corr.AveragePrice.sort_values(ascending=False)
corr_AveragePrice.plot(kind='bar', color='blue')
plt.xlabel('Correlation')
plt.ylabel('')
plt.xticks(rotation=75)
plt.legend([])
plt.show()
plt.savefig("corr_avocado.pdf")<jupyter_output><empty_output><jupyter_text>We can observe on the graph above that organic type avocado seems much more expensive than the conventionals. But apart from that, we don't see any high corelated feature.  Creating our models We define AveragePrice as our target Then, we split our data set in train and test data sets with a 33% ratio<jupyter_code>from sklearn.model_selection import train_test_split
X = df.drop(['AveragePrice'],axis=1)
y = df.AveragePrice
X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size=0.33, random_state=42)
from sklearn.ensemble import RandomForestClassifier
random_forest = RandomForestClassifier(n_estimators=10)
random_forest.fit(X_train, Y_train)
Y_pred = random_forest.predict(X_test)
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors = 3)
knn.fit(X_train, Y_train)
Y_pred = knn.predict(X_test)
from sklearn.naive_bayes import GaussianNB
gaussian = GaussianNB()
gaussian.fit(X_train, Y_train)
Y_pred = gaussian.predict(X_test)
from sklearn.tree import DecisionTreeClassifier
decision_tree = DecisionTreeClassifier()
decision_tree.fit(X_train, Y_train)
Y_pred = decision_tree.predict(X_test)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.linear_model import Lasso
lasso = make_pipeline(RobustScaler(), Lasso(alpha =0.0005, random_state=1))
lasso.fit(X_train,Y_train)
from sklearn.linear_model import ElasticNet
ENet = make_pipeline(RobustScaler(), ElasticNet(alpha=0.0005, l1_ratio=.9, random_state=3))
ENet.fit(X_train,Y_train)
from sklearn.kernel_ridge import KernelRidge
KRR = KernelRidge(alpha=0.6, kernel='polynomial', degree=2, coef0=2.5)
KRR.fit(X_train,Y_train)
from sklearn.ensemble import GradientBoostingRegressor
GBoost = GradientBoostingRegressor(n_estimators=3000, learning_rate=0.05,
                                   max_depth=4, max_features='sqrt',
                                   min_samples_leaf=15, min_samples_split=10, 
                                   loss='huber', random_state =5)
GBoost.fit(X_train,Y_train)<jupyter_output><empty_output><jupyter_text>Displaying our results<jupyter_code>We calculate the RMSE for each model
from sklearn.metrics import mean_squared_error
from math import sqrt  
lasso_rmse = sqrt(mean_squared_error(Y_test,lasso.predict(X_test)))
knn_rmse = sqrt(mean_squared_error(Y_test,knn.predict(X_test)))
random_forest_rmse = sqrt(mean_squared_error(Y_test,random_forest.predict(X_test)))
gaussian_rmse = sqrt(mean_squared_error(Y_test,gaussian.predict(X_test)))
decision_tree_rmse = sqrt(mean_squared_error(Y_test,decision_tree.predict(X_test)))
ENet_rmse = sqrt(mean_squared_error(Y_test,ENet.predict(X_test)))
KRR_rmse = sqrt(mean_squared_error(Y_test,KRR.predict(X_test)))
GBoost_rmse = sqrt(mean_squared_error(Y_test,GBoost.predict(X_test)))
models = pd.DataFrame({
    'Model': ['KNN', 
              'Random Forest', 'Naive Bayes', 
              'Decision Tree','Lasso','ENet','KernelRidge','GBoost'],
    'Score': [knn_rmse ,random_forest_rmse , gaussian_rmse , decision_tree_rmse ,lasso_rmse ,ENet_rmse ,KRR_rmse ,GBoost_rmse]})
models.sort_values(by='Score', ascending=True)<jupyter_output><empty_output><jupyter_text>Conclusion By creating a leaderboard with model by their accuracy, we observe:    - The Kernel Ridge model is doing extremly bad    - The average RMSE (removing the extreme one) is around 10   - GBoost have the best RMSE (5.87) We save our processed data set to use it with our Auto-ML models<jupyter_code>df.to_csv('../Data/avocado_price/no_nan_id.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12150901932057503, 'fraction_numerical': 0.03149894698287702, 'mean_word_length': 5.346310284718187, 'pattern_counts': {'":': 13, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '619633', 'license': 'Apache-2.0', 'n_tokens_mistral': 4024, 'n_tokens_neox': 3661, 'n_words': 999, 'path': 'Scripts/Avocado_price.ipynb', 'repo_name': 'Thomas92330/automlbench', 'size': '10921'}