<jupyter_start><jupyter_text>Business Understanding1 - What areas generate more revenue?2 - What kind of properties is best suited for these areas?3 - Can we predict future revenue?To discover this, we will explore data from airbnb from boston and seattle  Data UnderstandingLets take a look into the data to see what kind of information we have  Downloading the datasetsDownload the dataset from:[https://www.kaggle.com/airbnb/seattle/data](https://www.kaggle.com/airbnb/seattle/data)[https://www.kaggle.com/airbnb/boston](https://www.kaggle.com/airbnb/boston)The files of the two cities have the same names, so I renamed for _city (ex: calendar_boston.csv)* Listings, including full descriptions and average review score* Reviews, including unique id for each reviewer and detailed comments* Calendar, including listing id and the price and availability for that dayThere is more information about the data here:http://insideairbnb.com/about.htmland a dictionary here:https://docs.google.com/spreadsheets/d/1iWCNJcSutYqpULSQHlNyGInUvHg2BoUGoNRIGa6Szc4/editgid=982310896<jupyter_code>#importing the data
def import_df(path, prefix, separator='_', col_name='file'):
    '''
    Import all the csv files with the same prefix and concatenate them in the same DF. 
    Create a column with the second splitted element of the separator.
    path = path of the files
    prefix = prefix (name) of the csv files
    separator = the separator of the prefix and the city in the name of the csv
    col_name = name of the column of the splitted element that goes into the dataframe
    '''
    df = pd.DataFrame()
    for file in listdir(path):
        if (file.endswith('.csv') and file.startswith(prefix)):
            df_temp = pd.read_csv(path + file)
            df_temp[col_name] = re.split('_|\.csv', file)[1] #second element of split and remove extension            
            df = pd.concat([df, df_temp], ignore_index=True)
            print('Read file: {}    Shape: {}'.format(file, df_temp.shape))
            
    return df
#gathering the data
df_calendar = import_df(path, 'calendar')
df_listings = import_df(path, 'listings')
df_reviews = import_df(path, 'reviews')<jupyter_output>Read file: calendar_boston.csv    Shape: (1308890, 5)
Read file: calendar_seattle.csv    Shape: (1393570, 5)
Read file: listings_boston.csv    Shape: (3585, 96)
Read file: listings_seattle.csv    Shape: (3818, 93)
Read file: reviews_boston.csv    Shape: (68275, 7)
Read file: reviews_seattle.csv    Shape: (84849, 7)<jupyter_text>Let's have a quick assesment of the data<jupyter_code>df_calendar['date'] = pd.to_datetime(df_calendar['date'])
display(df_calendar.head(3))
df_calendar.groupby(['file']).agg({'date' : [np.min,np.max]}) #see timespan of each file
display(df_listings.head(1))
df_reviews['date'] = pd.to_datetime(df_reviews['date'])
display(df_reviews.head(3))
df_reviews.groupby(['file']).agg({'date' : [np.min,np.max]}) #see timespan of each file<jupyter_output><empty_output><jupyter_text>Lookind at the data, it seems that the boston file was generated on 2016-09-06 and the seattle file was generated on 2016-01-03.We have several useful information:* Geographic information such as lat/lon coordinates, street, neighbourhood, city, state, country and zipcode* Listing characteristics like property_type, room_type and others* Host characteristics such as host_since, host_total_listings_count, host_has_profile_pic, etc.* Review scores review_scores_value, review_scores_rating and more  Data PreparationWe need to clean the data first before trying to answer our question<jupyter_code>#Our Boston listing dataframe have 3 more columns than the Seattle one, and 
#both have a lot of columns that we don't need, so get only the columns that 
#are related to our question (location, property characteristics, price and reviews)
listings_cols = ['id', 'street', 'neighbourhood', 'neighbourhood_cleansed', 'neighbourhood_group_cleansed',
                'city', 'state', 'zipcode', 'market', 'smart_location', 'country', 'latitude', 
                'longitude', 'is_location_exact', 'property_type', 'room_type', 'accommodates', 
                'bathrooms', 'bedrooms', 'beds', 'bed_type', 'amenities', 'square_feet', 'price',
                'weekly_price', 'monthly_price', 'security_deposit', 'cleaning_fee', 'guests_included',
                'extra_people', 'minimum_nights', 'maximum_nights', 'number_of_reviews', 'review_scores_rating',
                'review_scores_accuracy', 'review_scores_cleanliness', 'review_scores_checkin', 'file',
                'review_scores_communication', 'review_scores_location', 'review_scores_value', 'cancellation_policy',
                'availability_365', 'calendar_last_scraped', 'availability_30', 'availability_60', 'availability_90']

df_listings = df_listings[listings_cols]

#Diving a little deeper into the data, we can see that we have 3 columns for neighbourhood
#so we will keep only the one with fewer missing values
missing_neighbourhood = df_listings[['neighbourhood','neighbourhood_cleansed', 'neighbourhood_group_cleansed']].count() / df_listings.shape[0]
print('Missing Values froom Neighbourhood Columns:\n{}'.format(missing_neighbourhood))
df_listings.drop(columns=['neighbourhood', 'neighbourhood_group_cleansed'], inplace=True)

#Since there is a lot of  other cities in the "city" and "smart_location" columns, we will replace it by the 
#values of the "file" column and drop it
print('\nCity Unique Values:\n{}'.format(df_listings['city'].unique()))
df_listings['city'] = df_listings['file']
df_listings.drop(columns=['file', 'smart_location'], inplace=True)

#Convert price related columns to number
price_cols = ['price', 'weekly_price', 'monthly_price', 'security_deposit', 'cleaning_fee', 'extra_people']
df_listings[price_cols] = df_listings[price_cols].replace('[\$,]','', regex=True)
df_listings[price_cols] = df_listings[price_cols].astype(float)

#The reviews dataframe is a little easier, let just keep the columns that we want.
#The comments column have very useful information, but since we won't do any text analysis 
#we won't keep it
reviews_cols = ['listing_id', 'id', 'date', 'reviewer_id', 'file']
df_reviews = df_reviews[reviews_cols]

#And rename the "file" for "city" also
df_reviews.rename(columns={'file' : 'city'}, inplace=True)
#For the calendar, we will convert the "available" flag for 1(true) and 0(false)
df_calendar['available'] = (df_calendar['available'] == 't')*1

#And rename the "file" for "city" also
df_calendar.rename(columns={'file' : 'city'}, inplace=True)<jupyter_output><empty_output><jupyter_text>Calculating occupancyInside Airbnb have an occupancy model that we will replicate, it consist mainly of these 3 characteristics:* **Convert review rate to estimated booking**Accordingly with Airbnb, 72% of the bookings leave reviews, the Budget and Legislative Analyst's Office made their analysis with this value in addition to other of 30.5%. We will go with Inside Airbnb and consider that 50% of the bookings lead to reviews* **Avarage length of stay**We will consider an avarage length of stay of 3 days. If the minimum nights of a listing is higher, then this value is used* **Occupancy rate was capped at 70%**font: http://insideairbnb.com/about.html<jupyter_code>df_booking = df_reviews.copy()
df_booking['month'] = pd.DatetimeIndex(df_booking['date']).month
df_booking['year'] = pd.DatetimeIndex(df_booking['date']).year
df_booking['days_in_month'] = pd.DatetimeIndex(df_booking['date']).daysinmonth
df_booking = df_booking.groupby(['listing_id', 'city', 'month', 'year', 'days_in_month'],
                                as_index=False).agg({'id' : 'count'})
df_booking.rename(columns={'id' : 'reviews'}, inplace=True)
df_booking['booking'] = df_booking['reviews'] * 2 #only 50% leave reviews


df_listings2 = df_listings[['id', 'neighbourhood_cleansed', 'property_type', 'room_type', 'accommodates', 
                            'bathrooms', 'bedrooms', 'beds', 'bed_type', 'square_feet', 'price', 'cancellation_policy',
                            'minimum_nights']]
df_revenue = df_booking.merge(df_listings2, how='left', left_on='listing_id', right_on='id')
df_revenue['minimum_nights'].where(df_revenue['minimum_nights'] > 3, 3, inplace=True)
df_revenue['max_days'] = df_booking['days_in_month'] * 0.7 #capped occupancy
df_revenue['days'] = df_revenue['booking'] * df_revenue['minimum_nights']
df_revenue['days'] = df_revenue[['days', 'max_days']].min(axis=1)                         
df_revenue['occupancy_rate'] = df_revenue['days'] / df_booking['days_in_month']
df_revenue['estimated_revenue'] = df_revenue['days'] * df_revenue['price']<jupyter_output><empty_output><jupyter_text>Analysing the data to answer questions 1 and 2Visualizing the data<jupyter_code>def plot_revenue_neighbourhood(year = None):
    '''
    Plot stacked barchart of Top 10 Seattle and Boston Neighbourhood by Property Type
    year = filter specific year, if nothing is passed it doesn't filter any year
    '''

    if year is not None:
        df_filtered = df_revenue[df_revenue['year'] == year]
        year_title = year
    else:
        df_filtered = df_revenue
        year_title = 'All Years'
        
    df_revenue_plot = df_filtered.groupby(['neighbourhood_cleansed', 'city', 'property_type'
                                         ]).agg({'estimated_revenue' : 'sum', 'listing_id' : 'count'})
    df_revenue_plot.rename(columns={'listing_id' : 'listing_quantity'}, inplace=True)
    df_revenue_plot.reset_index(inplace=True)

    #ranking and top 10 for each city
    df_revenue_temp = df_filtered.groupby(['neighbourhood_cleansed', 'city'
                                         ]).agg({'estimated_revenue' : 'sum'})
    df_revenue_temp.reset_index(inplace=True)
    df_revenue_temp.sort_values('estimated_revenue', ascending=False, inplace=True)
    df_revenue_temp['rank'] = df_revenue_temp.groupby(['city'])['estimated_revenue'].rank(ascending=False, method='dense')
    df_revenue_temp.drop(columns=['estimated_revenue'], inplace=True)
    df_revenue_plot = df_revenue_plot.merge(df_revenue_temp, how='left', on=['city','neighbourhood_cleansed'])
    df_revenue_plot.sort_values('rank', inplace=True)

    df_revenue_plot_boston = df_revenue_plot[(df_revenue_plot['city'] == 'boston') & (df_revenue_plot['rank'] <= 10)]
    df_revenue_plot_seattle = df_revenue_plot[(df_revenue_plot['city'] == 'seattle') & (df_revenue_plot['rank'] <= 10)]

    #get xlabels
    boston_labels = list(df_revenue_plot_boston['neighbourhood_cleansed'].drop_duplicates())
    seattle_labels = list(df_revenue_plot_seattle['neighbourhood_cleansed'].drop_duplicates())

    df_pivot_boston = df_revenue_plot_boston.pivot_table(index=['rank'], columns=['property_type'],
                                      values=['estimated_revenue'], fill_value=0, aggfunc='sum', margins=True)
    df_pivot_boston.sort_values(by='All', ascending=False, axis=1, inplace=True) #sort columns by total value 
    df_pivot_boston.drop(columns=['All'], level=1, inplace=True)
    df_pivot_boston.drop('All', axis=0, inplace=True)
    df_pivot_boston = df_pivot_boston.droplevel(0,axis=1)

    df_pivot_seattle = df_revenue_plot_seattle.pivot_table(index=['rank'], columns=['property_type'],
                                      values=['estimated_revenue'], fill_value=0, aggfunc='sum', margins=True)
    df_pivot_seattle.sort_values(by='All', ascending=False, axis=1, inplace=True)
    df_pivot_seattle.drop(columns=['All'], level=1, inplace=True)
    df_pivot_seattle.drop('All', axis=0, inplace=True)
    df_pivot_seattle = df_pivot_seattle.droplevel(0,axis=1)


    #plotting
    fig, ax = plt.subplots(1,2, figsize=(15,7), sharey=True)

    df_pivot_seattle.plot.bar(stacked=True, ax=ax[0], legend=True, colormap='tab20')
    df_pivot_boston.plot.bar(stacked=True, ax=ax[1], legend=False, colormap='tab20')


    #adjusting appearance and labels
    ax[0].legend()
    ax[1].legend()
    ax[0].set_xticklabels(seattle_labels, rotation=70, fontsize=11)
    ax[1].set_xticklabels(boston_labels, rotation=70, fontsize=11)
    ax[0].set_ylabel('Estimated Revenue in Million USD', fontsize=13)
    ax[0].set_xlabel('Neighbourhood', fontsize=13)
    ax[1].set_xlabel('')
    ax[0].set_title('Seattle by Property Type', fontsize=16)
    ax[1].set_title('Boston by Property Type', fontsize=16)
    ax[0].xaxis.set_label_coords(1.05, -0.3)
    ax[0].spines['right'].set_visible(False)
    ax[0].spines['top'].set_visible(False)
    ax[1].spines['right'].set_visible(False)
    ax[1].spines['top'].set_visible(False)
    fig.suptitle('Top 10 Neighbourhood ' + str(year_title), fontsize=20)

    plt.tight_layout()
    plt.show()
#answering question 1:
#lets see a 2015 comparison since is the last full year of both cities
plot_revenue_neighbourhood(2015)<jupyter_output><empty_output><jupyter_text>Here we can see that Broadway is the leader with it’s estimated revenue above 2.5 million USD followed by South End.We can identify that Seattle’s revenue is concentrated in two main neighbourhoods, Boston on the other hand is a little more uniformly distributed.<jupyter_code>#and for all years
plot_revenue_neighbourhood()<jupyter_output><empty_output><jupyter_text>This indicates to us that Boston has proably had a headstart with Airbnb rentals comparing with Seattle, since most of the neighbourhoods have a larger estimated revenue<jupyter_code>def top_neighbourhood(year = None):
    '''
    Return DataFrame with top 10 neighbourhood and the top 1 property type for each city
    year = filter specific year, if nothing is passed it doesn't filter any year
    '''
    if year is not None:
        df_filtered = df_revenue[df_revenue['year'] == year]
        year_title = year
    else:
        df_filtered = df_revenue
        year_title = 'All Years'

    df_neighbourhood = df_filtered.groupby(['neighbourhood_cleansed', 'city', 'property_type'
                                             ]).agg({'estimated_revenue' : 'sum', 'listing_id' : 'count'})
    df_neighbourhood.rename(columns={'listing_id' : 'listing_quantity'}, inplace=True)
    df_neighbourhood.reset_index(inplace=True)

    #ranking for each city and top 10
    df_neighbourhood_temp = df_filtered.groupby(['neighbourhood_cleansed', 'city'
                                         ]).agg({'estimated_revenue' : 'sum'})
    df_neighbourhood_temp.reset_index(inplace=True)
    df_neighbourhood_temp.sort_values('estimated_revenue', ascending=False, inplace=True)
    df_neighbourhood_temp['rank'] = df_neighbourhood_temp.groupby(['city'])['estimated_revenue'].rank(ascending=False, method='dense')
    df_neighbourhood_temp.drop(columns=['estimated_revenue'], inplace=True)
    df_neighbourhood = df_neighbourhood.merge(df_neighbourhood_temp, how='left', on=['city','neighbourhood_cleansed'])
    df_neighbourhood.sort_values('rank', inplace=True)

    df_neighbourhood_boston = df_neighbourhood[(df_neighbourhood['city'] == 'boston') & (df_neighbourhood['rank'] <= 10)]
    df_neighbourhood_seattle = df_neighbourhood[(df_neighbourhood['city'] == 'seattle') & (df_neighbourhood['rank'] <= 10)]

    idx_boston = df_neighbourhood_boston.groupby(['neighbourhood_cleansed', 'city'])['estimated_revenue'].transform(max) == df_neighbourhood_boston['estimated_revenue']
    df_neighbourhood_boston = df_neighbourhood_boston[idx_boston]
    df_neighbourhood_boston.drop(columns=['city'], inplace=True)
    df_neighbourhood_boston['Average Revenue'] = (df_neighbourhood_boston['estimated_revenue']
                                                   / df_neighbourhood_boston['listing_quantity']).map('${:,.2f}'.format)
    df_neighbourhood_boston['estimated_revenue'] = df_neighbourhood_boston['estimated_revenue'].map('${:,.2f}'.format)
    df_neighbourhood_boston.rename(columns={'neighbourhood_cleansed' : 'Boston Neighbourhood', 
                                            'property_type' : 'Boston Property Type',
                                            'estimated_revenue' : 'Boston Estimated Revenue',
                                            'listing_quantity' : 'Boston Listing Quantity',
                                            'Average Revenue' : 'Boston Average Revenue',
                                            'rank' : 'Rank ' + str(year_title)}, inplace=True)
    df_neighbourhood_boston.set_index('Rank ' + str(year_title), inplace=True)


    idx_seattle = df_neighbourhood_seattle.groupby(['neighbourhood_cleansed', 'city'])['estimated_revenue'].transform(max) == df_neighbourhood_seattle['estimated_revenue']
    df_neighbourhood_seattle = df_neighbourhood_seattle[idx_seattle]
    df_neighbourhood_seattle.drop(columns=['city'], inplace=True)
    df_neighbourhood_seattle['Average Revenue'] = (df_neighbourhood_seattle['estimated_revenue']
                                                   / df_neighbourhood_seattle['listing_quantity']).map('${:,.2f}'.format)
    df_neighbourhood_seattle['estimated_revenue'] = df_neighbourhood_seattle['estimated_revenue'].map('${:,.2f}'.format)
    df_neighbourhood_seattle.rename(columns={'neighbourhood_cleansed' : 'Seattle Neighbourhood', 
                                            'property_type' : 'Seattle Property Type',
                                            'estimated_revenue' : 'Seattle Estimated Revenue',
                                            'listing_quantity' : 'Seattle Listing Quantity',
                                            'Average Revenue' : 'Seattle Average Revenue',
                                            'rank' : 'Rank ' + str(year_title)}, inplace=True)
    df_neighbourhood_seattle.set_index('Rank ' + str(year_title), inplace=True)


    df_neighbourhood_return = pd.concat([df_neighbourhood_seattle, df_neighbourhood_boston], axis=1)
    
    return df_neighbourhood_return
#answering question 2:
#getting the top property type for 2015
top_neighbourhood(2015)
#and for all years
top_neighbourhood()<jupyter_output><empty_output><jupyter_text>Now lets see if we can create a model to predict future revenueFor this, we need to modify our dataset to include our main features that contribute to the predictionour target variable will be the estimated_revenue<jupyter_code>df_revenue_ml = df_revenue.drop(columns=['listing_id', 'id', 'occupancy_rate', 'reviews', 'booking']) #we won't have occupancy rate, reviews or booking for new listings

#dealing with missing data
print('Columns missing count: \n{}'.format(df_revenue_ml.isnull().sum()))
print('\nSquare_feet % of missing: {:.2f}'.format(df_revenue_ml['square_feet'].isnull().sum()  / df_revenue_ml.shape[0]))
#since only 5% of the square_feet have data, we will drop it
df_revenue_ml.drop(columns=['square_feet'], inplace=True)
#for property_type, we will create e new classification "not_specified"
df_revenue_ml['property_type'].fillna('Not Specified', inplace=True)
#filling the na for bathrooms with the mean considering property type
#we are filling this way because different properties may have different means
#for example, is very likely that a house have more bathrooms than an camper, so
#by doing this we tend to preserve the features differences between them and still 
#keep it simple to do and understand
bathroom_dict = df_revenue_ml[['property_type', 'bathrooms']].groupby(['property_type']).median()['bathrooms'].to_dict()
df_revenue_ml['bathrooms'].fillna(df_revenue_ml['property_type'].map(bathroom_dict), inplace=True)
#same for bedrooms
bedroom_dic = df_revenue_ml[['property_type', 'bedrooms']].groupby(['property_type']).median()['bedrooms'].to_dict()
df_revenue_ml['bedrooms'].fillna(df_revenue_ml['property_type'].map(bedroom_dic), inplace=True)
#and for beds
beds_dic = df_revenue_ml[['property_type', 'beds']].groupby(['property_type']).median()['beds'].to_dict()
df_revenue_ml['beds'].fillna(df_revenue_ml['property_type'].map(beds_dic), inplace=True)
#verifying if all missing values have been filled
print('\nColumns missing count after filling: \n{}'.format(df_revenue_ml.isnull().sum()))
#dealing with categorical data
print('Columns type: \n{}'.format(df_revenue_ml.dtypes))

#for each categorical data column (object type) we will do a hot encode (create a column for each value of each column)
print('\nNumber of columns to be create: \n{}'.format(df_revenue_ml.select_dtypes('object').nunique()))
#since is not a huge amount of columns, we are good to go

cat_list = list(df_revenue_ml.select_dtypes('object').columns)
df_revenue_ml_encoded =  pd.get_dummies(df_revenue_ml, prefix=cat_list, columns=cat_list)
print('\nDataframe hot encoded shape: {}'.format(df_revenue_ml_encoded.shape))<jupyter_output>Columns type: 
city                       object
month                       int64
year                        int64
days_in_month               int64
neighbourhood_cleansed     object
property_type              object
room_type                  object
accommodates                int64
bathrooms                 float64
bedrooms                  float64
beds                      float64
bed_type                   object
price                     float64
cancellation_policy        object
minimum_nights              int64
max_days                  float64
days                      float64
estimated_revenue         float64
dtype: object

Number of columns to be create: 
city                        2
neighbourhood_cleansed    111
property_type              20
room_type                   3
bed_type                    5
cancellation_policy         4
dtype: int64

Dataframe hot encoded shape: (48888, 157)<jupyter_text>ModelingNow its time to model our algorithm<jupyter_code>#lets take a look in the correlations
fig = sns.pairplot(df_revenue_ml, diag_kind='kde')
plt.show()
#splitting the data
X = df_revenue_ml_encoded.drop(columns=['estimated_revenue'])
y = df_revenue_ml_encoded['estimated_revenue']
X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.3, random_state=0)


def grid_lasso():
    '''
    Does a grid search to optimze hyperparameters of the Lasso regression, return the best hyperparameters, the
    mean score and mean time to fit
    '''
    #GridSearch
    grid_values = {'alpha' : [0.01, 0.1, 1, 10, 100],
                   'warm_start' : [True, False],
                   'positive' : [True, False],
                   'selection' : ['cyclic', 'random']}
    scoring = 'r2'
    ncv = 3

    lasso = Lasso(random_state=0)
    grid_lasso = GridSearchCV(lasso, param_grid=grid_values, scoring=scoring, cv=ncv).fit(X_train, y_train)
    df_grid = pd.DataFrame(grid_lasso.cv_results_)
    df_grid = df_grid[['mean_fit_time', 'param_alpha','param_positive','param_selection',
                       'param_warm_start','mean_test_score']]
    df_grid.sort_values('mean_test_score', ascending=False, inplace=True)
    df_grid = df_grid.iloc[0]
    
    return df_grid

def grid_ridge():
    '''
    Does a grid search to optimze hyperparameters of the Lasso regression, return the best hyperparameters, the
    mean score and mean time to fit
    '''
    #GridSearch
    grid_values = {'alpha' : [0.01, 0.1, 1, 10, 100],
                   'solver' : ['auto', 'svd', 'cholesky', 'sparse_cg', 'lsqr', 'sag', 'saga']}
    scoring = 'r2'
    ncv = 3

    ridge = Ridge(random_state=0)
    grid_ridge = GridSearchCV(ridge, param_grid=grid_values, scoring=scoring, cv=ncv).fit(X_train, y_train)
    df_grid = pd.DataFrame(grid_ridge.cv_results_)
    df_grid = df_grid[['mean_fit_time', 'param_alpha', 'param_solver', 'mean_test_score']]
    df_grid.sort_values('mean_test_score', ascending=False, inplace=True)
    df_grid = df_grid.iloc[0]
    
    return df_grid

lasso_gs = grid_lasso()
print('Lasso gridsearch results:\n{}'.format(lasso_gs))
ridge_gs = grid_ridge()
print('Ridge gridsearch results:\n{}'.format(ridge_gs))

#training and predicting with lasso
start_lasso = time.time()
lasso = Lasso(alpha=0.1, positive=False, selection='random',
              warm_start=False, random_state=0).fit(X_train, y_train)
y_predict_train_lasso = lasso.predict(X_train)
y_predict_lasso = lasso.predict(X_test)
end_lasso = time.time()
print('Lasso Duration: {}'.format(end_lasso - start_lasso))

#training and predicting with Ridge
start_ridge = time.time()
ridge = Ridge(alpha=10, solver='cholesky', random_state=0).fit(X_train, y_train)
y_predict_train_ridge = ridge.predict(X_train)
y_predict_ridge = ridge.predict(X_test)
end_ridge = time.time()
print('Ridge Duration: {}'.format(end_ridge - start_ridge))<jupyter_output>Lasso Duration: 0.5944063663482666
Ridge Duration: 0.11015677452087402<jupyter_text>Evaluation<jupyter_code>print('Lasso Evaluation:')
#using r2_score to evaluate performance
r2_train = r2_score(y_train, y_predict_train_lasso)
r2_test = r2_score(y_test, y_predict_lasso)
print('R2 train score: {}\nR2 test score: {}'.format(r2_train, r2_test))

#using cross validation to estimate scores
cv_scores = cross_val_score(lasso, X, y, cv=5, scoring='r2')
print('Cross Validation R2 mean: {}\nCross Validation R2 standard deviation: {}'.format(cv_scores.mean(),cv_scores.std()))


print('\nRidge Evaluation:')
#using r2_score to evaluate performance
r2_train_ridge = r2_score(y_train, y_predict_train_ridge)
r2_test_ridge = r2_score(y_test, y_predict_ridge)
print('R2 train score: {}\nR2 test score: {}'.format(r2_train_ridge, r2_test_ridge))

#using cross validation to estimate scores
cv_scores_ridge = cross_val_score(ridge, X, y, cv=5, scoring='r2')
print('Cross Validation R2 mean: {}\nCross Validation R2 standard deviation: {}'.format(cv_scores_ridge.mean(),cv_scores_ridge.std()))<jupyter_output>Lasso Evaluation:
R2 train score: 0.85156950972085
R2 test score: 0.8470433348956904
Cross Validation R2 mean: 0.8350424187389971
Cross Validation R2 standard deviation: 0.020411665666851028

Ridge Evaluation:
R2 train score: 0.851689295982942
R2 test score: 0.8471187077080635
Cross Validation R2 mean: 0.834778600016311
Cross Validation R2 standard deviation: 0.020414035609031635
{'fraction_non_alphanumeric': 0.1156271629623933, 'fraction_numerical': 0.019533953702991616, 'mean_word_length': 4.446492146596858, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 38, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '507863', 'license': 'MIT', 'n_tokens_mistral': 8829, 'n_tokens_neox': 8302, 'n_words': 1954, 'path': 'Udacity_Airbnb.ipynb', 'repo_name': 'gmarafon/Udacity_Airbnb', 'size': '26006'}