<jupyter_start><jupyter_text>Section 1: Business Understanding AirBnB is a company, which provides internet services for customer, who temporary wants to share their home with others.  This Notebook adresses both people, who are interested in unterstanding data provided by AirBnB and people, who are interested to use AirBnb in Seattle and to get some information about the situation in the city.    With the given data, following question will be adressed:1. how ist the AirBnB market in Seattle? 1. how many listings do exist? 2. what ist the average price, how is its distribution over the year? 3. what is the average availability, how is its distribution over the year? 4. is there a correlation between price, availabilty and the climate in Seattle (which would give a view on the	tourist potential of the city) 5. how ist the distribution of the listings over the neighborhoods and in general over the city in terms of numbers, prices, ratings and property type?2. how can written reviews used for data analysis?  1. is it possible to quantify them using a "SentimentIntensityAnalyzer"?  2. what are the pitfalls if doing so?  3. is there a correlation between the polarity_scores from a SentimentIntensityAnalyzer and the direct rating scores from the customer?3. how can the price be predicted using a data-based model?  1. which data best fit to the model?  The answers can be found in section 5: Evaluation      Section 2: Data Understanding   To start the data analysis, necessary packages will be loaded and setting are done:<jupyter_code># Import statements
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython import display
import seaborn as sns
import nltk
from nltk.sentiment.vader import SentimentIntensityAnalyzer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

%matplotlib inline
sns.set_theme(style="darkgrid")<jupyter_output><empty_output><jupyter_text>definition of some helper-functions to process string-based operations:<jupyter_code># definitions of functions for data-processing

def char2Bool(input_col):
    '''
    INPUT
    input_col - pandas series 
    
    OUTPUT
    output_col - pandas series 
    
    This function changes input_col using the following steps:
    1. replace all character 't' by the bool True
    2. replace all character 'f' by the bool False
    '''
    
    input_col = input_col.replace('t', True)
    output_col = input_col.replace('f', False)
    return (output_col)
    
def price2numeric(input_col):
    '''
    INPUT
    input_col - pandas series 
    
    OUTPUT
    output_col - pandas series 
    
    This function changes input_col using the following steps:
    1. delete all character ','
    2. delete all character '$'
    3. delete all character '.00'
    4. change dtype of values from string to numeric
    '''
    input_col = input_col.str.replace(',', '')
    input_col = input_col.str.replace('$', '')
    output_col = input_col.str.replace('.00', '')
    return (pd.to_numeric(output_col))

def clean_str(input_col):
    '''
    INPUT
    input_col - pandas series 
    
    OUTPUT
    output_col - pandas series 
    
    This function changes input_col using the following steps:
    1. delete all character '{'
    2. delete all character '}'
    3. delete all character '"'
    '''
    input_col = input_col.str.replace('{', '')
    input_col = input_col.str.replace('}', '')
    output_col = input_col.str.replace('"', '')
    return (output_col)<jupyter_output><empty_output><jupyter_text>Now the data can be loaded and a very first look on the data can be made.  First the **calender** data set: gather data<jupyter_code># load calendar data set
df_calendar = pd.read_csv('./seattle/calendar.csv')

# show first 5 rows
df_calendar.head()
# number of rows and columns
print("Number of rows and columns: ", df_calendar.shape)
# how many values are missing?
print(df_calendar.isnull().sum())
# Describe
print(df_calendar.describe(include='all'))<jupyter_output>listing_id        date available    price
count   1.393570e+06     1393570   1393570   934542
unique           NaN         365         2      669
top              NaN  2016-11-24         t  $150.00
freq             NaN        3818    934542    36646
mean    5.550111e+06         NaN       NaN      NaN
std     2.962274e+06         NaN       NaN      NaN
min     3.335000e+03         NaN       NaN      NaN
25%     3.258213e+06         NaN       NaN      NaN
50%     6.118244e+06         NaN       NaN      NaN
75%     8.035212e+06         NaN       NaN      NaN
max     1.034016e+07         NaN       NaN      NaN<jupyter_text>**listings** data set:<jupyter_code># load listings data set
df_listings = pd.read_csv('./seattle/listings.csv')

# set id as index (is needed later when outer data is joined...)
df_listings = df_listings.set_index('id')

# show first 5 rows
print (df_listings.head())   
# how many values are missing?
print(df_listings.isnull().sum())
# Describe
print(df_listings.describe(include='all'))<jupyter_output>listing_url     scrape_id last_scraped  \
count                                   3818  3.818000e+03         3818   
unique                                  3818           NaN            1   
top     https://www.airbnb.com/rooms/3147706           NaN   2016-01-04   
freq                                       1           NaN         3818   
mean                                     NaN  2.016010e+13          NaN   
std                                      NaN  0.000000e+00          NaN   
min                                      NaN  2.016010e+13          NaN   
25%                                      NaN  2.016010e+13          NaN   
50%                                      NaN  2.016010e+13          NaN   
75%                                      NaN  2.016010e+13          NaN   
max                                      NaN  2.016010e+13          NaN   

                          name  \
count                     3818   
unique                    3792 [...]<jupyter_text>**Review** data set:<jupyter_code># load reviews data set
df_reviews = pd.read_csv('./seattle/reviews.csv')

# show first 5 rows
print(df_reviews.head())
# number of rows and columns
print("Number of rows and columns: ", df_reviews.shape)
# how many values are missing?
print(df_reviews.isnull().sum())
# Describe
print(df_reviews.describe(include='all'))<jupyter_output>listing_id            id        date   reviewer_id reviewer_name  \
count   8.484900e+04  8.484900e+04       84849  8.484900e+04         84849   
unique           NaN           NaN        1930           NaN         14380   
top              NaN           NaN  2015-09-21           NaN         David   
freq             NaN           NaN         580           NaN           870   
mean    3.005067e+06  3.058765e+07         NaN  1.701301e+07           NaN   
std     2.472877e+06  1.636613e+07         NaN  1.353704e+07           NaN   
min     4.291000e+03  3.721000e+03         NaN  1.500000e+01           NaN   
25%     7.946330e+05  1.725127e+07         NaN  5.053141e+06           NaN   
50%     2.488228e+06  3.228809e+07         NaN  1.413476e+07           NaN   
75%     4.694479e+06  4.457648e+07         NaN  2.762402e+07           NaN   
max     1.024814e+07  5.873651e+07         NaN  5.281274e+07           NaN   

                                                 comments  
cou[...]<jupyter_text>As we have now a better overview about the data, we take a closer look.We first focus on the data set "calendar".To prepare the data for the next questions, new colums for month and year are extracted from column "date".We also need the data for the price in numeric format (also in data-set "listings", as we need them later).Also, we convert the data for "availability" to real boolean values.Finally, as the timerange of the data is 01/2016 - 01/2017 and we want to have the overview over 1 year, we drop the data for 2017:<jupyter_code># extract year and month from date
df_calendar['year'] = pd.DatetimeIndex(df_calendar['date']).year
df_calendar['month'] = pd.DatetimeIndex(df_calendar['date']).month

# price to numeric
df_calendar['price'] = price2numeric(df_calendar['price'])
df_listings['price'] = price2numeric(df_listings['price'])

# availability to bool
df_calendar['available'] = char2Bool(df_calendar['available'])<jupyter_output><empty_output><jupyter_text>transform human-written reviews into polarity_scores:<jupyter_code># create analyzer
analyzer = SentimentIntensityAnalyzer()

# function to calculate polarity_scores
def get_polarity_scores(review_text):
    '''
    INPUT
    review_text - text to be assessed (string )
    
    OUTPUT
    output_col - polarity_scores (float 0 - 1)
    
    This function calculates the polarity_scores based on the input-text
    using the SentimentIntensityAnalyzer from nltk.sentiment.vader
    '''
    return analyzer.polarity_scores(review_text)['compound']

# remove NANs from reviews
df_reviews = df_reviews.dropna(subset = ['comments'])<jupyter_output><empty_output><jupyter_text>- create new column with polarity_scores<jupyter_code># add polarity_scores of reviews
df_reviews['polarity_scores'] = df_reviews['comments'].apply(get_polarity_scores)<jupyter_output><empty_output><jupyter_text>Now let's compare polarity_scores to the ratings of the listings:  For that, we have to:  - calculate the mean polarity_score for each listing and- join these polarity_scores to the linting data set<jupyter_code># mean polarity_scores per listing
df_reviews_mean_per_listing = df_reviews.groupby(['listing_id']).mean()

# add polarity_scores to listings
df_listings = df_listings.join(df_reviews_mean_per_listing['polarity_scores'])<jupyter_output><empty_output><jupyter_text>Section 3: Data PreparationAfter we got a good overview about the data we are dealing with, data has to be prepared, so that it can be fed to a linear model, which will predict prices from given data.The following question will be adressed:1. which data can be used?    a. how to prepare the target data?    b. which numerical data can be used as features?    c. which categorical data can be used as features?    c. how do we deal with missing data?    d...    2. which data really make sense to use?    a. how is the correllation to the cvalue to be predicted?    b. how many differend numerical and categoical  data shall be used, to get the best results    c. how many different categorys within a categorical data should be used?    3. what can we do to avoid overfitting?As stated above, the goal is to predict the prices using relevant data as features.To use the prices as targets, they need to be in a numerical format. We did that already when analysing the data.We only will use listing with valid prices:<jupyter_code># drop all rows with price as NAN
df_listings = df_listings.dropna(subset=['price'])<jupyter_output><empty_output><jupyter_text>to get a good set of features, we can drop all non-categorical and non-relevant numerical data:<jupyter_code># drop all non-numerical- or categorical columns:'neighbourhood_group_cleansed',
non_num_cat = ['name', 'summary', 'space', 'description', 'experiences_offered',
       'neighborhood_overview', 'notes', 'transit', 'picture_url', 'host_name',
       'host_since', 'host_location', 'host_about','listing_url','last_scraped',
       'thumbnail_url','medium_url','xl_picture_url','host_url','host_thumbnail_url',
       'street','zipcode','smart_location','country','calendar_updated',
       'calendar_last_scraped','first_review','last_review','host_picture_url', 'host_neighbourhood', 'neighbourhood',  'city', 'weekly_price',
       'monthly_price', 'security_deposit', 'cleaning_fee', 'extra_people', 'jurisdiction_names',
       'state', 'market', 'country_code', 'is_location_exact']
df_listings = df_listings.drop(non_num_cat, axis=1)

# non-relevant numerical values
non_rel_num = ['scrape_id','host_id','host_verifications','square_feet','license']
df_listings = df_listings.drop(non_rel_num, axis=1)<jupyter_output><empty_output><jupyter_text>Now we want to find out, which numerical data we can use as features to predict the price as target.For that, we visualize the correllation between features and target:<jupyter_code># filter for numerical features
df_listings_num = df_listings.select_dtypes(exclude=['object'])

# normalize numerical features
df_listings_num = (df_listings_num - df_listings_num.mean())/df_listings_num.std()

# calculate correlation matrix to see, which numeric values are correllated to the price
df_listings_num_cov = df_listings_num.cov()

# display correllation as heat-map
sns.heatmap(df_listings_num_cov)<jupyter_output><empty_output><jupyter_text>we can see now the TOP10 of the features, which are correlated to the price. We will use the best correlated feattures to train our model.<jupyter_code>df_listings_num_cov.sort_values(['price'],ascending = False).index[1:10]
key_mean = df_listings_num.columns<jupyter_output><empty_output><jupyter_text>Are there any missing numerical data?<jupyter_code>df_listings[key_mean].isnull().sum()<jupyter_output><empty_output><jupyter_text>As there are no several features with missing data, we replace them by the mean of all the data. Otherwise we would have to drop the entire data row, and we would loos other information, we want to use.<jupyter_code>df_listings[key_mean] = df_listings[key_mean].fillna(df_listings[key_mean].mean())<jupyter_output><empty_output><jupyter_text>Now we can deal with the missing data within the categorical data. First lets have an overview about categorical data:<jupyter_code># overview about categories:
df_listings.select_dtypes(include=['object']).columns<jupyter_output><empty_output><jupyter_text>In order not to get to much features, only the most relevant features will be used. Here the missing data can be replaced by their mean:<jupyter_code>key_mode = ['host_response_time','host_response_rate', 'host_is_superhost','host_has_profile_pic','host_identity_verified',
            'property_type']

# remove NAN in categorical data by mode()
df_listings[key_mode] = df_listings[key_mode].fillna(df_listings[key_mode].mode().squeeze())<jupyter_output><empty_output><jupyter_text>to be able to use the categorical data in our model, they need to be converted into dummy-variables. To make sure that we only use relevant dummy-variables, we have alook at them:<jupyter_code># plot pie
plt.figure(figsize=(5*len(key_mode), 5))
for i,key in enumerate(key_mode):
    plt.subplot(100+10*len(key_mode)+i+1)
    df_listings[key].value_counts().plot.pie(y=df_listings.columns[i+1])<jupyter_output><empty_output><jupyter_text>here we see:- several dummy-variables have very few data, they will be removed, if they have less than 100 occurences.- 'host_has_profile_pic' will be removed, because there is not enough variation in the feature.- these measures will help to prevent over-fitting<jupyter_code>df_listings.head()
# choose categorical values
df_listings = pd.get_dummies(df_listings, columns = key_mode, prefix_sep = '_d_')<jupyter_output><empty_output><jupyter_text>we also create dummy-variables from 'anemety'<jupyter_code># create dummies from 'amenities'
# clean data
df_listings_a = clean_str(df_listings['amenities'])

# create anemety
df_listings_a_d = df_listings_a.str.get_dummies(sep=',')

# adapt columns, so that they can be handeld together with the other dummies
df_listings_a_d.columns = df_listings_a_d.columns.astype(str) + '_d_'

# join df_listings and df_listings_a_d
df_listings = df_listings.join(df_listings_a_d)<jupyter_output><empty_output><jupyter_text>the available anemety are:<jupyter_code># all dummy-headers from 'anemety'
df_listings_a_d.columns
# all dummy-headers
dh = df_listings.columns[df_listings.columns.str.contains('_d_')]
dh<jupyter_output><empty_output><jupyter_text>these are to many dummy-variables. Remove the ones with low occurence-rates (<100)<jupyter_code># dummy-header with less that 100 ocurences
most_rel_cat = df_listings[dh].columns[df_listings[dh].sum() > 100]<jupyter_output><empty_output><jupyter_text>all relevant dummy-variables are:<jupyter_code>most_rel_cat<jupyter_output><empty_output><jupyter_text>we now have a set of relevant dummy-variables, that we want to use to train our model.Now, as we have the most relevant numerical and categorical features, we can set up and train our model.  Section 4: ModelingA standard, linear model will be used.Different approchas with differne input data will be used:1. different numbers of only numerical features:  - we will iterate throu 2 to 15 different features, to see which set will have the best train- and test-scores2. additional categorical data  - we will then add the categorical features, to see, if it improves.<jupyter_code># set up DataFrame for the results
result = pd.DataFrame(index = ['number of vars', 'train_score', 'test_score'])

# Instantiate linear regression model
lm_model = LinearRegression(normalize=True) 

for i in range(3,15):
    
    # take the i most relevant numerical features
    most_rel_num = df_listings_num_cov.sort_values(['price'],ascending = False).index[1:i].values
    
    #most_rel = np.append(most_rel_num,most_rel_cat)
    
    # define data sets
    X = df_listings[most_rel_num]
    y = df_listings['price']
    
    # Create train and test data sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .10, random_state=42) 
        
    # fit model to training data set
    lm_model.fit(X_train, y_train) 
            
    # Predict and score the model
    y_train_preds = lm_model.predict(X_train)
    y_test_preds = lm_model.predict(X_test) 
    train_score = r2_score(y_train, y_train_preds)
    test_score = r2_score(y_test,y_test_preds)
    
    result[str(i)] = [i,train_score,test_score]
most_rel_num<jupyter_output><empty_output><jupyter_text>we can see that even with 3 features the results are not to bad, but they get better if we use more features.  Now we will add the categorical features to see, if the score will be better:<jupyter_code># take the 14 most relevant numerical features
most_rel_num = df_listings_num_cov.sort_values(['price'],ascending = False).index[1:14].values
most_rel = np.append(most_rel_num,most_rel_cat)
    
# define data sets
X = df_listings[most_rel]
y = df_listings['price']
    
# Create train and test data sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .10, random_state=42) 
        
# fit model to training data set
lm_model.fit(X_train, y_train) 
            
# Predict and score the model
y_train_preds = lm_model.predict(X_train)
y_test_preds = lm_model.predict(X_test) 
train_score = r2_score(y_train, y_train_preds)
test_score = r2_score(y_test,y_test_preds)
    
result[str(15)] = [i,train_score,test_score]<jupyter_output><empty_output><jupyter_text>Section 5: Evaluation  Question block 1: how ist the AirBnB market in Seattle?! Question A: how many listings do exist?can already be answered:<jupyter_code>print("Number of listings: ", df_calendar.shape[0])<jupyter_output>Number of listings:  1393570<jupyter_text>Question B: what ist the average price, how is its distribution over the year? Question C:what is the average availability, how is its distribution over the year?To be able to answer the next questions, we need to sort the data by the month:<jupyter_code># only take 2016
df_calendar_16 = df_calendar[df_calendar['year'] == 2016]# groupby month

df_calendar_16_gb_p = df_calendar_16.groupby('month').mean()<jupyter_output><empty_output><jupyter_text>So we get an overview about the price and availability per month:<jupyter_code>print ("the average price in Seattle is:",df_calendar_16_gb_p['price'].mean(),"US$")
# barplots:
# price per month:
df_calendar_16_gb_p['price'].plot.bar(y = 'price in $', ylim = [100,150])
print ("the average availability in Seattle is:",df_calendar_16_gb_p['available'].mean()*100,"%")
# availability per month:
df_calendar_16_gb_p['available'].plot.bar(y = 'relativ availability', ylim = [0.4,0.8])<jupyter_output><empty_output><jupyter_text>Question D: is there a correlation between price, availabilty and the climate in SeattleTo get an idea, when it is a good idea to travel to Seattle, we need to see these 2 plot together with weather data:<jupyter_code># get climate data from https://www.bestplaces.net/climate/city/washington/seattle
comfort_index = np.array([5.6,6.1,6.5,7.2,8.2,9,9.6,9.7,9.1,7.5,6,5.3])

# normalize price, availibility and comfort data for comparison
df_calendar_16_gb_p['comfort_index_n'] = comfort_index/max(comfort_index)
df_calendar_16_gb_p['price_n'] = df_calendar_16_gb_p['price']/df_calendar_16_gb_p['price'].max()
df_calendar_16_gb_p['available_n'] = df_calendar_16_gb_p['available']/df_calendar_16_gb_p['available'].max()

# plot comparison
df_calendar_16_gb_p[['price_n','available_n','comfort_index_n']].plot.bar(ylim = [0.4,1.1])<jupyter_output><empty_output><jupyter_text>In this diagramm we can see a certain correlation between the price with the comfort-index. The best month to travel would be August, because only there the comfort-index is higher than the price ;-).It also turns out that the availability doesn't seem to correlate to strong with the price or comfort-index.  Question E: how ist the distribution of the listings over the neighborhoods and in general over the city in terms of numbers, prices, ratings and property type?Besides the time-related behavior of the data it could be interesting to get more information about local differences.For that we now turn to the listings-data-set.First, let's get an overview about the over-all price-structure:<jupyter_code># histogram of price
df_listings['price'].plot.hist(bins=20)<jupyter_output><empty_output><jupyter_text>we can see here, that the price of most of the listings is less than 400US$, so these listings will be shown in a geographical context:<jupyter_code># price-distribution geographicly
# clip price, otherwise the color distribution is bad.
pr400 = df_listings['price'].clip(0,400)

# plot listings at their position
plt.scatter(df_listings.latitude, df_listings.longitude, c= pr400/pr400.max(), alpha=0.3)<jupyter_output><empty_output><jupyter_text>the brighter a listing is, the more expensive it is. Apparently there are no "hot spots" with expensive listings.  We also can look at neigborhoods.How are the differences in term of price between the different neigborhoods?<jupyter_code>df_ng = df_listings.groupby(['neighbourhood_group_cleansed']).mean()['price'].sort_values()
df_ng.plot.bar(figsize=[10,5])<jupyter_output><empty_output><jupyter_text>so we see that there are differences in the mean of the prices of the differend neigborhoods.  But are they relevant?  To find out, we show the distribution of the prices within the neigborhoods:<jupyter_code># visualize the variance of the prices in the neig.
sns.set(rc={'figure.figsize':(16,8)})
sns.violinplot(x="neighbourhood_group_cleansed", y="price", data=df_listings, order = df_ng.index)<jupyter_output><empty_output><jupyter_text>so we see, that the variance of the prices is relativly high.  This shows, that the prices of the listing is well distributed over the neigborhoods. We have already seen that in our geographical overview.  now let's see how many listings are in which neigborhoods:<jupyter_code>df_ng = df_listings.groupby(['neighbourhood_group_cleansed']).count()['review_scores_rating'].sort_values()
df_ng.plot.bar(figsize=[10,5])<jupyter_output><empty_output><jupyter_text>And finaly, which neigborhood has the best ratings?<jupyter_code>df_ng = df_listings.groupby(['neighbourhood_group_cleansed']).mean()['review_scores_rating'].sort_values()
df_ng.plot.bar(figsize=[10,5],ylim = [80,100])<jupyter_output><empty_output><jupyter_text>Question block 2: how can written reviews used for data analysis? Question is it possible to quantify them using a "SentimentIntensityAnalyzer"? Question what are the pitfalls if doing so?Now we will deal also with the review data.The general questions are:- is it possible to process the human-written review data?- will it help us in understanding the data?We use the SentimentIntensityAnalyzer to translate the human-written reviews into polarity scores.These scores can be between -1 and 1 and should represent the sentiment intensity of the reviewer. High rates shows a good feeling, low rates show bad feelings. To get an idea, if the calculated polarity_scores make sense, we have a look at the TOP-10:<jupyter_code>df_reviews[['polarity_scores', 'comments']].sort_values('polarity_scores', ascending = False)[:10]<jupyter_output><empty_output><jupyter_text>we can see, that the reviews with a high polarity_score indeed are positiv reviews.  So we have a look at the Bottom-10:<jupyter_code>df_reviews[['polarity_scores', 'comments']].sort_values('polarity_scores', ascending = True)[:10]<jupyter_output><empty_output><jupyter_text>here we see some very bad review (e.g. "Staying at Robert’s place was a nightmare"), so that is fine.  On the other hand, we see mostly German reviews. These are not specific bad, one is very good: "Die Unterkunft war sehr schön und genau so, wi..." (in english: "the appartment was very nice").  So in this case, the SentimentIntensityAnalyzer didn't worked good, very likly because of the foreign language. Question C:is there a correlation between the polarity_scores from a SentimentIntensityAnalyzer and the direct rating scores from the customer?<jupyter_code># is there a correlation between polarity_scores of the reviews and the ratings of the listings?
rev_corr = df_listings['review_scores_rating'].corr(df_listings['polarity_scores'])
print ("the correlation between polarity_scores of the reviews and the ratings of the listings is:",rev_corr)<jupyter_output>the correlation between polarity_scores of the reviews and the ratings of the listings is: 0.3844318918809194<jupyter_text>We can have a look on the scatter-plot:<jupyter_code># scatterplot between polarity_scores of the reviews and the ratings of the listings
df_listings.plot.scatter(x='review_scores_rating', y='polarity_scores', c='Blue')<jupyter_output><empty_output><jupyter_text>We see a correlation between the two scores, which shows, that:- the SentimentIntensityAnalyzer works, and- the ratings and the review of the customer are implied.We can also see the correlation to the other ratings in the heatmap below.  Question 3: how can the price be predicted using a data-based model?   - which data best fit to the model?Finally we have a look at the coefficients of the model we created, to see the influence of these<jupyter_code>def coef_weights(coefficients, X_train):
    '''
    Function to get coefficients back for each of the features
    INPUT:
    coefficients - the coefficients of the linear model 
    X_train - the training data, so the column names can be used
    OUTPUT:
    coefs_df - a dataframe holding the coefficient, estimate, and abs(estimate)
    
    Provides a dataframe that can be used to understand the most influential coefficients
    in a linear model by providing the coefficient estimates along with the name of the 
    variable attached to the coefficient.
    '''
    coefs_df = pd.DataFrame()
    coefs_df['est_int'] = X_train.columns
    coefs_df['coefs'] = lm_model.coef_
    coefs_df['abs_coefs'] = np.abs(lm_model.coef_)
    coefs_df = coefs_df.sort_values('abs_coefs', ascending=False)
    return coefs_df

#Use the function
coef_df = coef_weights(lm_model.coef_, X_train)

#A quick look at the top results
coef_df.head(10)<jupyter_output><empty_output><jupyter_text>Now let's have a look at the results of linear model:  - results number 3-14 are the scores of the linear model only using 3-14 numerical features. the more features we use, the better is the model. But too many feature would increase the danger of overfitting.  - result number 15 uses 14 numerical features and also 6 categorical features. This model is better due to the information, it gets from the additional categorical features.<jupyter_code>result<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08315292614543777, 'fraction_numerical': 0.028108031373105725, 'mean_word_length': 4.176924490170862, 'pattern_counts': {'":': 0, '<': 161, '<?xml version=': 0, '>': 161, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '41', 'dir': 'jupyter-structured-clean-dedup', 'id': '489353', 'license': 'CNRI-Python', 'n_tokens_mistral': 9219, 'n_tokens_neox': 8486, 'n_words': 2851, 'path': 'project_1.ipynb', 'repo_name': 'stone12de/airbnb-seattle', 'size': '28177'}