<jupyter_start><jupyter_text>Preamble This project was inspired by the work of : [scrapfishies](https://github.com/scrapfishies) Prediction of rental prices  For this project we're gonna use an API in order to extract our data and frame into a dataset to exploitThis study will be based with the help of the various tools presented below : **Web scraping*** Request * BeautifulSoup  **Tools** * Pandas* Numpy **Modeling** * sklearn* statsmodels **Visualizations** * Seaborn*Matplotlib Step 1: Web scraping from craiglist*  Scrape the page will be focused on listing the date when the anounce was posted, title, url, rent amount, square footage, neighborhood and number of bedrooms.* Append all those url together into a dataframe* Repeat the process until the number of instance is obtained First let's begin by importing the librairies that we'll use to initialize this study. In this case we're going to scrape 25 pages of craiglist containing apartment and housing listing.**Date and time of scrape :** >22 January 2022**Setting for scraping from craiglist :**> SF bay area **>** san francisco **>** housing **>** apartments / housing for rent<jupyter_code>from bs4 import BeautifulSoup
import requests

import pandas as pd
import numpy as np

from random import randint
from time import sleep
%load scrape_cl.py
from scrape_cl import *
start_url = 'https://sfbay.craigslist.org/search/sfc/apa?sort=date&bundleDuplicates=1'
import six
import sys
sys.modules['sklearn.externals.six'] = six
from mlxtend.classifier import StackingCVClassifier
sf=full_listings_scrape(start_url)
#Drop extra index
sf = sf.drop(['index'], axis=1)
sf.tail()
#Encapsulate our Data into a csv file
sf.to_csv('raw_sf_scrape.csv', index=False)<jupyter_output><empty_output><jupyter_text>Step 2 : Cleaning our Data With the help of the scraple process we have now a decent dataset to exploit. A first move to make will be to clean it the best we can to further progress to the predictive part.  Let's begin first by : * Importing the librairies that we will use on our dataset<jupyter_code>import pandas as pd
import numpy as np

import matplotlib as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>* Importing the Data<jupyter_code>sf_r=pd.read_csv('raw_sf_scrape.csv')<jupyter_output><empty_output><jupyter_text>* Type of data to be handled<jupyter_code>sf_r.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3101 entries, 0 to 3100
Data columns (total 9 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   date       3101 non-null   object 
 1   title      3101 non-null   object 
 2   link       3101 non-null   object 
 3   price      3101 non-null   int64  
 4   brs        2632 non-null   float64
 5   sqft       1437 non-null   float64
 6   hood       3101 non-null   object 
 7   bath       3099 non-null   object 
 8   amenities  3099 non-null   object 
dtypes: float64(2), int64(1), object(6)
memory usage: 218.2+ KB<jupyter_text>* First and last component of our dataset<jupyter_code>sf_r.head()
sf_r.tail()<jupyter_output><empty_output><jupyter_text>* There's chances to fall into some duplicates coming from the fact that a user may list multiple times the same announce to give it great chances to be seen by apartement hunters.> Knowing that the user keeps the same title, we're gonna give a new dataframe where we're ordering the titles a way we can catch repetition easily.<jupyter_code>sf=sf_r.drop(['date','link'],axis=1)
sf.head()
sf.sort_values("title",inplace=True)
sf.head()
sf.drop_duplicates(keep=False,inplace=True)<jupyter_output><empty_output><jupyter_text>* Showing the size of our dataset after mofication<jupyter_code>sf.shape
sf.head()<jupyter_output><empty_output><jupyter_text>Cleaning variable bathrooms* In this phase we're going to better resume the number of bathroom in the house by erasing any extra word from the numerical value<jupyter_code>sf.bath.unique()<jupyter_output><empty_output><jupyter_text>* After establishing the value that we may encounter in this variable we make some changes to better adapt it<jupyter_code>shared_baths=sf_r[(sf_r.bath=="sharedBa")]
shared_baths_links=list(shared_baths.link)
#Number of user's sharing the same value 'sharedBa'
len(shared_baths_links)<jupyter_output><empty_output><jupyter_text>* In order to better understand the *sharedBa* value let's investigate the links<jupyter_code>for link in shared_baths_links[:5]:
  print(link)
  print("")<jupyter_output>https://sfbay.craigslist.org/sfc/apa/d/1200-private-rm-not-share-for-rent/7431680580.html

https://sfbay.craigslist.org/sfc/apa/d/san-francisco-cozy-studio-in-prime/7437214610.html

https://sfbay.craigslist.org/sfc/apa/d/san-francisco-1050-special-private/7437205341.html

https://sfbay.craigslist.org/sfc/apa/d/san-francisco-rent-this-single-room-in/7435784205.html

https://sfbay.craigslist.org/sfc/apa/d/san-francisco-amazing-value-for-sro/7437116433.html<jupyter_text>* After quick verification the **sharedBa** stands for not full apartment, we're going then to skip them in our study<jupyter_code>sf=sf[sf.bath!='sharedBa']<jupyter_output><empty_output><jupyter_text>* Let's see now the changes that we've commited<jupyter_code>sf.bath.unique()<jupyter_output><empty_output><jupyter_text>* Another modification will be for **splitBa** where this time we're going to switch to the value 1<jupyter_code>sf["bath"]=sf["bath"].replace("splitBa",'1Ba')
sf.bath.unique()<jupyter_output><empty_output><jupyter_text>* Let's see now for the the missing value's cases where here we have **nan** and **'0Ba'** ?<jupyter_code>sf.to_csv('sf_scrape.csv', index=False)
miss_bath_info=sf_r[(sf_r.bath == np.nan)|(sf_r.bath=="0Ba")]
len(miss_bath_info)<jupyter_output><empty_output><jupyter_text>* Access as previously seen the links to better understand the interpretation of **nan** and **'0Ba'**<jupyter_code>for link in miss_bath_info.link:
  print(link)
  print("")<jupyter_output><empty_output><jupyter_text>* The post specify that the place contains one bathroom so we replace those values by 1<jupyter_code>sf["bath"]=sf["bath"].replace(np.nan,'1Ba')
sf["bath"]=sf["bath"].replace('0Ba','1Ba')
sf.bath.unique()<jupyter_output><empty_output><jupyter_text>* Now let's erase the suffix **'Ba'** to better computate our data in what will come after<jupyter_code>sf["bath"]=sf["bath"].str.replace("Ba",'').astype(float)
sf.head()<jupyter_output><empty_output><jupyter_text>Dealing with missing values* In order to enhance the prediction that will come further after we deal with the missing values<jupyter_code>sf.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2436 entries, 1381 to 579
Data columns (total 7 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   title      2436 non-null   object 
 1   price      2436 non-null   int64  
 2   brs        2045 non-null   float64
 3   sqft       1198 non-null   float64
 4   hood       2436 non-null   object 
 5   bath       2436 non-null   float64
 6   amenities  2434 non-null   object 
dtypes: float64(3), int64(1), object(3)
memory usage: 152.2+ KB<jupyter_text>* Let's first see the variables that present missing values<jupyter_code>sf.isnull().sum()<jupyter_output><empty_output><jupyter_text>* We select to begin with the lines that does not suffer of missing values<jupyter_code>sf=sf[sf["sqft"].notna()]
sf.shape
sf.head()<jupyter_output><empty_output><jupyter_text>* Let's focus now on the **bedrooms**<jupyter_code>sf.brs.unique()
len(sf.brs.unique())<jupyter_output><empty_output><jupyter_text>* Let's see how many **NaN** values do we have<jupyter_code>miss_brs=sf[sf.brs.isnull()]
miss_brs.head(3)
print('Number of missing bedroom rows :',len(miss_brs))<jupyter_output>Number of missing bedroom rows : 167<jupyter_text>* As done before let's see the meaning of this missing value by checking the link<jupyter_code>br_nan=sf_r[sf_r['brs'].isnull()]
br_nan_links=list(br_nan.link)
for link in br_nan_links[:4]:
  print(link)
  print("")<jupyter_output>https://sfbay.craigslist.org/sfc/apa/d/san-francisco-nice-studio-available-now/7436185394.html

https://sfbay.craigslist.org/sfc/apa/d/san-francisco-large-remodeled-art-deco/7433050028.html

https://sfbay.craigslist.org/sfc/apa/d/san-francisco-bright-top-floor-studio/7432088404.html

https://sfbay.craigslist.org/sfc/apa/d/san-francisco-top-floor-renovated-and/7434393942.html<jupyter_text>* We see that the information was not filled but it's still contained in the title, let's deal with that<jupyter_code>def replace_missing_brs(post_title):
    if 'studio' in post_title.lower():
        return 0
    elif '1br' in post_title.lower().replace(' ', ''):
        return 1
    elif '1bed' in post_title.lower().replace(' ', ''):
        return 1
    elif 'onebed' in post_title.lower().replace(' ', ''):
        return 1
    elif '2br' in post_title.lower().replace(' ', ''):
        return 2
    elif '3br' in post_title.lower().replace(' ', ''):
        return 3
    elif '4br' in post_title.lower().replace(' ', ''):
        return 4
    elif '4bd' in post_title.lower().replace(' ', ''):
        return 4
    elif '4bed' in post_title.lower().replace(' ', ''):
        return 4
    else:
        pass<jupyter_output><empty_output><jupyter_text>* Before applying the function let's fill the NaN values with the character **'missing'**<jupyter_code>sf["brs"]=sf['brs'].fillna('missing')
sf["beds"]=sf.apply(lambda row : replace_missing_brs(row['title']) 
                                                    if row['brs'] == 'missing' 
                                                    else row['brs'], axis=1)
sf.head()
miss_brs_mod=sf[sf['beds'].isnull()]
print("Number of missing bedrooms at start: ", len(miss_brs))
print("Number of missing bedrooms after title parse: ",len(miss_brs_mod))
print("Number of recoverd bedrooms: ", len(miss_brs) - len(miss_brs_mod))
print("Percent recovered: ", ((len(miss_brs) - len(miss_brs_mod)) / len(miss_brs)) * 100, '%')<jupyter_output>Number of missing bedrooms at start:  167
Number of missing bedrooms after title parse:  49
Number of recoverd bedrooms:  118
Percent recovered:  70.65868263473054 %<jupyter_text>* We saved 118 bedrooms, now let's drop the missing values<jupyter_code>sf = sf[['title', 'price', 'sqft', 'beds', 'bath', 'hood', 'amenities']]
sf=sf[sf['beds'].notna()]<jupyter_output><empty_output><jupyter_text>* Now let's check our data cleaned<jupyter_code>sf.head()<jupyter_output><empty_output><jupyter_text>Amenities re-adaptation * As it can be seen the Amenities columns suffer from a high cluster of data all packed together. In this part we will enhance our dataset by separting them in new columns. In order to proceed we're gonna call the library **ast**<jupyter_code>from ast import literal_eval
def f(x):
    try:
        return literal_eval(str(x))   
    except Exception as e:
        print(e)
        return []

sf['amens_list'] = sf.amenities.apply(lambda x: f(x))
sf.head()<jupyter_output><empty_output><jupyter_text>* Let's test our results before the big changes<jupyter_code>amen_eg=sf.loc[1381,'amens_list']
type(amen_eg)
'condo' in amen_eg<jupyter_output><empty_output><jupyter_text>* We can now create columns in order to better separate the values in **amenities**<jupyter_code>sf.drop(['amenities'],axis=1)
sf.head()<jupyter_output><empty_output><jupyter_text>Laundry* We will group this variable into three different classes<jupyter_code>def laundry_parse(amen_list):
    if 'w/d in unit' in amen_list:
        return '(a) in-unit'
    elif 'laundry in bldg' in amen_list:
        return '(b) on-site'
    elif 'laundry on site' in amen_list:
        return '(b) on-site'
    else:
        return '(c) no laundry'
sf['laundry'] = sf['amens_list'].apply(lambda amen_list: laundry_parse(amen_list))<jupyter_output><empty_output><jupyter_text>Pets * In this case we're going to segment our variables into four categories<jupyter_code>def pets_allowed(amen_list):
    if 'dogs are OK - wooof' in amen_list and 'cats are OK - purrr' in amen_list:
        return '(a) both'
    elif 'dogs are OK - wooof' in amen_list:
        return '(b) dogs'
    elif 'cats are OK - purrr' in amen_list:
        return '(c) cats'
    else:
        return '(d) no pets'
sf['pets']=sf['amens_list'].apply(lambda amen_list: pets_allowed(amen_list))<jupyter_output><empty_output><jupyter_text>Housing TypeWe're now going to groupe following three categories<jupyter_code>def housing_type(amen_list):
    if 'cottage/cabin' in amen_list:
        return '(a) single'
    elif ' duplex' in amen_list:
        return '(b) double'
    elif 'house' in amen_list:
        return '(a) single'
    elif 'in-law' in amen_list:
        return '(b) double'
    elif 'townhouse' in amen_list:
        return '(a) single'
    else:
        return '(c) multi'
sf['housing_type']=sf['amens_list'].apply(lambda amen_list:housing_type(amen_list))<jupyter_output><empty_output><jupyter_text>Parking* This variable will be grouped into four categories<jupyter_code>def parking_situation(amen_list):
    if 'attached garage' in amen_list:
        return '(b) protected'
    elif 'valet parking' in amen_list:
        return '(a) valet'
    elif 'carport' in amen_list:
        return '(b) protected'
    elif 'detatched garage' in amen_list:
        return '(b) protected'
    elif 'off-street parking' in amen_list:
        return '(c) off-street'
    else:
        return '(d) no parking'
sf['parking']=sf['amens_list'].apply(lambda amen_list: parking_situation(amen_list))
sf=sf.drop(['amens_list'],axis=1)
sf.shape
sf.head()<jupyter_output><empty_output><jupyter_text>Neighborhoods* The main objective for this part will be to re-adapt the neighborhoods in order to use regression models and avoiding too many neighborhoods<jupyter_code>cl_locations = ['alamo square / nopa', 'bayview', 'bernal heights', 
               'castro / upper market', 'cole valley / ashbury hts','downtown / civic / van ness',
               'excelsior / outer mission','financial district','glen park','haight ashbury','hayes valley',
               'ingleside / SFSU / CCSF','inner richmond','inner sunset / UCSF', 'laurel hts / presidio',
               'lower haight','lower nob hill','lower pac hts','marina / cow hollow','mission district',
               'nob hill','noe valley','north beach / telegraph hill','pacific heights','portola district',
               'potrero hill','richmond / seacliff', 'russian hill','SOMA / south beach','sunset / parkside',
               'tenderloin','treasure island','twin peaks / diamond hts','USF / panhandle','visitacion valley',
               'west portal / forest hill', 'western addition']
len(cl_locations)
hood_list=list(sf.hood.unique())
print("Number of unique neighborhoods provided: ", len(hood_list))
print("Number of NON-CL Hoods proviced: ", len(hood_list) - len(cl_locations))
extra_hoods = []

for hood in hood_list:
    if hood not in cl_locations:
        extra_hoods.append(hood)
        
print(extra_hoods)<jupyter_output>['San Francsico city of san francisco ', 'San Francisco city of san francisco ', 'russian hill city of san francisco ', 'San Francisco Civic Center city of san francisco ', ' city of san francisco ', 'Dog Patch city of san francisco ', '601 Niantic Ave Daly City, CA city of san francisco ', 'Los Banos city of san francisco ', '2228 Union St city of san francisco ', 'South of Market city of san francisco ', 'SAN JOSE city of san francisco ', 'SoMa city of san francisco ', '801 E Jones St city of san francisco ', 'Pacific Heights city of san francisco ', 'Bernal Heights city of san francisco ', 'Outer Richmond District city of san francisco ', 'Geary @ 41st Ave. city of san francisco ', 'Mission & Cortland city of san francisco ', 'Richmond District city of san francisco ', 'Ocean Ave city of san francisco ', '***Saratoga, CA*** city of san francisco ', 'Mission Bay city of san francisco ', 'Sausalito city of san francisco ', 'Showplace Square city of san francisco ', 'South Park city of[...]<jupyter_text>* Now we will re-order the different neighborhoods following the district keys<jupyter_code>sf_map_dict = {1: ['inner richmond', 'richmond / seacliff', 'San Franciso Richmond District' 
               ],
           2: ['inner sunset / UCSF', 'sunset / parkside', 'Golden Gate Heights, San Francisco' 
              ], 
           3: ['ingleside / SFSU / CCSF' 
              ], 
           4: ['twin peaks / diamond hts', 'west portal / forest hill', 'San Francisco/SunnySide'
              ], 
           5: ['alamo square / nopa', 'castro / upper market', 'cole valley / ashbury hts', 
               'glen park', 'haight ashbury', 'noe valley', 'The Castro', 'Glen Park',
               'North Panhandle', 'Eureka Valley' 
              ], 
           6: ['hayes valley', 'lower haight', 'USF / panhandle', 'western addition', 
              'NOPA', 'Hayes Valley', 'Lower Pacific Heights'
              ], 
           7: ['laurel hts / presidio', 'lower pac hts', 'marina / cow hollow', 'pacific heights',
              'Pacific Heights', 'Marina District', 'Marina'
              ], 
           8: ['downtown / civic / van ness', 'financial district', 'lower nob hill', 'nob hill', 
              'north beach / telegraph hill', 'russian hill', 'tenderloin', 'Nob Hill', 'Lower Nob Hill', 
               'North Beach', 'Civic Center, Downtown, Van Ness', 'Telegraph Hill', 
               'Embarcadero / North Waterfront', 'San Francisco North Waterfront' 
              ], 
           9: ['bernal heights', 'mission district', 'potrero hill', 'SOMA / south beach', 
              'Mission Bay', 'Mission District', 'South Park', 'South Beach', 'Rincon Hill, San Francisco', 
              'Rincon Hill', 'SoMa', 'SOMA', 'South of Market' 
              ], 
           10: ['bayview', 'excelsior / outer mission', 'portola district', 
                'visitacion valley', 'Bayview' 
               ]
           }
sf.head()
def set_hood_district(hood):
  for dist,hood_list in sf_map_dict.items():
    if hood in hood_list:
      return dist
sf['hood_district']=sf['hood'].apply(lambda x : set_hood_district(x))
sf.head()<jupyter_output><empty_output><jupyter_text>Dropping rows missing values<jupyter_code>sf.info()
sf.isnull().sum()
sf=sf[sf['hood_district'].notna()]
sf.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1091 entries, 1381 to 579
Data columns (total 12 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   title          1091 non-null   object 
 1   price          1091 non-null   int64  
 2   sqft           1091 non-null   float64
 3   beds           1091 non-null   float64
 4   bath           1091 non-null   float64
 5   hood           1091 non-null   object 
 6   amenities      1090 non-null   object 
 7   laundry        1091 non-null   object 
 8   pets           1091 non-null   object 
 9   housing_type   1091 non-null   object 
 10  parking        1091 non-null   object 
 11  hood_district  1091 non-null   float64
dtypes: float64(4), int64(1), object(7)
memory usage: 110.8+ KB<jupyter_text>Outliers* Before developping model for prediction purposes we will check the outliers to see if the posts were legitimate<jupyter_code>sf.describe()<jupyter_output><empty_output><jupyter_text>Price* The table above we see that **75%** are over **4K**, knowing that the maximum value is **28500**. Let's see if our data are accurate<jupyter_code>high_price_sf=sf[sf.price>10000]
len(high_price_sf)
high_price_sf.head()<jupyter_output><empty_output><jupyter_text>* In order to investigate our dataset we will need to check the links from **Craiglist** for that we use **sf_r** the first dataset before erasing the link columns<jupyter_code>high_prices=sf_r[sf_r.price>10000]
len(high_prices)
high_prices.describe()
high_prices.head()
price_28500=high_prices[high_prices.price==28500]
price_28500<jupyter_output><empty_output><jupyter_text>* After checking the maximum value for rent everything seems pretty legit so we will keep everything for our study  Square footage<jupyter_code>high_sqft_sf=sf[sf.sqft>5000]
len(high_sqft_sf)
high_sqft_sf.head()<jupyter_output><empty_output><jupyter_text>* In order to better filtrate our data we will look for square footage above **3K** (the 75 percentile is 1200) and less than **5K**<jupyter_code>high_sqft_sf=sf[(sf.sqft>3000)&(sf.sqft<5000)]
len(high_sqft_sf)
high_sqft_sf.head()
high_sqft_sf.describe()<jupyter_output><empty_output><jupyter_text>* In order to reduce the dispersion of our data we crop the maximal square footage to **4000**<jupyter_code>sf=sf[sf.sqft<4000]
sf.describe()<jupyter_output><empty_output><jupyter_text>* We follow the same steps for the lowest square footage<jupyter_code>low_sqft_sf=sf[ sf.sqft <250 ]
low_sqft_sf.head()
len(low_sqft_sf)
low_sqft_index=list(low_sqft_sf.index)
low_sqft_index
for link in sf_r.iloc[low_sqft_index].link:
  print(link)
  print('')
sf.drop([156],inplace=True)
sf.head()
sf.describe()<jupyter_output><empty_output><jupyter_text>Final Cleaning and exporting<jupyter_code>sf_clean = sf[['price', 'sqft', 'beds', 'bath', 'laundry', 'pets', 'housing_type', 'parking', 'hood_district']].reset_index(drop=True)
sf_clean.head()
sf_clean.info()
sf_clean.to_csv('sf_clean.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11962660443407235, 'fraction_numerical': 0.023710618436406067, 'mean_word_length': 4.136897626468473, 'pattern_counts': {'":': 0, '<': 179, '<?xml version=': 0, '>': 186, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '174931', 'license': 'MIT', 'n_tokens_mistral': 7556, 'n_tokens_neox': 7027, 'n_words': 2023, 'path': 'BD_project.ipynb', 'repo_name': 'Yassine-Boukhari/Big_Data_Project_Forecasting_rent_prices', 'size': '21425'}