<jupyter_start><jupyter_text>Imports<jupyter_code>import pandas as pd
import numpy as np
import patsy
import datetime as dt
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import RidgeCV

'''
The output of plotting commands is displayed 'inline within 
frontends' with the result of the plot being placed directly
below the code cell that produced it.
'''
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>df = pd.read_csv('/Users/johnmetzger/Desktop/Coding/Project2/base_scrape.csv')

## Check the size and datatypes
df.info()<jupyter_output><empty_output><jupyter_text>Data type conversion Feature: Make datetime data numerical.  * Take column to be converted to datetime using '.to_datetime()'* Add it to the dataframe but **NOTE** it may need to have a matching index with the dataframe it was derived from.* QC call of new data frame to make sure the new column is there<jupyter_code>df['datetimes']=pd.to_datetime(df['datetimes'])
df_hour= df['datetimes'].dt.hour
dave_hour = dict(df_hour)
df["hours"] = pd.Series(dave_hour)
df<jupyter_output><empty_output><jupyter_text>Feature: Day of the month---Same as above but * data is day of the month (here: January 2021).* makes a dictionary and adds it as a column.<jupyter_code>df_day= df['datetimes'].dt.day
mikey_day = dict(df_day)
df["day"] = pd.Series(mikey_day)
df<jupyter_output><empty_output><jupyter_text>Feature: Title length---Same as above, but* it counts the characters in the title cell* makes a dictionary and adds it as a column.<jupyter_code>title_char_count = df['title'].str.len()
colin_count = dict(title_char_count)
df['title_char_count']= pd.Series(colin_count)
df<jupyter_output><empty_output><jupyter_text>Feature: Has pictures---Same as above, but* it changes the boolean to a binary 0-1 using the '*1'    - NOTE: You can only do this '*1" when you have True-False data* does **not** make a dictionary* adds binary 0-1 data as a new column.<jupyter_code>john = df['pics']*1
df['has_pics']= pd.Series(john)
df<jupyter_output><empty_output><jupyter_text>Target: Price* 'price' feature is cleaned up using ".str.replace('','')"* Convert price to int64<jupyter_code>df['price'] = df['price'].str.replace('$', '')
df['price'] = df['price'].str.replace(',', '')

df['price']=np.array(df['price'], dtype=int)<jupyter_output><empty_output><jupyter_text>Set asside dataframe with all columns---Save this new dataframe as the [dataframe = original dataframe + new columns (i.e., new columns added to original but none removed)].<jupyter_code># df_init_all_col is short for 'data frame, initial, all columns'
df_init_all_col = df
df_init_all_col<jupyter_output><empty_output><jupyter_text>Delete columns that won't be used in stats modeling<jupyter_code>del df['datetimes']
del df['pics']
del df['hood']
del df['title']<jupyter_output><ipython-input-8-f9a400414950>:5: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will*not* be treated as literal strings when regex=True.
  df['price'] = df['price'].str.replace('$', '')<jupyter_text>Data inspection---  Data frame for analysis<jupyter_code>df<jupyter_output><empty_output><jupyter_text>**NOTE** I detected a bug in seaborn that would crash because of 1 or 2 data points that were mulitple orders of magnitude larger than the rest of the data.  The unique values here were prices that were $10^6$ and  $10^8$ USD.The filtering method is placed at the end, but in reality may need to be plotting if future data also triggers this bug.  Correlation matrix<jupyter_code>df.corr()<jupyter_output><empty_output><jupyter_text>--- Heatmap of target and all features<jupyter_code>doot = sns.heatmap(df.corr(), cmap="seismic", annot=True, vmin=-1, vmax=1, fmt=".2f")<jupyter_output><empty_output><jupyter_text>Saves the figure.<jupyter_code>doot.figure.savefig("doot.pdf", format="pdf")<jupyter_output><empty_output><jupyter_text>--- Pairplots of target and all features<jupyter_code>dino = sns.pairplot(df)<jupyter_output><empty_output><jupyter_text>Saves the figure.  * There is a bug where '.figure.savefig()' doesn't work with seaborn or matplotlib. Here you can to use the format'.fig.savefig()'<jupyter_code>dino.fig.savefig("dino.pdf", format="pdf")<jupyter_output><empty_output><jupyter_text>Histogram of price frequency* X-axis scaled for clarity as few data items are >$10,000$ USD.* Can be reasonably scaled down to $3,000$ USD for this project<jupyter_code>yeet = sns.histplot(df['price'])
plt.xlim(0, 10000)<jupyter_output><empty_output><jupyter_text>Saves the figure.<jupyter_code>yeet.figure.savefig("yeet10000.svg", format="svg")<jupyter_output><empty_output><jupyter_text>Export csv for modeling<jupyter_code>df.to_csv('/Users/johnmetzger/Desktop/Coding/Project2/df.csv', index = False)<jupyter_output><empty_output><jupyter_text>Appendix  Isolated plot of price vs. hours.* Helps see the lower posted prices during the late night/early morning hours.<jupyter_code>dances = sns.scatterplot(data=df, x="hours", y="price")
dances.set(ylim=(0, 3000))<jupyter_output><empty_output><jupyter_text>Saves the figure.<jupyter_code>dances.figure.savefig("dances_3000.png", format="png")<jupyter_output><empty_output><jupyter_text>Outlier check<jupyter_code>df['price'].unique()<jupyter_output><empty_output><jupyter_text>Outliers detected.Select range to filter them.*Here, 50,000 USD was chosen.<jupyter_code>df.drop(df.loc[df['price']>50000].index, inplace=True)
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12248888888888888, 'fraction_numerical': 0.01368888888888889, 'mean_word_length': 7.153623188405797, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 91, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '210019', 'license': 'MIT', 'n_tokens_mistral': 1984, 'n_tokens_neox': 1891, 'n_words': 518, 'path': 'Working Notebooks/Base_scrape_2_regression.ipynb', 'repo_name': 'JGarrechtMetzger/Project2', 'size': '5625'}