<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 1, Module 2*---  Regression 2 AssignmentYou'll continue to **predict how much it costs to rent an apartment in NYC,** using the dataset from renthop.com.- [X] Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.- [X] Engineer at least two new features. (See below for explanation & ideas.)- [X] Fit a linear regression model with at least two features.- [X] Get the model's coefficients and intercept.- [X] Get regression metrics RMSE, MAE, and $R^2$, for both the train and test data.- [X] What's the best test MAE you can get? Share your score and features used with your cohort on Slack!- [X] As always, commit your notebook to your fork of the GitHub repo. [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)> "Some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used." — <NAME>, ["A Few Useful Things to Know about Machine Learning"](https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf)> "Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering." — <NAME>, [Machine Learning and AI via Brain simulations](https://forum.stanford.edu/events/2011/2011slides/plenary/2011plenaryNg.pdf) > Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work.  Feature Ideas- Does the apartment have a description?- How long is the description?- How many total perks does each apartment have?- Are cats _or_ dogs allowed?- Are cats _and_ dogs allowed?- Total number of rooms (beds + baths)- Ratio of beds to baths- What's the neighborhood, based on address or latitude & longitude? Stretch Goals- [X] If you want more math, skim [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf),  Chapter 3.1, Simple Linear Regression, & Chapter 3.2, Multiple Linear Regression- [X] If you want more introduction, watch [<NAME>, Statistics 101: Simple Linear Regression](https://www.youtube.com/watch?v=ZkjP5RJLQF4)(20 minutes, over 1 million views)- [X] Add your own stretch goal(s) !<jupyter_code>%%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'
    !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'
    
# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
import numpy as np
import pandas as pd

# Read New York City apartment rental listing data
df = pd.read_csv(DATA_PATH+'apartments/renthop-nyc.csv')
assert df.shape == (49352, 34)

# Remove the most extreme 1% prices,
# the most extreme .1% latitudes, &
# the most extreme .1% longitudes
df = df[(df['price'] >= np.percentile(df['price'], 0.5)) & 
        (df['price'] <= np.percentile(df['price'], 99.5)) & 
        (df['latitude'] >= np.percentile(df['latitude'], 0.05)) & 
        (df['latitude'] < np.percentile(df['latitude'], 99.95)) &
        (df['longitude'] >= np.percentile(df['longitude'], 0.05)) & 
        (df['longitude'] <= np.percentile(df['longitude'], 99.95))]
df = df.dropna(subset=['description', 'display_address', 'street_address'], axis=0)
# These rows have no data on Mapbox
df = df.drop([380, 5149, 45372])<jupyter_output><empty_output><jupyter_text>Engineer at least two new features<jupyter_code># Generate feature: does the apartment have a description?
no_description = ['        ', '<p><a  website_redacted ', ' ']
condition = df['description'].isin(no_description)
df.loc[condition, 'has_description'] = 0
df.loc[~condition, 'has_description'] = 1
df['has_description'] = df['has_description'].apply(int)
# Generate feature: length of the apartment description
condition = (df['has_description'] == 0)
df.loc[condition, 'len_description'] = 0
df.loc[~condition, 'len_description'] = df.loc[~condition, 'description'].apply(len)
df['len_description'] = df['len_description'].apply(int)
# Generate feature: number of total perks
df['num_perks'] = df.iloc[:, 10:34].sum(axis=1)
# Generate feature: cats or dogs allowed
condition = (df['cats_allowed'] == 1) | (df['dogs_allowed'] == 1)
df.loc[condition, 'cats_or_dogs_allowed'] = 1
df.loc[~condition, 'cats_or_dogs_allowed'] = 0
df['cats_or_dogs_allowed'] = df['cats_or_dogs_allowed'].apply(int)
# Generate feature: cats and dogs allowed
condition = (df['cats_allowed'] == 1) & (df['dogs_allowed'] == 1)
df.loc[condition, 'cats_and_dogs_allowed'] = 1
df.loc[~condition, 'cats_and_dogs_allowed'] = 0
df['cats_and_dogs_allowed'] = df['cats_and_dogs_allowed'].apply(int)
# Generate feature: number of bathrooms and bedrooms
df['total_rooms'] = df['bathrooms'] + df['bedrooms']
# Generate feature: ratio of bathrooms to bedrooms
df['ratio_bath_to_bed'] = df['bathrooms'] / df['bedrooms']
df.loc[df['bedrooms'] == 0, 'ratio_bath_to_bed'] = 0
# Generate feature: interest level number
df['interest_level_num'] = df['interest_level'].replace({'low': 1, 'medium': 2, 'high': 3})<jupyter_output><empty_output><jupyter_text>Generate feature: get the neighborhood based on latitude and longitude<jupyter_code>import requests
API_KEY = '?access_token=<KEY>'
url_init = 'https://api.mapbox.com/geocoding/v5/'
mode = 'mapbox.places/'
types = '&types=neighborhood'
def mapbox_get(latitude, longitude, print_url=False):
    lon_lat = str(longitude) + '%2C' + str(latitude) + '.json'
    return get_neighborhood(lon_lat, print_url)
def get_neighborhood(lon_lat, print_url):
    url = url_init + mode + lon_lat + API_KEY + types
    if print_url:
        print(url)
    response = requests.get(url)
    return response
r = mapbox_get(df['latitude'][0], df['longitude'][0], print_url=True)
r.status_code
import json

def jprint(obj):
    # create a formatted string of the Python JSON object
    text = json.dumps(obj, sort_keys=True, indent=4)
    print(text)

jprint(r.json())
jprint(r.json()['features'][0]['text'])
neighborhoods = []
import time
for i in df.index.tolist()[len(neighborhoods):]:
    t = time.time()
    r = mapbox_get(df['latitude'][i], df['longitude'][i])
    nbh = r.json()['features'][0]['text']
    neighborhoods.append(nbh)
    elapsed = time.time() - t
    print(f'Row number: {i}, time elapsed: {elapsed} sec')
# df.insert(9, 'neighborhood', neighborhoods)
df.tail()
df[(df['latitude'] == 40.7813) & (df['longitude'] == -74.0094)]
df.index.tolist()[len(neighborhoods)]
# Take the neighborhoods list and export to CSV
import csv
import os

def WriteListToCSV(csv_file,csv_columns,data_list):
    with open(csv_file, 'w') as csvfile:
        writer = csv.writer(csvfile, dialect='excel', quoting=csv.QUOTE_NONNUMERIC)
        writer.writerow(csv_columns)
        for data in data_list:
            writer.writerow(data)
    return

csv_columns = ['neighborhood']
csv_data_list = [['Neighborhood'] + neighborhoods]

currentPath = os.getcwd()
csv_file = currentPath + "/csv/neighborhoods.csv"


WriteListToCSV(csv_file,csv_columns,csv_data_list)<jupyter_output><empty_output><jupyter_text>Check our work<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Do train-test split<jupyter_code># Split the data into two sets based on date
train = df[df['created'] < '2016-06']
test = df[df['created'] > '2016-06']
train.head()
test.head()<jupyter_output><empty_output><jupyter_text>Fit a multiple regression with at least two features<jupyter_code># Create correlation matrix
corr_matrix = df.corr().abs()

# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))

# Find feature columns where correlation is greater than 0.9
to_drop = [column for column in upper.columns.values if any(upper[column] > 0.9)]
print(to_drop)
# 1. Import the appropriate estimator class from Scikit-Learn
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import GridSearchCV

# 2. Instantiate this class
model = LinearRegression()

# 3. Arrange the X features matrices and y target vectors
features = ['bathrooms', 'bedrooms'] + train.columns.values[10:].tolist()
for feature in to_drop:
    features.remove(feature)
feature_print = "\n".join(features)
print(f'Linear Regression, dependent on:\n{feature_print}\n')

X_train = train[features]
X_test = test[features]

target = 'price'
y_train = train[target]
y_test = test[target]

# Cross-validation
# param_grid = {'fit_intercept': [True, False],
#               'normalize': [True, False]}

# grid = GridSearchCV(LinearRegression(), param_grid, cv=7)

# grid.fit(X_train, y_train)
# print(grid.best_params_)

# 4. Fit the model
# model = grid.best_estimator_
model.fit(X_train, y_train)
y_pred_train = model.predict(X_train)
mae = mean_absolute_error(y_train, y_pred_train)
print(f'Train Error: ${mae:.0f}')

# 5. Apply the model to new data
y_pred_test = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred_test)
print(f'Test Error: ${mae:.0f}')<jupyter_output>Linear Regression, dependent on:
bathrooms
bedrooms
elevator
cats_allowed
hardwood_floors
dogs_allowed
doorman
dishwasher
no_fee
laundry_in_building
fitness_center
pre-war
laundry_in_unit
roof_deck
outdoor_space
dining_room
high_speed_internet
balcony
swimming_pool
new_construction
terrace
exclusive
loft
garden_patio
wheelchair_access
common_outdoor_space
interest_level_num
has_description
len_description
num_perks
ratio_bath_to_bed

Train Error: $725
Test Error: $725<jupyter_text>Get the model's coefficients and intercept<jupyter_code>print('Intercept:', '\t'*2, f'{model.intercept_:.6f}')
coefficients = pd.Series(model.coef_, features)
print('\nCoefficients:')
print(coefficients.to_string())<jupyter_output>Intercept: 		 1264.899439

Coefficients:
bathrooms               1684.767483
bedrooms                 470.032431
elevator                  81.498517
cats_allowed            -153.395791
hardwood_floors         -165.096850
dogs_allowed              91.657317
doorman                  492.177040
dishwasher                61.581625
no_fee                  -176.741048
laundry_in_building     -134.842651
fitness_center           126.333281
pre-war                 -122.181700
laundry_in_unit          476.871980
roof_deck               -200.364926
outdoor_space           -145.560882
dining_room              126.274898
high_speed_internet     -330.802849
balcony                 -128.979920
swimming_pool            -29.992715
new_construction        -193.326305
terrace                  164.776986
exclusive                107.166897
loft                     177.440841
garden_patio            -109.737939
wheelchair_access         79.371920
common_outdoor_space     -37.033476
interest_level_num     [...]<jupyter_text>Get regression metrics RMSE, MAE, and $R^2$ , for both the train and test data.<jupyter_code>from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Calculate regression metrics
mse = mean_squared_error(y_train, y_pred_train)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_train, y_pred_train)
r2 = r2_score(y_train, y_pred_train)

# Print regression metrics
print('=======Training Set=======')
print(f'Root Mean Squared Error: ${rmse:.0f}')
print(f'Mean Absolute Error: ${mae:.0f}')
print('R^2:', r2)

# Calculate regression metrics
mse = mean_squared_error(y_test, y_pred_test)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred_test)
r2 = r2_score(y_test, y_pred_test)

# Print regression metrics
print('\n=======Testing Set=======')
print(f'Root Mean Squared Error: ${rmse:.0f}')
print(f'Mean Absolute Error: ${mae:.0f}')
print('R^2:', r2)<jupyter_output>=======Training Set=======
Root Mean Squared Error: 1116.5943457168876
Mean Absolute Error: 725.2054606757093
R^2: 0.599396392394393

=======Testing Set=======
Root Mean Squared Error: 1097.096925005977
Mean Absolute Error: 724.5369280866752
R^2: 0.6145446116393236
{'fraction_non_alphanumeric': 0.1264311814859927, 'fraction_numerical': 0.04198132358911896, 'mean_word_length': 4.85918173168411, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 43, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '84180', 'license': 'MIT', 'n_tokens_mistral': 4529, 'n_tokens_neox': 4038, 'n_words': 1086, 'path': 'module2-regression-2/LS_DS_212_assignment.ipynb', 'repo_name': 'jwross24/DS-Unit-2-Linear-Models', 'size': '12416'}