<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 1, Module 2*---  Regression 2 AssignmentYou'll continue to **predict how much it costs to rent an apartment in NYC,** using the dataset from renthop.com.- [ ] Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.- [ ] Engineer at least two new features. (See below for explanation & ideas.)- [ ] Fit a linear regression model with at least two features.- [ ] Get the model's coefficients and intercept.- [ ] Get regression metrics RMSE, MAE, and $R^2$, for both the train and test data.- [ ] What's the best test MAE you can get? Share your score and features used with your cohort on Slack!- [ ] As always, commit your notebook to your fork of the GitHub repo. [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)> "Some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used." — <NAME>, ["A Few Useful Things to Know about Machine Learning"](https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf)> "Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering." — <NAME>, [Machine Learning and AI via Brain simulations](https://forum.stanford.edu/events/2011/2011slides/plenary/2011plenaryNg.pdf) > Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work.  Feature Ideas- Does the apartment have a description?- How long is the description?- How many total perks does each apartment have?- Are cats _or_ dogs allowed?- Are cats _and_ dogs allowed?- Total number of rooms (beds + baths)- Ratio of beds to baths- What's the neighborhood, based on address or latitude & longitude? Stretch Goals- [ ] If you want more math, skim [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf),  Chapter 3.1, Simple Linear Regression, & Chapter 3.2, Multiple Linear Regression- [ ] If you want more introduction, watch [<NAME>, Statistics 101: Simple Linear Regression](https://www.youtube.com/watch?v=ZkjP5RJLQF4)(20 minutes, over 1 million views)- [ ] Add your own stretch goal(s) !<jupyter_code>%%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'
    !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'
    
# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
import numpy as np
import pandas as pd

# Read New York City apartment rental listing data
df = pd.read_csv(DATA_PATH+'apartments/renthop-nyc.csv')
assert df.shape == (49352, 34)

# Remove the most extreme 1% prices,
# the most extreme .1% latitudes, &
# the most extreme .1% longitudes
df = df[(df['price'] >= np.percentile(df['price'], 0.5)) & 
        (df['price'] <= np.percentile(df['price'], 99.5)) & 
        (df['latitude'] >= np.percentile(df['latitude'], 0.05)) & 
        (df['latitude'] < np.percentile(df['latitude'], 99.95)) &
        (df['longitude'] >= np.percentile(df['longitude'], 0.05)) & 
        (df['longitude'] <= np.percentile(df['longitude'], 99.95))]<jupyter_output><empty_output><jupyter_text>Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
print(df.shape)
df.head(3)
# April and May into train data
train = df[df['created'].str.contains('-04-') | 
           df['created'].str.contains('-05-')]
print(train.shape)
train.head(3)
train['created'].str.contains('-06-').sum()
# Looks like June has been succesfully removed
a = df['created'].str.contains('-04-').sum()
b = df['created'].str.contains('-05-').sum()

a+b
# Looks like I still have appropriate number of rows, 
# so train data should be g2g
# June into test data
test = df[df['created'].str.contains('-06-')] 
print(test.shape)
test.head(3)
df['created'].str.contains('-06-').sum()
# Looks like test data (June) has appropriate number of rows as well<jupyter_output><empty_output><jupyter_text>Engineer at least two new features.<jupyter_code># I'll make a column for total number of pets allowed (cats/dogs)
train['total_pets'] = train['cats_allowed'] + train['dogs_allowed']
train.head(1)
# I'll make a column that is the total number of amenities

train['amenities'] = train['elevator'] + train['doorman'] + train['dishwasher'] 
+ train['laundry_in_building'] + train['fitness_center'] + train['laundry_in_unit']
+ train['roof_deck'] + train['outdoor_space'] + train['high_speed_internet']
+ train['balcony'] + train['swimming_pool'] + train['terrace'] + train['garden_patio']
+ train['wheelchair_access'] + train['common_outdoor_space'] 

train.head(1)
# I'll make a column for total number of rooms

train['total_rooms'] = train['bathrooms'] + train['bedrooms']
train.head(1)
# I also think a column for "special location" could be interesting, meaning
# pre-war, exclusive or loft - I would think having a few of these or all 3 
# could impact price

train['spec_loc'] = train['pre-war'] + train['exclusive'] + train['loft']
train.head(1)<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy<jupyter_text>Fit a linear regression model with at least two features.<jupyter_code># First, I'll establish a baseline which is simply the mean rental price

target = 'price'
y_train = train[target]
y_test = test[target]
guess = y_train.mean()
print('Mean baseline:', guess)
# And I'll establish my baseline error

y_pred = [guess] * len(y_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Baseline train error: ${mae:.0f}')
y_pred = [guess] * len(y_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Baseline test error: ${mae:.0f}')
# Now I'll fit a linear regression model

model = LinearRegression()

features = ['amenities','total_rooms','total_pets','spec_loc','bedrooms',
            'bathrooms']

X_train = train[features]
y_train = train[target]

model.fit(X_train, y_train)
y_pred = model.predict(X_train)
mae = mean_absolute_error(y_train, y_pred)
print(f'Train error: ${mae:.0f}')

# Looks like an improvement of about -$500 error
# I'll need to add those features to the test data in order to compare

test['total_pets'] = test['cats_allowed'] + test['dogs_allowed']

test['total_rooms'] = test['bathrooms'] + test['bedrooms']

test['spec_loc'] = test['pre-war'] + test['exclusive'] + test['loft']

test['amenities'] = test['elevator'] + test['doorman'] + test['dishwasher'] 
+ test['laundry_in_building'] + test['fitness_center'] + test['laundry_in_unit']
+ test['roof_deck'] + test['outdoor_space'] + test['high_speed_internet']
+ test['balcony'] + test['swimming_pool'] + test['terrace'] + test['garden_patio']
+ test['wheelchair_access'] + test['common_outdoor_space'] 
X_test = test[features]

y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test error: ${mae:.0f}')

# Similar improvement as the train error<jupyter_output>Test error: $782<jupyter_text>Get the model's coefficients and intercept.<jupyter_code>print('Intercept', model.intercept_)
print(f'Amenities coefficient: ${model.coef_[0]:.0f}')
print(f'Total rooms coefficient: ${model.coef_[1]:.0f}')
print(f'Total pets coefficient: ${model.coef_[2]:.0f}')
print(f'Special location coefficient: ${model.coef_[3]:.0f}')
print(f'Bedrooms coefficient: ${model.coef_[4]:.0f}')
print(f'Bathrooms coefficient: ${model.coef_[5]:.0f}')<jupyter_output>Intercept 192.89845792840106
Amenities coefficient: $288
Total rooms coefficient: $771
Total pets coefficient: $70
Special location coefficient: $-2
Bedrooms coefficient: $-351
Bathrooms coefficient: $1122<jupyter_text>Get regression metrics RMSE, MAE, and  𝑅2 , for both the train and test data.<jupyter_code># Regression metrics for TRAIN data

y_pred = model.predict(X_train)
mse = mean_squared_error(y_train, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_train, y_pred)
r2 = r2_score(y_train, y_pred)
print(f'Root Mean Squared Error: ${rmse:.0f}')
print(f'Mean Absolute Error: ${mae:.0f}')
print('R^2:', r2)
# Regression metrics for TEST data

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'Root Mean Squared Error: ${rmse:.0f}')
print(f'Mean Absolute Error: ${mae:.0f}')
print('R^2:', r2)<jupyter_output>Root Mean Squared Error: $1174
Mean Absolute Error: $782
R^2: 0.5566894643923536<jupyter_text>Stretch - how low can we go?<jupyter_code># I'll try encoding the interest level to see if that has an impact
test.columns
test['interest_level'].value_counts()
train['interest_level'] = train['interest_level'].str.replace('low','1')
train['interest_level'] = train['interest_level'].str.replace('medium','2')
train['interest_level'] = train['interest_level'].str.replace('high','3')
train['interest_level'] = train['interest_level'].astype(int)
test['interest_level'] = test['interest_level'].str.replace('low','1')
test['interest_level'] = test['interest_level'].str.replace('medium','2')
test['interest_level'] = test['interest_level'].str.replace('high','3')
test['interest_level'] = test['interest_level'].astype(int)
# test.head()
# Now I can view interest level in the correlation matrix
test.corr()
# Then re-fit the model

features = ['amenities','total_rooms','total_pets','spec_loc','bedrooms',
            'bathrooms','interest_level']

X_train = train[features]
y_train = train[target]

model.fit(X_train, y_train)
X_test = test[features]

y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test error: ${mae:.0f}')

# Test error went from $782 to $742, so a slight improvement ...
# Let's try adding longitude ...
features = ['amenities','total_rooms','total_pets','spec_loc','bedrooms',
            'bathrooms','interest_level','longitude']

X_train = train[features]
y_train = train[target]

model.fit(X_train, y_train)
X_test = test[features]

y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test error: ${mae:.0f}')

# Down to $689
# Dining-room looked like it had some correlation, and I didn't include that as 
# an 'amenity'
features = ['amenities','total_rooms','total_pets','spec_loc','bedrooms',
            'bathrooms','interest_level','longitude','dining_room']

X_train = train[features]
y_train = train[target]

model.fit(X_train, y_train)
X_test = test[features]

y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
print(f'Test error: ${mae:.0f}')

# No difference, so I'll be satisfied with $689 for now<jupyter_output>Test error: $689
{'fraction_non_alphanumeric': 0.14234407848633496, 'fraction_numerical': 0.01979677645409951, 'mean_word_length': 5.751626256652868, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 33, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '552303', 'license': 'MIT', 'n_tokens_mistral': 4001, 'n_tokens_neox': 3742, 'n_words': 1053, 'path': 'module2-regression-2/2_1_2_A_Copy_of_LS_DS_212_assignment.ipynb', 'repo_name': 'ekselan/DS-Unit-2-Linear-Models', 'size': '11434'}