<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 1, Module 2*---  Regression 2 AssignmentYou'll continue to **predict how much it costs to rent an apartment in NYC,** using the dataset from renthop.com.- [ ] Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.- [ ] Engineer at least two new features. (See below for explanation & ideas.)- [ ] Fit a linear regression model with at least two features.- [ ] Get the model's coefficients and intercept.- [ ] Get regression metrics RMSE, MAE, and $R^2$, for both the train and test data.- [ ] What's the best test MAE you can get? Share your score and features used with your cohort on Slack!- [ ] As always, commit your notebook to your fork of the GitHub repo. [Feature Engineering](https://en.wikipedia.org/wiki/Feature_engineering)> "Some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used." — <NAME>, ["A Few Useful Things to Know about Machine Learning"](https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf)> "Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering." — <NAME>, [Machine Learning and AI via Brain simulations](https://forum.stanford.edu/events/2011/2011slides/plenary/2011plenaryNg.pdf) > Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work.  Feature Ideas- Does the apartment have a description?- How long is the description?- How many total perks does each apartment have?- Are cats _or_ dogs allowed?- Are cats _and_ dogs allowed?- Total number of rooms (beds + baths)- Ratio of beds to baths- What's the neighborhood, based on address or latitude & longitude? Stretch Goals- [ ] If you want more math, skim [_An Introduction to Statistical Learning_](http://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf),  Chapter 3.1, Simple Linear Regression, & Chapter 3.2, Multiple Linear Regression- [ ] If you want more introduction, watch [<NAME>, Statistics 101: Simple Linear Regression](https://www.youtube.com/watch?v=ZkjP5RJLQF4)(20 minutes, over 1 million views)- [ ] Add your own stretch goal(s) !<jupyter_code>%%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Applied-Modeling/master/data/'
    !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'
    
# Ignore this Numpy warning when using Plotly Express:
# FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
import warnings
warnings.filterwarnings(action='ignore', category=FutureWarning, module='numpy')
import numpy as np
import pandas as pd

# Read New York City apartment rental listing data
df = pd.read_csv(DATA_PATH+'apartments/renthop-nyc.csv')
assert df.shape == (49352, 34)<jupyter_output><empty_output><jupyter_text>Assignment- 1- Do train/test split. Use data from April & May 2016 to train. Use data from June 2016 to test.2- Engineer at least two new features. (See below for explanation & ideas.)3- Fit a linear regression model with at least two features.4- Get the model's coefficients and intercept.5- Get regression metrics RMSE, MAE, and  𝑅2 , for both the train and test data.6- What's the best test MAE you can get? Share your score and features used with your cohort on Slack!**As always, commit your notebook to your fork of the GitHub repo.<jupyter_code>df.head()
# Removing outliers
df = df[(df['price'] >= np.percentile(df['price'], 0.5)) & 
        (df['price'] <= np.percentile(df['price'], 99.5)) & 
        (df['latitude'] >= np.percentile(df['latitude'], 0.05)) & 
        (df['latitude'] < np.percentile(df['latitude'], 99.95)) &
        (df['longitude'] >= np.percentile(df['longitude'], 0.05)) & 
        (df['longitude'] <= np.percentile(df['longitude'], 99.95))]
# I don't need string columns for my models
df = df.drop(['description', 'display_address', 'street_address', 'interest_level'], axis = 1)
df.head()
# To divide test and train data, i need to convert this column to datetime format
import datetime as dt
df['created'] = df['created'].astype('datetime64[ns]')
df.dtypes
df = df.sort_values(by=['created'])
## Feature 1:
#I create a feature with my own rent preferencies
df_me = df[['dishwasher', 'laundry_in_unit', 'fitness_center', 'high_speed_internet','terrace', 'cats_allowed']]
df['my_preferencies'] = df_me.sum(axis=1)
print(df.shape)
df.head()
## Feature 2:
df['core'] = df['bathrooms'] + df['bedrooms'] + df['elevator'] + df['terrace']
df['bed&bath'] = df['bedrooms'] + df['bathrooms']
df.head()
# Now i split my data to train and test sets
train = df[df['created'] <= '2016-05-31']
test = df[df['created'] >= '2016-06-01']
train
test
# I drop the time column in already filtered test and train
train = train.drop(['created'], axis=1)
test = test.drop(['created'], axis=1)
train
# Never mind, this didn't work

# from scipy import stats
# train = train[(np.abs(stats.zscore(train)) < 3).all(axis=1)]
train.shape
# Make a model with two features I created
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
model = LinearRegression()
features = ['core', 'my_preferencies']
target = ['price']
X_train = train[features]
X_test = test[features]
y_train = train[target]
y_test = test[target]
# Baseline
y_train['price'].mean()
# Fit the model
model.fit(X_train, y_train)
# Mean absolute error for training
y_pred_train = model.predict(X_train)
mean_absolute_error(y_train, y_pred_train)
# Mean absolute error for testing
y_pred_test = model.predict(X_test)
mean_absolute_error(y_test, y_pred_test)
# Coefficients
model.coef_[0]
# Train results for the model
mse_train = mean_squared_error(y_train, y_pred_train)
rmse_train = np.sqrt(mse_train)
mae_train = mean_absolute_error(y_train, y_pred_train)
r2_train = r2_score(y_train, y_pred_train)
print('RMSE for train:', rmse_train)
print('MAE for train:', mae_train)
print('R2 for train:', r2_train)
# Test results for the model
mse_test = mean_squared_error(y_test, y_pred_test)
rmse_test = np.sqrt(mse_test)
mae_test = mean_absolute_error(y_test, y_pred_test)
r2_test = r2_score(y_test, y_pred_test)
print('RMSE for test:', rmse_test)
print('MAE for test:', mae_test)
print('R2 for test:', r2_test)
# Let's try another model with different features
model2 = LinearRegression()
features = ['bed&bath', 'my_preferencies']
target = ['price']
X_train = train[features]
X_test = test[features]
y_train = train[target]
y_test = test[target]
model2.fit(X_train, y_train)
# MAE of the second model is higher, so my first worked better
y_pred_train2 = model2.predict(X_train)
mean_absolute_error(y_train, y_pred_train)
model2.coef_[0]
# Train results for model2
mse_train2 = mean_squared_error(y_train, y_pred_train2)
rmse_train2 = np.sqrt(mse_train)
mae_train2 = mean_absolute_error(y_train, y_pred_train2)
r2_train2 = r2_score(y_train, y_pred_train2)
print('RMSE for train:', rmse_train2)
print('MAE for train:', mae_train2)
print('R2 for train:', r2_train2)
y_pred_test2 = model2.predict(X_test)
mean_absolute_error(y_test, y_pred_test2)
# Test results for model2
mse_test2 = mean_squared_error(y_test, y_pred_test2)
rmse_test2 = np.sqrt(mse_test)
mae_test2 = mean_absolute_error(y_test, y_pred_test2)
r2_test2 = r2_score(y_test, y_pred_test2)
print('RMSE for test:', rmse_test2)
print('MAE for test:', mae_test2)
print('R2 for test:', r2_test2)<jupyter_output>RMSE for test: 1291.5740365908905
MAE for test: 871.1754393566712
R2 for test: 0.4593194738127827
{'fraction_non_alphanumeric': 0.1180920630850109, 'fraction_numerical': 0.02718297217591999, 'mean_word_length': 5.4891846921797, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 18, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '461426', 'license': 'MIT', 'n_tokens_mistral': 2739, 'n_tokens_neox': 2536, 'n_words': 829, 'path': 'module2-regression-2/LS_DS_212_assignment.ipynb', 'repo_name': 'KathyRoma/DS-Unit-2-Linear-Models', 'size': '7817'}