<jupyter_start><jupyter_text>Creating heatmap and paiplot for numerical features<jupyter_code>df.corr()
sns.heatmap(df.corr(), cmap="seismic", annot=True, vmin=-1, vmax=1);
sns.pairplot(df, height=1.2, aspect=1.5);<jupyter_output><empty_output><jupyter_text>Simple Linear Regression for target price and feature: number of accessories<jupyter_code>fig, ax = plt.subplots(figsize=(9, 7))

SWISU_x = df[['Ac_Count']]
SWISU_y = df['Price']

ax.scatter(SWISU_x, SWISU_y, alpha=.5)
ax.set_xlabel('Quanity of accesories')
ax.set_ylabel('Price')
ax.set_title('Price vs Quanity of accesories')

lr = LinearRegression()

lr.fit(SWISU_x, SWISU_y)

lr.score(SWISU_x, SWISU_y) #R2
preds = lr.predict(SWISU_x)

np.sqrt(mean_squared_error(SWISU_y, preds))
r2_score(SWISU_y, preds)
fig, ax = plt.subplots(figsize=(9, 7))

SWISU_x = df[['Ac_Count']]
SWISU_y = df['Price']
ax.scatter(SWISU_x, SWISU_y, alpha=.5)

_x = np.linspace(SWISU_x.min(), SWISU_x.max())
_y = lr.predict(_x)

ax.plot(_x, _y, color='orange')<jupyter_output><empty_output><jupyter_text>Result:R^2 is very low, I'll try to do prediction for numerical and categorical features  Regressions for numerical and categorical features Creating copy of df and mapping some categorical features<jupyter_code>df_dummies = df.copy()<jupyter_output><empty_output><jupyter_text>For this category, let's map an "other" category before adding the feature to our feature set and running a new benchmark.<jupyter_code>#brands to others
brnd_counts = df_dummies.Brand.value_counts()
other_brands = list(brnd_counts[brnd_counts <= 8].index)
df_dummies['Brand'] = df_dummies['Brand'].replace(other_brands, 'Other')
#Exterior Color to others
ec_counts = df_dummies.Exterior_Color.value_counts()
other_ec = list(ec_counts[ec_counts < 10].index)
df_dummies['Exterior_Color'] = df_dummies['Exterior_Color'].replace(other_ec, 'Other')
#Exterior Material to others
em_counts = df_dummies.Exterior_Material.value_counts()

other_em = list(em_counts[em_counts < 10].index)

df_dummies['Exterior_Material'] = df_dummies['Exterior_Material'].replace('Canvas (Coated)', 'Canvas Coated')
df_dummies['Exterior_Material'] = df_dummies['Exterior_Material'].replace('Canvas (Plain)', 'Canvas Plain')
df_dummies['Exterior_Material'] = df_dummies['Exterior_Material'].replace(other_em, 'Other')
#Interior Color to others
ic_counts = df_dummies.Interior_Color.value_counts()
other_ic = list(ic_counts[ic_counts < 10].index)
df_dummies['Interior_Color'] = df_dummies['Interior_Color'].replace(other_ic, 'Other')
#Interior Material to others
im_counts = df_dummies.Interior_Material.value_counts()
other_im = list(im_counts[im_counts < 10].index)
df_dummies['Interior_Material'] = df_dummies['Interior_Material'].replace(other_im, 'Other')
df_dummies = df_dummies[df_dummies.Price < 8000] #remove outliers<jupyter_output><empty_output><jupyter_text>Boxplots for every cateforical features and target(price)<jupyter_code>ax = sns.boxplot(x="Condition", y="Price", data=df_dummies)
ax = sns.boxplot(x="Brand", y="Price", data=df_dummies)
ax.set_xticklabels(ax.get_xticklabels(),rotation=90)
ax = sns.boxplot(x="Exterior_Color", y="Price", data=df_dummies)
ax.set_xticklabels(ax.get_xticklabels(),rotation=90)
ax = sns.boxplot(x="Exterior_Material", y="Price", data=df_dummies)
ax.set_xticklabels(ax.get_xticklabels(),rotation=90)
ax = sns.boxplot(x="Interior_Color", y="Price", data=df_dummies)
ax = sns.boxplot(x="Interior_Material", y="Price", data=df_dummies)
ax.set_xticklabels(ax.get_xticklabels(),rotation=90)<jupyter_output><empty_output><jupyter_text>Cross Validations and regressions<jupyter_code>X, y = pd.get_dummies(df_dummies.drop('Price',axis=1)), df_dummies['Price']

# hold out 20% of the data for final testing
X_, X_test, y_, y_test = train_test_split(X, y, test_size=.2, random_state=10)<jupyter_output><empty_output><jupyter_text>1 Validation<jupyter_code>def split_and_validate(X, y):
    '''
    For a set of features and target X, y, perform a 80/20 train/val split, 
    fit and validate a linear regression model, and report results
    '''
    
    # perform train/val split
    X_train, X_val, y_train, y_val = \
        train_test_split(X, y, test_size=0.2, random_state=42)
    
    # fit linear regression to training data
    lr_model = LinearRegression()
    lr_model.fit(X_train, y_train)
    
    # score fit model on validation data
    val_score = lr_model.score(X_val, y_val)
    
    # report results
    print('\nValidation R^2 score was:', val_score)
    print('Feature coefficient results: \n')
    for feature, coef in zip(X.columns, lr_model.coef_):
        print(feature, ':', f'{coef:.2f}') 
split_and_validate(X_, y_)<jupyter_output>Validation R^2 score was: 0.4551968648246444
Feature coefficient results: 

Volume : 0.38
Ac_Count : 86.86
Brand_Balenciaga : -890.03
Brand_Bottega Veneta : -532.63
Brand_Celine : -584.87
Brand_Chanel : 1740.55
Brand_Chloe : -710.18
Brand_<NAME> : 662.94
Brand_Fendi : -264.32
Brand_Gucci : -289.11
Brand_Hermes : 2073.47
Brand_<NAME> : 336.58
Brand_Other : -771.75
Brand_Prada : -624.43
Brand_Saint Laurent : -146.23
Condition_Excellent : 455.79
Condition_Fair : -290.34
Condition_Good : -448.88
Condition_Great : 315.64
Condition_Pristine : -62.33
Condition_Very Good : 30.13
Exterior_Color_Black : -127.82
Exterior_Color_Blue : 124.85
Exterior_Color_Brown : -38.78
Exterior_Color_Gray : 118.83
Exterior_Color_Green : -65.06
Exterior_Color_Neutral : 187.15
Exterior_Color_Orange : -51.58
Exterior_Color_Other : 109.02
Exterior_Color_Pink : -145.18
Exterior_Color_Purple : 210.53
Exterior_Color_Red : -418.91
Exterior_Color_White : -175.83
Exterior_Color_Yellow : 272.79
Exterior_Mat[...]<jupyter_text>Cross Validation on Linear Regression<jupyter_code>from sklearn.model_selection import cross_val_score
lm = LinearRegression()

r2_5 = cross_val_score(lm, X_, y_, # estimator, features, target
                cv=5, # number of folds 
                scoring='r2') # scoring metric
print(r2_5)
print(r2_5.mean())<jupyter_output>[0.45274819 0.47667696 0.4696716  0.44797821 0.41662466]
0.45273992331345037<jupyter_text>Polynomial regression:<jupyter_code>X, y = pd.get_dummies(df_dummies.drop('Price',axis=1)), df_dummies['Price']

# hold out 20% of the data for final testing
X_, X_test, y_, y_test = train_test_split(X, y, test_size=.2, random_state=10)

#this helps with the way kf will generate indices below
X, y = np.array(X), np.array(y)
kf = KFold(n_splits=5, shuffle=True, random_state = 71)
cv_lm_poly_r2s = []

for train_ind, val_ind in kf.split(X,y):    
    #poly with degree 2
    poly = PolynomialFeatures(degree=2, interaction_only=True)
    
    X_train, y_train = X[train_ind], y[train_ind]
    X_val, y_val = X[val_ind], y[val_ind] 

    X_train_poly = poly.fit_transform(X_train)
    X_val_poly = poly.transform(X_val)

    lm_poly = LinearRegression()
    
    lm_poly.fit(X_train_poly, y_train)
    cv_lm_poly_r2s.append(round(lm_poly.score(X_val_poly, y_val), 3))
    
print('Poly scores: ', cv_lm_poly_r2s, '\n')

print(f'Poly mean cv r^2: {np.mean(cv_lm_poly_r2s):.3f} +- {np.std(cv_lm_poly_r2s):.3f}')<jupyter_output>Poly scores:  [-19.678, -14.761, -41.516, -1210265931564651.8, -108.731] 

Poly mean cv r^2: -242053186312967.312 +- 484106372625842.250<jupyter_text>There's so different R^2 on each validation set it's mean model has high varince overfit  There were overfitting model. Too complex model(too many features), high variance Should be used regularization for overfit model! I use Lasso regularization (it's a lot of features after OneHotEncoding, so Lasso will help to get rid of not meaningful features)<jupyter_code>X, y = pd.get_dummies(df_dummies.drop('Price',axis=1)), df_dummies['Price']
#Split the data 60 - 20 - 20 train/val/test

X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.2,random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=.25, random_state=43)
## Scale the data as before
std = StandardScaler()
std.fit(X_train.values)
## Scale the Predictors on both the train and test set
X_tr = std.transform(X_train.values)
X_te = std.transform(X_test.values)
# Run the cross validation, find the best alpha, refit the model on all the data with that alpha

alphavec = 10**np.linspace(-2,2,200)

lasso_model = LassoCV(alphas = alphavec, cv=5)
lasso_model.fit(X_tr, y_train)
# This is the best alpha value it found - not far from the value
# selected using simple validation
lasso_model.alpha_
# These are the (standardized) coefficients found
# when it refit using that best alpha
list(zip(X_train.columns, lasso_model.coef_))
# Make predictions on the test set using the new model
test_set_pred = lasso_model.predict(X_te)
#Mean Absolute Error (MAE)
def mae(y_true, y_pred):
    return np.mean(np.abs(y_pred - y_true)) 

mae(y_test, test_set_pred)
# Find the MAE and R^2 on the test set using this model
mae(y_test, test_set_pred)
r2_score(y_test, test_set_pred)<jupyter_output><empty_output><jupyter_text>Plotting:<jupyter_code>## Scale the variables
std = StandardScaler()
std.fit(X_train.values)
X_tr = std.transform(X_train.values)
## Note: lars_path takes numpy matrices, not pandas dataframes

print("Computing regularization path using the LARS ...")
alphas, _, coefs = lars_path(X_tr, y_train.values, method='lasso')
# plotting the LARS path

xx = np.sum(np.abs(coefs.T), axis=1)
xx /= xx[-1]

plt.figure(figsize=(20,20))
plt.plot(xx, coefs.T)
ymin, ymax = plt.ylim()
plt.vlines(xx, ymin, ymax, linestyle='dashed')
plt.xlabel('|coef| / max|coef|')
plt.ylabel('Coefficients')
plt.title('LASSO Path (no retail price)')
plt.axis('tight')
#plt.legend(X_train.columns)
plt.show()<jupyter_output><empty_output><jupyter_text>Because R^2 was low(0.48) I decided to add retail price too for analysis, to make better prediction model  Creating DF Adding retail price feature<jupyter_code>df_retail= df_all.loc[:,['Brand', 'Price', 'Condition', 'Retail_Price', 'Volume',
       'Exterior_Color', 'Exterior_Material', 'Interior_Color',
       'Interior_Material', 'Ac_Count']]
df_retail = df_retail.dropna()
#brands to others
brnd_counts = df_retail.Brand.value_counts()
other_brands = list(brnd_counts[brnd_counts <= 8].index)
df_retail['Brand'] = df_retail['Brand'].replace(other_brands, 'Other')

#Exterior Color to others
ec_counts = df_retail.Exterior_Color.value_counts()
other_ec = list(ec_counts[ec_counts < 10].index)
df_retail['Exterior_Color'] = df_retail['Exterior_Color'].replace(other_ec, 'Other')


#Exterior Material to others
em_counts = df_retail.Exterior_Material.value_counts()
other_em = list(em_counts[em_counts < 10].index)
df_retail['Exterior_Material'] = df_retail['Exterior_Material'].replace('Canvas (Coated)', 'Canvas Coated')
df_retail['Exterior_Material'] = df_retail['Exterior_Material'].replace('Canvas (Plain)', 'Canvas Plain')
df_retail['Exterior_Material'] = df_retail['Exterior_Material'].replace(other_em, 'Other')

#Interior Color to others
ic_counts = df_retail.Interior_Color.value_counts()
other_ic = list(ic_counts[ic_counts < 10].index)
df_retail['Interior_Color'] = df_retail['Interior_Color'].replace(other_ic, 'Other')

#Interior Material to others
im_counts = df_retail.Interior_Material.value_counts()
other_im = list(im_counts[im_counts < 10].index)
df_retail['Interior_Material'] = df_retail['Interior_Material'].replace(other_im, 'Other')<jupyter_output><empty_output><jupyter_text>Correlation for price and numerical fetures after adding retail price<jupyter_code>df_retail.corr()
sns.heatmap(df_retail.corr(), cmap="seismic", annot=True, vmin=-1, vmax=1);
sns.pairplot(df_retail, height=1.2, aspect=1.5);
sns.lmplot(x="Price", y="Retail_Price", data = df_retail)<jupyter_output><empty_output><jupyter_text>Lasso regularization after adding retail price<jupyter_code>X, y = pd.get_dummies(df_retail.drop('Price',axis=1)), df_retail['Price']
#Split the data 60 - 20 - 20 train/val/test
X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.2,random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=.25, random_state=43)
## Scale the data as before
std = StandardScaler()
std.fit(X_train.values)
## Scale the Predictors on both the train and test set
X_tr = std.transform(X_train.values)
X_te = std.transform(X_test.values)
# Run the cross validation, find the best alpha, refit the model on all the data with that alpha

alphavec = 10**np.linspace(-2,2,200)

lasso_model = LassoCV(alphas = alphavec, cv=5)
lasso_model.fit(X_tr, y_train)
# This is the best alpha value it found - not far from the value
# selected using simple validation
lasso_model.alpha_
# These are the (standardized) coefficients found
# when it refit using that best alpha
list(zip(X_train.columns, lasso_model.coef_))
# Make predictions on the test set using the new model
test_set_pred = lasso_model.predict(X_te)
#Mean Absolute Error (MAE)
def mae(y_true, y_pred):
    return np.mean(np.abs(y_pred - y_true)) 

mae(y_test, test_set_pred)
r2_score(y_test, test_set_pred)<jupyter_output><empty_output><jupyter_text>After adding retil value I got R^2 = 0.78Much better!MAE = 393 which means prediction could be off(plus or minus) $393 for each bag Plot<jupyter_code>## Scale the variables
std = StandardScaler()
std.fit(X_train.values)
X_tr = std.transform(X_train.values)
## Note: lars_path takes numpy matrices, not pandas dataframes

print("Computing regularization path using the LARS ...")
alphas, _, coefs = lars_path(X_tr, y_train.values, method='lasso')
# plotting the LARS path

xx = np.sum(np.abs(coefs.T), axis=1)
xx /= xx[-1]

plt.figure(figsize=(10,10))
plt.plot(xx, coefs.T)
ymin, ymax = plt.ylim()
plt.vlines(xx, ymin, ymax, linestyle='dashed')
plt.xlabel('|coef| / max|coef|')
plt.ylabel('Coefficients')
plt.title('LASSO Path (w/retail price)')
plt.axis('tight')
#plt.legend(X_train.columns)
plt.show()<jupyter_output><empty_output><jupyter_text>Prediction graphic:<jupyter_code>fig, ax = plt.subplots(figsize=(9, 9))

#SWISU_x = y_train_val
#SWISU_y = y_test

ax.scatter(test_set_pred, y_test, alpha=.5)
ax.set_xlabel('Prediction')
ax.set_ylabel('Actual')
#ax.set_title('Predictable price & Actual')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12714005051922536, 'fraction_numerical': 0.034661801852371596, 'mean_word_length': 6.369700103412616, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '402295', 'license': 'MIT', 'n_tokens_mistral': 5853, 'n_tokens_neox': 5406, 'n_words': 1148, 'path': 'Regressions_notebook.ipynb', 'repo_name': 'albasitdikova/Project2', 'size': '14267'}