<jupyter_start><jupyter_text>Problem 1: Data preparation<jupyter_code>wine_white=pd.read_csv('winequality-white.csv',sep=';')
wine_red=pd.read_csv('winequality-red.csv',sep=';')
combined=wine_white.append(wine_red)
X=combined.loc[:,combined.columns!='quality']
y=combined['quality']
X_normalize=normalize(X)
X_train,X_test,y_train,y_test=train_test_split(X_normalize,y,test_size=0.20, random_state=42)<jupyter_output><empty_output><jupyter_text>Problem 2: Linear regression<jupyter_code>model = LinearRegression().fit(X_train, y_train)
scores = cross_val_score(model, X_train, y_train, cv=10)
print(scores)
print('Average Performance is: %.4f'%np.mean(scores))
#r_sq = model.score(x, y)
#print('coefficient of determination:', r_sq)<jupyter_output>[0.19664201 0.20913544 0.14349591 0.22385844 0.20474605 0.1526096
 0.13917479 0.16686326 0.18906068 0.20693535]
Average Performance is: 0.1833<jupyter_text>Average performance on train set is 18% and there is no wild oscillation in performance measure.<jupyter_code>model = LinearRegression().fit(X_train, y_train)
r_sq = model.score(X_train, y_train)
print(r_sq)
y_pred=model.predict(X_test)
r2_test=r2_score(y_test, y_pred)
print(r2_test)<jupyter_output>0.17157030719137134<jupyter_text>There is no big difference between the average performance of train and test set. However, R2 of both sets are low in that it only accounts for 18% (17%) of variance in dependent variable for training set (test set).  Problem 3. Ridge and Lasso Regression<jupyter_code>parameters = [{'alpha': np.logspace(-6,3,num=10) }]
ridge=Ridge()
grid_search_ridge = GridSearchCV(ridge, param_grid = parameters,
                                 n_jobs = -1)
grid_search_ridge.fit(X_train,y_train)
print(grid_search_ridge.best_params_)
print(grid_search_ridge.best_score_)
Ridgeregression = Ridge(random_state=3, **grid_search_ridge.best_params_)
all_accuracies_ridge = cross_val_score(estimator=Ridgeregression, X=X_train, y=y_train, cv=10)
print(all_accuracies_ridge)
print(all_accuracies_ridge.mean())
Ridgeregression = Ridge(random_state=3, **grid_search_ridge.best_params_)
all_accuracies_ridge_test = cross_val_score(estimator=Ridgeregression, X=X_test, y=y_test, cv=10)
print(all_accuracies_ridge_test)
print(all_accuracies_ridge_test.mean())
model_ridge = RidgeCV(alphas = np.logspace(-6,3,num=10)).fit(X_train, y_train)
coef_ridge = pd.Series(model_ridge.coef_, index = pd.DataFrame(X_train).columns)
coef_ridge
lasso=Lasso(max_iter = 100000)
grid_search_lasso = GridSearchCV(lasso, param_grid = parameters,
                           n_jobs = -1)
grid_search_lasso.fit(X_train,y_train)
print(grid_search_lasso.best_params_)
print(grid_search_lasso.best_score_)
lassoregression = Lasso(random_state=3, **grid_search_lasso.best_params_)
all_accuracies_lasso = cross_val_score(estimator=lassoregression, X=X_train, y=y_train, cv=10)
print(all_accuracies_lasso)
print(all_accuracies_lasso.mean())
lassoregression = Lasso(random_state=3, **grid_search_lasso.best_params_)
all_accuracies_lasso_test = cross_val_score(estimator=lassoregression, X=X_test, y=y_test, cv=10)
print(all_accuracies_lasso_test)
print(all_accuracies_lasso_test.mean())
model_lasso = LassoCV(alphas = np.logspace(-6,3,num=10)).fit(X_train, y_train)
coef_lasso = pd.Series(model_lasso.coef_, index = pd.DataFrame(X_train).columns)
coef_lasso<jupyter_output><empty_output><jupyter_text>After applying grid search on training in Ridge and Lasso regression, the mean accuracy is 0.18 for Ridge and 0.15 for Lasso. What is striking is the fluctuation of the performance in test set for both models. However, the mean of accuracy on test set for both models is close to what we found on train set.  Problem 4. Visualization Performance<jupyter_code>plt.figure(figsize=(18,10))
plt.subplot(2,2,1)
plt.plot(np.arange(1,11,1), all_accuracies_ridge, color='blue', label='training ridge score')
plt.legend(loc='best')
plt.xlabel('Number of Cross Validation')
plt.ylabel('Scores')
plt.subplot(2,2,2)
plt.plot(np.arange(1,11,1), all_accuracies_ridge_test, color='orange', label='test ridge score')
plt.legend(loc='best')
plt.xlabel('Number of Cross Validation')
plt.ylabel('Scores')
plt.subplot(2,2,3)
plt.plot(np.arange(1,11,1), all_accuracies_lasso, color='red', label='training lasso score')
plt.legend(loc='best')
plt.xlabel('Number of Cross Validation')
plt.ylabel('Scores')
plt.subplot(2,2,4)
plt.plot(np.arange(1,11,1), all_accuracies_lasso_test, color='green', label='test lasso score')
plt.legend(loc='best')
plt.xlabel('Alpha')
plt.ylabel('Scores')<jupyter_output><empty_output><jupyter_text>This graph tells us that the performance of the models vary based on cross validation. For instance, Ridge regression on train set has lowest performance in 7th chunk and best performance in 4th chunk. Performance on test set is different from train set but the performance of both models are same on train and test set. Optimum alpha values are 0.0001 and  1e-06 for Ridge and Lasso, respectively and the corresponding coefficients of determination are 0.18292294734456932 and 0.18292671374284009  for Ridge and Lasso, respectively. The bottomline is Ridge outperforms Lasso by margin.  Problem 5. Visualize the coefficients<jupyter_code>import matplotlib
imp_coef = pd.concat([coef_ridge.sort_values()])
matplotlib.rcParams['figure.figsize'] = (8.0, 10.0)
imp_coef.plot(kind = "barh")
plt.title("Coefficients in the Ridge Model")
imp_coef = pd.concat([coef_lasso.sort_values()])
matplotlib.rcParams['figure.figsize'] = (8.0, 10.0)
imp_coef.plot(kind = "barh")
plt.title("Coefficients in the Lasso Model")<jupyter_output><empty_output><jupyter_text>These plots indicate that the importance of the estimated coefficients vary and some coefficients are very close to 0 implying very low impact on the wine quality.  As it is L1, Lasso has the property of feature selection but as we do not have a coefficient with 0 value, none of the coefficient can be dropped.  6. Classification Problem<jupyter_code>def categorize(data):
    if data<=5:
        return 'BAD'
    elif data>5 & data<=7:
        return 'AVERAGE'
    else:  
        return 'GOOD'
combined['quality_bin'] = combined['quality'].apply(categorize)
combined[combined['quality_bin']=='GOOD']
del combined['quality']
combined.head()
X=combined.loc[:,combined.columns!='quality_bin']
y=combined['quality_bin']
X_normalize=normalize(X)
X_train,X_test,y_train,y_test=train_test_split(X_normalize,y,test_size=0.20, random_state=42)
from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
seed = 7
models = []
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))

results = []
names = []
scoring = 'accuracy'
for name, model in models:
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    cv_results = model_selection.cross_val_score(model, X_train, y_train, cv=kfold, scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)
fig = plt.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11581321305386044, 'fraction_numerical': 0.037012470151233746, 'mean_word_length': 7.212418300653595, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '370352', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 2847, 'n_tokens_neox': 2609, 'n_words': 583, 'path': 'regression_practice.ipynb', 'repo_name': 'mouax209/mouax209', 'size': '7538'}