<jupyter_start><jupyter_text>We combined both csv files together and created a "type" column to see whether the type of the wine will affect the judgement of the quality. Because sometimes, human's perception of wine type may affect the independent scoring on the wine quality, so that's why we added a binary feature to account for this factor.<jupyter_code>train_df, test_df = train_test_split(wine, test_size=0.2, random_state=123)
X_train = train_df.drop(columns=["quality"])
X_test = test_df.drop(columns=["quality"])

y_train = train_df["quality"]
y_test = test_df["quality"]
numeric_feats = X_train.select_dtypes(include=[np.number]).columns.values.tolist()
binary_feats = ["type"]

numeric_transformer = make_pipeline(StandardScaler())
binary_transformer = make_pipeline(OneHotEncoder(drop="if_binary", dtype=int))

preprocessor = make_column_transformer(
    (numeric_transformer, numeric_feats),
    (binary_transformer, binary_feats)
)<jupyter_output><empty_output><jupyter_text>In our data file, we only have numerical features originally; however we need to scale those number before we feed them in the model. And we engineered a binary feature ("type"), we used `OneHotEncoder` with `drop="if_binary"` argument to form the transformer.<jupyter_code>column_name = numeric_feats + binary_feats
pd.DataFrame(preprocessor.fit_transform(X_train, y_train), columns = column_name)
# Imported from DSCI 573 Lecture Notes from UBC
def mean_std_cross_val_scores(model, X_train, y_train, **kwargs):
    """
    Returns mean and std of cross validation

    Parameters
    ----------
    model :
        scikit-learn model
    X_train : numpy array or pandas DataFrame
        X in the training data
    y_train :
        y in the training data

    Returns
    ----------
        pandas Series with mean scores from cross_validation
    """

    scores = cross_validate(model, X_train, y_train, **kwargs)

    mean_scores = pd.DataFrame(scores).mean()
    std_scores = pd.DataFrame(scores).std()
    out_col = []

    for i in range(len(mean_scores)):
        out_col.append((f"%0.3f (+/- %0.3f)" % (mean_scores[i], std_scores[i])))

    return pd.Series(data=out_col, index=mean_scores.index)
# Imported from DSCI 573 Lecture Notes from UBC
def mape(true, pred):
    return 100.0 * np.mean(np.abs((pred - true) / true))


# make a scorer function that we can pass into cross-validation
mape_scorer = make_scorer(mape, greater_is_better=False)

score_types_reg = {
    "neg_mean_squared_error": "neg_mean_squared_error",
    "neg_root_mean_squared_error": "neg_root_mean_squared_error",
    "neg_mean_absolute_error": "neg_mean_absolute_error",
    "r2": "r2",
    "mape_scorer": mape_scorer,
}

models = {
    "Ridge": Ridge(max_iter=2000),
    "SVR": SVR(),
    "OneVsRest":OneVsRestClassifier(LogisticRegression()),
    "Random Forest": RandomForestRegressor(random_state=123)
}
import imblearn
from imblearn.pipeline import make_pipeline as make_imb_pipeline
from imblearn.over_sampling import RandomOverSampler
results_comb={}
for keys in models.keys():
    pipe_comb = make_imb_pipeline(RandomOverSampler(sampling_strategy="minority"), preprocessor, models[keys])
    results_comb[keys]=mean_std_cross_val_scores(
        pipe_comb, X_train, y_train, return_train_score=True, scoring=score_types_reg
    )
pd.DataFrame(results_comb)<jupyter_output><empty_output><jupyter_text>After comparing different regression models by using various matrix, we found the best model is Random Forest, because we got highest cross-validation score. However, we figured out that we may encounter some overfitting issue with Random Forest model as the difference between train score and validation score is quite wide. So, we further conduct feature selections and hyper-parameter optimization as follows.<jupyter_code>rfe = RFE(RandomForestRegressor(random_state=123), n_features_to_select=10)

pipe_rf_rfe = make_imb_pipeline(RandomOverSampler(sampling_strategy="minority"), preprocessor, rfe, RandomForestRegressor(random_state=123))

results_comb['Random Forest_rfe'] = mean_std_cross_val_scores(pipe_rf_rfe, X_train, y_train, return_train_score=True, scoring=score_types_reg)
pd.DataFrame(results_comb)
pipe_rf_rfe.fit(X_train, y_train)
pipe_rf_rfe.named_steps['rfe'].ranking_
rfe_fs = pipe_rf_rfe.named_steps["rfe"].support_
rfe_fs
rfe_selected_feats = X_train.columns[rfe_fs]
rfe_selected_feats<jupyter_output><empty_output><jupyter_text>By using Recursive Feature Elimination algorithm, we chose to drop "type" and "density" features. Because by dropping those two features, even though we did not get much better scores, we are able to achieve the same scores with lesser features. It is great because it will simplify our model and save cost for future data collection.<jupyter_code>from scipy.stats import randint
param_dist = {"randomforestregressor__max_depth": randint(low=5, high=1000),
             "randomforestregressor__max_leaf_nodes": randint(low=5, high=1000),
             "randomforestregressor__n_estimators": randint(low=5, high=1000),}

random_search = RandomizedSearchCV(
    pipe_rf_rfe,
    param_distributions=param_dist,
    n_jobs=-1,
    n_iter=10,
    cv=5,
    random_state=123
)

random_search.fit(X_train, y_train)
random_search.best_params_
random_search.best_score_
random_search.score(X_train, y_train)
random_search.score(X_test, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09366901147723065, 'fraction_numerical': 0.010736764161421697, 'mean_word_length': 5.662145499383477, 'pattern_counts': {'":': 12, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '169341', 'license': 'MIT', 'n_tokens_mistral': 1797, 'n_tokens_neox': 1705, 'n_words': 444, 'path': 'misc/model_fitting_raw_code.ipynb', 'repo_name': 'PavelLevchenko/DSCI_522_Group19_Wine_Quality_Score_Predictor', 'size': '5402'}