<jupyter_start><jupyter_text>Data Cleaning<jupyter_code># Reading merged data Arabica 98% and Robusta 2% of records
df_quality = pd.read_csv("data/merged_data_cleaned.csv")
df_quality.head()
################### Quality Measures - Features ####################
# Aroma
# Flavor
# Aftertaste
# Acidity
# Body
# Balance
# Uniformity
# Cup Cleanliness
# Sweetness
# Moisture
# Defects
################### Bean Metadata - Labels ####################
# Species (arabica / robusta)
# Country of Origin
# Processing.Method'

df_quality = df_quality[['Processing.Method', 'Species', 'Aroma', 'Flavor', 'Aftertaste', 'Acidity', 
                         'Body', 'Balance', 'Uniformity', 'Clean.Cup', 'Sweetness',
                         'Moisture', 'Category.One.Defects', 'Category.Two.Defects']]
# check for null values in data
df_quality.isnull().sum()
df_quality.dropna(inplace=True)
# checking again for null values in data
df_quality.isnull().sum()<jupyter_output><empty_output><jupyter_text>Data is not fully normaly distributed. There is skewness. Algorithms will perform better with a normalized or scaled dataset. There are outliers in data, so better to select Models which are insensitive to outliers like SVC, Decision Tree, and Ensembling Techniques  Label Encoding columns with not numeric values<jupyter_code>df_qual = df_quality.copy()
from sklearn.preprocessing import LabelEncoder

df_qual = df_qual.apply(LabelEncoder().fit_transform)
# correlation matrix
corr_mat = df_qual.corr()
top_corr_features = corr_mat.index
fig = plt.figure(figsize=(10,10))
fig.patch.set_facecolor('white')
# heatmap
sns.heatmap(df_qual[top_corr_features].corr(), annot=True, cmap="RdYlGn")
# From the Correlation heatmap we can see Flavor column is highly correlated to many other features
# So we can remove that feature from training the model. It improved overall accuracy by 2% 
# but after SMOTE it doesn't matter

df_quality.drop('Flavor', axis = 1, inplace = True) 
df_quality.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1169 entries, 0 to 1338
Data columns (total 13 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   Processing.Method     1169 non-null   object 
 1   Species               1169 non-null   object 
 2   Aroma                 1169 non-null   float64
 3   Aftertaste            1169 non-null   float64
 4   Acidity               1169 non-null   float64
 5   Body                  1169 non-null   float64
 6   Balance               1169 non-null   float64
 7   Uniformity            1169 non-null   float64
 8   Clean.Cup             1169 non-null   float64
 9   Sweetness             1169 non-null   float64
 10  Moisture              1169 non-null   float64
 11  Category.One.Defects  1169 non-null   int64  
 12  Category.Two.Defects  1169 non-null   int64  
dtypes: float64(9), int64(2), object(2)
memory usage: 127.9+ KB<jupyter_text>Select your features (columns)<jupyter_code># Set features. This will also be used as your x values.
X = df_quality.iloc[:, 1:13]
y = df_quality.iloc[:, 0]
print(X.shape, y.shape)
# Checking for imbalanced Dataset
df_quality['Processing.Method'].value_counts()
X_encoded = X.apply(LabelEncoder().fit_transform)<jupyter_output><empty_output><jupyter_text>Train Test Split<jupyter_code># Creating train test split on oversampled data
from sklearn.model_selection import train_test_split

X_train_full, X_test, y_train_full, y_test = train_test_split(X_encoded, y, random_state=13, test_size=0.2)<jupyter_output><empty_output><jupyter_text>Pre-processingScale and Normalize the data using the StandardScaler<jupyter_code>from sklearn.preprocessing import StandardScaler
# Create a StandardScater model and fit it to the training data
X_scaler = StandardScaler().fit(X_train_full)

# Transform the training and testing data using the X_scaler
X_train_scaled = X_scaler.transform(X_train_full)
X_test_scaled = X_scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Imbalanced Dataset Synthethetic Minority Over sampling Technique (SMOTE)<jupyter_code># Over sampling
from imblearn.over_sampling import SMOTE
sampling_strategy= {'Washed / Wet': 1000, 'Natural / Dry': 1000, 'Semi-washed / Semi-pulped': 1000, 'Other': 1000, 'Pulped natural / honey': 1000}
synth = SMOTE(random_state=13, sampling_strategy=sampling_strategy)
X_resampled, y_resampled = synth.fit_resample(X_train_scaled, y_train_full)

from collections import Counter
print("Before SMOTE count:" , Counter(y_train_full))
print("After SMOTE count:" , Counter(y_resampled))<jupyter_output>Before SMOTE count: Counter({'Washed / Wet': 644, 'Natural / Dry': 217, 'Semi-washed / Semi-pulped': 45, 'Other': 18, 'Pulped natural / honey': 11})
After SMOTE count: Counter({'Washed / Wet': 1000, 'Natural / Dry': 1000, 'Semi-washed / Semi-pulped': 1000, 'Other': 1000, 'Pulped natural / honey': 1000})<jupyter_text>Create a Train Validation Split After SMOTE<jupyter_code># Creating train test split on oversampled data
from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(X_resampled, y_resampled, random_state=13, test_size=0.1)<jupyter_output><empty_output><jupyter_text>Train the Model using SVC<jupyter_code># Support vector machine linear classifier
from sklearn.svm import SVC 
model = SVC(kernel='linear')
model.fit(X_train, y_train)

print(f"Training Data Score: {model.score(X_train, y_train)}")
print(f"Testing Data Score: {model.score(X_val, y_val)}")<jupyter_output>Training Data Score: 0.528
Testing Data Score: 0.524<jupyter_text>Evaluate the Model using Confusion Matrix<jupyter_code># Make predictions with the model
predictions = model.predict(X_val)  

from sklearn.metrics import confusion_matrix

conf_mat = confusion_matrix(y_true=y_val, y_pred=predictions)
print('Confusion matrix:\n', conf_mat)

labels = df_quality['Processing.Method'].unique()
fig = plt.figure()
fig.patch.set_facecolor('white')
ax = fig.add_subplot(111)
cax = ax.matshow(conf_mat, cmap=plt.cm.Blues)
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('Expected')
plt.show()<jupyter_output>Confusion matrix:
 [[34 33 10 26  7]
 [ 9 69  2 14  1]
 [ 6  9 85  0  5]
 [ 5 21 13 54  3]
 [11 21 13 29 20]]<jupyter_text>Hyperparameter Tuning for SVC Model<jupyter_code># Create the GridSearch estimator along with a parameter object containing the values to adjust
from sklearn.model_selection import GridSearchCV
param_grid = {'C': [50,100],
              'kernel': ['rbf','linear'],
              'gamma': [0.001, 0.01]}
grid = GridSearchCV(model, param_grid, cv=5)
# # Train the model with GridSearch
# # This will take the SVC model and try each combination of parameters

# grid.fit(X_train, y_train)
# print(grid.best_params_)
# print(grid.best_score_)<jupyter_output><empty_output><jupyter_text>Decision Tree Model<jupyter_code>from sklearn import tree

# Create and score a decision tree classifier
clfTree = tree.DecisionTreeClassifier(min_samples_split=4)
clf = clfTree.fit(X_train, y_train)

# Make predictions with the hypertuned model
treePredictions = clf.predict(X_val)
# Calculate classification report
from sklearn.metrics import classification_report
print(classification_report(y_val, treePredictions))
print(f"Test Accuracy: {clf.score(X_test_scaled, y_test)}")<jupyter_output>Test Accuracy: 0.452991452991453<jupyter_text>Random Forest Grid Search<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

model_params = {
    'model': RandomForestClassifier(random_state=5),
    'params' : {
        'n_estimators': [100,1000],  # no of trees
        'min_samples_split':[3, 4],
        'max_samples': [0.2, 0.5]
    }
}

RFclf =  GridSearchCV(model_params['model'], model_params['params'], cv=5, n_jobs=-1, return_train_score=False)
RFclf.fit(X_train, y_train)

print(RFclf.best_params_)
print(RFclf.best_score_)
print(f"Train Accuracy: {RFclf.score(X_train, y_train)}")
print(f"Validation Accuracy: {RFclf.score(X_val, y_val)}")
print(f"Test Accuracy: {RFclf.score(X_test_scaled, y_test)}")<jupyter_output>Test Accuracy: 0.6538461538461539<jupyter_text>Save the Random Forest Model<jupyter_code>import joblib
joblib.dump(RFclf, 'models/model_process_method.sav')
joblib.dump(X_scaler, 'models/model_method_scaler.sav')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11487446707721459, 'fraction_numerical': 0.03588346755092373, 'mean_word_length': 5.023537803138374, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '317383', 'license': 'MIT', 'n_tokens_mistral': 3129, 'n_tokens_neox': 2812, 'n_words': 717, 'path': 'predict_Method.ipynb', 'repo_name': 'CicilyGeorge/Coffee-Analysis-and-Predictions', 'size': '8444'}