<jupyter_start><jupyter_text>Fetch data<jupyter_code>X, y = fetch_openml('mnist_784', version=1, return_X_y=True)
sorted(set(y))<jupyter_output><empty_output><jupyter_text>Depending on the Depending on the versions of pandas and sklearn that you're using, you may need to expressly convert these to numpy arrays.<jupyter_code>X = np.array(X)
y = np.array(y)<jupyter_output><empty_output><jupyter_text>Predifine classes and number of classes<jupyter_code>classes = sorted(set(y))
nclasses = len(classes)
classes<jupyter_output><empty_output><jupyter_text>Plot random sample<jupyter_code>clf_util.plot_sample(X, y, classes, 5)<jupyter_output><empty_output><jupyter_text>Create training and test dataset¶<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, # Our data
                                                    y, # Labels
                                                    random_state=9, # To get the same sample if we do it again
                                                    train_size=7500, #0.8 to get %
                                                    test_size=2500)  #0.2 to get %
X_train_scaled[10]<jupyter_output><empty_output><jupyter_text>__Scaling the features__I'm choosing here to scale the features between 0-1Why am I doing that?<jupyter_code>#scaling the features
X_train_scaled = X_train/255.0
X_test_scaled = X_test/255.0<jupyter_output><empty_output><jupyter_text>Train a logistic regression model We're going to use a logistic regression model<jupyter_code>clf = LogisticRegression(penalty='none', 
                         tol=0.1, 
                         solver='saga',
                         multi_class='multinomial').fit(X_train_scaled, y_train)
help(LogisticRegression)
#to check the shape of the coefficient matrix
clf.coef_.shape<jupyter_output><empty_output><jupyter_text>Interpreting the coefficients What features of each class are most informative?Again, there's a util function called plot<jupyter_code>clf_util.plot_coefs(clf.coef_, nclasses)<jupyter_output><empty_output><jupyter_text>Test on individual point We can then use this model to predict the class of an individual data point.We can inspect it using the util function pot_individual().<jupyter_code>clf_util.plot_individual(X_test_scaled, y_test, 30)<jupyter_output><empty_output><jupyter_text>Plot conditional probabilities We can compute the conditional probability for each class<jupyter_code>clf_util.plot_probs(X_test_scaled, 30, clf, classes)<jupyter_output>I think that this is class 5<jupyter_text>We can find the class with the highest probability.If this matches the actual label for the first test sample, then our prediction is correct.  Evaluate on the whole training data<jupyter_code>y_pred = clf.predict(X_test_scaled) # Use the model to predict
y_pred[:10]
len(y_pred)<jupyter_output><empty_output><jupyter_text>We then calculate metrics for accuracy based on comparing the predicted labels with the actual labels.<jupyter_code>#method 2
accuracy = accuracy_score(y_test, y_pred)
print(accuracy)<jupyter_output>0.9088<jupyter_text>We can use classification_report() from sklearn to construct a detailed report showing different evaluation metrics<jupyter_code>cm = metrics.classification_report(y_test, y_pred)
print(cm)<jupyter_output>precision    recall  f1-score   support

           0       0.97      0.97      0.97       244
           1       0.91      0.97      0.94       287
           2       0.91      0.90      0.90       235
           3       0.90      0.88      0.89       281
           4       0.90      0.94      0.92       213
           5       0.88      0.84      0.86       215
           6       0.95      0.93      0.94       225
           7       0.93      0.91      0.92       257
           8       0.84      0.87      0.85       253
           9       0.91      0.88      0.89       290

    accuracy                           0.91      2500
   macro avg       0.91      0.91      0.91      2500
weighted avg       0.91      0.91      0.91      2500<jupyter_text>This can be visualised using a confusion matrix<jupyter_code>clf_util.plot_cm(y_test, y_pred, normalized= False)<jupyter_output><empty_output><jupyter_text>We can also normalize the confusion matrix to show percentages instead of raw values<jupyter_code>clf_util.plot_cm(y_test, y_pred, normalized=True)
import cv2
import matplotlib.pyplot as plt
os.getcwd()
filepath = os.path.join("data", "test.png")
test_image =cv2.imread(filepath)
gray = cv2.bitwise_not(cv2.cvtColor(test_image, cv2.COLOR_BGR2GRAY))
plt.imshow(gray, cmap=plt.cm.gray)

compressed = cv2.resize(gray, (28, 28), interpolation=cv2.INTER_AREA)
plt.imshow(compressed, cmap=plt.cm.gray)
clf_util.predict_unseen(compressed, clf, classes)

clf_util.prediction_coefficients(compressed, clf, classes)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07942088934850051, 'fraction_numerical': 0.04570837642192348, 'mean_word_length': 3.0232945091514143, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '559344', 'license': 'MIT', 'n_tokens_mistral': 1691, 'n_tokens_neox': 1545, 'n_words': 388, 'path': 'notebooks/Session 6.ipynb', 'repo_name': 'aksroa/Visual-portfolio', 'size': '4835'}