<jupyter_start><jupyter_text>Task 1> Use PCA to investigate the dimensionality of $𝑋_{train}$ and plot the first 16 PCA modes as 16 ×16 images<jupyter_code># Analyzing the first 100 singular values
pca = PCA()
pca.fit(X_train)
fig, ax = plt.subplots(1, 1, figsize=(8, 8))

ax.plot(np.log(pca.singular_values_[0:100]))
ax.set_xlabel('index $j$')
ax.set_ylabel('$\log(\sigma_j)$')
plot_digits(pca.components_, 4, 'First 16 PCA modes')<jupyter_output><empty_output><jupyter_text>Task 2> How many PCA modes do you need to keep in order to approximate $𝑋_{train}$ up to 60%,80% and 90% in the Frobenius norm? Do you need the entire 16 ×16 image for each data point?<jupyter_code># define function to get the num of modes needed for keeping specific percent of Frobenius norm
F_norm = np.sqrt(sum(pca.singular_values_**2))


def find_num_of_modes(percent: float) -> int:
    for _ in range(100):
        approx_F_norm = np.sqrt(sum(pca.singular_values_[:_]**2))
        if approx_F_norm >= F_norm * percent:
            return _


percents = [0.6, 0.8, 0.9]
print('number of modes needed respectively:',
      *list(map(find_num_of_modes, percents)))<jupyter_output>number of modes needed respectively: 3 7 14<jupyter_text>From this, we can see that not the entire 16$\times$16 image is needed for each data point. Only the first few or several modes are needed for approximation.  Task 3> Train a classifier to distinguish the digits 1 and 8 via the following steps:* First, you need to write a function that extracts the features and labels of the digits 1 and 8 fromthe training data set. Let us call these 𝑋(1,8) and 𝑌(1,8).* Then project 𝑋(1,8) on the first 16 PCA modes of $𝑋_{train}$ computed in step 1, this should give you a matrix $𝐴_{train}$ which has 16 columns corresponding to the PCA coefficients of each feature and 455 rows corresponding to the total number of 1’s and 8’s in the training set. * Assign label −1 to the images of the digit 1 and label +1 to the images of the digit 8. This should result in a vector $𝑏_{train}$ ∈{−1,+1}$^{455}$.* Use Ridge regression or least squares to train a predictor for the vector $b_{train}$ by linearly combining the columns of $A_{train}$.* Report the training mean squared error (MSE) of your classifier$$MSE_{train}(1,8) = \frac{1}{\text{length of }𝑏_{train}}×‖𝐴_{train} \hat{𝛽} −𝑏_{train}‖_2^2$$* Report the testing MSE of your classifier $$MSE_{test}(1,8) = \frac{1}{\text{length of }𝑏_{test}}×‖𝐴_{test} \hat{𝛽} −𝑏_{test}‖_2^2$$ where you need to construct analogously the matrix $𝐴_{test}$ and $𝑏_{test}$ corresponding to the digits 1 and 8 from the test data set.<jupyter_code># perform 16 modes PCA
pca_16 = PCA(n_components=16)

# define fuction to extract relevant features with labels n1 ,n2
# return A for 16 modes PCA transforming A_train
# return b for b_train of labeling -1 to 1 and 1 to 8


def extract_relavent_features(n1: int, n2: int, X: np.array,
                              Y: np.array) -> Tuple[np.array, list]:
    N = len(X)
    index = [i for i in range(N) if Y[i] == n1 or Y[i] == n2]
    A = pca_16.fit_transform(X)[index]
    b = list(map(lambda x: -1 if Y[index][x] == n1 else 1, range(len(index))))
    return A, b


# get the A_train, b_train as well as the A_test, b_test
A_train, b_train = extract_relavent_features(1, 8, X_train, Y_train)
A_test, b_test = extract_relavent_features(1, 8, X_test, Y_test)
# two Ridge regression for different alphas, 1 and 0.02

reg1 = Ridge(alpha=1)
reg1.fit(A_train, b_train)

reg2 = Ridge(alpha=0.02)
reg2.fit(A_train, b_train)
# calculating MSE for train and test set for alpha = 1
b = np.insert(reg1.coef_, 0, reg1.intercept_)

MSE_train = 1/len(b_train) * \
    np.linalg.norm((np.dot(A_train, b[1:])+b[0]-b_train), 2)**2
MSE_test = 1/len(b_test) * \
    np.linalg.norm((np.dot(A_test, b[1:])+b[0]-b_test), 2)**2

print(
    f'For alpha = 1, the MSE for train set is {MSE_train} and {MSE_test} for test set'
)
# calculating MSE for train and test set for alpha = 0.02
b = np.insert(reg2.coef_, 0, reg2.intercept_)

MSE_train = 1/len(b_train) * \
    np.linalg.norm((np.dot(A_train, b[1:])+b[0]-b_train), 2)**2
MSE_test = 1/len(b_test) * \
    np.linalg.norm((np.dot(A_test, b[1:])+b[0]-b_test), 2)**2

print(
    f'For alpha = 0.02, the MSE for train set is {MSE_train} and {MSE_test} for test set'
)<jupyter_output>For alpha = 0.02, the MSE for train set is 0.07459894314853126 and 0.13507635517360442 for test set<jupyter_text>We see that two different alphas give results that are very close. The smaller alpha =0.02 gives a slightly smaller MSE. So for the next part, we use alpha =0.02.  Task 4> Use your code from step 3 to train classifiers for the pairs of digits (3,8) and (2,7) and report the training and test MSE’s. Can you explain the performance variations?<jupyter_code># for number 3 and 8

A_train, b_train = extract_relavent_features(3, 8, X_train, Y_train)
A_test, b_test = extract_relavent_features(3, 8, X_test, Y_test)

reg = Ridge(alpha=0.02)
reg.fit(A_train, b_train)

# calculating MSE for train and test set for alpha = 0.02
b = np.insert(reg.coef_, 0, reg.intercept_)

MSE_train = 1/len(b_train) * \
    np.linalg.norm((np.dot(A_train, b[1:])+b[0]-b_train), 2)**2
MSE_test = 1/len(b_test) * \
    np.linalg.norm((np.dot(A_test, b[1:])+b[0]-b_test), 2)**2

print(
    f'For alpha = 0.02, the MSE for train set is {MSE_train} and {MSE_test} for test set'
)
# for number 2 and 7

A_train, b_train = extract_relavent_features(2, 7, X_train, Y_train)
A_test, b_test = extract_relavent_features(2, 7, X_test, Y_test)

reg = Ridge(alpha=0.02)
reg.fit(A_train, b_train)

# calculating MSE for train and test set for alpha = 0.02
b = np.insert(reg.coef_, 0, reg.intercept_)

MSE_train = 1/len(b_train) * \
    np.linalg.norm((np.dot(A_train, b[1:])+b[0]-b_train), 2)**2
MSE_test = 1/len(b_test) * \
    np.linalg.norm((np.dot(A_test, b[1:])+b[0]-b_test), 2)**2

print(
    f'For alpha = 0.02, the MSE for train set is {MSE_train} and {MSE_test} for test set'
)<jupyter_output>For alpha = 0.02, the MSE for train set is 0.09177852736527603 and 0.3742354809625476 for test set
{'fraction_non_alphanumeric': 0.1261437908496732, 'fraction_numerical': 0.05016339869281046, 'mean_word_length': 4.383465259454705, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '213904', 'license': 'MIT', 'n_tokens_mistral': 2560, 'n_tokens_neox': 2334, 'n_words': 764, 'path': 'reports/2/hw2.ipynb', 'repo_name': 'raph651/Amath-582-Data-Analysis', 'size': '6120'}